May 2018 - tor-commits - lists.torproject.org

[metrics-web/master] Bump version to 1.1.0-dev.
by karsten＠torproject.org 30 May '18

30 May '18

commit 9ff54625841590904b0198dbac0b6e12d041938e Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed May 30 15:46:39 2018 +0200 Bump version to 1.1.0-dev. --- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.xml b/build.xml index 8baa764..acf7f51 100644 --- a/build.xml +++ b/build.xml @@ -8,7 +8,7 @@ <property name="javadoc-title" value="MetricsWeb API Documentation"/> <property name="implementation-title" value="metrics-web" /> - <property name="release.version" value="1.1.0" /> + <property name="release.version" value="1.1.0-dev" /> <property name="metricslibversion" value="2.4.0" /> <property name="jetty.version" value="-9.2.21.v20170120" /> <property name="warfile"

1 0

[metrics-web/release] Adds extended introduction and services to the home page (See: #23169)
by karsten＠torproject.org 30 May '18

30 May '18

commit 8f55bff0cc8fa70cce1ce8a87e0bb54a1bf739e4 Author: Iain R. Learmonth <irl(a)fsfe.org> Date: Thu Mar 1 13:44:20 2018 +0000 Adds extended introduction and services to the home page (See: #23169) --- src/main/resources/web/css/style.css | 68 +++++++++++++++++++++-------- src/main/resources/web/jsps/index.jsp | 81 +++++++++++++++++++++++++++-------- 2 files changed, 115 insertions(+), 34 deletions(-) diff --git a/src/main/resources/web/css/style.css b/src/main/resources/web/css/style.css index b2485ff..97e91b8 100644 --- a/src/main/resources/web/css/style.css +++ b/src/main/resources/web/css/style.css @@ -14,7 +14,7 @@ body #wrapper { /* Page header - override bootstrap defaults */ .page-header { - padding: 0em 2.0em 2.0em 2.0em; + padding: 0em 2.0em 2.0em 2.0em; margin: 0; border-bottom: 0; color:#fff; @@ -33,20 +33,20 @@ body #wrapper { max-width:40%; height:auto; } -.page-header img#metrics-logo { - height:75px; - width:auto !important; -} +.page-header img#metrics-logo { + height:75px; + width:auto !important; +} .page-header div { text-align:right; position:absolute; right:2.5em; - bottom: 0.9em; - max-width: 400px; + bottom: 0.9em; + max-width: 400px; } .page-header div p { - font-size: 14px; - line-height: 16px; + font-size: 14px; + line-height: 16px; } @media (max-width: 767.99999px) { .page-header { @@ -59,7 +59,7 @@ body #wrapper { display:none; } } -@media (max-width: 1199.99999px) { +@media (max-width: 1199.99999px) { .page-header div { max-width:330px; } @@ -136,9 +136,9 @@ body { .navbar-secondary .section-header:first-of-type { margin-top:0px; } -.navbar-secondary .navbar-nav>li>a { - margin-bottom: 0px !important; -} +.navbar-secondary .navbar-nav>li>a { + margin-bottom: 0px !important; +} /* images */ @@ -336,10 +336,27 @@ body .topButton { color: #5bc0de; } +/* home jumbotron */ +.jumbotron { + padding-bottom: 10px; +} +.jumbotron h1 { + margin-top: -20px; + font-size: 50px; +} +@media only screen and (max-width : 991px) { .jumbotron h1 { font-size: 46px; } } +@media only screen and (max-width : 352px) { + .jumbotron h1 { + font-size: 38px; + } + .jumbotron p { + display: none; + } +} /* home dashboard page */ .dashboard { - padding:3em 5em 0 5em; + padding:0em 5em 0 5em; } .dashboard .col-sm-4 { margin-bottom:2em; @@ -353,10 +370,27 @@ body .topButton { padding:2em 2em 1em 2em; color:#999; } -.dashboard h2, .dashboard p { - color:#999; +.dashboard h2 { + font-size: 42px; + color: rgb(125, 70, 152); + padding-top: 0; + margin-top: 0; +} +.dashboard h3 { + font-size: 26px; +} +@media (max-width: 992px) { + .dashboard h3 { + font-size: 20px; + } +} +.dashboard h3, .dashboard p { + color: #999; +} +.dashboard p.lead { + color: #333; } -.dashboard a[href], .dashboard a[href]:hover, .dashboard a[href]:focus, .dashboard a[href]:active, .dashboard a[href] h2 { +.dashboard a[href], .dashboard a[href]:hover, .dashboard a[href]:focus, .dashboard a[href]:active, .dashboard a[href] h3 { color:#68b030; } .dashboard a[href]:hover { diff --git a/src/main/resources/web/jsps/index.jsp b/src/main/resources/web/jsps/index.jsp index 56e2afa..5968727 100644 --- a/src/main/resources/web/jsps/index.jsp +++ b/src/main/resources/web/jsps/index.jsp @@ -5,32 +5,79 @@ <jsp:param name="navActive" value="Home"/> </jsp:include> -  - <div class="container"> - <ul class="breadcrumb"> - <li class="active"> </li> - </ul> - </div> - <div class="container"> - <h1>Welcome!</h1> - <p> - What would you like to know about the <a href="https://www.torproject.org/" target="_blank">Tor</a> network? - </p> + <div class="jumbotron"> + <h1>Welcome to Tor Metrics!</h1> + <div class="row"> + <div class="col-md-6"> + <p>The <a href="https://www.torproject.org/" target="_blank">Tor</a> + network is one of the largest deployed anonymity networks, consisting of <a + href="/networksize.html"><span + id="latest_relay_count">thousands</span></a> of volunteer-run + relays and <a + href="/userstats-relay-country.html"><span + id="latest_user_count">millions</span></a> of users. + Users, advocates, relay operators, and journalists can better understand the Tor network + through data and analysis made available by Tor Metrics.</p> + </div> + <div class="col-md-6"> + <p>Analyzing a live anonymity system must be performed with great care so that + the users' privacy is not put at risk. Any metrics collected <i>must + not</i> undermine the anonymity or security properties of the Tor + network. <a href="/about.html">Read more »</a></p> + </div> + </div> </div> - <div class="container dashboard"> + <div class="dashboard"> + <h2><i class="fa fa-line-chart"></i> Analysis</h2> + <p class="lead">View visualizations of statistics collected from the public Tor network and from Tor Project infrastructure.</p> <div class="row"> - <c:forEach var="category" items="${categories}"> <div class="col-sm-4"> - <a<c:if test="${fn:length(category[0]) > 0}"> href="${category[0]}.html"</c:if>><i class="fa ${category[3]} fa-fw fa-4x" aria-hidden="true"></i> <h2>${category[1]}</h2> <p>${category[2]}</p></a> + <a href="userstats-relay-country.html"><i class="fa fa-users fa-fw fa-4x" aria-hidden="true"></i> <h3>Users</h3> <p>Where Tor users are from and how they connect to Tor.</p></a> + </div> + + <div class="col-sm-4"> + <a href="networksize.html"><i class="fa fa-server fa-fw fa-4x" aria-hidden="true"></i> <h3>Servers</h3> <p>How many relays and bridges are online and what we know about them.</p></a> + </div> + + <div class="col-sm-4"> + <a href="bandwidth.html"><i class="fa fa-road fa-fw fa-4x" aria-hidden="true"></i> <h3>Traffic</h3> <p>How much traffic the Tor network can handle and how much traffic there is.</p></a> + </div> + + <div class="col-sm-4"> + <a href="torperf.html"><i class="fa fa-dashboard fa-fw fa-4x" aria-hidden="true"></i> <h3>Performance</h3> <p>How fast and reliable the Tor network is.</p></a> + </div> + + <div class="col-sm-4"> + <a href="hidserv-dir-onions-seen.html"><i class="fa fa-map-signs fa-fw fa-4x" aria-hidden="true"></i> <h3>Onion Services</h3> <p>How many onion services there are and how much traffic they pull.</p></a> </div> - </c:forEach> + + <div class="col-sm-4"> + <a href="webstats-tb.html"><i class="fa fa-download fa-fw fa-4x" aria-hidden="true"></i> <h3>Applications</h3> <p>How many Tor applications, like Tor Browser, have been downloaded or updated.</p></a> + </div> + </div> </div> - <div class="container"> - <p><a href="about.html#contact">Let us know</a> if we're missing anything, or if we should measure something else.</p> + <div class="dashboard"> + <h2><i class="fa fa-cogs"></i> Services</h2> + <p class="lead">Perform interactive queries for more detailed information relating to relays or bridges in the public Tor network.</p> + <div class="row"> + <div class="col-sm-4"> + <a href="/rs.html#search"><i class="fa fa-search fa-fw fa-4x" aria-hidden="true"></i> <h3>Relay Search</h3> <p>Look up information on a particular Tor relay or bridge.</p></a> + </div> + + <div class="col-sm-4"> + <a href="/rs.html#aggregate"><i class="fa fa-compress fa-fw fa-4x" aria-hidden="true"></i> <h3>Aggregated Relay Search</h3> <p>Look up aggregated statistics on groups of relays.</p></a> + </div> + + <div class="col-sm-4"> + <a href="https://exonerator.torproject.org/"><i class="fa fa-history fa-fw fa-4x" aria-hidden="true"></i> <h3>Network Archive</h3> <p>Look up if a particular IP address was used as a Tor relay on a particular date.</p></a> + </div> + + </div> + </div> </div> <jsp:include page="bottom.jsp"/>

1 0

[metrics-web/release] Take out old news from Onionoo's specification.
by karsten＠torproject.org 30 May '18

30 May '18

commit 75258264b84b01e2d4e43b4642f493bb652da828 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Mar 1 14:49:54 2018 +0100 Take out old news from Onionoo's specification. --- src/main/resources/web/jsps/onionoo.jsp | 36 ++++++++++----------------------- 1 file changed, 11 insertions(+), 25 deletions(-) diff --git a/src/main/resources/web/jsps/onionoo.jsp b/src/main/resources/web/jsps/onionoo.jsp index fc80677..0f29507 100644 --- a/src/main/resources/web/jsps/onionoo.jsp +++ b/src/main/resources/web/jsps/onionoo.jsp @@ -569,19 +569,18 @@ a family. <li> <a id="parameters_version"></a> -<b>version <span class="label label-primary">updated</span></b> +<b>version</b> <a href="#parameters_version" class="anchor">#</a> <p> Return only relays or bridges running a Tor version that starts with the parameter value <i>without</i> leading <code>"Tor"</code>. Searches are case-insensitive. -<span class="blue">Extended to bridges on November 28, 2017.</span> </p> </li> <li> <a id="parameters_host_name"></a> -<b>host_name <span class="label label-primary">new</span></b> +<b>host_name</b> <a href="#parameters_host_name" class="anchor">#</a> <p> Return only relays with a domain name <i>ending</i> in the given (partial) @@ -590,13 +589,12 @@ Searches for subdomains of a specific domain should ideally be prefixed with a period, for example: ".csail.mit.edu". Non-ASCII host name characters must be encoded as punycode. Filtering by host name is case-insensitive. -<span class="blue">Added on November 17, 2017.</span> </p> </li> <li> <a id="parameters_recommended_version"></a> -<b>recommended_version <span class="label label-primary">new</span></b> +<b>recommended_version</b> <a href="#parameters_recommended_version" class="anchor">#</a> <p> Return only relays and bridges running a Tor software version that is @@ -606,7 +604,6 @@ Uses the version in the consensus or bridge network status. Relays and bridges are not contained in either result, if the version they are running is not known. Parameter values are case-insensitive. -<span class="blue">Added on November 28, 2017.</span> </p> </li> @@ -740,14 +737,13 @@ Omitted if no major protocol changes are planned. <li> <a id="responses_build_revision"></a> -<b>build_revision <span class="label label-primary">new</span></b> +<b>build_revision</b> <code class="typeof">string</code> <span class="required-false">optional</span> <a href="#responses_build_revision" class="anchor">#</a> <p> Git revision of the Onionoo instance's software used to write this response, which will be omitted if unknown. -<span class="blue">Added on October 10, 2017.</span> </p> </li> @@ -1455,7 +1451,7 @@ found. <li> <a id="details_relay_version"></a> -<b>version <span class="label label-primary">new</span></b> +<b>version</b> <code class="typeof">string</code> <span class="required-false">optional</span> <a href="#details_relay_version" class="anchor">#</a> @@ -1465,7 +1461,6 @@ by the directory authorities in the "v" line of the consensus. Omitted if either the directory authorities or the relay did not report which version the relay runs or if the relay runs an alternative Tor implementation. -<span class="blue">Added on November 28, 2017.</span> </p> </li> @@ -1486,7 +1481,7 @@ the relay did not report which version it runs. <li> <a id="details_relay_effective_family"></a> -<b>effective_family <span class="label label-primary">updated</span></b> +<b>effective_family</b> <code class="typeof">array of strings</code> <span class="required-false">optional</span> <a href="#details_relay_effective_family" class="anchor">#</a> @@ -1497,14 +1492,12 @@ These relays are part of this relay's family and they consider this relay to be part of their family. Omitted if empty or if descriptor containing this information cannot be found. -<span class="blue">Removed the $ prefix from fingerprintson December 20, -2017.</span> </p> </li> <li> <a id="details_relay_alleged_family"></a> -<b>alleged_family <span class="label label-primary">updated</span></b> +<b>alleged_family</b> <code class="typeof">array of strings</code> <span class="required-false">optional</span> <a href="#details_relay_alleged_family" class="anchor">#</a> @@ -1515,14 +1508,12 @@ These relays are part of this relay's family but they don't consider this relay to be part of their family. Omitted if empty or if descriptor containing this information cannot be found. -<span class="blue">Removed the $ prefix from fingerprintson December 20, -2017.</span> </p> </li> <li> <a id="details_relay_indirect_family"></a> -<b>indirect_family <span class="label label-primary">updated</span></b> +<b>indirect_family</b> <code class="typeof">array of strings</code> <span class="required-false">optional</span> <a href="#details_relay_indirect_family" class="anchor">#</a> @@ -1532,8 +1523,6 @@ mutual family relationship with this relay but that can be reached by following effective, mutual family relationships starting at this relay. Omitted if empty or if descriptor containing this information cannot be found. -<span class="blue">Removed the $ prefix from fingerprintson December 20, -2017.</span> </p> </li> @@ -1619,7 +1608,7 @@ contain measurement information. <li> <a id="details_relay_unreachable_or_addresses"></a> -<b>unreachable_or_addresses <span class="label label-primary">new</span></b> +<b>unreachable_or_addresses</b> <code class="typeof">array of strings</code> <span class="required-false">optional</span> <a href="#details_relay_unreachable_or_addresses" class="anchor">#</a> @@ -1640,7 +1629,6 @@ relays with unreachable addresses will be included here. Addresses are in arbitrary order. IPv6 hex characters are all lower-case. Omitted if empty. -<span class="blue">Added on November 17, 2017.</span> </p> </li> @@ -1801,7 +1789,7 @@ information cannot be found. <li> <a id="details_bridge_version"></a> -<b>version <span class="label label-primary">new</span></b> +<b>version</b> <code class="typeof">string</code> <span class="required-false">optional</span> <a href="#details_bridge_version" class="anchor">#</a> @@ -1811,13 +1799,12 @@ by the bridge in the "platform" line of its server descriptor. Omitted if not provided by the bridge, if the descriptor containing this information cannot be found, or if the bridge runs an alternative Tor implementation. -<span class="blue">Added on November 28, 2017.</span> </p> </li> <li> <a id="details_bridge_recommended_version"></a> -<b>recommended_version <span class="label label-primary">new</span></b> +<b>recommended_version</b> <code class="typeof">boolean</code> <span class="required-false">optional</span> <a href="#details_bridge_recommended_version" class="anchor">#</a> @@ -1827,7 +1814,6 @@ recommended by the directory authorities or not. Uses the bridge version in the bridge network status. Omitted if either the directory authorities did not recommend versions, or the bridge did not report which version it runs. -<span class="blue">Added on November 28, 2017.</span> </p> </li>

1 0

[metrics-web/release] Give up on reshape in favor of tidyr.
by karsten＠torproject.org 30 May '18

30 May '18

commit 055229303076be6275ba64e9d2864d1ebbed004e Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Mar 1 09:49:42 2018 +0100 Give up on reshape in favor of tidyr. --- src/main/R/rserver/graphs.R | 14 +++++++------- src/main/R/rserver/rserve-init.R | 1 - 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R index fd5201d..8847967 100644 --- a/src/main/R/rserver/graphs.R +++ b/src/main/R/rserver/graphs.R @@ -348,7 +348,7 @@ plot_networksize <- function(start, end, path) { s <- rbind(s, data.frame(date = as.Date(missing, origin = "1970-01-01"), relays = NA, bridges = NA)) - networksize <- melt(s, id = "date") + networksize <- gather(s, variable, value, -date) ggplot(networksize, aes(x = as.Date(date, "%Y-%m-%d"), y = value, colour = variable)) + geom_line() + scale_x_date(name = "", breaks = custom_breaks, @@ -456,7 +456,7 @@ prepare_bandwidth <- function(start, end) { plot_bandwidth <- function(start, end, path) { b <- prepare_bandwidth(start, end) - bandwidth <- melt(b, id = "date") + bandwidth <- gather(b, variable, value, -date) ggplot(bandwidth, aes(x = as.Date(date, "%Y-%m-%d"), y = value, colour = variable)) + geom_line() + @@ -544,7 +544,7 @@ prepare_dirbytes <- function(start, end, path) { plot_dirbytes <- function(start, end, path) { b <- prepare_dirbytes(start, end) - dir <- melt(b, id = "date") + dir <- gather(b, variable, value, -date) ggplot(dir, aes(x = as.Date(date, "%Y-%m-%d"), y = value, colour = variable)) + geom_line() + @@ -699,7 +699,7 @@ plot_torperf_failures <- function(start, end, source, server, filesize, path) { torperf <- rbind(torperf, data.frame(date = as.Date(missing, origin = "1970-01-01"), timeouts = NA, failures = NA)) - torperf <- melt(torperf, id = "date") + torperf <- gather(torperf, variable, value, -date) filesizes <- data.frame(filesizes = c("5mb", "1mb", "50kb"), label = c("5 MiB", "1 MiB", "50 KiB"), stringsAsFactors = FALSE) filesizeStr <- filesizes[filesizes$filesize == filesize, "label"] @@ -734,7 +734,7 @@ prepare_connbidirect <- function(start, end) { levels = c("both", "write", "read")), quantile = paste("X", c$quantile, sep = ""), fraction = c$fraction / 100) - c <- cast(c, date + direction ~ quantile, value = "fraction") + c <- spread(c, quantile, fraction) c } @@ -743,7 +743,7 @@ plot_connbidirect <- function(start, end, path) { ggplot(c, aes(x = date, y = X0.5, colour = direction)) + geom_line(size = 0.75) + geom_ribbon(aes(x = date, ymin = X0.25, ymax = X0.75, - fill = direction), alpha = 0.5, show_guide = FALSE) + + fill = direction), alpha = 0.5, show.legend = FALSE) + scale_x_date(name = "", breaks = custom_breaks, labels = custom_labels, minor_breaks = custom_minor_breaks) + scale_y_continuous(name = "", labels = percent, limits = c(0, NA)) + @@ -787,7 +787,7 @@ prepare_bandwidth_flags <- function(start, end) { b <- aggregate(list(advbw = b$advbw, bwhist = b$bwhist), by = list(date = b$date, flag = b$flag), FUN = sum, na.rm = TRUE, na.action = NULL) - b <- melt(b, id.vars = c("date", "flag"), variable_name = "type") + b <- gather(b, type, value, -c(date, flag)) bandwidth <- b[b$value > 0, ] dates <- seq(from = as.Date(start, "%Y-%m-%d"), to = as.Date(end, "%Y-%m-%d"), by = "1 day") diff --git a/src/main/R/rserver/rserve-init.R b/src/main/R/rserver/rserve-init.R index ede7c73..b9a1d3b 100644 --- a/src/main/R/rserver/rserve-init.R +++ b/src/main/R/rserver/rserve-init.R @@ -1,7 +1,6 @@ ##Pre-loaded libraries and graphing functions to speed things up library("ggplot2") -library("reshape") library("RColorBrewer") library("scales") library(dplyr)

1 0

[metrics-web/release] Adds text about Metrics ecosystem to about page (See: #23169)
by karsten＠torproject.org 30 May '18

30 May '18

commit 28d5eada13f1f8b2bb509e280afba01da5770ad3 Author: Iain R. Learmonth <irl(a)fsfe.org> Date: Thu Mar 1 13:45:44 2018 +0000 Adds text about Metrics ecosystem to about page (See: #23169) --- src/main/resources/graphviz/ecosystem.gv | 56 ++++++++++++++++++++++ src/main/resources/web/images/ecosystem.png | Bin 0 -> 73561 bytes src/main/resources/web/jsps/about.jsp | 69 ++++++++++++++++++++++++---- 3 files changed, 115 insertions(+), 10 deletions(-) diff --git a/src/main/resources/graphviz/ecosystem.gv b/src/main/resources/graphviz/ecosystem.gv new file mode 100644 index 0000000..c5cdd5c --- /dev/null +++ b/src/main/resources/graphviz/ecosystem.gv @@ -0,0 +1,56 @@ +digraph ecosystem { + node [fontname="Source Sans Pro"]; + subgraph cluster_network { + label="Public Tor Network"; + fontname="Source Sans Pro Semi-Bold"; + node [color="#59316b",fillcolor="#e1d3e8",style="filled"]; + relays [label="Relays"]; + bridges [label="Bridges"]; + dirauths [label="Directory Authorities"]; + bridgeauth [label="Bridge Authority"]; + } + subgraph cluster_metrics { + label="Tor Metrics"; + fontname="Source Sans Pro Semi-Bold"; + labelloc=b; + subgraph resdev { + node [color="#6b5831",style="filled",fillcolor="#ede9e1"]; + collector [label="CollecTor"]; + onionoo [label="Onionoo"]; + subgraph lib { + node [shape=box,margin="0.11,0.11",height=0,width=0,fontsize=8]; + lib1 [label="metrics-lib"]; + lib2 [label="metrics-lib"]; + lib3 [label="metrics-lib"]; + } + } + subgraph user { + node [color="#436b31",style="filled",fillcolor="#d4e5cc"]; + statistics [label="Tor Metrics Graphs"]; + exonerator [label="Exonerator"]; + rs [label="Relay Search"]; + bot [label="metrics-bot"]; + consensushealth [label="Consensus Health"]; + } + } + relays -> dirauths; + bridges -> bridgeauth; + dirauths -> collector; + dirauths -> consensushealth; + bridgeauth -> collector; + collector -> lib1; + lib1 -> statistics; + collector -> lib2; + lib2 -> onionoo; + collector -> lib3; + lib3 -> exonerator; + onionoo -> rs; + onionoo -> bot; + legend [rank=sink,fontsize=10,shape=none,label= + <<table cellspacing="0" cellborder="1"> + <tr><td>Legend</td></tr> + <tr><td bgcolor="#e1d3e8" color="#59316b">Public Tor Network Servers</td></tr> + <tr><td bgcolor="#ede9e1" color="#6b5831">Tools for Researchers and Developers</td></tr> + <tr><td bgcolor="#d4e5cc" color="#436b31">Tools for End-Users</td></tr> + </table>>]; +} diff --git a/src/main/resources/web/images/ecosystem.png b/src/main/resources/web/images/ecosystem.png new file mode 100644 index 0000000..3b80ed1 Binary files /dev/null and b/src/main/resources/web/images/ecosystem.png differ diff --git a/src/main/resources/web/jsps/about.jsp b/src/main/resources/web/jsps/about.jsp index 7a4d303..bc14b08 100644 --- a/src/main/resources/web/jsps/about.jsp +++ b/src/main/resources/web/jsps/about.jsp @@ -14,20 +14,71 @@ <div class="container"> <h1>About <a href="#about" name="about" class="anchor">#</a></h1> - <p>You're curious who has put together all this information, and you're thinking about contributing? Here we tell you a bit about the Tor metrics team and give you some pointers for joining our team.</p> - </div> + <p>Tor Metrics archives historical data about the Tor ecosystem, collects + data from the public Tor network and related services, and assists in + developing novel approaches to safe, privacy preserving data + collection.</p> - <div class="container"> <h2>Philosophy <a href="#philosophy" name="philosophy" class="anchor">#</a></h2> - <p>We only use public, non-sensitive data for metrics. Each metric goes through a rigorous review and discussion process before appearing here. We never publish statistics—or aggregate statistics—of sensitive data, such as unencrypted contents of traffic.</p> - </div> + <p>We only use public, non-sensitive data for metrics. Each metric goes + through a rigorous review and discussion process before appearing here. + We never publish statistics—or aggregate statistics—of + sensitive data, such as unencrypted contents of traffic.</p> + + <h2>Ecosystem <a href="#ecosystem" name="ecosystem" class="anchor">#</a></h2> + <p>Tor relays and bridges collect aggregated statistics about their usage + including bandwidth and connecting clients per country. Source + aggregation is used to protect the privacy of connecting + users—discarding IP addresses and only reporting country + information from a local database mapping IP address ranges to countries. + These statistics are sent periodically to the directory authorities.</p> + <p><a href="/collector.html">CollecTor</a> + downloads the latest server descriptors, extra info descriptors + containing the aggregated statistics, and consensus documents from the + directory authorities and archives them. This archive is public and the + <a href="/metrics-lib.html">metrics-lib</a> + Java library can be used to parse the contents of the archive to perform + analysis of the data.</p> + <p>In order to provide easy access to visualizations of the historical + data archived, the Tor Metrics website contains a number of customizable + plots to show <a + href="/userstats-relay-country.html">user</a>, + <a href="/bandwidth.html">traffic</a>, <a + href="/relayflags.html">relay</a>, <a + href="/bridges-ipv6.html">bridge</a>, and + <a href="/webstats-tb.html">application + download</a> statistics over a requested time period and filtered to a + particular country.</p> + <p>In order to provide easy access to current information about the + public Tor network, <a + href="/onionoo.html">Onionoo</a> implements + a protocol to serve JSON documents over HTTP that can be consumed by + applications that would like to display information about relays along + with historical bandwidth, uptime, and consensus weight information.</p> + <p>An example of one such application is <a + href="/rs.html">Relay Search</a> which is + used by relay operators, those monitoring the health of the network, and + developers of software using the Tor network. Another example of such an + application is <a href="https://people.torproject.org/~irl/metrics-bot" + target="_blank">metrics-bot</a> which posts regular snapshots to <a + href="https://twitter.com/TorAtlas" target="_blank">Twitter</a> and <a + href="https://botsin.space/@metricsbot" target="_blank">Mastodon</a> + including country statistics and a world map plotting known relays.</p> + <p>The diagram below shows how data is collected, archived, analyzed, and + presented to users through services operated by Tor Metrics. The majority + of our services use metrics-lib to parse the descriptors that have been + collected by CollecTor as their source of raw data about the public Tor + network.</p> + <figure> + <img alt="Data flows from the public Tor network to the end user tools + Tor Metrics Graphs and Exonerator via CollecTor, and Relay Search and + metrics-bot via CollecTor and Onionoo." src="/images/ecosystem.png" + class="thumbnail img-responsive center-block"> + </figure> - <div class="container"> <h2>Contributing <a href="#contributing" name="contributing" class="anchor">#</a></h2> <p>Collecting and processing new data won't likely happen without your help! If you really want to see something measured here, we would be happy to work with you. Learn more about contributing on our <a href="https://trac.torproject.org/projects/tor/wiki/org/teams/MetricsTeam" target="_blank">team wiki page</a>.</p> - </div> - <div class="container"> <h2>Contact <a href="#contact" name="contact" class="anchor">#</a></h2> <p>If you have any questions or suggestions, contact us at <a href="mailto:metrics-team@lists.torproject.org">metrics-team(a)lists.torproject.org</a>, which is a <a href="https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team" target="_blank">public mailing list</a>.</p> @@ -35,9 +86,7 @@ <pre>The Tor Project, Inc. 217 1st Ave South #4903 Seattle, WA 98194 USA</pre> - </div> - <div class="container"> <h2>Testimonials <a href="#testimonials" name="testimonials" class="anchor">#</a></h2> <blockquote> <p class="mb-0">

1 0

[metrics-web/release] Fix first batch of style issues found by lintr.
by karsten＠torproject.org 30 May '18

30 May '18

commit debef097e14c83523a9ae4e077bf1bb1dd945720 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Mar 1 22:30:34 2018 +0100 Fix first batch of style issues found by lintr. --- src/main/R/rserver/graphs.R | 218 ++++++++++++++++++++++---------------------- 1 file changed, 109 insertions(+), 109 deletions(-) diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R index 93ff919..a31790d 100644 --- a/src/main/R/rserver/graphs.R +++ b/src/main/R/rserver/graphs.R @@ -308,7 +308,7 @@ custom_labels <- function(breaks) { # Helper function to format numbers in non-scientific notation with spaces as # thousands separator. formatter <- function(x, ...) { - format(x, ..., scientific = FALSE, big.mark = ' ') + format(x, ..., scientific = FALSE, big.mark = " ") } theme_update( @@ -322,17 +322,17 @@ theme_update( # Set the default line size of geom_line() to 1. update_geom_defaults("line", list(size = 1)) -copyright_notice = "The Tor Project - https://metrics.torproject.org/" +copyright_notice <- "The Tor Project - https://metrics.torproject.org/" -stats_dir = "/srv/metrics.torproject.org/metrics/shared/stats/" +stats_dir <- "/srv/metrics.torproject.org/metrics/shared/stats/" -rdata_dir = "/srv/metrics.torproject.org/metrics/shared/RData/" +rdata_dir <- "/srv/metrics.torproject.org/metrics/shared/RData/" prepare_networksize <- function(start, end) { read.csv(paste(stats_dir, "servers.csv", sep = ""), colClasses = c("date" = "Date")) %>% - filter(date >= as.Date(start), date <= as.Date(end), flag == '', - country == '', version == '', platform == '', ec2bridge == '') %>% + filter(date >= as.Date(start), date <= as.Date(end), flag == "", + country == "", version == "", platform == "", ec2bridge == "") %>% select(date, relays, bridges) } @@ -361,8 +361,8 @@ write_networksize <- function(start, end, path) { prepare_versions <- function(start, end) { read.csv(paste(stats_dir, "servers.csv", sep = ""), colClasses = c("date" = "Date")) %>% - filter(date >= as.Date(start), date <= as.Date(end), flag == '', - country == '', version != '', platform == '', ec2bridge == '') %>% + filter(date >= as.Date(start), date <= as.Date(end), flag == "", + country == "", version != "", platform == "", ec2bridge == "") %>% select(date, version, relays) } @@ -371,7 +371,7 @@ plot_versions <- function(start, end, path) { known_versions <- c("Other", "0.1.0", "0.1.1", "0.1.2", "0.2.0", "0.2.1", "0.2.2", "0.2.3", "0.2.4", "0.2.5", "0.2.6", "0.2.7", "0.2.8", "0.2.9", "0.3.0", "0.3.1", "0.3.2", "0.3.3") - getPalette = colorRampPalette(brewer.pal(12, "Paired")) + getPalette <- colorRampPalette(brewer.pal(12, "Paired")) colours <- data.frame(breaks = known_versions, values = rep(brewer.pal(min(12, length(known_versions)), "Paired"), len = length(known_versions)), @@ -400,8 +400,8 @@ write_versions <- function(start, end, path) { prepare_platforms <- function(start, end) { read.csv(paste(stats_dir, "servers.csv", sep = ""), colClasses = c("date" = "Date")) %>% - filter(date >= as.Date(start), date <= as.Date(end), flag == '', - country == '', version == '', platform != '', ec2bridge == '') %>% + filter(date >= as.Date(start), date <= as.Date(end), flag == "", + country == "", version == "", platform != "", ec2bridge == "") %>% select(date, platform, relays) %>% mutate(platform = ifelse(platform == "Darwin", "macOS", as.character(platform))) @@ -431,8 +431,8 @@ write_platforms <- function(start, end, path) { prepare_bandwidth <- function(start, end) { read.csv(paste(stats_dir, "bandwidth.csv", sep = ""), colClasses = c("date" = "Date")) %>% - filter(date >= as.Date(start), date <= as.Date(end), isexit == '', - isguard == '') %>% + filter(date >= as.Date(start), date <= as.Date(end), isexit == "", + isguard == "") %>% mutate(advbw = advbw * 8 / 1e9) %>% mutate(bwhist = (bwread + bwwrite) * 8 / 2e9) %>% select(date, advbw, bwhist) @@ -464,11 +464,11 @@ write_bandwidth <- function(start, end, path) { prepare_bwhist_flags <- function(start, end) { read.csv(paste(stats_dir, "bandwidth.csv", sep = ""), colClasses = c("date" = "Date")) %>% - filter(date >= as.Date(start), date <= as.Date(end), isexit != '', - isguard != '') %>% - mutate(variable = ifelse(isexit == 't', - ifelse(isguard == 't', "guard_and_exit", "exit_only"), - ifelse(isguard == 't', "guard_only", "middle_only")), + filter(date >= as.Date(start), date <= as.Date(end), isexit != "", + isguard != "") %>% + mutate(variable = ifelse(isexit == "t", + ifelse(isguard == "t", "guard_and_exit", "exit_only"), + ifelse(isguard == "t", "guard_only", "middle_only")), value = (bwread + bwwrite) * 8 / 2e9) %>% select(date, variable, value) } @@ -502,8 +502,8 @@ write_bwhist_flags <- function(start, end, path) { prepare_dirbytes <- function(start, end, path) { read.csv(paste(stats_dir, "bandwidth.csv", sep = ""), colClasses = c("date" = "Date")) %>% - filter(date >= as.Date(start), date <= as.Date(end), isexit == '', - isguard == '') %>% + filter(date >= as.Date(start), date <= as.Date(end), isexit == "", + isguard == "") %>% mutate(dirread = dirread * 8 / 1e9, dirwrite = dirwrite * 8 / 1e9) %>% select(date, dirread, dirwrite) @@ -535,9 +535,9 @@ write_dirbytes <- function(start, end, path) { prepare_relayflags <- function(start, end, flags) { read.csv(paste(stats_dir, "servers.csv", sep = ""), colClasses = c("date" = "Date")) %>% - filter(date >= as.Date(start), date <= as.Date(end), country == '', - version == '', platform == '', ec2bridge == '') %>% - mutate(flag = ifelse(flag == '', 'Running', as.character(flag))) %>% + filter(date >= as.Date(start), date <= as.Date(end), country == "", + version == "", platform == "", ec2bridge == "") %>% + mutate(flag = ifelse(flag == "", "Running", as.character(flag))) %>% filter(flag %in% flags) %>% select(date, flag, relays) } @@ -566,8 +566,8 @@ write_relayflags <- function(start, end, flags, path) { } plot_torperf <- function(start, end, source, server, filesize, path) { - filesizeVal <- ifelse(filesize == '50kb', 50 * 1024, - ifelse(filesize == '1mb', 1024 * 1024, 5 * 1024 * 1024)) + filesize_val <- ifelse(filesize == "50kb", 50 * 1024, + ifelse(filesize == "1mb", 1024 * 1024, 5 * 1024 * 1024)) t <- read.csv(paste(stats_dir, "torperf-1.1.csv", sep = ""), colClasses = c("date" = "Date")) known_sources <- c("all", unique(t[t$source != "", "source"])) @@ -577,10 +577,10 @@ plot_torperf <- function(start, end, source, server, filesize, path) { colour <- colours[colours$source == source, "colour"] filesizes <- data.frame(filesizes = c("5mb", "1mb", "50kb"), label = c("5 MiB", "1 MiB", "50 KiB"), stringsAsFactors = FALSE) - filesizeStr <- filesizes[filesizes$filesize == filesize, "label"] + filesize_str <- filesizes[filesizes$filesize == filesize, "label"] t[t$date >= as.Date(start) & t$date <= as.Date(end) & - t$filesize == filesizeVal & - t$source == ifelse(source == 'all', '', source) & + t$filesize == filesize_val & + t$source == ifelse(source == "all", "", source) & t$server == server, ] %>% transmute(date, q1 = q1 / 1e3, md = md / 1e3, q3 = q3 / 1e3) %>% complete(date = full_seq(date, period = 1)) %>% @@ -596,7 +596,7 @@ plot_torperf <- function(start, end, source, server, filesize, path) { breaks = c("line", "ribbon"), labels = c("Median", "1st to 3rd quartile"), values = paste(colour, c("", "66"), sep = "")) + - ggtitle(paste("Time to complete", filesizeStr, + ggtitle(paste("Time to complete", filesize_str, "request to", server, "server")) + labs(caption = copyright_notice) + theme(legend.position = "top") @@ -612,21 +612,21 @@ write_torperf <- function(start, end, source_, server_, filesize_, path) { read.csv(paste(stats_dir, "torperf-1.1.csv", sep = ""), colClasses = c("date" = "Date")) %>% filter(date >= as.Date(start), date <= as.Date(end), - filesize == ifelse(filesize_ == '50kb', 50 * 1024, - ifelse(filesize_ == '1mb', 1024 * 1024, 5 * 1024 * 1024)), - source == ifelse(source_ == 'all', '', source_), + filesize == ifelse(filesize_ == "50kb", 50 * 1024, + ifelse(filesize_ == "1mb", 1024 * 1024, 5 * 1024 * 1024)), + source == ifelse(source_ == "all", "", source_), server == server_) %>% transmute(date, q1 = q1 / 1e3, md = md / 1e3, q3 = q3 / 1e3) %>% write.csv(path, quote = FALSE, row.names = FALSE) } prepare_torperf_failures <- function(start, end, source, server, filesize) { - filesizeVal <- ifelse(filesize == '50kb', 50 * 1024, - ifelse(filesize == '1mb', 1024 * 1024, 5 * 1024 * 1024)) + filesize_val <- ifelse(filesize == "50kb", 50 * 1024, + ifelse(filesize == "1mb", 1024 * 1024, 5 * 1024 * 1024)) t <- read.csv(paste(stats_dir, "torperf-1.1.csv", sep = ""), colClasses = c("date" = "Date")) - t[t$date >= start & t$date <= end & t$filesize == filesizeVal & - t$source == ifelse(source == 'all', '', source) & + t[t$date >= start & t$date <= end & t$filesize == filesize_val & + t$source == ifelse(source == "all", "", source) & t$server == server & t$requests > 0, ] %>% transmute(date, timeouts = timeouts / requests, failures = failures / requests) @@ -635,7 +635,7 @@ prepare_torperf_failures <- function(start, end, source, server, filesize) { plot_torperf_failures <- function(start, end, source, server, filesize, path) { filesizes <- data.frame(filesizes = c("5mb", "1mb", "50kb"), label = c("5 MiB", "1 MiB", "50 KiB"), stringsAsFactors = FALSE) - filesizeStr <- filesizes[filesizes$filesize == filesize, "label"] + filesize_str <- filesizes[filesizes$filesize == filesize, "label"] prepare_torperf_failures(start, end, source, server, filesize) %>% gather(variable, value, -date) %>% ggplot(aes(x = date, y = value, colour = variable)) + @@ -647,7 +647,7 @@ plot_torperf_failures <- function(start, end, source, server, filesize, path) { ifelse(source == "all", "all sources", source)), h.start = 45, breaks = c("timeouts", "failures"), labels = c("Timeouts", "Failures")) + - ggtitle(paste("Timeouts and failures of", filesizeStr, + ggtitle(paste("Timeouts and failures of", filesize_str, "requests to", server, "server")) + labs(caption = copyright_notice) + theme(legend.position = "top") @@ -703,10 +703,10 @@ write_connbidirect <- function(start, end, path) { prepare_bandwidth_flags <- function(start, end) { b <- read.csv(paste(stats_dir, "bandwidth.csv", sep = ""), colClasses = c("date" = "Date")) - b <- b[b$date >= start & b$date <= end & b$isexit != '' & - b$isguard != '', ] + b <- b[b$date >= start & b$date <= end & b$isexit != "" & + b$isguard != "", ] b <- data.frame(date = b$date, - isexit = b$isexit == 't', isguard = b$isguard == 't', + isexit = b$isexit == "t", isguard = b$isguard == "t", advbw = b$advbw * 8 / 1e9, bwhist = (b$bwread + b$bwwrite) * 8 / 2e9) b <- rbind( @@ -761,16 +761,16 @@ plot_userstats <- function(start, end, node, variable, value, events, c <- data u <- c[c$date >= start & c$date <= end, ] u <- rbind(u, data.frame(date = start, - country = ifelse(variable == 'country' & value != 'all', value, ''), - transport = ifelse(variable == 'transport', value, ''), - version = ifelse(variable == 'version', value, ''), + country = ifelse(variable == "country" & value != "all", value, ""), + transport = ifelse(variable == "transport", value, ""), + version = ifelse(variable == "version", value, ""), lower = 0, upper = 0, clients = 0)) - if (node == 'relay') { - if (value != 'all') { + if (node == "relay") { + if (value != "all") { u <- u[u$country == value, ] title <- paste("Directly connecting users from", countryname(value)) } else { - u <- u[u$country == '', ] + u <- u[u$country == "", ] title <- "Directly connecting users" } u <- aggregate(list(lower = u$lower, upper = u$upper, @@ -778,16 +778,16 @@ plot_userstats <- function(start, end, node, variable, value, events, by = list(date = as.Date(u$date, "%Y-%m-%d"), value = u$country), FUN = sum) - } else if (variable == 'transport') { - if ('!<OR>' %in% value) { - n <- u[u$transport != '' & u$transport != '<OR>', ] + } else if (variable == "transport") { + if ("!<OR>" %in% value) { + n <- u[u$transport != "" & u$transport != "<OR>", ] n <- aggregate(list(lower = n$lower, upper = n$upper, clients = n$clients), by = list(date = n$date), FUN = sum) u <- rbind(u, data.frame(date = n$date, - country = '', transport = '!<OR>', - version = '', lower = n$lower, + country = "", transport = "!<OR>", + version = "", lower = n$lower, upper = n$upper, clients = n$clients)) } if (length(value) > 1) { @@ -806,14 +806,14 @@ plot_userstats <- function(start, end, node, variable, value, events, value = u$transport), FUN = sum) title <- paste("Bridge users using", - ifelse(value == '<??>', 'unknown pluggable transport(s)', - ifelse(value == '<OR>', 'default OR protocol', - ifelse(value == '!<OR>', 'any pluggable transport', - ifelse(value == 'fte', 'FTE', - ifelse(value == 'websocket', 'Flash proxy/websocket', - paste('transport', value))))))) + ifelse(value == "<??>", "unknown pluggable transport(s)", + ifelse(value == "<OR>", "default OR protocol", + ifelse(value == "!<OR>", "any pluggable transport", + ifelse(value == "fte", "FTE", + ifelse(value == "websocket", "Flash proxy/websocket", + paste("transport", value))))))) } - } else if (variable == 'version') { + } else if (variable == "version") { u <- u[u$version == value, ] title <- paste("Bridge users using IP", value, sep = "") u <- aggregate(list(lower = u$lower, upper = u$upper, @@ -822,11 +822,11 @@ plot_userstats <- function(start, end, node, variable, value, events, value = u$version), FUN = sum) } else { - if (value != 'all') { + if (value != "all") { u <- u[u$country == value, ] title <- paste("Bridge users from", countryname(value)) } else { - u <- u[u$country == '' & u$transport == '' & u$version == '', ] + u <- u[u$country == "" & u$transport == "" & u$version == "", ] title <- "Bridge users" } u <- aggregate(list(lower = u$lower, upper = u$upper, @@ -837,15 +837,15 @@ plot_userstats <- function(start, end, node, variable, value, events, } u <- merge(x = u, all.y = TRUE, y = data.frame(expand.grid( date = seq(from = as.Date(start, "%Y-%m-%d"), - to = as.Date(end, "%Y-%m-%d"), by="1 day"), - value = ifelse(value == 'all', '', value)))) + to = as.Date(end, "%Y-%m-%d"), by = "1 day"), + value = ifelse(value == "all", "", value)))) if (length(value) > 1) { plot <- ggplot(u, aes(x = date, y = users, colour = value)) } else { plot <- ggplot(u, aes(x = date, y = users)) } if (length(na.omit(u$users)) > 0 & events != "off" & - variable == 'country' & length(value) == 1 && value != "all") { + variable == "country" & length(value) == 1 && value != "all") { upturns <- u[u$users > u$upper, c("date", "users")] downturns <- u[u$users < u$lower, c("date", "users")] if (events == "on") { @@ -872,11 +872,11 @@ plot_userstats <- function(start, end, node, variable, value, events, if (length(value) > 1) { plot <- plot + scale_colour_hue(name = "", breaks = value, - labels = ifelse(value == '<??>', 'Unknown PT', - ifelse(value == '<OR>', 'Default OR protocol', - ifelse(value == '!<OR>', 'Any PT', - ifelse(value == 'fte', 'FTE', - ifelse(value == 'websocket', 'Flash proxy/websocket', + labels = ifelse(value == "<??>", "Unknown PT", + ifelse(value == "<OR>", "Default OR protocol", + ifelse(value == "!<OR>", "Any PT", + ifelse(value == "fte", "FTE", + ifelse(value == "websocket", "Flash proxy/websocket", value)))))) } ggsave(filename = path, width = 8, height = 5, dpi = 150) @@ -884,20 +884,20 @@ plot_userstats <- function(start, end, node, variable, value, events, plot_userstats_relay_country <- function(start, end, country, events, path) { - plot_userstats(start, end, 'relay', 'country', country, events, path) + plot_userstats(start, end, "relay", "country", country, events, path) } plot_userstats_bridge_country <- function(start, end, country, path) { - plot_userstats(start, end, 'bridge', 'country', country, 'off', path) + plot_userstats(start, end, "bridge", "country", country, "off", path) } plot_userstats_bridge_transport <- function(start, end, transport, path) { - plot_userstats(start, end, 'bridge', 'transport', transport, 'off', + plot_userstats(start, end, "bridge", "transport", transport, "off", path) } plot_userstats_bridge_version <- function(start, end, version, path) { - plot_userstats(start, end, 'bridge', 'version', version, 'off', path) + plot_userstats(start, end, "bridge", "version", version, "off", path) } write_userstats_relay_country <- function(start, end, country_, events, @@ -905,13 +905,13 @@ write_userstats_relay_country <- function(start, end, country_, events, load(paste(rdata_dir, "clients-relay.RData", sep = "")) u <- data %>% filter(date >= as.Date(start), date <= as.Date(end), - country == ifelse(country_ == 'all', '', country_), transport == '', - version == '') - if (country_ != 'all' && events == 'on') { + country == ifelse(country_ == "all", "", country_), transport == "", + version == "") + if (country_ != "all" && events == "on") { u <- u %>% mutate(downturns = clients < u$lower, upturns = clients > upper) %>% select(date, clients, downturns, upturns, lower, upper) - } else if (country_ != 'all' && events != 'off') { + } else if (country_ != "all" && events != "off") { u <- u %>% mutate(downturns = clients < u$lower, upturns = clients > upper) %>% select(date, clients, downturns, upturns) @@ -928,8 +928,8 @@ write_userstats_bridge_country <- function(start, end, country_, path) { load(paste(rdata_dir, "clients-bridge.RData", sep = "")) data %>% filter(date >= as.Date(start), date <= as.Date(end), - country == ifelse(country_ == 'all', '', country_), transport == '', - version == '') %>% + country == ifelse(country_ == "all", "", country_), transport == "", + version == "") %>% select(date, clients) %>% rename(users = clients) %>% write.csv(path, quote = FALSE, row.names = FALSE) @@ -939,21 +939,21 @@ write_userstats_bridge_transport <- function(start, end, transports, path) { load(paste(rdata_dir, "clients-bridge.RData", sep = "")) u <- data %>% filter(date >= as.Date(start), date <= as.Date(end), - country == '', version == '', transport != '') %>% + country == "", version == "", transport != "") %>% select(date, transport, clients) - if ('!<OR>' %in% transports) { + if ("!<OR>" %in% transports) { n <- u %>% - filter(transport != '<OR>') %>% + filter(transport != "<OR>") %>% group_by(date) %>% summarize(clients = sum(clients)) - u <- rbind(u, data.frame(date = n$date, transport = '!<OR>', + u <- rbind(u, data.frame(date = n$date, transport = "!<OR>", clients = n$clients)) } u %>% filter(transport %in% transports) %>% - mutate(transport = ifelse(transport == '<OR>', 'default_or_protocol', - ifelse(transport == '!<OR>', 'any_pt', - ifelse(transport == '<??>', 'unknown_pluggable_transports', + mutate(transport = ifelse(transport == "<OR>", "default_or_protocol", + ifelse(transport == "!<OR>", "any_pt", + ifelse(transport == "<??>", "unknown_pluggable_transports", transport)))) %>% group_by(date, transport) %>% select(date, transport, clients) %>% @@ -965,7 +965,7 @@ write_userstats_bridge_version <- function(start, end, version_, path) { load(paste(rdata_dir, "clients-bridge.RData", sep = "")) data %>% filter(date >= as.Date(start), date <= as.Date(end), - country == '', transport == '', version == version_) %>% + country == "", transport == "", version == version_) %>% select(date, clients) %>% rename(users = clients) %>% write.csv(path, quote = FALSE, row.names = FALSE) @@ -1013,8 +1013,8 @@ write_userstats_bridge_combined <- function(start, end, country, path) { } else { prepare_userstats_bridge_combined(start, end, country) %>% select(date, transport, low, high) %>% - mutate(transport = ifelse(transport == '<OR>', - 'default_or_protocol', transport)) %>% + mutate(transport = ifelse(transport == "<OR>", + "default_or_protocol", transport)) %>% gather(variable, value, -(date:transport)) %>% unite(temp, transport, variable) %>% spread(temp, value) %>% @@ -1185,10 +1185,10 @@ prepare_webstats_tb <- function(start, end) { plot_webstats_tb <- function(start, end, path) { d <- prepare_webstats_tb(start, end) levels(d$request_type) <- list( - 'Initial downloads' = 'tbid', - 'Signature downloads' = 'tbsd', - 'Update pings' = 'tbup', - 'Update requests' = 'tbur') + "Initial downloads" = "tbid", + "Signature downloads" = "tbsd", + "Update pings" = "tbup", + "Update requests" = "tbur") ggplot(d, aes(x = log_date, y = count)) + geom_point() + geom_line() + @@ -1216,7 +1216,7 @@ prepare_webstats_tb_platform <- function(start, end) { read.csv(paste(stats_dir, "webstats.csv", sep = ""), colClasses = c("log_date" = "Date")) %>% filter(log_date >= as.Date(start), log_date <= as.Date(end), - request_type == 'tbid') %>% + request_type == "tbid") %>% group_by(log_date, platform) %>% summarize(count = sum(count)) } @@ -1250,13 +1250,13 @@ write_webstats_tb_platform <- function(start, end, path) { plot_webstats_tb_locale <- function(start, end, path) { d <- read.csv(paste(stats_dir, "webstats.csv", sep = ""), colClasses = c("log_date" = "Date")) - d <- d[d$log_date >= start & d$log_date <= end & d$request_type == 'tbid', ] + d <- d[d$log_date >= start & d$log_date <= end & d$request_type == "tbid", ] e <- d e <- aggregate(list(count = e$count), by = list(locale = e$locale), FUN = sum) e <- e[order(e$count, decreasing = TRUE), ] e <- e[1:5, ] d <- aggregate(list(count = d$count), by = list(log_date = d$log_date, - locale = ifelse(d$locale %in% e$locale, d$locale, '(other)')), FUN = sum) + locale = ifelse(d$locale %in% e$locale, d$locale, "(other)")), FUN = sum) ggplot(d, aes(x = log_date, y = count, colour = locale)) + geom_point() + geom_line() + @@ -1281,13 +1281,13 @@ plot_webstats_tb_locale <- function(start, end, path) { write_webstats_tb_locale <- function(start, end, path) { d <- read.csv(paste(stats_dir, "webstats.csv", sep = ""), colClasses = c("log_date" = "Date")) - d <- d[d$log_date >= start & d$log_date <= end & d$request_type == 'tbid', ] + d <- d[d$log_date >= start & d$log_date <= end & d$request_type == "tbid", ] e <- d e <- aggregate(list(count = e$count), by = list(locale = e$locale), FUN = sum) e <- e[order(e$count, decreasing = TRUE), ] e <- e[1:5, ] d <- aggregate(list(count = d$count), by = list(log_date = d$log_date, - locale = ifelse(d$locale %in% e$locale, d$locale, 'other')), FUN = sum) + locale = ifelse(d$locale %in% e$locale, d$locale, "other")), FUN = sum) d %>% mutate(locale = tolower(locale)) %>% rename(date = log_date) %>% @@ -1305,8 +1305,8 @@ prepare_webstats_tm <- function(start, end) { plot_webstats_tm <- function(start, end, path) { d <- prepare_webstats_tm(start, end) levels(d$request_type) <- list( - 'Initial downloads' = 'tmid', - 'Update pings' = 'tmup') + "Initial downloads" = "tmid", + "Update pings" = "tmup") ggplot(d, aes(x = log_date, y = count)) + geom_point() + geom_line() + @@ -1336,9 +1336,9 @@ prepare_relays_ipv6 <- function(start, end) { valid_after_date <= as.Date(end), server == "relay") %>% group_by(valid_after_date) %>% summarize(total = sum(server_count_sum_avg), - announced = sum(server_count_sum_avg[announced_ipv6 == 't']), - reachable = sum(server_count_sum_avg[reachable_ipv6_relay == 't']), - exiting = sum(server_count_sum_avg[exiting_ipv6_relay == 't'])) %>% + announced = sum(server_count_sum_avg[announced_ipv6 == "t"]), + reachable = sum(server_count_sum_avg[reachable_ipv6_relay == "t"]), + exiting = sum(server_count_sum_avg[exiting_ipv6_relay == "t"])) %>% complete(valid_after_date = full_seq(valid_after_date, period = 1)) %>% gather(total, announced, reachable, exiting, key = "category", value = "count") @@ -1375,7 +1375,7 @@ prepare_bridges_ipv6 <- function(start, end) { valid_after_date <= as.Date(end), server == "bridge") %>% group_by(valid_after_date) %>% summarize(total = sum(server_count_sum_avg), - announced = sum(server_count_sum_avg[announced_ipv6 == 't'])) %>% + announced = sum(server_count_sum_avg[announced_ipv6 == "t"])) %>% complete(valid_after_date = full_seq(valid_after_date, period = 1)) %>% gather(total, announced, key = "category", value = "count") } @@ -1410,14 +1410,14 @@ prepare_advbw_ipv6 <- function(start, end) { valid_after_date <= as.Date(end), server == "relay") %>% group_by(valid_after_date) %>% summarize(total = sum(advertised_bandwidth_bytes_sum_avg), - total_guard = sum(advertised_bandwidth_bytes_sum_avg[guard_relay != 'f']), - total_exit = sum(advertised_bandwidth_bytes_sum_avg[exit_relay != 'f']), + total_guard = sum(advertised_bandwidth_bytes_sum_avg[guard_relay != "f"]), + total_exit = sum(advertised_bandwidth_bytes_sum_avg[exit_relay != "f"]), reachable_guard = sum(advertised_bandwidth_bytes_sum_avg[ - reachable_ipv6_relay != 'f' & guard_relay != 'f']), + reachable_ipv6_relay != "f" & guard_relay != "f"]), reachable_exit = sum(advertised_bandwidth_bytes_sum_avg[ - reachable_ipv6_relay != 'f' & exit_relay != 'f']), + reachable_ipv6_relay != "f" & exit_relay != "f"]), exiting = sum(advertised_bandwidth_bytes_sum_avg[ - exiting_ipv6_relay != 'f'])) %>% + exiting_ipv6_relay != "f"])) %>% complete(valid_after_date = full_seq(valid_after_date, period = 1)) %>% gather(total, total_guard, total_exit, reachable_guard, reachable_exit, exiting, key = "category", value = "advbw") %>%

1 0

[metrics-web/release] Expand Philosophy text on about page (See: #23169)
by karsten＠torproject.org 30 May '18

30 May '18

commit 4a6b9deb39fe1011963066a78ee1512d20bad64f Author: Iain R. Learmonth <irl(a)fsfe.org> Date: Thu Mar 1 14:21:06 2018 +0000 Expand Philosophy text on about page (See: #23169) --- src/main/resources/web/jsps/about.jsp | 37 +++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/src/main/resources/web/jsps/about.jsp b/src/main/resources/web/jsps/about.jsp index bc14b08..dbd1be9 100644 --- a/src/main/resources/web/jsps/about.jsp +++ b/src/main/resources/web/jsps/about.jsp @@ -24,6 +24,43 @@ through a rigorous review and discussion process before appearing here. We never publish statistics—or aggregate statistics—of sensitive data, such as unencrypted contents of traffic.</p> + <p>The goals of a privacy and anonymity network like Tor are not easily + combined with extensive data gathering, but at the same time data is needed for + monitoring, understanding, and improving the network. Data can be used to + detect possible censorship events or attacks against the network. Safety and + privacy concerns regarding data collection by Tor Metrics are guided by the <a + href="https://research.torproject.org/safetyboard.html#guidelines">Tor Research + Safety Board's guidelines</a>. Safety and privacy assessment is usually done + openly by discussion during the proposal process for changes to the Tor source, + and/or supported by closer analysis in form of <a + href="https://research.torproject.org/techreports.html">Tor Technical + Reports</a>.</p> + <p>For data we collect from the public Tor network, we will always follow + three main guidelines:</p> + <dl class="dl-horizontal"> + <dt>Data minimalism</dt> + <dd>The first and most important guideline is that only the minimum + amount of statistical data should be gathered to solve a given problem. + The level of detail of measured data should be as small as possible.</dd> + <dt>Source aggregation</dt> + <dd>Possibly sensitive data should exist for as short a time + as possible. Data should be aggregated at its source, including + categorizing single events and memorizing category counts only, summing + up event counts over large time frames, and being imprecise regarding + exact event counts.</dd> + <dt>Transparency</dt> + <dd>All algorithms to gather statistical data need to be discussed + publicly before deploying them. All measured statistical data should be + made publicly available as a safeguard to not gather data that is too + sensitive.</dd> + </dl> + <p>You can read more about safety considerations when collecting data in + the Tor network in "<a + href="https://www.freehaven.net/anonbib/#wecsr10measuring-tor">A Case + Study on Measuring Statistical Data in the Tor Anonymity Network</a>" by + Karsten Loesing, Steven J. Murdoch, and Roger Dingledine. <i>In the + Proceedings of the Workshop on Ethics in Computer Security Research + (WECSR 2010), Tenerife, Canary Islands, Spain, January 2010.</i></p> <h2>Ecosystem <a href="#ecosystem" name="ecosystem" class="anchor">#</a></h2> <p>Tor relays and bridges collect aggregated statistics about their usage

1 0

[metrics-web/release] Bumps copyright year to 2018 in common JSP footer
by karsten＠torproject.org 30 May '18

30 May '18

commit 187cb1e86808eda2555eee7ac25fe953452e766b Author: Iain R. Learmonth <irl(a)fsfe.org> Date: Fri Mar 2 13:08:09 2018 +0000 Bumps copyright year to 2018 in common JSP footer --- src/main/resources/web/jsps/bottom.jsp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/resources/web/jsps/bottom.jsp b/src/main/resources/web/jsps/bottom.jsp index 6719adc..85db407 100644 --- a/src/main/resources/web/jsps/bottom.jsp +++ b/src/main/resources/web/jsps/bottom.jsp @@ -3,7 +3,7 @@ <div class="container no-gutter"> <div class="col-xs-6"> <p> - © 2009–2017 <a href="https://www.torproject.org/" target="_blank">The Tor Project</a> + © 2009–2018 <a href="https://www.torproject.org/" target="_blank">The Tor Project</a> </p> </div> <div class="col-xs-6">

1 0

[metrics-web/release] Adds an Ant target to generate PNGs using Graphviz
by karsten＠torproject.org 30 May '18

30 May '18

commit 252d4635df29db2a984f4bf75e0b0a8910e44370 Author: Iain R. Learmonth <irl(a)fsfe.org> Date: Sat Mar 3 06:35:34 2018 +0000 Adds an Ant target to generate PNGs using Graphviz --- build.xml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/build.xml b/build.xml index 4b67ae6..59f3720 100644 --- a/build.xml +++ b/build.xml @@ -220,6 +220,21 @@ </exec> </target> + <target name="generate-graphviz-pngs" > + <apply executable="dot" + dest="${basedir}/src/main/resources/web/images" + verbose="true" > + <arg value="-Tpng" /> + <arg value="-o" /> + <targetfile /> + <srcfile /> + <fileset dir="${basedir}/src/main/resources/graphviz" > + <include name="*.gv" /> + </fileset> + <mapper type="glob" from="*.gv" to="*.png" /> + </apply> + </target> +  <property name="prepare.deployment" value="/srv/metrics.torproject.org/metrics/work" />

1 0

[metrics-web/release] Update to latest metrics-lib.
by karsten＠torproject.org 30 May '18

30 May '18

commit ff1fbdd2f208049df061c8449ba86ceda961be91 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Mar 1 14:43:32 2018 +0100 Update to latest metrics-lib. --- src/submods/metrics-lib | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/submods/metrics-lib b/src/submods/metrics-lib index 9f2db9a..7648780 160000 --- a/src/submods/metrics-lib +++ b/src/submods/metrics-lib @@ -1 +1 @@ -Subproject commit 9f2db9a194d5e290b456765f83cc6eeaf745e619 +Subproject commit 7648780844f809f3068d3e625973fe43c78ccbb3

1 0