September 2018 - tor-commits - lists.torproject.org

[metrics-web/release] Fix a couple HTML issues.
by karsten＠torproject.org 26 Sep '18

26 Sep '18

commit 74f7e3d736815a0966fab482538315e2b8a6c740 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Sep 11 15:00:11 2018 +0200 Fix a couple HTML issues. --- src/main/resources/web/jsps/reproducible-metrics.jsp | 4 ++-- src/main/resources/web/jsps/stats.jsp | 7 ++++++- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/src/main/resources/web/jsps/reproducible-metrics.jsp b/src/main/resources/web/jsps/reproducible-metrics.jsp index 87f692e..8c72ffc 100644 --- a/src/main/resources/web/jsps/reproducible-metrics.jsp +++ b/src/main/resources/web/jsps/reproducible-metrics.jsp @@ -304,7 +304,7 @@ If the <code>"platform"</code> line contains neither of these substrings, memori If the platform line is missing, we skip this descriptor, which later leads to not counting this relay at all rather than including it in the "Other" group, which is slightly wrong.</li> <li>IPv6 announced OR: Parse any <code>"or-address"</code> lines and memorize whether at least one of them contains an IPv6 address. This indicates that the relay announced an IPv6 address.</li> <li>IPv6 exiting: Parse the <code>"ipv6-policy"</code> line, if present, and memorize whether it's different from "reject 1-65535". This indicates whether the relay permitted exiting to IPv6 targets. If the line is not present, memorize that the relay does not permit exiting to IPv6 targets.</li> -<li>Server descriptor digest: Compute the SHA-1 digest, or determine it from the file name in case of archived descriptor tarballs. +<li>Server descriptor digest: Compute the SHA-1 digest, or determine it from the file name in case of archived descriptor tarballs.</li> </ul> <h4>Step 3: Compute daily averages</h4> @@ -429,7 +429,7 @@ Refer to the <a href="https://gitweb.torproject.org/torspec.git/tree/dir-spec.tx <ul> <li>Advertised bandwidth: Parse the three values (or just two in very old descriptors) from the <code>"bandwidth"</code> line. These values stand for the average bandwidth, burst bandwidth, and observed bandwidth. The advertised bandwidth is the minimum of these values.</li> -<li>Server descriptor digest: Compute the SHA-1 digest, or determine it from the file name in case of archived descriptor tarballs. +<li>Server descriptor digest: Compute the SHA-1 digest, or determine it from the file name in case of archived descriptor tarballs.</li> </ul>  diff --git a/src/main/resources/web/jsps/stats.jsp b/src/main/resources/web/jsps/stats.jsp index 565f235..52e0bb5 100644 --- a/src/main/resources/web/jsps/stats.jsp +++ b/src/main/resources/web/jsps/stats.jsp @@ -171,6 +171,8 @@ using bridges, which can be either "v4" or "v6".</li> <li>frac: Fraction of bridges in percent that the estimate is based on.</li> </ul> +</div> + <div class="container"> <h2> Servers <a href="#servers" name="servers" class="anchor">#</a></h2> @@ -393,6 +395,7 @@ Traffic <a href="#traffic" name="traffic" class="anchor">#</a></h2> <li>p: Percentile as value between 0 and 100.</li> <li>all: Advertised bandwidth in Gbit/s of the p-th percentile of all relays.</li> <li>exits: Advertised bandwidth in Gbit/s of the p-th percentile of relays with the "Exit" relay flag.</li> +</ul> <h3>Advertised bandwidth of n-th fastest relays <a href="/advbwdist-relay.html" class="btn btn-primary btn-xs"> graph</a> @@ -696,7 +699,7 @@ Applications <a href="#applications" name="applications" class="anchor">#</a></h <h4>Columns</h4> -</ul> +<ul> <li>date: UTC date (YYYY-MM-DD) when requests to <code>torproject.org</code> web servers have been logged.</li> <li>locale: Locale, like "en-US" for English (United States), "de" for German, etc., and "??" for unrecognized locales.</li> <li>initial_downloads: Number of Tor Browser initial downloads.</li> @@ -731,6 +734,8 @@ Applications <a href="#applications" name="applications" class="anchor">#</a></h These files will no longer be publicly available after September 15, 2018. All relevant contained data will still be available via the statistics files specified above. +</div> + <div class="container"> <h3>Number of relays and bridges (deprecated) <a href="#servers" name="servers" class="anchor">#</a></h3>

1 0

[metrics-web/release] Prepare for 1.2.0 release.
by karsten＠torproject.org 26 Sep '18

26 Sep '18

commit 581c62e94b5b983193764aa39d917be89821bc67 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Sep 25 09:30:45 2018 +0200 Prepare for 1.2.0 release. --- CERT | 18 +++++++++--------- CHANGELOG.md | 6 ++++++ build.xml | 4 ++-- 3 files changed, 17 insertions(+), 11 deletions(-) diff --git a/CERT b/CERT index 61fa7c6..b90b397 100644 --- a/CERT +++ b/CERT @@ -1,8 +1,8 @@ -----BEGIN CERTIFICATE----- -MIIDaTCCAlGgAwIBAgIENNGkczANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJV +MIIDaTCCAlGgAwIBAgIELle0dTANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJV UzELMAkGA1UECBMCV0ExEDAOBgNVBAcTB1NlYXR0bGUxHTAbBgNVBAoTFFRoZSBU b3IgUHJvamVjdCwgSW5jMRgwFgYDVQQDEw9LYXJzdGVuIExvZXNpbmcwHhcNMTgw -NTIzMTUxNDU2WhcNMTgwODIxMTUxNDU2WjBlMQswCQYDVQQGEwJVUzELMAkGA1UE +ODI4MDcwNjM2WhcNMTgxMTI2MDcwNjM2WjBlMQswCQYDVQQGEwJVUzELMAkGA1UE CBMCV0ExEDAOBgNVBAcTB1NlYXR0bGUxHTAbBgNVBAoTFFRoZSBUb3IgUHJvamVj dCwgSW5jMRgwFgYDVQQDEw9LYXJzdGVuIExvZXNpbmcwggEiMA0GCSqGSIb3DQEB AQUAA4IBDwAwggEKAoIBAQChXn+IUp+o6G+k4ffxk3TkxZb3iXfiG7byNsG63olU @@ -11,11 +11,11 @@ Qw+VAhKTcEIv4yiR0BWapQyR07pgmKirYVjN6s6ef8NJzUptpxLlaYJ3ZfQfc4aE MXzScgaccwDFIWQ661lzLGCfeSxxa3Xy4wWsGwzNzLITYrrABcbg7yogLo2btNvD oEwGL3/baQdhl0dra6biVCZr9ydn3Hg57S55pUU0rBY25id78zUO8xrfNHw54wwX lOblGt75OOkahP/ZZSBxxoiknJ6y5VQV8y+noA4vigXFAgMBAAGjITAfMB0GA1Ud -DgQWBBSeh60M+/wMYyYhlxtuff2Hk9n7bzANBgkqhkiG9w0BAQsFAAOCAQEAJtEQ -B2AVpVtjTGU7uujUtEjSn1ICv7QLW/37TmHn+m1SWvXOVlSG81eb5JwX449Pn3w5 -K1Bx0JJmJ6Kec2kHtoR1r5/DUlTMtRqKQ5ZHQbvYMx+Ifq8TRALHjcw8p5Vw+gep -0zYbSQycklzFaFqLB8Cus0ICb+UY54HTddpezQ3IXS/vc4Vy07ocm8rUCz1s8L/r -ehTOxSym7G7+kcRzNplFOLL5iO8o6uHPyLR1TIAIXa0XZ41ogNl2q+6CvXG+UI/j -qkzSXD6FJK1Px2nxNFIe/w9NL+chSytIGnV3CImyiORuV+1OxMglyQspSGEbl1XP -OfiTEYHnp12BYMeRyw== +DgQWBBSeh60M+/wMYyYhlxtuff2Hk9n7bzANBgkqhkiG9w0BAQsFAAOCAQEAkXZs +3T3GTkZ+EGvZG5puzKdgZiSsLgIy25xdWsIx147AIZEJFKjEAtbu0osMpkTa96B6 +a+BHf7PTjQUuH3YOEmeW9ab8pwu5SRijCq2qkuvjjSLBcJzWnalcKDYYvoQte1// +Di8JqpRXCw20WY2bldTiafyG80E0RGfiX2I8vbDiPIhjwz9Wox8Q1rw1c9T/vRn9 +pI8FrHgTnDO6R54yD25QSpsj+hC+IDkFKO17vGCIaJrPG5o6th438ijEwJsG+LRB +4zKKKsFTby7UJI3Ag8xolIhsBkRZO2j4Na35i15SZ7QJNj9J5g171z8RyOmyIQbg +q7OXN2iiRIxiIJwoQw== -----END CERTIFICATE----- diff --git a/CHANGELOG.md b/CHANGELOG.md index a4fee6f..7890d2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,9 @@ +# Changes in version 1.2.0 - 2018-08-25 + + * Medium changes + - Add ExoneraTor 4.0.0 thin jar as dependency. + + # Changes in version 1.1.0 - 2018-05-29 * Medium changes diff --git a/build.xml b/build.xml index a97cb01..759901e 100644 --- a/build.xml +++ b/build.xml @@ -8,9 +8,9 @@ <property name="javadoc-title" value="MetricsWeb API Documentation"/> <property name="implementation-title" value="metrics-web" /> - <property name="release.version" value="1.1.0-dev" /> + <property name="release.version" value="1.2.0" /> <property name="metricslibversion" value="2.4.0" /> - <property name="exoneratorversion" value="3.0.1" /> + <property name="exoneratorversion" value="4.0.0" /> <property name="jetty.version" value="-9.2.21.v20170120" /> <property name="warfile" value="metrics-web-${release.version}.war"/>

1 0

[metrics-web/release] Add missing double quote.
by karsten＠torproject.org 26 Sep '18

26 Sep '18

commit c3afbfcd03330b69898ed3877a2cdfc4e7f1766b Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Sep 11 15:57:46 2018 +0200 Add missing double quote. --- src/main/resources/web/jsps/stats.jsp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/resources/web/jsps/stats.jsp b/src/main/resources/web/jsps/stats.jsp index 4736309..9c8d158 100644 --- a/src/main/resources/web/jsps/stats.jsp +++ b/src/main/resources/web/jsps/stats.jsp @@ -112,7 +112,7 @@ Users <a href="#users" name="users" class="anchor">#</a></h2> <ul> <li>start: First UTC date (YYYY-MM-DD) to include in the file.</li> <li>end: Last UTC date (YYYY-MM-DD) to include in the file.</li> -<li>transport: Lower-case transport name to include in the file. Possible values include "obfs4", "websocket" for Flash proxy/websocket, "fte" for FTE, "!%3COR%3E" (percent encoding of "!<OR>") for any pluggable transport, "%3C??%3E (percent encoding of "<??>") for unknown pluggable transport(s), or "%3COR%3E" (percent encoding of "<OR>") for the default OR protocol. Can be given multiple times.</li> +<li>transport: Lower-case transport name to include in the file. Possible values include "obfs4", "websocket" for Flash proxy/websocket, "fte" for FTE, "!%3COR%3E" (percent encoding of "!<OR>") for any pluggable transport, "%3C??%3E" (percent encoding of "<??>") for unknown pluggable transport(s), or "%3COR%3E" (percent encoding of "<OR>") for the default OR protocol. Can be given multiple times.</li> </ul> <h4>Columns</h4>

1 0

[metrics-web/release] Use "as" in place of "as_number" field for RS
by karsten＠torproject.org 26 Sep '18

26 Sep '18

commit 9e49336e2cc3c4f94824157aa19754598e2123b5 Author: Iain R. Learmonth <irl(a)fsfe.org> Date: Wed Sep 12 10:22:11 2018 +0100 Use "as" in place of "as_number" field for RS Fixes: #27652 --- src/main/resources/web/js/rs/collections/aggregates.js | 14 +++++++------- src/main/resources/web/js/rs/models/relay.js | 2 +- src/main/resources/web/templates/rs/details/router.html | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/src/main/resources/web/js/rs/collections/aggregates.js b/src/main/resources/web/js/rs/collections/aggregates.js index 75c5e22..bbf5a45 100644 --- a/src/main/resources/web/js/rs/collections/aggregates.js +++ b/src/main/resources/web/js/rs/collections/aggregates.js @@ -7,7 +7,7 @@ define([ ], function($, _, Backbone, aggregateModel){ var aggregatesCollection = Backbone.Collection.extend({ model: aggregateModel, - baseurl: 'https://onionoo.torproject.org/details?running=true&type=relay&fields=count…', + baseurl: 'https://onionoo.torproject.org/details?running=true&type=relay&fields=count…', url: '', aType: 'cc', lookup: function(options) { @@ -32,8 +32,8 @@ define([ This code will never be assigned for use with ISO 3166-1 and is "user-assigned". Fun fact: UN/LOCODE assigns XZ to represent installations in international waters. */ relay.country = ((typeof relay.country) == "undefined") ? "xz" : relay.country; - relay.as_number = ((typeof relay.as_number) == "undefined") ? "AS0" : relay.as_number; - if (relay.as_number == "AS0") relay.as_name = "Unknown"; + relay.as = ((typeof relay.as) == "undefined") ? "AS0" : relay.as; + if (relay.as == "AS0") relay.as_name = "Unknown"; var ccAggregate = false; var asAggregate = false; @@ -45,13 +45,13 @@ define([ aggregateKey = relay.country; ccAggregate = true; } else if (collection.aType == "as") { - aggregateKey = relay.as_number; + aggregateKey = relay.as; asAggregate = true; } else if (collection.aType == "version") { aggregateKey = relay.version.split(".").slice(0, 3).join(".") + "."; versionAggregate = true; } else { - aggregateKey = relay.country + "/" + relay.as_number; + aggregateKey = relay.country + "/" + relay.as; ccAggregate = asAggregate = true; } @@ -63,7 +63,7 @@ define([ aggregates[aggregateKey].country = new Set(); } if (asAggregate) { - aggregates[aggregateKey].as = relay.as_number; + aggregates[aggregateKey].as = relay.as; } else { aggregates[aggregateKey].as = new Set(); } @@ -77,7 +77,7 @@ define([ if (relay.country !== "xz") aggregates[aggregateKey].country.add(relay.country); } if (!asAggregate) { - if (relay.as_number !== "AS0") aggregates[aggregateKey].as.add(relay.as_number); + if (relay.as !== "AS0") aggregates[aggregateKey].as.add(relay.as); } aggregates[aggregateKey].relays++; if ((typeof relay.guard_probability) !== "undefined") aggregates[aggregateKey].guard_probability += relay.guard_probability; diff --git a/src/main/resources/web/js/rs/models/relay.js b/src/main/resources/web/js/rs/models/relay.js index b5a6b8e..eeb0569 100644 --- a/src/main/resources/web/js/rs/models/relay.js +++ b/src/main/resources/web/js/rs/models/relay.js @@ -234,7 +234,7 @@ define([ relay.uptime = relay.last_restarted ? relay.uptime.millisecs : null; relay.last_restarted = relay.last_restarted ? relay.last_restarted : null; relay.downtime = relay.last_seen ? model.parsedate(relay.last_seen).hrfull : null; - relay.as_number = relay.as_number ? relay.as_number : null; + relay.as = relay.as ? relay.as : null; relay.as_name = relay.as_name ? relay.as_name : null; relay.transports = relay.transports ? relay.transports : null; relay.fingerprint = relay.hashed_fingerprint ? relay.hashed_fingerprint : relay.fingerprint; diff --git a/src/main/resources/web/templates/rs/details/router.html b/src/main/resources/web/templates/rs/details/router.html index c746269..d7596ca 100644 --- a/src/main/resources/web/templates/rs/details/router.html +++ b/src/main/resources/web/templates/rs/details/router.html @@ -153,7 +153,7 @@ should update it as soon as possible.</div> <dd><% if (relay.get('country')) { %><img class="inline country" title="<%= relay.get('countryname') %>" alt="" src="/images/cc/<%= relay.get('country') %>.png"/> <a href="#search/country:<%= relay.get('country') %>"><%= relay.get('countryname') %></a> (<a href="#search/country:<%= relay.get('country') %>%20flag:exit"><img src="/images/flags/exit.png" title="All exits in <%= relay.get('countryname') %>"></a>)<% } else { %>Unknown<% } %></dd> <dt>AS Number</dt> - <dd><% if (relay.get('as_number') !== null) { %><a href="#search/as:<%= relay.get('as_number') %>"><%= _.escape(relay.get('as_number')) %></a><% } else { %>unknown<% }; %></dd> + <dd><% if (relay.get('as') !== null) { %><a href="#search/as:<%= relay.get('as') %>"><%= _.escape(relay.get('as')) %></a><% } else { %>unknown<% }; %></dd> <dt>AS Name</dt> <dd><%= relay.get('as_name') ? _.escape(relay.get('as_name')) : "unknown" %></dd>

1 0

[metrics-web/release] Link to Statistics and Reproducible Metrics from more places.
by karsten＠torproject.org 26 Sep '18

26 Sep '18

commit 2fc1f3c757a44d1f15efbda1883429fee23437a7 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Sep 11 16:09:05 2018 +0200 Link to Statistics and Reproducible Metrics from more places. Still part of #26857. --- src/main/java/org/torproject/metrics/web/GraphServlet.java | 1 + src/main/resources/web/json/categories.json | 2 +- src/main/resources/web/jsps/graph.jsp | 2 ++ src/main/resources/web/jsps/sources.jsp | 1 + 4 files changed, 5 insertions(+), 1 deletion(-) diff --git a/src/main/java/org/torproject/metrics/web/GraphServlet.java b/src/main/java/org/torproject/metrics/web/GraphServlet.java index 73aaec7..2f35320 100644 --- a/src/main/java/org/torproject/metrics/web/GraphServlet.java +++ b/src/main/java/org/torproject/metrics/web/GraphServlet.java @@ -141,6 +141,7 @@ public class GraphServlet extends MetricServlet { request.setAttribute("title", this.titles.get(requestedId)); if (this.categoriesById.containsKey(requestedId)) { Category category = this.categoriesById.get(requestedId); + request.setAttribute("categoryId", category.getId()); request.setAttribute("categoryHeader", category.getHeader()); request.setAttribute("categoryDescription", category.getDescription()); List<String[]> categoryTabs = new ArrayList<>(); diff --git a/src/main/resources/web/json/categories.json b/src/main/resources/web/json/categories.json index a5a4fcf..af05085 100644 --- a/src/main/resources/web/json/categories.json +++ b/src/main/resources/web/json/categories.json @@ -1,6 +1,6 @@ [ { - "id": "clients", + "id": "users", "icon": "fa-users", "header": "Users", "summary": "Where Tor users are from and how they connect to Tor.", diff --git a/src/main/resources/web/jsps/graph.jsp b/src/main/resources/web/jsps/graph.jsp index 06582ab..c30481f 100644 --- a/src/main/resources/web/jsps/graph.jsp +++ b/src/main/resources/web/jsps/graph.jsp @@ -161,6 +161,8 @@ Download data as <a href="${id}.csv${parameters}">CSV</a>. +Learn more about the CSV data <a href="/stats.html#${id}">format</a> or how to <a href="/reproducible-metrics.html#${categoryId}">reproduce</a> the graph data. + </div> </div> diff --git a/src/main/resources/web/jsps/sources.jsp b/src/main/resources/web/jsps/sources.jsp index f8f9e07..a43d681 100644 --- a/src/main/resources/web/jsps/sources.jsp +++ b/src/main/resources/web/jsps/sources.jsp @@ -24,6 +24,7 @@ <li><a href="collector.html">CollecTor</a> collects and archives data from various nodes and services in the public Tor network.</li> <li><a href="stats.html">Statistics</a> used for visualizations on this site are available in the CSV format.</li> <li><a href="onionoo.html">Onionoo</a> provides current and historical data about relays and bridges via a web-based API.</li> + <li><a href="reproducible-metrics.html">Reproducible Metrics</a> specifies how to reproduce the data behind the graphs on this site.</li> </ul> </div>

1 0

[metrics-web/release] Stop providing pre-aggregated CSV files.
by karsten＠torproject.org 26 Sep '18

26 Sep '18

commit e503aa40b1f4528e2baec4f8ca7b56cfce3b6ec0 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Sat Sep 15 20:29:07 2018 +0200 Stop providing pre-aggregated CSV files. Implements #27000. --- .../metrics/web/ResearchStatsServlet.java | 105 ---- src/main/resources/web.xml | 17 - src/main/resources/web/jsps/stats.jsp | 589 +-------------------- 3 files changed, 3 insertions(+), 708 deletions(-) diff --git a/src/main/java/org/torproject/metrics/web/ResearchStatsServlet.java b/src/main/java/org/torproject/metrics/web/ResearchStatsServlet.java deleted file mode 100644 index e4d665c..0000000 --- a/src/main/java/org/torproject/metrics/web/ResearchStatsServlet.java +++ /dev/null @@ -1,105 +0,0 @@ -/* Copyright 2013--2018 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.metrics.web; - -import java.io.BufferedInputStream; -import java.io.BufferedOutputStream; -import java.io.File; -import java.io.FileInputStream; -import java.io.IOException; -import java.util.SortedSet; -import java.util.TreeSet; - -import javax.servlet.ServletConfig; -import javax.servlet.ServletException; -import javax.servlet.http.HttpServlet; -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; - -public class ResearchStatsServlet extends HttpServlet { - - private static final long serialVersionUID = 3346710354297653810L; - - private File statsDir; - - private SortedSet<String> availableStatisticsFiles; - - @Override - public void init(ServletConfig config) throws ServletException { - super.init(config); - this.statsDir = new File(config.getInitParameter("statsDir")); - this.availableStatisticsFiles = new TreeSet<>(); - this.availableStatisticsFiles.add("servers"); - this.availableStatisticsFiles.add("ipv6servers"); - this.availableStatisticsFiles.add("bandwidth"); - this.availableStatisticsFiles.add("clients"); - this.availableStatisticsFiles.add("userstats-combined"); - this.availableStatisticsFiles.add("torperf"); - this.availableStatisticsFiles.add("torperf-1.1"); - this.availableStatisticsFiles.add("connbidirect2"); - this.availableStatisticsFiles.add("advbwdist"); - this.availableStatisticsFiles.add("hidserv"); - this.availableStatisticsFiles.add("webstats"); - } - - @Override - public long getLastModified(HttpServletRequest request) { - File statsFile = this.determineStatsFile(request); - if (statsFile == null || !statsFile.exists()) { - return -1L; - } else { - return statsFile.lastModified(); - } - } - - @Override - public void doGet(HttpServletRequest request, - HttpServletResponse response) throws IOException { - File statsFile = this.determineStatsFile(request); - if (statsFile == null) { - response.sendError(HttpServletResponse.SC_NOT_FOUND); - } else if (!statsFile.exists()) { - response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR); - } else { - this.writeStatsFile(statsFile, response); - } - } - - private File determineStatsFile(HttpServletRequest request) { - String requestedStatsFile = request.getRequestURI(); - if (requestedStatsFile.endsWith(".csv")) { - requestedStatsFile = requestedStatsFile.substring(0, - requestedStatsFile.length() - ".csv".length()); - } - if (requestedStatsFile.contains("/")) { - requestedStatsFile = requestedStatsFile.substring( - requestedStatsFile.lastIndexOf("/") + 1); - } - if (!availableStatisticsFiles.contains(requestedStatsFile)) { - return null; - } else { - return new File(this.statsDir, requestedStatsFile + ".csv"); - } - } - - private void writeStatsFile(File statsFile, - HttpServletResponse response) throws IOException { - response.setContentType("text/csv"); - response.setHeader("Content-Length", String.valueOf( - statsFile.length())); - response.setHeader("Content-Disposition", - "inline; filename=\"" + statsFile.getName() + "\""); - try (BufferedInputStream bis = new BufferedInputStream( - new FileInputStream(statsFile), 8192); - BufferedOutputStream bos = new BufferedOutputStream( - response.getOutputStream())) { - byte[] buffer = new byte[8192]; - int length; - while ((length = bis.read(buffer)) > 0) { - bos.write(buffer, 0, length); - } - } - } -} - diff --git a/src/main/resources/web.xml b/src/main/resources/web.xml index 5c88564..0c79916 100644 --- a/src/main/resources/web.xml +++ b/src/main/resources/web.xml @@ -96,23 +96,6 @@ </servlet-mapping> <servlet> - <servlet-name>ResearchStats</servlet-name> - <servlet-class> - org.torproject.metrics.web.ResearchStatsServlet - </servlet-class> - <init-param> - <param-name>statsDir</param-name> - <param-value> - /srv/metrics.torproject.org/metrics/shared/stats/ - </param-value> - </init-param> - </servlet> - <servlet-mapping> - <servlet-name>ResearchStats</servlet-name> - <url-pattern>/stats/*</url-pattern> - </servlet-mapping> - - <servlet> <servlet-name>GraphImage</servlet-name> <servlet-class> org.torproject.metrics.web.GraphImageServlet diff --git a/src/main/resources/web/jsps/stats.jsp b/src/main/resources/web/jsps/stats.jsp index b737bf7..3559994 100644 --- a/src/main/resources/web/jsps/stats.jsp +++ b/src/main/resources/web/jsps/stats.jsp @@ -21,7 +21,7 @@ <h3>Parameters</h3> -All per-graph statistics files (excluding the now deprecated pre-aggregated files) are available for download via an URL of the form: +All per-graph statistics files are available for download via an URL of the form: <pre> https://metrics.torproject.org/identifier.csv @@ -31,7 +31,7 @@ https://metrics.torproject.org/identifier.csv <h3>Columns</h3> -Each per-graph statistics file (again, excluding the deprecated pre-aggregated files) starts with a comment section, followed by a header line and then the actual data lines. Columns are pre-defined and specified further down below. The rule of thumb for columns is that neither the choice of parameters nor availability of data should affect the set of columns, but that only a code change can add, change, or remove a column. This rule of thumb is not yet implemented for all per-graph statistics files. +Each per-graph statistics file starts with a comment section, followed by a header line and then the actual data lines. Columns are pre-defined and specified further down below. The rule of thumb for columns is that neither the choice of parameters nor availability of data should affect the set of columns, but that only a code change can add, change, or remove a column. This rule of thumb is not yet implemented for all per-graph statistics files. Applications must not rely on the order of columns, as this order may change when columns are removed. Instead, applications should refer to columns by their name. Applications should be able to handle newly added columns and fail gracefully in case of removed columns. @@ -46,7 +46,7 @@ https://metrics.torproject.org/identifier.csv <li>May 29, 2018: Made all parameters of per-graph CSV files optional to support providing both pre-filtered and complete data sets.</li> <li>July 31, 2018: Announced pending changes to per-graph CSV files to become effective on August 15 and pre-aggregated CSV files to be removed by September 15.</li> <li>August 15, 2018: Made the first batch of changes to per-graph CSV files.</li> -<li>September 15, 2018 (scheduled): Remove all pre-aggregated CSV files.</li> +<li>September 15, 2018: Removed all pre-aggregated CSV files.</li> </ul> </div> @@ -727,588 +727,5 @@ Applications <a href="#applications" name="applications" class="anchor">#</a></h </div> -<div class="container"> -<h2>Pre-aggregated statistics files used on this website (deprecated) <a href="#stats" name="stats" class="anchor">#</a></h2> - -The remaining part of this page contains specifications and links to pre-aggregated statistics files. -These files will no longer be publicly available after September 15, 2018. -All relevant contained data will still be available via the statistics files specified above. - -</div> - -<div class="container"> -<h3>Number of relays and bridges (deprecated) <a href="#servers" name="servers" class="anchor">#</a></h3> - -<div class="bs-callout bs-callout-danger"> -<h3>Deprecation notice</h3> -The following file will no longer be publicly available after September 15, 2018. -</div> - -The following data file contains the number of running <a -href="glossary.html#relay">relays</a> and <a href="glossary.html#bridge">bridges</a> -in the network. Statistics include subsets of relays or bridges by <a -href="glossary.html#relay-flag">relay flag</a> (relays only), country code (relays -only, and only until February 2013), tor software version (relays only), -operating system (relays only), and by whether or not they are running in the -EC2 cloud (bridges only). The data file contains daily (mean) averages of relay -and bridge numbers. - -Download as <a href="stats/servers.csv">CSV file</a>. - -The statistics file contains the following columns: -<ul> - -<li>date: UTC date (YYYY-MM-DD) when relays or bridges have been listed -as running.</li> - -<li>flag: Relay flag assigned by the directory authorities. Examples are -"Exit", "Guard", "Fast", "Stable", and -"HSDir". Relays can have none, some, or all these relay flags assigned. -Relays that don't have the "Running" flag are not included in these -statistics regardless of their other flags. If this column contains the empty -string, all running relays are included, regardless of assigned flags. There -are no statistics on the number of bridges by relay flag.</li> - -<li>country: Two-letter lower-case country code as found in a GeoIP -database by resolving the relay's first onion-routing IP address, or "??" -if an IP addresses could not be resolved. If this column contains the empty -string, all running relays are included, regardless of their resolved country -code. Statistics on relays by country code are only available until January 31, -2013. There are no statistics on the number of bridges by country code.</li> - -<li>version: First three dotted numbers of the Tor software version as -reported by the relay. An example is "0.2.5". If this column contains -the empty string, all running relays are included, regardless of the Tor -software version they run. There are no statistics on the number of bridges by -Tor software version.</li> - -<li>platform: Operating system as reported by the relay. Examples are -"Linux", "Darwin" (macOS), "BSD", "Windows", and -"Other". If this column contains the empty string, all running relays -are included, regardless of the operating system they run on. There are no -statistics on the number of bridges by operating system.</li> - -<li>ec2bridge: Whether bridges are running in the EC2 cloud or not. More -precisely, bridges in the EC2 cloud running an image provided by Tor by default -set their nickname to "ec2bridger" plus 8 random hex characters. This -column either contains "t" for bridges matching this naming scheme, or -the empty string for all bridges regardless of their nickname. There are no -statistics on the number of relays running in the EC2 cloud.</li> - -<li>relays: The average number of relays matching the criteria in the -previous columns. If the values in previous columns are specific to bridges -only, this column contains the empty string.</li> - -<li>bridges: The average number of bridges matching the criteria in the -previous columns. If the values in previous columns are specific to relays -only, this column contains the empty string.</li> - -</ul> - -</div> - -<div class="container"> -<h3>Bandwidth provided and consumed by relays (deprecated) <a href="#bandwidth-deprecated" name="bandwidth-deprecated" class="anchor">#</a></h3> - -<div class="bs-callout bs-callout-danger"> -<h3>Deprecation notice</h3> -The following file will no longer be publicly available after September 15, 2018. -</div> - -The following data file contains statistics on <a -href="glossary.html#advertised-bandwidth">advertised</a> and <a -href="glossary.html#bandwidth-history">consumed bandwidth</a> of <a -href="glossary.html#relay">relays</a> in the network. Statistics on advertised -bandwidth include any kind of traffic handled by a relay, whereas statistics on -consumed bandwidth are available either for all traffic combined, or -specifically for directory traffic. Some of the statistics are available for -subsets of relays that have the "Exit" and/or the "Guard" <a -href="glossary.html#relay-flag">flag</a>. The data file contains daily (mean) -averages of bandwidth numbers. - -Download as <a href="stats/bandwidth.csv">CSV file</a>. - -The statistics file contains the following columns: -<ul> - -<li>date: UTC date (YYYY-MM-DD) that relays reported bandwidth data -for.</li> - -<li>isexit: Whether relays included in this line have the "Exit" -relay flag or not, which can be "t" or "f". If this column -contains the empty string, bandwidth data from all running relays are included, -regardless of assigned relay flags.</li> - -<li>isguard: Whether relays included in this line have the "Guard" -relay flag or not, which can be "t" or "f". If this column -contains the empty string, bandwidth data from all running relays are included, -regardless of assigned relay flags.</li> - -<li>advbw: Total advertised bandwidth in bytes per second that relays are -capable to provide.</li> - -<li>bwread: Total bandwidth in bytes per second that relays have read. -This metric includes any kind of traffic.</li> - -<li>bwwrite: Similar to bwread, but for traffic written by -relays.</li> - -<li>dirread: Bandwidth in bytes per second that directory mirrors have -read when serving directory data. -This metric is not available for subsets of relays with certain relay -flags, so that this column will contain the empty string if either isexit -or isguard is non-empty.</li> - -<li>dirwrite: Similar to dirread, but for traffic written by -directory mirrors when serving directory data.</li> - -</ul> - -</div> - -<div class="container"> -<h3>Relays and bridges supporting IPv6 (deprecated) <a href="#ipv6servers-deprecated" name="ipv6servers-deprecated" class="anchor">#</a></h3> - -<div class="bs-callout bs-callout-danger"> -<h3>Deprecation notice</h3> -The following file will no longer be publicly available after September 15, 2018. -</div> - -The following data file contains statistics on <a href="glossary.html#relay">relays</a> and <a href="glossary.html#bridge">bridges</a> supporting IPv6. -A relay can support IPv6 by announcing an IPv6 address and port for the OR protocol, which may then be confirmed as reachable by the <a href="glossary.html#directory-authority">directory authorities</a>. -It can further permit exiting to IPv6 targets. -A bridge can support IPv6 by announcing an IPv6 address and port for the OR protocol. -The <a href="glossary.html#bridge-authority">bridge authority</a> may test -reachability of IPv6 OR addresses similar to directory authorities, however it -does not explicitly list reachable IPv6 OR addresses of bridges. -Some of the statistics are available for subsets of relays that got the "Guard" and/or "Exit" <a href="glossary.html#relay-flag">relay flags</a>. -The data file contains daily (mean) averages of relay or server numbers and advertised bandwidth numbers. - -Download as <a href="stats/ipv6servers.csv">CSV file</a>. - -The statistics file contains the following columns: -<ul> - -<li>valid_after_date: UTC date (YYYY-MM-DD) when relays or bridges have -been listed as running.</li> - -<li>server: Server type, which can be either "relay" or -"bridge".</li> - -<li>guard_relay: Whether relays included in this line have the -"Guard" relay flag ("t") or not ("f"). -If this column contains the empty string, all running relays are included, -regardless of assigned relay flags. -Always the empty string for bridges.</li> - -<li>exit_relay: Whether relays included in this line have the -"Exit" relay flag ("t") or not ("f"). -If this column contains the empty string, all running relays are included, -regardless of assigned relay flags. -Always the empty string for bridges.</li> - -<li>announced_ipv6: Whether relays or bridges have announced an IPv6 -address in their server descriptor ("t") or not ("f"). -If this column contains the empty string, all running relays or bridges are -included, regardless of whether they have announced an IPv6 address.</li> - -<li>exiting_ipv6_relay: Whether relays have announced an IPv6 exit policy -other than "reject 1-65535" in their server descriptor ("t") or -not ("f"). -If this column contains the empty string, all running relays are included, -regardless of IPv6 exit policy. -Always the empty string for bridges.</li> - -<li>reachable_ipv6_relay: Whether enough directory authorities have -confirmed reachability of an IPv6 OR address announced by a relay, by including -an "a" line in the consensus ("t") or not ("f"). -If this column contains the empty string, all running relays are included, -regardless of whether their IPv6 address was found reachable. -Always the empty string for bridges.</li> - -<li>server_count_sum_avg: Number of relays or bridges matching the -criteria in the previous columns, computed as the mean value over all statuses -published on the given date.</li> - -<li>advertised_bandwidth_bytes_sum_avg: Total advertised bandwidth of all -relays matching the previous criteria, computed as the mean value over all -statuses published on the given date. -Always the empty string for bridges.</li> - -</ul> - -</div> - -<div class="container"> -<h3>Advertised bandwidth distribution and n-th fastest relays (deprecated) <a href="#advbwdist-deprecated" name="advbwdist-deprecated" class="anchor">#</a></h3> - -<div class="bs-callout bs-callout-danger"> -<h3>Deprecation notice</h3> -The following file will no longer be publicly available after September 15, 2018. -</div> - -The following data file contains statistics on the distribution of <a -href="glossary.html#advertised-bandwidth">advertised bandwidth</a> of relays in the -network. These statistics include advertised bandwidth percentiles and -advertised bandwidth values of the n-th fastest relays. All values are obtained -from advertised bandwidths of running relays in a <a -href="glossary.html#consensus">network status consensus</a>. The data file -contains daily (median) averages of percentiles and n-th largest values. - -Download as <a href="stats/advbwdist.csv">CSV file</a>. - -The statistics file contains the following columns: -<ul> - -<li>date: UTC date (YYYY-MM-DD) when relays have been listed as -running.</li> - -<li>isexit: Whether relays included in this line have the "Exit" -relay flag, which would be indicated as "t". If this column contains the -empty string, advertised bandwidths from all running relays are included, -regardless of assigned relay flags.</li> - -<li>relay: Position of the relay in an ordered list of all advertised -bandwidths, starting at 1 for the fastest relay in the network. May be the -empty string if this line contains advertised bandwidth by percentile.</li> - -<li>percentile: Advertised bandwidth percentile given in this line. May -be the empty string if this line contains advertised bandwidth by fastest -relays.</li> - -<li>advbw: Advertised bandwidth in B/s.</li> - -</ul> - -</div> - -<div class="container"> -<h3>Estimated number of clients in the Tor network (deprecated) <a href="#clients-deprecated" name="clients-deprecated" class="anchor">#</a></h3> - -<div class="bs-callout bs-callout-danger"> -<h3>Deprecation notice</h3> -The following file will no longer be publicly available after September 15, 2018. -</div> - -The following data file contains estimates on the number of <a -href="glossary.html#client">clients</a> in the network. These numbers are derived -from directory requests counted on <a -href="glossary.html#directory-authority">directory authorities</a>, <a -href="glossary.html#directory-mirror">directory mirrors</a>, and <a -href="glossary.html#bridge">bridges</a>. Statistics are available for clients -connecting directly relays and clients connecting via bridges. There are -statistics available by country (for both directly-connecting clients and -clients connecting via bridges), by transport protocol (only for clients -connecting via bridges), and by IP version (only for clients connecting via -bridges). Statistics also include predicted client numbers from past -observations, which can be used to detect censorship events. - -Download as <a href="stats/clients.csv">CSV file</a>. - -The statistics file contains the following columns: -<ul> - -<li>date: UTC date (YYYY-MM-DD) for which client numbers are -estimated.</li> - -<li>node: The node type to which clients connect first, which can be -either "relay" or "bridge".</li> - -<li>country: Two-letter lower-case country code as found in a GeoIP -database by resolving clients' IP addresses, or "??" if client IP -addresses could not be resolved. If this column contains the empty string, all -clients are included, regardless of their country code.</li> - -<li>transport: Transport name used by clients to connect to the Tor -network using bridges. Examples are "obfs2", "obfs3", -"websocket", or "<OR>" (original onion routing protocol). -If this column contains the empty string, all clients are included, regardless -of their transport. There are no statistics on the number of clients by -transport that connect to the Tor network via relays.</li> - -<li>version: IP version used by clients to connect to the Tor network -using bridges. Examples are "v4" and "v6". If this column -contains the empty string, all clients are included, regardless of their IP -version. There are no statistics on the number of clients by IP version that -connect directly to the Tor network using relays.</li> - -<li>lower: Lower number of expected clients under the assumption that -there has been no censorship event. If this column contains the empty string, -there are no expectations on the number of clients.</li> - -<li>upper: Upper number of expected clients under the assumption that -there has been no release of censorship. If this column contains the empty -string, there are no expectations on the number of clients.</li> - -<li>clients: Estimated number of clients.</li> - -<li>frac: Fraction of relays or bridges in percent that the estimate is -based on. The higher this value, the more reliable is the estimate. Values -above 50 can be considered reliable enough for most purposes, lower values -should be handled with more care.</li> - -</ul> - -</div> - -<div class="container"> -<h3>Estimated number of clients by country and transport (deprecated) <a href="#userstats-combined-deprecated" name="userstats-combined-deprecated" class="anchor">#</a></h3> - -<div class="bs-callout bs-callout-danger"> -<h3>Deprecation notice</h3> -The following file will no longer be publicly available after September 15, 2018. -</div> - -The following data file contains additional statistics on the number of <a -href="glossary.html#client">clients</a> in the network. This data file is related -to the <a href="stats.html#clients">clients-data file</a> that contains estimates -on the number of clients by country and by transport protocol. This data file -enhances these statistics by containing estimates of clients connecting to <a -href="glossary.html#bridge">bridges</a> by a given country and using a given <a -href="glossary.html#pluggable-transport">transport protocol</a>. Even though -bridges don't report a combination of clients by country and transport, it's -possible to derive lower and upper bounds from existing usage statistics. - -Download as <a href="stats/userstats-combined.csv">CSV file</a>. - -The statistics file contains the following columns: -<ul> - -<li>date: UTC date (YYYY-MM-DD) for which client numbers are -estimated.</li> - -<li>node: The node type to which clients connect first, which is always -"bridge", because relays don't report responses by transport.</li> - -<li>country: Two-letter lower-case country code as found in a GeoIP -database by resolving clients' IP addresses, or "??" if client IP -addresses could not be resolved.</li> - -<li>transport: Transport name used by clients to connect to the Tor -network using bridges. Examples are "obfs2", "obfs3", -"websocket", or "<OR>" (original onion routing -protocol).</li> - -<li>version: IP version used by clients to connect to the Tor network -using bridges. This column always contains the empty string and is only -included for compatibility reasons.</li> - -<li>frac: Fraction of relays or bridges in percent that the estimate is -based on. The higher this value, the more reliable is the estimate. Values -above 50 can be considered reliable enough for most purposes, lower values -should be handled with more care.</li> - -<li>low: Lower bound of users by country and transport, calculated as sum -over all bridges having reports for the given country and transport, that is, -the sum of M(b), where for each bridge b define M(b) := max(0, -C(b) + T(b) - S(b)) using the following definitions: C(b) is the -number of users from a given country reported by b; T(b) is the -number of users using a given transport reported by b; and S(b) is -the total numbers of users reported by b. Reasoning: If the sum C(b) -+ T(b) exceeds the total number of users from all countries and transports -S(b), there must be users from that country and transport. And if that -is not the case, 0 is the lower limit.</li> - -<li>high: Upper bound of users by country and transport, calculated as -sum over all bridges having reports for the given country and transport, that -is, the sum of m(b), where for each bridge b define -m(b):=min(C(b), T(b)) where we use the definitions from low -(above). Reasoning: there cannot be more users by country and transport than -there are users by either of the two numbers.</li> - -</ul> - -</div> - -<div class="container"> -<h3>Performance of downloading static files over Tor (deprecated) <a href="#torperf-1.1-deprecated" name="torperf-1.1-deprecated" class="anchor">#</a></h3> - -<div class="bs-callout bs-callout-danger"> -<h3>Deprecation notice</h3> -The following file will no longer be publicly available after September 15, 2018. -</div> - -The following data file contains aggregate statistics on performance when -downloading static files of different sizes over Tor. These statistics are -generated by <a href="https://github.com/robgjansen/onionperf">OnionPerf</a> and -its predecessor <a href="https://gitweb.torproject.org/torperf.git">Torperf</a>, -which both periodically fetch static files over Tor and record several -timestamps in the process. The data file contains daily medians and quartiles -as well as total numbers of requests, timeouts, and failures. Raw Onionperf and -Torperf measurement data is specified and available for download on the <a -href="/collector.html#torperf">CollecTor</a> page. - -Download as <a href="stats/torperf-1.1.csv">CSV file</a>. - -The statistics file contains the following columns: -<ul> - -<li>date: UTC date (YYYY-MM-DD) when download performance was -measured.</li> - -<li>filesize: Size of the downloaded file in bytes.</li> - -<li>source: Name of the OnionPerf or Torperf service performing -measurements. If this column contains the empty string, all measurements are -included, regardless of which service performed them.</li> - -<li>server: Either "public" if the request was made to a server on -the public internet, or "onion" if the request was made to a version 2 -onion server.</li> - -<li>q1: First quartile of time until receiving the last byte in -milliseconds.</li> - -<li>md: Median of time until receiving the last byte in -milliseconds.</li> - -<li>q3: Third quartile of time until receiving the last byte in -milliseconds.</li> - -<li>timeouts: Number of timeouts that occurred when attempting to -download the static file over Tor.</li> - -<li>failures: Number of failures that occurred when attempting to -download the static file over Tor.</li> - -<li>requests: Total number of requests made to download the static file -over Tor.</li> - -</ul> - -</div> - -<div class="container"> -<h3>Fraction of connections used uni-/bidirectionally (deprecated) <a href="#connbidirect2-deprecated" name="connbidirect2-deprecated" class="anchor">#</a></h3> - -<div class="bs-callout bs-callout-danger"> -<h3>Deprecation notice</h3> -The following file will no longer be publicly available after September 15, 2018. -</div> - -The following data file contains statistics on the fraction of direct -connections between a <a href="glossary.html#relay">relay</a> and other nodes in -the network that are used uni- or bidirectionally. Every 10 seconds, relays -determine for every direct connection whether they read and wrote less than a -threshold of 20 KiB. Connections below this threshold are excluded from the -statistics file. For the remaining connections, relays determine whether they -read/wrote at least 10 times as many bytes as they wrote/read. If so, they -classify a connection as "mostly reading" or "mostly writing", respectively. -All other connections are classified as "both reading and writing". After -classifying connections, read and write counters are reset for the next -10-second interval. The data file contains daily medians and quartiles of -reported fractions. - -Download as <a href="stats/connbidirect2.csv">CSV file</a>. - -The statistics file contains the following columns: -<ul> - -<li>date: UTC date (YYYY-MM-DD) for which statistics on -uni-/bidirectional connection usage were reported.</li> - -<li>direction: Direction of reported fraction, which can be -"read", "write", or "both" for connections classified as -"mostly reading", "mostly writing", or "both reading and writing". Connections -below the threshold have been removed from this statistics file entirely.</li> - -<li>quantile: Quantile of the reported fraction when considering all -statistics reported for this date. Examples are "0.5" for the median and -"0.25" and "0.75" for the lower and upper quartile.</li> - -<li>fraction: Fraction of connections in percent for the given date, -direction, and quantile. For each daily statistic reported by a relay, -fractions for the three directions "read", "write", and "both" sum up to exactly -100.</li> - -</ul> - -</div> - -<div class="container"> -<h3>Onion-service statistics (deprecated) <a href="#hidserv-deprecated" name="hidserv-deprecated" class="anchor">#</a></h3> - -<div class="bs-callout bs-callout-danger"> -<h3>Deprecation notice</h3> -The following file will no longer be publicly available after September 15, 2018. -</div> - -The following data file contains <a -href="glossary.html#onion-service">onion-service</a> statistics gathered by a -small subset of <a href="glossary.html#relay">relays</a> and extrapolated to -network totals. Statistics include the amount of onion-service traffic and the -number of onion-service addresses in the network per day. For more details on -the extrapolation algorithm, see <a -href="https://blog.torproject.org/blog/some-statistics-about-onions" target="_blank">this blog -post</a> and <a -href="https://research.torproject.org/techreports/extrapolating-hidserv-stats-201…" target="_blank">this -technical report</a>. - -Download as <a href="stats/hidserv.csv">CSV file</a>. - -The statistics file contains the following columns: -<ul> - -<li>date: UTC date (YYYY-MM-DD) when relays or bridges have been listed -as running.</li> - -<li>type: Type of onion-service statistic reported by relays and -extrapolated to network totals. Examples include "rend-relayed-cells" -for the number of cells on rendezvous circuits observed by rendezvous points and -"dir-onions-seen" for the number of unique .onion addresses observed by -onion-service directories.</li> - -<li>wmean: Weighted mean of extrapolated network totals.</li> - -<li>wmedian: Weighted median of extrapolated network totals.</li> - -<li>wiqm: Weighted interquartile mean of extrapolated network -totals.</li> - -<li>frac: Total network fraction of reported statistics.</li> - -<li>stats: Number of reported statistics with non-zero computed network -fraction.</li> - -</ul> - -</div> - -<div class="container"> -<h3>Requests to <code>torproject.org</code> web servers (deprecated) <a href="#webstats-deprecated" name="webstats-deprecated" class="anchor">#</a></h3> - -<div class="bs-callout bs-callout-danger"> -<h3>Deprecation notice</h3> -The following file will no longer be publicly available after September 15, 2018. -</div> - -The following data file contains aggregate statistics on requests to <code>torproject.org</code> web servers. - -Download as <a href="stats/webstats.csv">CSV file</a>. - -The statistics file contains the following columns: -<ul> -<li>log_date: UTC date (YYYY-MM-DD) when requests to <code>torproject.org</code> web servers have been logged.</li> -<li>request_type: Request type with fixed identifiers as follows: -<ul> -<li>"tbid": Tor Browser initial downloads: GET requests to all sites with resource strings <code>'%/torbrowser/%.exe'</code>, <code>'%/torbrowser/%.dmg'</code>, and <code>'%/torbrowser/%.tar.xz'</code> and response code 200.</li> -<li>"tbsd": Tor Browser signature downloads: GET requests to all sites with resource strings <code>'%/torbrowser/%.exe.asc'</code>, <code>'%/torbrowser/%.dmg.asc'</code>, and <code>'%/torbrowser/%.tar.xz.asc'</code> and response code 200.</li> -<li>"tbup": Tor Browser update pings: GET requests to all sites with resource strings <code>'%/torbrowser/update\__/%'</code> and response code 200.</li> -<li>"tbur": Tor Browser update requests: GET requests to all sites with resource strings <code>'%/torbrowser/%.mar'</code> and response code 302.</li> -<li>"tmid": Tor Messenger initial downloads: GET requests to all sites with resource strings <code>'%/tormessenger/%.exe'</code>, <code>'%/tormessenger/%.dmg'</code>, and <code>'%/tormessenger/%.tar.xz'</code> and response code 200.</li> -<li>"tmup": Tor Messenger update pings: GET requests to all sites with resource strings <code>'%/tormessenger/update\__/%'</code> and response code 200.</li> -<li>"twhph": Tor website home page hits: GET requests to sites <code>'torproject.org'</code> and <code>'www.torproject.org'</code> with resource strings <code>'/'</code> and <code>'/index%'</code> and response code 200.</li> -<li>"twdph": Tor website download page hits: GET requests to sites <code>'torproject.org'</code> and <code>'www.torproject.org'</code> with resource strings <code>'/download/download%'</code> and <code>'/projects/torbrowser.html%'</code> and response code 200.</li> -</ul> -</li> -<li>platform: Platform string, like "w" for Windows, "m" for macOS, or "l" for Linux, "o" for other platforms, and the empty string for all platforms.</li> -<li>channel: Release channel, like "r" for stable releases, "a" for alpha releases, "h" for hardened releases, and the empty string for all channels.</li> -<li>locale: Locale, like "en-US" for English (United States), "de" for German, etc., "??" for unrecognized locales, and the empty string for all locales.</li> -<li>incremental: Incremental update, with "t" for incremental updates, "f" for non-incremental (full) updates, and the empty string for all updates types.</li> -<li>count: Number of request for the given request type, platform, etc.</li> -</ul> - -</div> - <jsp:include page="bottom.jsp"/>

1 0

[metrics-web/release] Update news.json to version 268 of doc/MetricsTimeline.
by karsten＠torproject.org 26 Sep '18

26 Sep '18

commit 968fe6e1d3cb13100bfb7ef5ecd19a592199db36 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jun 6 10:02:37 2018 +0200 Update news.json to version 268 of doc/MetricsTimeline. --- src/main/resources/web/json/news.json | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/src/main/resources/web/json/news.json b/src/main/resources/web/json/news.json index b8f481e..1dd5c3e 100644 --- a/src/main/resources/web/json/news.json +++ b/src/main/resources/web/json/news.json @@ -3675,4 +3675,33 @@ "target" : "https://bugs.torproject.org/24669" } ], "unknown" : true +}, { + "start" : "2018-04-15", + "ongoing" : true, + "protocols" : [ "bridge" ], + "short_description" : "Doubling of bridge users over about 45 days", + "description" : "Doubling of bridge users over about 45 days. Relay users unaffected.", + "links" : [ { + "label" : "relay graph", + "target" : "https://metrics.torproject.org/userstats-relay-country.html?start=2018-03-0…" + }, { + "label" : "bridge graph", + "target" : "https://metrics.torproject.org/userstats-bridge-country.html?start=2018-03-…" + } ], + "unknown" : true +}, { + "start" : "2018-05-23", + "ongoing" : true, + "places" : [ "bd", "pk" ], + "protocols" : [ "bridge" ], + "short_description" : "10× increase of bridge users in Bangldesh and Pakistan", + "description" : "10× increase of bridge users in Bangldesh and Pakistan (and possibly other places). Relay users unaffected.", + "links" : [ { + "label" : "relay graph", + "target" : "https://metrics.torproject.org/userstats-relay-country.html?start=2018-04-0…" + }, { + "label" : "bridge graph", + "target" : "https://metrics.torproject.org/userstats-bridge-country.html?start=2018-04-…" + } ], + "unknown" : true } ] \ No newline at end of file

1 0

[metrics-web/release] Rename flags_p to flag_p.
by karsten＠torproject.org 26 Sep '18

26 Sep '18

commit 8c4c3e644b63cb0f7a9d5d70b93018a021e98f55 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Jun 4 10:09:49 2018 +0200 Rename flags_p to flag_p. Even though this parameter expects a list of flags, we refer to it by flag or flag_p. Fixes #26278. --- src/main/R/rserver/graphs.R | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R index 9a2b939..d5eeff5 100644 --- a/src/main/R/rserver/graphs.R +++ b/src/main/R/rserver/graphs.R @@ -574,7 +574,7 @@ write_dirbytes <- function(start_p = NULL, end_p = NULL, path_p) { write.csv(path_p, quote = FALSE, row.names = FALSE, na = "") } -prepare_relayflags <- function(start_p, end_p, flags_p) { +prepare_relayflags <- function(start_p, end_p, flag_p) { read.csv(paste(stats_dir, "servers.csv", sep = ""), colClasses = c("date" = "Date")) %>% filter(if (!is.null(start_p)) date >= as.Date(start_p) else TRUE) %>% @@ -584,12 +584,12 @@ prepare_relayflags <- function(start_p, end_p, flags_p) { filter(platform == "") %>% filter(ec2bridge == "") %>% mutate(flag = ifelse(flag == "", "Running", as.character(flag))) %>% - filter(if (!is.null(flags_p)) flag %in% flags_p else TRUE) %>% + filter(if (!is.null(flag_p)) flag %in% flag_p else TRUE) %>% select(date, flag, relays) } -plot_relayflags <- function(start_p, end_p, flags_p, path_p) { - prepare_relayflags(start_p, end_p, flags_p) %>% +plot_relayflags <- function(start_p, end_p, flag_p, path_p) { + prepare_relayflags(start_p, end_p, flag_p) %>% complete(date = full_seq(date, period = 1), flag = unique(flag)) %>% ggplot(aes(x = date, y = relays, colour = as.factor(flag))) + geom_line() + @@ -598,15 +598,15 @@ plot_relayflags <- function(start_p, end_p, flags_p, path_p) { scale_y_continuous(name = "", labels = formatter, limits = c(0, NA)) + scale_colour_manual(name = "Relay flags", values = c("#E69F00", "#56B4E9", "#009E73", "#EE6A50", "#000000", "#0072B2"), - breaks = flags_p, labels = flags_p) + + breaks = flag_p, labels = flag_p) + ggtitle("Number of relays with relay flags assigned") + labs(caption = copyright_notice) ggsave(filename = path_p, width = 8, height = 5, dpi = 150) } -write_relayflags <- function(start_p = NULL, end_p = NULL, flags_p = NULL, +write_relayflags <- function(start_p = NULL, end_p = NULL, flag_p = NULL, path_p) { - prepare_relayflags(start_p, end_p, flags_p) %>% + prepare_relayflags(start_p, end_p, flag_p) %>% mutate(flag = tolower(flag)) %>% spread(flag, relays) %>% write.csv(path_p, quote = FALSE, row.names = FALSE, na = "")

1 0

[metrics-web/release] Update news.json to version 266 of doc/MetricsTimeline.
by karsten＠torproject.org 26 Sep '18

26 Sep '18

commit 0410899b81f5107c047621bc41de4786dd6ec1f2 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Jun 4 10:28:46 2018 +0200 Update news.json to version 266 of doc/MetricsTimeline. --- src/main/resources/web/json/news.json | 36 ++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/src/main/resources/web/json/news.json b/src/main/resources/web/json/news.json index 69c3d96..b8f481e 100644 --- a/src/main/resources/web/json/news.json +++ b/src/main/resources/web/json/news.json @@ -2981,13 +2981,31 @@ } ] }, { "start" : "2018-04-16", - "ongoing" : true, + "end" : "2018-05-08", "places" : [ "ru" ], - "short_description" : "Russia tries to block Telegram; disrupts many services on Amazon and Google cloud services.", - "description" : "Russia tries to block Telegram; disrupts many services on Amazon and Google cloud services.", + "short_description" : "Russia tries to block Telegram; blocks about 18 million IP addresses including some belonging to Amazon and Google cloud services.", + "description" : "Russia tries to block Telegram; blocks about 18 million IP addresses including some belonging to Amazon and Google cloud services.", "links" : [ { "label" : "article", "target" : "https://www.bleepingcomputer.com/news/government/russia-bans-18-million-ama…" + }, { + "label" : "relay graph", + "target" : "https://metrics.torproject.org/userstats-relay-country.html?start=2018-03-0…" + }, { + "label" : "bridge graph", + "target" : "https://metrics.torproject.org/userstats-bridge-country.html?start=2018-03-…" + } ] +}, { + "start" : "2018-04-28", + "places" : [ "ru" ], + "short_description" : "Russia unblocks about 3 million IP addresses belonging to Amazon and OVH.", + "description" : "Russia unblocks about 3 million IP addresses belonging to Amazon and OVH.", + "links" : [ { + "label" : "Roskomnadzor press release", + "target" : "http://rkn.gov.ru/news/rsoc/news57288.htm" + }, { + "label" : "monitoring page", + "target" : "https://usher2.club/en/" } ] }, { "start" : "2018-04-30", @@ -3010,6 +3028,18 @@ } ] }, { "start" : "2018-05-08", + "places" : [ "ru" ], + "short_description" : "Russia unblocks about 3.7 million IP addresses belonging to Google and Digital Ocean.", + "description" : "Russia unblocks about 3.7 million IP addresses belonging to Google and Digital Ocean.", + "links" : [ { + "label" : "Roskomnadzor press release", + "target" : "http://rkn.gov.ru/news/rsoc/news57454.htm" + }, { + "label" : "monitoring page", + "target" : "https://usher2.club/en/" + } ] +}, { + "start" : "2018-05-08", "protocols" : [ "meek" ], "short_description" : "The meek-amazon bridge imposes a rate limit of 1 MB/s.", "description" : "The <a href=\"https://metrics.torproject.org/rs.html#details/F4AD82B2032EDEF6C02C5A529C42…">meek-amazon bridge</a> imposes a rate limit of 1 MB/s."

1 0

[metrics-web/release] Update news.json to version 269 of doc/MetricsTimeline.
by karsten＠torproject.org 26 Sep '18

26 Sep '18

commit bcc04460ebcff02da1884b3894ce3db285fcec04 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Jun 12 09:26:15 2018 +0200 Update news.json to version 269 of doc/MetricsTimeline. --- src/main/resources/web/json/news.json | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/src/main/resources/web/json/news.json b/src/main/resources/web/json/news.json index 1dd5c3e..f69edad 100644 --- a/src/main/resources/web/json/news.json +++ b/src/main/resources/web/json/news.json @@ -3068,6 +3068,24 @@ "target" : "https://bugs.torproject.org/26111" } ] }, { + "start" : "2018-06-09", + "protocols" : [ "meek" ], + "short_description" : "Release of Tor Browser 7.5.5, removes meek-amazon option.", + "description" : "Release of Tor Browser 7.5.5, removes meek-amazon option.", + "links" : [ { + "label" : "blog post", + "target" : "https://blog.torproject.org/tor-browser-755-released" + } ] +}, { + "start" : "2018-06-10", + "protocols" : [ "meek" ], + "short_description" : "Release of Tor Browser 8.0a8, removes meek-amazon option.", + "description" : "Release of Tor Browser 8.0a8, removes meek-amazon option.", + "links" : [ { + "label" : "blog post", + "target" : "https://blog.torproject.org/tor-browser-80a8-released" + } ] +}, { "start" : "2016-02-24", "places" : [ "tm" ], "protocols" : [ "<OR>" ],

1 0