commit c05e23cc466ee538f5fb5797d1b26dd4bd528ef0 Author: Karsten Loesing <karsten.loesing@gmx.net> Date: Tue Nov 25 14:18:53 2014 +0100 Move data files to their own subpages. --- doc/stats-spec.txt | 256 +---------------- website/etc/web.xml | 6 + .../metrics/web/graphs/GraphsSubpagesServlet.java | 12 + website/web/WEB-INF/advbwdist-data.jsp | 56 ++++ website/web/WEB-INF/bandwidth-data.jsp | 68 +++++ website/web/WEB-INF/bandwidth.jsp | 2 + website/web/WEB-INF/clients-data.jsp | 84 ++++++ website/web/WEB-INF/connbidirect-data.jsp | 58 ++++ website/web/WEB-INF/network.jsp | 1 + website/web/WEB-INF/performance.jsp | 2 + website/web/WEB-INF/servers-data.jsp | 90 ++++++ website/web/WEB-INF/stats.jsp | 292 +------------------- website/web/WEB-INF/torperf-data.jsp | 59 ++++ website/web/WEB-INF/users.jsp | 1 + 14 files changed, 452 insertions(+), 535 deletions(-) diff --git a/doc/stats-spec.txt b/doc/stats-spec.txt index a0c45c3..85bb78e 100644 --- a/doc/stats-spec.txt +++ b/doc/stats-spec.txt @@ -5,260 +5,10 @@ Tor Metrics aggregates large amounts of Tor network data and visualizes results in customizable graphs and tables. All aggregated data are also available for download, so that people can easily plot their own graphs or even develop a prettier metrics website without writing their own data -aggregation code. Data formats of aggregate statistics are specified -below. +aggregation code. -Statistics files are available for download at: +Statistics files and data formats are available at: - https://metrics.torproject.org/stats/ + https://metrics.torproject.org/ -Number of relays and bridges ----------------------------- - -Statistics file servers.csv contains the average number of relays and -bridges in the Tor network. All averages are calculated per day by -evaluating the relay and bridge lists published by the directory -authorities. Statistics include subsets of relays or bridges by relay -flag (only relays), country code (only relays, only until February 2013), -Tor software version (only relays), operating system (only relays), and -EC2 cloud (only bridges). The statistics file contains the following -columns: - - - date: UTC date (YYYY-MM-DD) when relays or bridges have been listed as - running. - - - flag: Relay flag assigned by the directory authorities. Examples are - "Exit", "Guard", "Fast", "Stable", and "HSDir". Relays can have none, - some, or all these relay flags assigned. Relays that don't have the - "Running" flag are not included in these statistics regardless of their - other flags. If this column contains the empty string, all running - relays are included, regardless of assigned flags. There are no - statistics on the number of bridges by relay flag. - - - country: Two-letter lower-case country code as found in a GeoIP - database by resolving the relay's first onion-routing IP address, or - "??" if an IP addresses could not be resolved. If this column contains - the empty string, all running relays are included, regardless of their - resolved country code. Statistics on relays by country code are only - available until January 31, 2013. There are no statistics on the - number of bridges by country code. - - - version: First three dotted numbers of the Tor software version as - reported by the relay. An example is "0.2.5". If this column contains - the empty string, all running relays are included, regardless of the - Tor software version they run. There are no statistics on the number - of bridges by Tor software version. - - - platform: Operating system as reported by the relay. Examples are - "Linux", "Darwin" (Mac OS X), "FreeBSD", "Windows", and "Other". If - this column contains the empty string, all running relays are included, - regardless of the operating system they run on. There are no - statistics on the number of bridges by operating system. - - - ec2bridge: Whether bridges are running in the EC2 cloud or not. More - precisely, bridges in the EC2 cloud running an image provided by Tor by - default set their nickname to "ec2bridger" plus 8 random hex - characters. This column either contains "t" for bridges matching this - naming scheme, or the empty string for all bridges regardless of their - nickname. There are no statistics on the number of relays running in - the EC2 cloud. - - - relays: The average number of relays matching the criteria in the - previous columns. If the values in previous columns are specific to - bridges only, this column contains the empty string. - - - bridges: The average number of bridges matching the criteria in the - previous columns. If the values in previous columns are specific to - relays only, this column contains the empty string. - - -Bandwidth provided and consumed by relays ------------------------------------------ - -Statistics on bandwidth provided and consumed by relays are contained in -file bandwidth.csv. This file contains three different bandwidth metrics: -(1) bandwidth that relays are capable to provide and bandwidth that relays -report to have consumed, either (2) for any traffic, or (3) only traffic -from serving directory data. Relays providing bandwidth statistics are -categorized by having the "Exit" and "Guard" relay flag, having both, or -not having either. The statistics file contains the following columns: - - - date: UTC date (YYYY-MM-DD) that relays reported bandwidth data for. - - - isexit: Whether relays included in this line have the "Exit" relay flag - or not, which can be "t" or "f". If this column contains the empty - string, bandwidth data from all running relays are included, regardless - of assigned relay flags. - - - isguard: Whether relays included in this line have the "Guard" relay - flag or not, which can be "t" or "f". If this column contains the - empty string, bandwidth data from all running relays are included, - regardless of assigned relay flags. - - - advbw: Total advertised bandwidth in bytes per second that relays are - capable to provide. - - - bwread: Total bandwidth in bytes per second that relays have read. - This metric includes any kind of traffic. - - - bwwrite: Similar to bwread, but for traffic written by relays. - - - dirread: Bandwidth in bytes per second that relays have read when - serving directory data. Not all relays report how many bytes they read - when serving directory data which is why this value is an estimate from - the available data. This metric is not available for subsets of relays - with certain relay flags, so that this column will contain the empty - string if either isexit or isguard is non-empty. - - - dirwrite: Similar to dirread, but for traffic written by relays when - serving directory data. - - -Advertised bandwidth distribution and n-th fastest relays ---------------------------------------------------------- - -Statistics file advbwdist.csv contains statistics on the advertised -bandwidth of relays in the network. These statistics include advertised -bandwidth percentiles and advertised bandwidth values of the n-th fastest -relays. The statistics file contains the following columns: - - - date: UTC date (YYYY-MM-DD) when relays have been listed as running. - - - isexit: Whether relays included in this line have the "Exit" relay - flag, which would be indicated as "t". If this column contains the - empty string, advertised bandwidths from all running relays are - included, regardless of assigned relay flags. - - - relay: Position of the relay in an ordered list of all advertised - bandwidths, starting at 1 for the fastest relay in the network. May be - the empty string if this line contains advertised bandwidth by - percentile. - - - percentile: Advertised bandwidth percentile given in this line. May be - the empty string if this line contains advertised bandwidth by fastest - relays. - - - advbw: Advertised bandwidth in B/s. - - -Estimated number of clients in the Tor network ----------------------------------------------- - -Statistics file clients.csv contains estimates on the number of clients in -the Tor network. These estimates are based on the number of directory -requests counted on directory mirrors and bridges. Statistics are -available for clients connecting directly to the Tor network and clients -connecting via bridges. For relays, there exist statistics on the number -of clients by country, and for bridges, statistics are available by -country, by transport, and by IP version. Statistics further include -expected client numbers from past observations which can be used to detect -censorship or release of censorship. The statistics file contains the -following columns: - - - date: UTC date (YYYY-MM-DD) for which client numbers are estimated. - - - node: The node type to which clients connect first, which can be either - "relay" or "bridge". - - - country: Two-letter lower-case country code as found in a GeoIP - database by resolving clients' IP addresses, or "??" if client IP - addresses could not be resolved. If this column contains the empty - string, all clients are included, regardless of their country code. - - - transport: Transport name used by clients to connect to the Tor network - using bridges. Examples are "obfs2", "obfs3", "websocket", or "<OR>" - (original onion routing protocol). If this column contains the empty - string, all clients are included, regardless of their transport. There - are no statistics on the number of clients by transport that connect to - the Tor network via relays. - - - version: IP version used by clients to connect to the Tor network using - bridges. Examples are "v4" and "v6". If this column contains the - empty string, all clients are included, regardless of their IP version. - There are no statistics on the number of clients by IP version that - connect directly to the Tor network using relays. - - - lower: Lower number of expected clients under the assumption that there - has been no censorship event. If this column contains the empty - string, there are no expectations on the number of clients. - - - upper: Upper number of expected clients under the assumption that there - has been no release of censorship. If this column contains the empty - string, there are no expectations on the number of clients. - - - clients: Estimated number of clients. - - - frac: Fraction of relays or bridges in percent that the estimate is - based on. The higher this value, the more reliable is the estimate. - Values above 50 can be considered reliable enough for most purposes, - lower values should be handled with more care. - - -Performance of downloading static files over Tor ------------------------------------------------- - -Statistics file torperf.csv contains aggregate statistics on download -performance over time. These statistics come from the Torperf service -that periodically downloads static files over Tor. The statistics file -contains the following columns: - - - date: UTC date (YYYY-MM-DD) when download performance was measured. - - - size: Size of the downloaded file in bytes. - - - source: Name of the Torperf service performing measurements. If this - column contains the empty string, all measurements are included, - regardless of which Torperf service performed them. Examples are - "moria", "siv", and "torperf". - - - q1: First quartile of time until receiving the last byte in - milliseconds. - - - md: Median of time until receiving the last byte in milliseconds. - - - q3: Third quartile of time until receiving the last byte in - milliseconds. - - - timeouts: Number of timeouts that occurred when attempting to download - the static file over Tor. - - - failures: Number of failures that occurred when attempting to download - the static file over Tor. - - - requests: Total number of requests made to download the static file - over Tor. - - -Fraction of connections used uni-/bidirectionally -------------------------------------------------- - -Statistics file connbidirect.csv contains statistics on the fraction of -connections that is used uni- or bidirectionally. Every 10 seconds, -relays determine for every connection whether they read and wrote less -than a threshold of 20 KiB. For the remaining connections, relays report -whether they read/wrote at least 10 times as many bytes as they -wrote/read. If so, they classify a connection as "mostly reading" or -"mostly writing," respectively. All other connections are classified as -"both reading and writing." After classifying connections, read and write -counters are reset for the next 10-second interval. Statistics are -aggregated over 24 hours. The statistics file contains the following -columns: - - - date: UTC date (YYYY-MM-DD) for which statistics on uni-/bidirectional - connection usage were reported. - - - source: Fingerprint of the relay reporting statistics. - - - below: Number of 10-second intervals of connections with less than - 20 KiB read and written data. - - - read: Number of 10-second intervals of connections with 10 times as - many read bytes as written bytes. - - - write: Number of 10-second intervals of connections with 10 times as - many written bytes as read bytes. - - - both: Number of 10-second intervals of connections with less than - 10 times as many written or read bytes as in the other direction. - diff --git a/website/etc/web.xml b/website/etc/web.xml index cc5d076..25f4e7d 100644 --- a/website/etc/web.xml +++ b/website/etc/web.xml @@ -58,6 +58,12 @@ <url-pattern>/torperf.html</url-pattern> <url-pattern>/torperf-failures.html</url-pattern> <url-pattern>/connbidirect.html</url-pattern> + <url-pattern>/servers-data.html</url-pattern> + <url-pattern>/bandwidth-data.html</url-pattern> + <url-pattern>/advbwdist-data.html</url-pattern> + <url-pattern>/clients-data.html</url-pattern> + <url-pattern>/torperf-data.html</url-pattern> + <url-pattern>/connbidirect-data.html</url-pattern> </servlet-mapping> <servlet> diff --git a/website/src/org/torproject/metrics/web/graphs/GraphsSubpagesServlet.java b/website/src/org/torproject/metrics/web/graphs/GraphsSubpagesServlet.java index 1d8a1e5..803969d 100644 --- a/website/src/org/torproject/metrics/web/graphs/GraphsSubpagesServlet.java +++ b/website/src/org/torproject/metrics/web/graphs/GraphsSubpagesServlet.java @@ -78,6 +78,18 @@ public class GraphsSubpagesServlet extends HttpServlet { "WEB-INF/torperf-failures.jsp"); this.availableGraphsSubpages.put("connbidirect.html", "WEB-INF/connbidirect.jsp"); + this.availableGraphsSubpages.put("servers-data.html", + "WEB-INF/servers-data.jsp"); + this.availableGraphsSubpages.put("bandwidth-data.html", + "WEB-INF/bandwidth-data.jsp"); + this.availableGraphsSubpages.put("advbwdist-data.html", + "WEB-INF/advbwdist-data.jsp"); + this.availableGraphsSubpages.put("clients-data.html", + "WEB-INF/clients-data.jsp"); + this.availableGraphsSubpages.put("torperf-data.html", + "WEB-INF/torperf-data.jsp"); + this.availableGraphsSubpages.put("connbidirect-data.html", + "WEB-INF/connbidirect-data.jsp"); this.availableGraphsSubpageTables = new HashMap<String, String>(); this.availableGraphsSubpageTables.put("userstats-relay-table.html", diff --git a/website/web/WEB-INF/advbwdist-data.jsp b/website/web/WEB-INF/advbwdist-data.jsp new file mode 100644 index 0000000..e6c655c --- /dev/null +++ b/website/web/WEB-INF/advbwdist-data.jsp @@ -0,0 +1,56 @@ +<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %> +<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> +<html> +<head> + <title>Tor Metrics: Advertised bandwidth distribution and n-th fastest relays</title> + <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> + <link href="/css/stylesheet-ltr.css" type="text/css" rel="stylesheet"> + <link href="/images/favicon.ico" type="image/x-icon" rel="shortcut icon"> +</head> +<body> + <div class="center"> + <%@ include file="banner.jsp"%> + <div class="main-column"> + +<h3>Tor Metrics: Advertised bandwidth distribution and n-th fastest +relays</h3> +<br> +<p>The following data file contains statistics on the distribution of +<a href="about.html#advertised-bandwidth">advertised bandwidth</a> of +relays in the network. +These statistics include advertised bandwidth percentiles and advertised +bandwidth values of the n-th fastest relays. +All values are obtained from advertised bandwidths of running relays in a +<a href="about.html#consensus">network status consensus</a>. +The data file contains daily (median) averages of percentiles and n-th +largest values.</p> + +<p><b>Download as <a href="stats/advbwdist.csv">CSV file</a>.</b></p> + +<p>The statistics file contains the following columns:</p> +<ul> +<li><b>date:</b> UTC date (YYYY-MM-DD) when relays have been listed as +running.</li> +<li><b>isexit:</b> Whether relays included in this line have the +<b>"Exit"</b> relay flag, which would be indicated as <b>"t"</b>. +If this column contains the empty string, advertised bandwidths from all +running relays are included, regardless of assigned relay flags.</li> +<li><b>relay:</b> Position of the relay in an ordered list of all +advertised bandwidths, starting at 1 for the fastest relay in the network. +May be the empty string if this line contains advertised bandwidth by +percentile.</li> +<li><b>percentile:</b> Advertised bandwidth percentile given in this line. +May be the empty string if this line contains advertised bandwidth by +fastest relays.</li> +<li><b>advbw:</b> Advertised bandwidth in B/s.</li> +</ul> + + </div> + </div> + <div class="bottom" id="bottom"> + <%@ include file="footer.jsp"%> + </div> +</body> +</html> + diff --git a/website/web/WEB-INF/bandwidth-data.jsp b/website/web/WEB-INF/bandwidth-data.jsp new file mode 100644 index 0000000..8bd2079 --- /dev/null +++ b/website/web/WEB-INF/bandwidth-data.jsp @@ -0,0 +1,68 @@ +<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %> +<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> +<html> +<head> + <title>Tor Metrics: Bandwidth provided and consumed by relays</title> + <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> + <link href="/css/stylesheet-ltr.css" type="text/css" rel="stylesheet"> + <link href="/images/favicon.ico" type="image/x-icon" rel="shortcut icon"> +</head> +<body> + <div class="center"> + <%@ include file="banner.jsp"%> + <div class="main-column"> + +<h3>Tor Metrics: Bandwidth provided and consumed by relays</h3> +<br> +<p>The following data file contains statistics on +<a href="about.html#advertised-bandwidth">advertised</a> and +<a href="about.html#bandwidth-history">consumed bandwidth</a> of +<a href="about.html#relay">relays</a> in the network. +Statistics on advertised bandwidth include any kind of traffic handled by +a relay, whereas statistics on consumed bandwidth are available either for +all traffic combined, or specifically for directory traffic. +Some of the statistics are available for subsets of relays that have the +"Exit" and/or the "Guard" <a href="about.html#relay-flag">flag</a>. +The data file contains daily (mean) averages of bandwidth numbers.</p> + +<p><b>Download as <a href="stats/bandwidth.csv">CSV file</a>.</b></p> + +<p>The statistics file contains the following columns:</p> +<ul> +<li><b>date:</b> UTC date (YYYY-MM-DD) that relays reported bandwidth data +for.</li> +<li><b>isexit:</b> Whether relays included in this line have the +<b>"Exit"</b> relay flag or not, which can be <b>"t"</b> or <b>"f"</b>. +If this column contains the empty string, bandwidth data from all running +relays are included, regardless of assigned relay flags.</li> +<li><b>isguard:</b> Whether relays included in this line have the +<b>"Guard"</b> relay flag or not, which can be <b>"t"</b> or <b>"f"</b>. +If this column contains the empty string, bandwidth data from all running +relays are included, regardless of assigned relay flags.</li> +<li><b>advbw:</b> Total advertised bandwidth in bytes per second that +relays are capable to provide.</li> +<li><b>bwread:</b> Total bandwidth in bytes per second that relays have +read. +This metric includes any kind of traffic.</li> +<li><b>bwwrite:</b> Similar to <b>bwread</b>, but for traffic written by +relays.</li> +<li><b>dirread:</b> Bandwidth in bytes per second that relays have read +when serving directory data. +Not all relays report how many bytes they read when serving directory data +which is why this value is an estimate from the available data. +This metric is not available for subsets of relays with certain relay +flags, so that this column will contain the empty string if either +<b>isexit</b> or <b>isguard</b> is non-empty.</li> +<li><b>dirwrite:</b> Similar to <b>dirread</b>, but for traffic written by +relays when serving directory data.</li> +</ul> + + </div> + </div> + <div class="bottom" id="bottom"> + <%@ include file="footer.jsp"%> + </div> +</body> +</html> + diff --git a/website/web/WEB-INF/bandwidth.jsp b/website/web/WEB-INF/bandwidth.jsp index 0e45a5f..d06d053 100644 --- a/website/web/WEB-INF/bandwidth.jsp +++ b/website/web/WEB-INF/bandwidth.jsp @@ -28,6 +28,8 @@ Sorry for any inconvenience caused by this.</p> <li><a href="dirbytes.html">Graph: Number of bytes spent on answering directory requests</a></li> <li><a href="advbwdist-perc.html">Graph: Advertised bandwidth distribution</a></li> <li><a href="advbwdist-relay.html">Graph: Advertised bandwidth of n-th fastest relays</a></li> +<li><a href="bandwidth-data.html">Data: Bandwidth provided and consumed by relays</a></li> +<li><a href="advbwdist-data.html">Data: Advertised bandwidth distribution and n-th fastest relays</a></li> </ul> <br> <hr> diff --git a/website/web/WEB-INF/clients-data.jsp b/website/web/WEB-INF/clients-data.jsp new file mode 100644 index 0000000..f315b1a --- /dev/null +++ b/website/web/WEB-INF/clients-data.jsp @@ -0,0 +1,84 @@ +<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %> +<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> +<html> +<head> + <title>Tor Metrics: Estimated number of clients in the Tor network</title> + <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> + <link href="/css/stylesheet-ltr.css" type="text/css" rel="stylesheet"> + <link href="/images/favicon.ico" type="image/x-icon" rel="shortcut icon"> +</head> +<body> + <div class="center"> + <%@ include file="banner.jsp"%> + <div class="main-column"> + +<h3>Tor Metrics: Estimated number of clients in the Tor network</h3> +<br> +<p>The following data file contains estimates on the number of +<a href="about.html#client">clients</a> in the network. +These numbers are derived from directory requests counted on +<a href="about.html#directory-authority">directory authorities</a>, +<a href="about.html#directory-mirror">directory mirrors</a>, and +<a href="about.html#bridge">bridges</a>. +Statistics are available for clients connecting directly relays and +clients connecting via bridges. +There are statistics available by country (for both directly-connecting +clients and clients connecting via bridges), by transport protocol (only +for clients connecting via bridges), and by IP version (only for clients +connecting via bridges). +Statistics also include predicted client numbers from past observations, +which can be used to detect censorship events.</p> + +<p><b>Download as <a href="stats/clients.csv">CSV file</a>.</b></p> + +<p>The statistics file contains the following columns:</p> +<ul> +<li><b>date:</b> UTC date (YYYY-MM-DD) for which client numbers are +estimated.</li> +<li><b>node:</b> The node type to which clients connect first, which can +be either <b>"relay"</b> or <b>"bridge"</b>.</li> +<li><b>country:</b> Two-letter lower-case country code as found in a GeoIP +database by resolving clients' IP addresses, or <b>"??"</b> if client IP +addresses could not be resolved. +If this column contains the empty string, all clients are included, +regardless of their country code.</li> +<li><b>transport:</b> Transport name used by clients to connect to the Tor +network using bridges. +Examples are <b>"obfs2"</b>, <b>"obfs3"</b>, <b>"websocket"</b>, or +<b>"<OR>"</b> (original onion routing protocol). +If this column contains the empty string, all clients are included, +regardless of their transport. +There are no statistics on the number of clients by transport that connect +to the Tor network via relays.</li> +<li><b>version:</b> IP version used by clients to connect to the Tor +network using bridges. +Examples are <b>"v4"</b> and <b>"v6"</b>. +If this column contains the empty string, all clients are included, +regardless of their IP version. +There are no statistics on the number of clients by IP version that connect +directly to the Tor network using relays.</li> +<li><b>lower:</b> Lower number of expected clients under the assumption +that there has been no censorship event. +If this column contains the empty string, there are no expectations on the +number of clients.</li> +<li><b>upper:</b> Upper number of expected clients under the assumption +that there has been no release of censorship. +If this column contains the empty string, there are no expectations on the +number of clients.</li> +<li><b>clients:</b> Estimated number of clients.</li> +<li><b>frac:</b> Fraction of relays or bridges in percent that the +estimate is based on. +The higher this value, the more reliable is the estimate. +Values above 50 can be considered reliable enough for most purposes, +lower values should be handled with more care.</li> +</ul> + + </div> + </div> + <div class="bottom" id="bottom"> + <%@ include file="footer.jsp"%> + </div> +</body> +</html> + diff --git a/website/web/WEB-INF/connbidirect-data.jsp b/website/web/WEB-INF/connbidirect-data.jsp new file mode 100644 index 0000000..83fee07 --- /dev/null +++ b/website/web/WEB-INF/connbidirect-data.jsp @@ -0,0 +1,58 @@ +<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %> +<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> +<html> +<head> + <title>Tor Metrics: Fraction of connections used uni-/bidirectionally</title> + <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> + <link href="/css/stylesheet-ltr.css" type="text/css" rel="stylesheet"> + <link href="/images/favicon.ico" type="image/x-icon" rel="shortcut icon"> +</head> +<body> + <div class="center"> + <%@ include file="banner.jsp"%> + <div class="main-column"> + +<h3>Tor Metrics: Fraction of connections used uni-/bidirectionally</h3> +<br> +<p>The following data file contains statistics on the fraction of direct +connections between a <a href="about.html#relay">relay</a> and other nodes +in the network that are used uni- or bidirectionally. +Every 10 seconds, relays determine for every direct connection whether +they read and wrote less than a threshold of 20 KiB. +For the remaining connections, relays determine whether they read/wrote at +least 10 times as many bytes as they wrote/read. +If so, they classify a connection as "mostly reading" or "mostly writing", +respectively. +All other connections are classified as "both reading and writing". +After classifying connections, read and write counters are reset for the +next 10-second interval. +The data file contains the absolute number of 10-second intervals per +relay, aggregated over 24-hour periods.</p> + +<p><b>Download as <a href="stats/connbidirect.csv">CSV file</a>.</b></p> + +<p>The statistics file contains the following columns:</p> +<ul> +<li><b>date:</b> UTC date (YYYY-MM-DD) for which statistics on +uni-/bidirectional connection usage were reported.</li> +<li><b>source:</b> Fingerprint of the relay reporting statistics.</li> +<li><b>below:</b> Number of 10-second intervals of connections with less +than 20 KiB read and written data.</li> +<li><b>read:</b> Number of 10-second intervals of connections with 10 +times as many read bytes as written bytes.</li> +<li><b>write:</b> Number of 10-second intervals of connections with 10 +times as many written bytes as read bytes.</li> +<li><b>both:</b> Number of 10-second intervals of connections with less +than 10 times as many written or read bytes as in the other +direction.</li> +</ul> + + </div> + </div> + <div class="bottom" id="bottom"> + <%@ include file="footer.jsp"%> + </div> +</body> +</html> + diff --git a/website/web/WEB-INF/network.jsp b/website/web/WEB-INF/network.jsp index ec18beb..83216b0 100644 --- a/website/web/WEB-INF/network.jsp +++ b/website/web/WEB-INF/network.jsp @@ -27,6 +27,7 @@ Sorry for any inconvenience caused by this.</p> <li><a href="versions.html">Graph: Relays by version</a></li> <li><a href="platforms.html">Graph: Relays by platform</a></li> <li><a href="cloudbridges.html">Graph: Tor Cloud bridges</a></li> +<li><a href="servers-data.html">Data: Number of relays and bridges</a></li> </ul> </div> diff --git a/website/web/WEB-INF/performance.jsp b/website/web/WEB-INF/performance.jsp index cc958b9..dd5d980 100644 --- a/website/web/WEB-INF/performance.jsp +++ b/website/web/WEB-INF/performance.jsp @@ -24,6 +24,8 @@ Sorry for any inconvenience caused by this.</p> <li><a href="torperf.html">Graph: Time to download files over Tor</a></li> <li><a href="torperf-failures.html">Graph: Timeouts and failures of downloading files over Tor</a></li> <li><a href="connbidirect.html">Graph: Fraction of connections used uni-/bidirectionally</a></li> +<li><a href="torperf-data.html">Data: Performance of downloading static files over Tor</a></li> +<li><a href="connbidirect-data.html">Data: Fraction of connections used uni-/bidirectionally</a></li> </ul> </div> diff --git a/website/web/WEB-INF/servers-data.jsp b/website/web/WEB-INF/servers-data.jsp new file mode 100644 index 0000000..307a77b --- /dev/null +++ b/website/web/WEB-INF/servers-data.jsp @@ -0,0 +1,90 @@ +<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %> +<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> +<html> +<head> + <title>Tor Metrics: Number of relays and bridges</title> + <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> + <link href="/css/stylesheet-ltr.css" type="text/css" rel="stylesheet"> + <link href="/images/favicon.ico" type="image/x-icon" rel="shortcut icon"> +</head> +<body> + <div class="center"> + <%@ include file="banner.jsp"%> + <div class="main-column"> + +<h3>Tor Metrics: Number of relays and bridges</h3> +<br> +<p>The following data file contains the number of running +<a href="about.html#relay">relays</a> and +<a href="about.html#bridge">bridges</a> in the network. +Statistics include subsets of relays or bridges by +<a href="about.html#relay-flag">relay flag</a> (relays only), country code +(relays only, and only until February 2013), tor software version (relays +only), operating system (relays only), and by whether or not they are +running in the EC2 cloud (bridges only). +The data file contains daily (mean) averages of relay and bridge +numbers.</p> + +<p><b>Download as <a href="stats/servers.csv">CSV file</a>.</b></p> + +<p>The statistics file contains the following columns:</p> +<ul> +<li><b>date:</b> UTC date (YYYY-MM-DD) when relays or bridges have been +listed as running.</li> +<li><b>flag:</b> Relay flag assigned by the directory authorities. +Examples are <b>"Exit"</b>, <b>"Guard"</b>, <b>"Fast"</b>, +<b>"Stable"</b>, and <b>"HSDir"</b>. +Relays can have none, some, or all these relay flags assigned. +Relays that don't have the <b>"Running"</b> flag are not included in these +statistics regardless of their other flags. +If this column contains the empty string, all running relays are included, +regardless of assigned flags. +There are no statistics on the number of bridges by relay flag.</li> +<li><b>country:</b> Two-letter lower-case country code as found in a GeoIP +database by resolving the relay's first onion-routing IP address, or +<b>"??"</b> if an IP addresses could not be resolved. +If this column contains the empty string, all running relays are included, +regardless of their resolved country code. +Statistics on relays by country code are only available until January 31, +2013. +There are no statistics on the number of bridges by country code.</li> +<li><b>version:</b> First three dotted numbers of the Tor software version +as reported by the relay. +An example is <b>"0.2.5"</b>. +If this column contains the empty string, all running relays are included, +regardless of the Tor software version they run. +There are no statistics on the number of bridges by Tor software +version.</li> +<li><b>platform:</b> Operating system as reported by the relay. +Examples are <b>"Linux"</b>, <b>"Darwin"</b> (Mac OS X), <b>"FreeBSD"</b>, +<b>"Windows"</b>, and <b>"Other"</b>. +If this column contains the empty string, all running relays are included, +regardless of the operating system they run on. +There are no statistics on the number of bridges by operating system.</li> +<li><b>ec2bridge:</b> Whether bridges are running in the EC2 cloud or not. +More precisely, bridges in the EC2 cloud running an image provided by Tor +by default set their nickname to <b>"ec2bridger"</b> plus 8 random hex +characters. +This column either contains <b>"t"</b> for bridges matching this naming +scheme, or the empty string for all bridges regardless of their nickname. +There are no statistics on the number of relays running in the EC2 +cloud.</li> +<li><b>relays:</b> The average number of relays matching the criteria in +the previous columns. +If the values in previous columns are specific to bridges only, this +column contains the empty string.</li> +<li><b>bridges:</b> The average number of bridges matching the criteria in +the previous columns. +If the values in previous columns are specific to relays only, this column +contains the empty string.</li> +</ul> + + </div> + </div> + <div class="bottom" id="bottom"> + <%@ include file="footer.jsp"%> + </div> +</body> +</html> + diff --git a/website/web/WEB-INF/stats.jsp b/website/web/WEB-INF/stats.jsp index 005235e..21a0e07 100644 --- a/website/web/WEB-INF/stats.jsp +++ b/website/web/WEB-INF/stats.jsp @@ -12,292 +12,20 @@ <div class="main-column"> <h2>Tor Metrics: Statistics</h2> <br> - -<p><font color="red"><b>Notice:</b> The specification on this page has -moved -<a href="https://gitweb.torproject.org/metrics-web.git/blob/HEAD:/doc/stats-spec.txt">here</a>. -This page will be removed after July 26, 2014.</font> -</p> - -<p>Tor Metrics aggregates large amounts of Tor network -<a href="data.html">data</a> and visualizes results in customizable -<a href="graphs.html">graphs</a> and tables. -All aggregated data are also available for download, so that people can -easily plot their own graphs or even develop a prettier metrics website -without writing their own data aggregation code. -Data formats of aggregate statistics are specified below.</p> -<hr> -<br> - -<a name="servers"></a> -<h3><a href="#servers" class="anchor">Number of relays and -bridges</a></h3> -<br> -<p>Statistics file <a href="stats/servers.csv">servers.csv</a> contains -the average number of relays and bridges in the Tor network. -All averages are calculated per day by evaluating the relay and bridge -lists published by the directory authorities. -Statistics include subsets of relays or bridges by relay flag (only -relays), country code (only relays, only until February 2013), Tor -software version (only relays), operating system (only relays), and EC2 -cloud (only bridges). -The statistics file contains the following columns:</p> - -<ul> -<li><b>date:</b> UTC date (YYYY-MM-DD) when relays or bridges have been -listed as running.</li> -<li><b>flag:</b> Relay flag assigned by the directory authorities. -Examples are <b>"Exit"</b>, <b>"Guard"</b>, <b>"Fast"</b>, -<b>"Stable"</b>, and <b>"HSDir"</b>. -Relays can have none, some, or all these relay flags assigned. -Relays that don't have the <b>"Running"</b> flag are not included in these -statistics regardless of their other flags. -If this column contains the empty string, all running relays are included, -regardless of assigned flags. -There are no statistics on the number of bridges by relay flag.</li> -<li><b>country:</b> Two-letter lower-case country code as found in a GeoIP -database by resolving the relay's first onion-routing IP address, or -<b>"??"</b> if an IP addresses could not be resolved. -If this column contains the empty string, all running relays are included, -regardless of their resolved country code. -Statistics on relays by country code are only available until January 31, -2013. -There are no statistics on the number of bridges by country code.</li> -<li><b>version:</b> First three dotted numbers of the Tor software version -as reported by the relay. -An example is <b>"0.2.5"</b>. -If this column contains the empty string, all running relays are included, -regardless of the Tor software version they run. -There are no statistics on the number of bridges by Tor software -version.</li> -<li><b>platform:</b> Operating system as reported by the relay. -Examples are <b>"Linux"</b>, <b>"Darwin"</b> (Mac OS X), <b>"FreeBSD"</b>, -<b>"Windows"</b>, and <b>"Other"</b>. -If this column contains the empty string, all running relays are included, -regardless of the operating system they run on. -There are no statistics on the number of bridges by operating system.</li> -<li><b>ec2bridge:</b> Whether bridges are running in the EC2 cloud or not. -More precisely, bridges in the EC2 cloud running an image provided by Tor -by default set their nickname to <b>"ec2bridger"</b> plus 8 random hex -characters. -This column either contains <b>"t"</b> for bridges matching this naming -scheme, or the empty string for all bridges regardless of their nickname. -There are no statistics on the number of relays running in the EC2 -cloud.</li> -<li><b>relays:</b> The average number of relays matching the criteria in -the previous columns. -If the values in previous columns are specific to bridges only, this -column contains the empty string.</li> -<li><b>bridges:</b> The average number of bridges matching the criteria in -the previous columns. -If the values in previous columns are specific to relays only, this column -contains the empty string.</li> -</ul> -<hr> -<br> - -<a name="bandwidth"></a> -<h3><a href="#bandwidth" class="anchor">Bandwidth provided and consumed by -relays</a></h3> -<br> -<p>Statistics on bandwidth provided and consumed by relays are contained -in file <a href="stats/bandwidth.csv">bandwidth.csv</a>. -This file contains three different bandwidth metrics: -(1) bandwidth that relays are capable to provide and bandwidth that relays -report to have consumed, either (2) for any traffic, or (3) only traffic -from serving directory data. -Relays providing bandwidth statistics are categorized by having the -<b>"Exit"</b> and <b>"Guard"</b> relay flag, having both, or not having -either. -The statistics file contains the following columns:</p> - -<ul> -<li><b>date:</b> UTC date (YYYY-MM-DD) that relays reported bandwidth data -for.</li> -<li><b>isexit:</b> Whether relays included in this line have the -<b>"Exit"</b> relay flag or not, which can be <b>"t"</b> or <b>"f"</b>. -If this column contains the empty string, bandwidth data from all running -relays are included, regardless of assigned relay flags.</li> -<li><b>isguard:</b> Whether relays included in this line have the -<b>"Guard"</b> relay flag or not, which can be <b>"t"</b> or <b>"f"</b>. -If this column contains the empty string, bandwidth data from all running -relays are included, regardless of assigned relay flags.</li> -<li><b>advbw:</b> Total advertised bandwidth in bytes per second that -relays are capable to provide.</li> -<li><b>bwread:</b> Total bandwidth in bytes per second that relays have -read. -This metric includes any kind of traffic.</li> -<li><b>bwwrite:</b> Similar to <b>bwread</b>, but for traffic written by -relays.</li> -<li><b>dirread:</b> Bandwidth in bytes per second that relays have read -when serving directory data. -Not all relays report how many bytes they read when serving directory data -which is why this value is an estimate from the available data. -This metric is not available for subsets of relays with certain relay -flags, so that this column will contain the empty string if either -<b>isexit</b> or <b>isguard</b> is non-empty.</li> -<li><b>dirwrite:</b> Similar to <b>dirread</b>, but for traffic written by -relays when serving directory data.</li> -</ul> -<hr> -<br> - -<a name="advbwdist"></a> -<h3><a href="#advbwdist" class="anchor">Advertised bandwidth distribution -and n-th fastest relays</a></h3> -<br> -<p>Statistics file <a href="stats/advbwdist.csv">advbwdist.csv</a> -contains statistics on the advertised bandwidth of relays in the network. -These statistics include advertised bandwidth percentiles and advertised -bandwidth values of the n-th fastest relays. -The statistics file contains the following columns:</p> +<p>The data file specifications on this page have moved to their own subpages. +This is part of an effort to make navigation on this website more intuitive. +In the future, this page will go away, and all data files can be searched more easily from the start page. +Sorry for any inconvenience caused by this.</p> <ul> -<li><b>date:</b> UTC date (YYYY-MM-DD) when relays have been listed as -running.</li> -<li><b>isexit:</b> Whether relays included in this line have the -<b>"Exit"</b> relay flag, which would be indicated as <b>"t"</b>. -If this column contains the empty string, advertised bandwidths from all -running relays are included, regardless of assigned relay flags.</li> -<li><b>relay:</b> Position of the relay in an ordered list of all -advertised bandwidths, starting at 1 for the fastest relay in the network. -May be the empty string if this line contains advertised bandwidth by -percentile.</li> -<li><b>percentile:</b> Advertised bandwidth percentile given in this line. -May be the empty string if this line contains advertised bandwidth by -fastest relays.</li> -<li><b>advbw:</b> Advertised bandwidth in B/s.</li> +<li><a href="servers-data.html">Data: Number of relays and bridges</a></li> +<li><a href="bandwidth-data.html">Data: Bandwidth provided and consumed by relays</a></li> +<li><a href="advbwdist-data.html">Data: Advertised bandwidth distribution and n-th fastest relays</a></li> +<li><a href="clients-data.html">Data: Estimated number of clients in the Tor network</a></li> +<li><a href="torperf-data.html">Data: Performance of downloading static files over Tor</a></li> +<li><a href="connbidirect-data.html">Data: Fraction of connections used uni-/bidirectionally</a></li> </ul> -<hr> -<br> - -<a name="clients"></a> -<h3><a href="#clients" class="anchor">Estimated number of clients in the -Tor network</a></h3> -<br> -<p>Statistics file <a href="stats/clients.csv">clients.csv</a> contains -estimates on the number of clients in the Tor network. -These estimates are based on the number of directory requests counted on -directory mirrors and bridges. -Statistics are available for clients connecting directly to the Tor -network and clients connecting via bridges. -For relays, there exist statistics on the number of clients by country, -and for bridges, statistics are available by country, by transport, and by -IP version. -Statistics further include expected client numbers from past observations -which can be used to detect censorship or release of censorship. -The statistics file contains the following columns:</p> - -<ul> -<li><b>date:</b> UTC date (YYYY-MM-DD) for which client numbers are -estimated.</li> -<li><b>node:</b> The node type to which clients connect first, which can -be either <b>"relay"</b> or <b>"bridge"</b>.</li> -<li><b>country:</b> Two-letter lower-case country code as found in a GeoIP -database by resolving clients' IP addresses, or <b>"??"</b> if client IP -addresses could not be resolved. -If this column contains the empty string, all clients are included, -regardless of their country code.</li> -<li><b>transport:</b> Transport name used by clients to connect to the Tor -network using bridges. -Examples are <b>"obfs2"</b>, <b>"obfs3"</b>, <b>"websocket"</b>, or -<b>"<OR>"</b> (original onion routing protocol). -If this column contains the empty string, all clients are included, -regardless of their transport. -There are no statistics on the number of clients by transport that connect -to the Tor network via relays.</li> -<li><b>version:</b> IP version used by clients to connect to the Tor -network using bridges. -Examples are <b>"v4"</b> and <b>"v6"</b>. -If this column contains the empty string, all clients are included, -regardless of their IP version. -There are no statistics on the number of clients by IP version that connect -directly to the Tor network using relays.</li> -<li><b>lower:</b> Lower number of expected clients under the assumption -that there has been no censorship event. -If this column contains the empty string, there are no expectations on the -number of clients.</li> -<li><b>upper:</b> Upper number of expected clients under the assumption -that there has been no release of censorship. -If this column contains the empty string, there are no expectations on the -number of clients.</li> -<li><b>clients:</b> Estimated number of clients.</li> -<li><b>frac:</b> Fraction of relays or bridges in percent that the -estimate is based on. -The higher this value, the more reliable is the estimate. -Values above 50 can be considered reliable enough for most purposes, -lower values should be handled with more care.</li> -</ul> -<hr> -<br> - -<a name="torperf"></a> -<h3><a href="#torperf" class="anchor">Performance of downloading static -files over Tor</a></h3> -<br> -<p>Statistics file <a href="stats/torperf.csv">torperf.csv</a> contains -aggregate statistics on download performance over time. -These statistics come from the Torperf service that periodically downloads -static files over Tor. -The statistics file contains the following columns:</p> -<ul> -<li><b>date:</b> UTC date (YYYY-MM-DD) when download performance was -measured.</li> -<li><b>size:</b> Size of the downloaded file in bytes.</li> -<li><b>source:</b> Name of the Torperf service performing measurements. -If this column contains the empty string, all measurements are included, -regardless of which Torperf service performed them. -Examples are <b>"moria"</b>, <b>"siv"</b>, and <b>"torperf"</b>.</li> -<li><b>q1:</b> First quartile of time until receiving the last byte in -milliseconds.</li> -<li><b>md:</b> Median of time until receiving the last byte in -milliseconds.</li> -<li><b>q3:</b> Third quartile of time until receiving the last byte in -milliseconds.</li> -<li><b>timeouts:</b> Number of timeouts that occurred when attempting to -download the static file over Tor.</li> -<li><b>failures:</b> Number of failures that occurred when attempting to -download the static file over Tor.</li> -<li><b>requests:</b> Total number of requests made to download the static -file over Tor.</li> -</ul> -<hr> -<br> - -<a name="connbidirect"></a> -<h3><a href="#connbidirect" class="anchor">Fraction of connections used -uni-/bidirectionally</a></h3> -<br> -<p>Statistics file <a href="stats/connbidirect.csv">connbidirect.csv</a> -contains statistics on the fraction of connections that is used uni- or -bidirectionally. -Every 10 seconds, relays determine for every connection whether they read -and wrote less than a threshold of 20 KiB. -For the remaining connections, relays report whether they read/wrote at -least 10 times as many bytes as they wrote/read. -If so, they classify a connection as "mostly reading" or "mostly writing," -respectively. -All other connections are classified as "both reading and writing." -After classifying connections, read and write counters are reset for the -next 10-second interval. -Statistics are aggregated over 24 hours. -The statistics file contains the following columns:</p> - -<ul> -<li><b>date:</b> UTC date (YYYY-MM-DD) for which statistics on -uni-/bidirectional connection usage were reported.</li> -<li><b>source:</b> Fingerprint of the relay reporting statistics.</li> -<li><b>below:</b> Number of 10-second intervals of connections with less -than 20 KiB read and written data.</li> -<li><b>read:</b> Number of 10-second intervals of connections with 10 -times as many read bytes as written bytes.</li> -<li><b>write:</b> Number of 10-second intervals of connections with 10 -times as many written bytes as read bytes.</li> -<li><b>both:</b> Number of 10-second intervals of connections with less -than 10 times as many written or read bytes as in the other -direction.</li> -</ul> </div> </div> <div class="bottom" id="bottom"> diff --git a/website/web/WEB-INF/torperf-data.jsp b/website/web/WEB-INF/torperf-data.jsp new file mode 100644 index 0000000..6b78270 --- /dev/null +++ b/website/web/WEB-INF/torperf-data.jsp @@ -0,0 +1,59 @@ +<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %> +<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> +<html> +<head> + <title>Tor Metrics: Performance of downloading static files over Tor</title> + <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> + <link href="/css/stylesheet-ltr.css" type="text/css" rel="stylesheet"> + <link href="/images/favicon.ico" type="image/x-icon" rel="shortcut icon"> +</head> +<body> + <div class="center"> + <%@ include file="banner.jsp"%> + <div class="main-column"> + +<h3>Tor Metrics: Performance of downloading static files over Tor</h3> +<br> +<p>The following data file contains aggregate statistics on performance +when downloading static files of different sizes over Tor. +These statistics are generated by the +<a href="https://gitweb.torproject.org/torperf.git">Torperf</a> tool, +which periodically fetches static files over Tor and records several +timestamps in the process. +The data file contains daily medians and quartiles as well as total +numbers of requests, timeouts, and failures.</p> + +<p><b>Download as <a href="stats/torperf.csv">CSV file</a>.</b></p> + +<p>The statistics file contains the following columns:</p> +<ul> +<li><b>date:</b> UTC date (YYYY-MM-DD) when download performance was +measured.</li> +<li><b>size:</b> Size of the downloaded file in bytes.</li> +<li><b>source:</b> Name of the Torperf service performing measurements. +If this column contains the empty string, all measurements are included, +regardless of which Torperf service performed them. +Examples are <b>"moria"</b>, <b>"siv"</b>, and <b>"torperf"</b>.</li> +<li><b>q1:</b> First quartile of time until receiving the last byte in +milliseconds.</li> +<li><b>md:</b> Median of time until receiving the last byte in +milliseconds.</li> +<li><b>q3:</b> Third quartile of time until receiving the last byte in +milliseconds.</li> +<li><b>timeouts:</b> Number of timeouts that occurred when attempting to +download the static file over Tor.</li> +<li><b>failures:</b> Number of failures that occurred when attempting to +download the static file over Tor.</li> +<li><b>requests:</b> Total number of requests made to download the static +file over Tor.</li> +</ul> + + </div> + </div> + <div class="bottom" id="bottom"> + <%@ include file="footer.jsp"%> + </div> +</body> +</html> + diff --git a/website/web/WEB-INF/users.jsp b/website/web/WEB-INF/users.jsp index 79ffd9d..015eea9 100644 --- a/website/web/WEB-INF/users.jsp +++ b/website/web/WEB-INF/users.jsp @@ -31,6 +31,7 @@ Sorry for any inconvenience caused by this.</p> <li><a href="userstats-bridge-transport.html">Graph: Bridge users by transport</a></li> <li><a href="userstats-bridge-version.html">Graph: Bridge users by IP version</a></li> <li><a href="oxford-anonymous-internet.html">Link: Tor users as percentage of larger Internet population</a></li> +<li><a href="clients-data.html">Data: Estimated number of clients in the Tor network</a></li> </ul> </div>