May 2018 - tor-commits - lists.torproject.org

[metrics-web/release] Fix typo on Statistics page.
by karsten＠torproject.org 30 May '18

30 May '18

commit 215f3639c245a4ed200158526bf03b3d07f25677 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Apr 23 12:41:29 2018 +0200 Fix typo on Statistics page. --- src/main/resources/web/jsps/stats.jsp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/resources/web/jsps/stats.jsp b/src/main/resources/web/jsps/stats.jsp index c224a06..2674ba0 100644 --- a/src/main/resources/web/jsps/stats.jsp +++ b/src/main/resources/web/jsps/stats.jsp @@ -452,7 +452,7 @@ uni-/bidirectional connection usage were reported.</li> <li>direction: Direction of reported fraction, which can be "read", "write", or "both" for connections classified as -"mostly reading", "mostly writing", or "both reading as writing". Connections +"mostly reading", "mostly writing", or "both reading and writing". Connections below the threshold have been removed from this statistics file entirely.</li> <li>quantile: Quantile of the reported fraction when considering all

1 0

[metrics-web/release] Comment out OOI link.
by karsten＠torproject.org 30 May '18

30 May '18

commit 1ac237e113434ca5beb4d0dc88a496a3f0a1c8bf Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Apr 26 14:39:51 2018 +0200 Comment out OOI link. This is a temporary measure to address #25926. --- src/main/resources/web/json/metrics.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/resources/web/json/metrics.json b/src/main/resources/web/json/metrics.json index 0739b88..7f04cb6 100644 --- a/src/main/resources/web/json/metrics.json +++ b/src/main/resources/web/json/metrics.json @@ -279,7 +279,7 @@ "id": "oxford-anonymous-internet", "title": "“The anonymous Internet”", "type": "Link", - "description": "The Oxford Internet Institute made a cartogram visualization of Tor users as compared to the overall Internet population. They used the average number of Tor <a href=\"glossary.html#client\">users</a> per country from August 2012 to August 2013 and put it in relation to total Internet users per country. More details and conclusions can be found on the <a href=\"http://geography.oii.ox.ac.uk/?page=tor\" target=\"_blank\">Information Geographies website at the Oxford Internet Institute</a>.<a href=\"http://geography.oii.ox.ac.uk/?page=tor\"><img src=\"images/oxford-anonymous-internet.png\" alt=\"The anonymous Internet\"></a>" + "description": "The Oxford Internet Institute made a cartogram visualization of Tor users as compared to the overall Internet population. They used the average number of Tor <a href=\"glossary.html#client\">users</a> per country from August 2012 to August 2013 and put it in relation to total Internet users per country. <img src=\"images/oxford-anonymous-internet.png\" alt=\"The anonymous Internet\">" }, { "id": "torperf",

1 0

[metrics-web/release] Update news.json to version 257 of doc/MetricsTimeline.
by karsten＠torproject.org 30 May '18

30 May '18

commit d41b8eb4165cfac4c9e2b415f58603152977ddb0 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon May 7 10:03:44 2018 +0200 Update news.json to version 257 of doc/MetricsTimeline. --- src/main/resources/web/json/news.json | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/src/main/resources/web/json/news.json b/src/main/resources/web/json/news.json index 32b0630..77d94aa 100644 --- a/src/main/resources/web/json/news.json +++ b/src/main/resources/web/json/news.json @@ -4569,6 +4569,25 @@ ] }, { + "start": "2018-03-28", + "ongoing": true, + "places": [ + "td" + ], + "shortDescription": "Chad blocks social messaging apps.", + "description": "Chad blocks social messaging apps.", + "links": [ + { + "label": "article", + "target": "https://qz.com/1247234/chad-has-blocked-social-messaging-apps-bbc-amid-poli…" + }, + { + "label": "graph", + "target": "https://metrics.torproject.org/userstats-relay-country.html?start=2018-02-0…" + } + ] + }, + { "start": "2018-04-02", "protocols": [ "snowflake"

1 0

[metrics-web/release] Reflect that webstats.tp.o is gone.
by karsten＠torproject.org 30 May '18

30 May '18

commit af48f3aeaf632089aeb50d0440cd1455e8aed0a3 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon May 7 10:10:50 2018 +0200 Reflect that webstats.tp.o is gone. --- src/main/resources/web/json/categories.json | 2 +- src/main/resources/web/jsps/sources.jsp | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/src/main/resources/web/json/categories.json b/src/main/resources/web/json/categories.json index 06efe8f..b323c79 100644 --- a/src/main/resources/web/json/categories.json +++ b/src/main/resources/web/json/categories.json @@ -81,7 +81,7 @@ "icon": "fa-download", "header": "Applications", "summary": "How many Tor applications, like Tor Browser, have been downloaded or updated.", - "description": "The following application statistics are based on the analysis of web server requests. All data comes from <a href=\"https://webstats.torproject.org/\" target=\"_blank\"><code>webstats.torproject.org</code></a> which collects logs from <code>torproject.org</code> web servers and provides them as a stripped-down version of Apache's \"combined\" log format without IP addresses, log times, HTTP parameters, referers, and user agent strings.", + "description": "The following application statistics are based on the analysis of web server requests. All data comes from <a href=\"/web-server-logs.html\">Tor web server logs</a> which are provided in a stripped-down version of Apache's \"combined\" log format without IP addresses, log times, HTTP parameters, referers, and user agent strings.", "metrics": [ "webstats-tb", "webstats-tb-platform", diff --git a/src/main/resources/web/jsps/sources.jsp b/src/main/resources/web/jsps/sources.jsp index c956208..f8f9e07 100644 --- a/src/main/resources/web/jsps/sources.jsp +++ b/src/main/resources/web/jsps/sources.jsp @@ -37,7 +37,6 @@ <li><a href="https://github.com/robgjansen/onionperf" target="_blank">OnionPerf</a> measures the performance of onion services.</li> <li><a href="https://ooni.torproject.org/" target="_blank">OONI</a> detects censorship, surveillance, and traffic manipulation on the internet.</li> <li><a href="https://gitweb.torproject.org/user/phw/sybilhunter.git/" target="_blank">Sybilhunter</a> attempts to detect Sybil attacks on the Tor network.</li> - <li><a href="https://webstats.torproject.org/" target="_blank">Webstats</a> collects logs from <code>torproject.org</code> web servers and provides them as a stripped-down version of Apache's "combined" log format without IP addresses, log times, HTTP parameters, referers, and user agent strings.</li> </ul> </div>

1 0

[metrics-web/release] Remove the PostgreSQL jar from our war file.
by karsten＠torproject.org 30 May '18

30 May '18

commit 5c8eec9d0ea3b3b39dfe99400622d8ad769c5559 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed May 16 14:41:56 2018 +0200 Remove the PostgreSQL jar from our war file. Another part of #25699. --- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.xml b/build.xml index bdb908b..81fa363 100644 --- a/build.xml +++ b/build.xml @@ -47,7 +47,6 @@ <include name="gson-2.4.jar" /> <include name="logback-core-1.1.9.jar" /> <include name="logback-classic-1.1.9.jar" /> - <include name="postgresql-9.4.1212.jar"/> <include name="REngine.jar"/> <include name="Rserve.jar"/> <include name="slf4j-api-1.7.22.jar"/> @@ -57,6 +56,7 @@ <patternset refid="common" /> <include name="metrics-lib-${metricslibversion}.jar"/> <include name="commons-compress-1.13.jar"/> + <include name="postgresql-9.4.1212.jar"/> <include name="servlet-api-3.1.jar"/> <include name="xz-1.6.jar"/> </patternset>

1 0

[metrics-web/release] Say that the Tor Messenger graph will soon disappear.
by karsten＠torproject.org 30 May '18

30 May '18

commit 0060ed5155f015d481f719c59aadb22114cd4dbf Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Fri May 11 17:37:33 2018 +0200 Say that the Tor Messenger graph will soon disappear. --- src/main/resources/web/json/metrics.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/resources/web/json/metrics.json b/src/main/resources/web/json/metrics.json index 7f04cb6..57b7c9c 100644 --- a/src/main/resources/web/json/metrics.json +++ b/src/main/resources/web/json/metrics.json @@ -408,7 +408,7 @@ "id": "webstats-tm", "title": "Tor Messenger downloads and updates", "type": "Graph", - "description": "This graph shows absolute numbers of requests to Tor's web servers related to Tor Messenger. Initial downloads are requests made by the user to download a Tor Messenger executable from the Tor website. Update pings are requests made by Tor Messenger to check whether a newer version is available.", + "description": "This graph shows absolute numbers of requests to Tor's web servers related to Tor Messenger. Initial downloads are requests made by the user to download a Tor Messenger executable from the Tor website. Update pings are requests made by Tor Messenger to check whether a newer version is available.Note that <a href=\"https://blog.torproject.org/sunsetting-tor-messenger\">Tor Messenger has been sunset</a> on April 2, 2018. This graph is soon going to disappear, too.", "function": "webstats_tm", "parameters": [ "start",

1 0

[metrics-web/release] Remove inconsistency between bandwidth history graphs.
by karsten＠torproject.org 30 May '18

30 May '18

commit 0eacbbecefecde7346b64629f3ade1403785213b Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed May 16 21:29:34 2018 +0200 Remove inconsistency between bandwidth history graphs. The number of bytes in all bandwidth histories differed from the sum of those numbers by Exit/Guard flag. The difference came from relays that reported bandwidth histories but that the directory authorities did not list as running. With this commit we stop producing bandwidth numbers that are not broken down by Exit/Guard flag, and we sum up those columns for the bandwidth graph. Implements #26015. --- src/main/R/rserver/graphs.R | 9 ++-- src/main/sql/legacy/tordir.sql | 112 +++-------------------------------------- 2 files changed, 13 insertions(+), 108 deletions(-) diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R index 79d5600..2ac2756 100644 --- a/src/main/R/rserver/graphs.R +++ b/src/main/R/rserver/graphs.R @@ -451,10 +451,11 @@ write_platforms <- function(start, end, path) { prepare_bandwidth <- function(start, end) { read.csv(paste(stats_dir, "bandwidth.csv", sep = ""), colClasses = c("date" = "Date")) %>% - filter(date >= as.Date(start), date <= as.Date(end), isexit == "", - isguard == "") %>% - mutate(advbw = advbw * 8 / 1e9) %>% - mutate(bwhist = (bwread + bwwrite) * 8 / 2e9) %>% + filter(date >= as.Date(start), date <= as.Date(end), isexit != "", + isguard != "") %>% + group_by(date) %>% + summarize(advbw = sum(advbw) * 8 / 1e9, + bwhist = sum(bwread + bwwrite) * 8 / 2e9) %>% select(date, advbw, bwhist) } diff --git a/src/main/sql/legacy/tordir.sql b/src/main/sql/legacy/tordir.sql index bd91ab6..16e7166 100644 --- a/src/main/sql/legacy/tordir.sql +++ b/src/main/sql/legacy/tordir.sql @@ -151,29 +151,6 @@ CREATE TABLE relay_versions ( CONSTRAINT relay_versions_pkey PRIMARY KEY(date, version) ); --- TABLE total_bandwidth --- Contains information for the whole network's total bandwidth which is --- used in the bandwidth graphs. -CREATE TABLE total_bandwidth ( - date DATE NOT NULL, - bwavg BIGINT NOT NULL, - bwburst BIGINT NOT NULL, - bwobserved BIGINT NOT NULL, - bwadvertised BIGINT NOT NULL, - CONSTRAINT total_bandwidth_pkey PRIMARY KEY(date) -); - --- TABLE total_bwhist --- Contains the total number of read/written and the number of dir bytes --- read/written by all relays in the network on a given day. The dir bytes --- are an estimate based on the subset of relays that count dir bytes. -CREATE TABLE total_bwhist ( - date DATE NOT NULL, - read BIGINT, - written BIGINT, - CONSTRAINT total_bwhist_pkey PRIMARY KEY(date) -); - -- TABLE bandwidth_flags CREATE TABLE bandwidth_flags ( date DATE NOT NULL, @@ -473,64 +450,6 @@ CREATE OR REPLACE FUNCTION refresh_relay_versions() RETURNS INTEGER AS $$ END; $$ LANGUAGE plpgsql; --- FUNCTION refresh_total_bandwidth() --- This keeps the table total_bandwidth up-to-date when necessary. -CREATE OR REPLACE FUNCTION refresh_total_bandwidth() RETURNS INTEGER AS $$ - DECLARE - min_date TIMESTAMP WITHOUT TIME ZONE; - max_date TIMESTAMP WITHOUT TIME ZONE; - BEGIN - - min_date := (SELECT MIN(date) FROM updates); - max_date := (SELECT MAX(date) + 1 FROM updates); - - DELETE FROM total_bandwidth - WHERE date IN (SELECT date FROM updates); - - EXECUTE ' - INSERT INTO total_bandwidth - (bwavg, bwburst, bwobserved, bwadvertised, date) - SELECT (SUM(bandwidthavg) - / relay_statuses_per_day.count)::BIGINT AS bwavg, - (SUM(bandwidthburst) - / relay_statuses_per_day.count)::BIGINT AS bwburst, - (SUM(bandwidthobserved) - / relay_statuses_per_day.count)::BIGINT AS bwobserved, - (SUM(LEAST(bandwidthavg, bandwidthobserved)) - / relay_statuses_per_day.count)::BIGINT AS bwadvertised, - DATE(validafter) - FROM descriptor RIGHT JOIN statusentry - ON descriptor.descriptor = statusentry.descriptor - JOIN relay_statuses_per_day - ON DATE(validafter) = relay_statuses_per_day.date - WHERE isrunning = TRUE - AND validafter >= ''' || min_date || ''' - AND validafter < ''' || max_date || ''' - AND DATE(validafter) IN (SELECT date FROM updates) - AND relay_statuses_per_day.date >= ''' || min_date || ''' - AND relay_statuses_per_day.date < ''' || max_date || ''' - AND DATE(relay_statuses_per_day.date) IN - (SELECT date FROM updates) - GROUP BY DATE(validafter), relay_statuses_per_day.count'; - - RETURN 1; - END; -$$ LANGUAGE plpgsql; - -CREATE OR REPLACE FUNCTION refresh_total_bwhist() RETURNS INTEGER AS $$ - BEGIN - DELETE FROM total_bwhist WHERE date IN (SELECT date FROM updates); - INSERT INTO total_bwhist (date, read, written) - SELECT date, SUM(read_sum) AS read, SUM(written_sum) AS written - FROM bwhist - WHERE date >= (SELECT MIN(date) FROM updates) - AND date <= (SELECT MAX(date) FROM updates) - AND date IN (SELECT date FROM updates) - GROUP BY date; - RETURN 1; - END; -$$ LANGUAGE plpgsql; - CREATE OR REPLACE FUNCTION refresh_bandwidth_flags() RETURNS INTEGER AS $$ DECLARE min_date TIMESTAMP WITHOUT TIME ZONE; @@ -693,12 +612,8 @@ CREATE OR REPLACE FUNCTION refresh_all() RETURNS INTEGER AS $$ RAISE NOTICE '% Refreshing relay versions.', timeofday(); PERFORM refresh_relay_versions(); RAISE NOTICE '% Refreshing total relay bandwidth.', timeofday(); - PERFORM refresh_total_bandwidth(); - RAISE NOTICE '% Refreshing relay bandwidth history.', timeofday(); - PERFORM refresh_total_bwhist(); - RAISE NOTICE '% Refreshing total relay bandwidth by flags.', timeofday(); PERFORM refresh_bandwidth_flags(); - RAISE NOTICE '% Refreshing bandwidth history by flags.', timeofday(); + RAISE NOTICE '% Refreshing bandwidth history.', timeofday(); PERFORM refresh_bwhist_flags(); RAISE NOTICE '% Refreshing user statistics.', timeofday(); PERFORM refresh_user_stats(); @@ -799,23 +714,12 @@ CREATE VIEW stats_bandwidth AS WHERE COALESCE(bandwidth_flags.date, bwhist_flags.date) < current_date - 2) UNION ALL - (SELECT COALESCE(total_bandwidth.date, total_bwhist.date, u.date) - AS date, NULL AS isexit, NULL AS isguard, - total_bandwidth.bwadvertised AS advbw, - CASE WHEN total_bwhist.read IS NOT NULL - THEN total_bwhist.read / 86400 END AS bwread, - CASE WHEN total_bwhist.written IS NOT NULL - THEN total_bwhist.written / 86400 END AS bwwrite, - CASE WHEN u.date IS NOT NULL - THEN FLOOR(CAST(u.dr AS NUMERIC) / CAST(86400 AS NUMERIC)) END AS dirread, - CASE WHEN u.date IS NOT NULL - THEN FLOOR(CAST(u.dw AS NUMERIC) / CAST(86400 AS NUMERIC)) END AS dirwrite - FROM total_bandwidth FULL OUTER JOIN total_bwhist - ON total_bandwidth.date = total_bwhist.date - FULL OUTER JOIN (SELECT * FROM user_stats WHERE country = 'zy' - AND bwp / bwd <= 3) u - ON COALESCE(total_bandwidth.date, total_bwhist.date) = u.date - WHERE COALESCE(total_bandwidth.date, total_bwhist.date, u.date) < - current_date - 2) + (SELECT date, NULL AS isexit, NULL AS isguard, NULL AS advbw, + NULL AS bwread, NULL AS bwwrite, + FLOOR(CAST(dr AS NUMERIC) / CAST(86400 AS NUMERIC)) AS dirread, + FLOOR(CAST(dw AS NUMERIC) / CAST(86400 AS NUMERIC)) AS dirwrite + FROM user_stats + WHERE country = 'zy' + AND date < current_date - 2) ORDER BY date, isexit, isguard;

1 0

[metrics-web/release] Update news.json to version 262 of doc/MetricsTimeline.
by karsten＠torproject.org 30 May '18

30 May '18

commit 8318bfcd1eb9bf5ee357c46fbd580c62d69707b7 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed May 16 09:06:06 2018 +0200 Update news.json to version 262 of doc/MetricsTimeline. --- src/main/resources/web/json/news.json | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/src/main/resources/web/json/news.json b/src/main/resources/web/json/news.json index d333e2a..554f094 100644 --- a/src/main/resources/web/json/news.json +++ b/src/main/resources/web/json/news.json @@ -4658,6 +4658,14 @@ ] }, { + "start": "2018-05-08", + "protocols": [ + "meek" + ], + "shortDescription": "The meek-amazon bridge imposes a rate limit of 1 MB/s.", + "description": "The <a href=\"https://metrics.torproject.org/rs.html#details/F4AD82B2032EDEF6C02C5A529C42…">meek-amazon bridge</a> imposes a rate limit of 1 MB/s." + }, + { "start": "2018-05-09", "protocols": [ "snowflake" @@ -4676,6 +4684,24 @@ ] }, { + "start": "2018-05-14", + "protocols": [ + "meek" + ], + "shortDescription": "Disabled the d2cly7j4zqgua7.cloudfront.net CDN endpoint for meek-amazon.", + "description": "Disabled the d2cly7j4zqgua7.cloudfront.net CDN endpoint for meek-amazon.", + "links": [ + { + "label": "Tor Browser ticket", + "target": "https://bugs.torproject.org/26098" + }, + { + "label": "Orbot ticket", + "target": "https://bugs.torproject.org/26111" + } + ] + }, + { "start": "2016-02-24", "places": [ "tm"

1 0

[metrics-web/release] Throw out Apache Commons Lang 2 in favor of 3.
by karsten＠torproject.org 30 May '18

30 May '18

commit f4c98122da6b21f93e00b149724e33f689cfc580 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed May 16 12:42:26 2018 +0200 Throw out Apache Commons Lang 2 in favor of 3. Part of #25699. --- build.xml | 1 - src/main/java/org/torproject/metrics/stats/ipv6servers/Parser.java | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/build.xml b/build.xml index 9b29452..bdb908b 100644 --- a/build.xml +++ b/build.xml @@ -44,7 +44,6 @@ <patternset id="common" > <include name="commons-codec-1.10.jar"/> <include name="commons-lang3-3.5.jar"/> - <include name="commons-lang-2.6.jar"/> <include name="gson-2.4.jar" /> <include name="logback-core-1.1.9.jar" /> <include name="logback-classic-1.1.9.jar" /> diff --git a/src/main/java/org/torproject/metrics/stats/ipv6servers/Parser.java b/src/main/java/org/torproject/metrics/stats/ipv6servers/Parser.java index 9959b93..cfbdd6c 100644 --- a/src/main/java/org/torproject/metrics/stats/ipv6servers/Parser.java +++ b/src/main/java/org/torproject/metrics/stats/ipv6servers/Parser.java @@ -9,7 +9,7 @@ import org.torproject.descriptor.RelayNetworkStatusConsensus; import org.torproject.descriptor.RelayServerDescriptor; import org.torproject.descriptor.ServerDescriptor; -import org.apache.commons.lang.StringUtils; +import org.apache.commons.lang3.StringUtils; import java.time.Instant; import java.time.ZoneId;

1 0

[metrics-web/release] Fix torperf graph for single source.
by karsten＠torproject.org 30 May '18

30 May '18

commit de1799779b3eb1cf0f94ff3c2ff5fb5e9df6daab Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon May 7 10:57:24 2018 +0200 Fix torperf graph for single source. --- src/main/R/rserver/graphs.R | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R index 332204b..b6eb221 100644 --- a/src/main/R/rserver/graphs.R +++ b/src/main/R/rserver/graphs.R @@ -589,7 +589,7 @@ plot_torperf <- function(start, end, source, server, filesize, path) { filesize_val <- ifelse(filesize == "50kb", 50 * 1024, ifelse(filesize == "1mb", 1024 * 1024, 5 * 1024 * 1024)) t <- read.csv(paste(stats_dir, "torperf-1.1.csv", sep = ""), - colClasses = c("date" = "Date")) + colClasses = c("date" = "Date", "source" = "character")) known_sources <- c("all", unique(t[t$source != "", "source"])) colours <- data.frame(source = known_sources, colour = brewer.pal(length(known_sources), "Paired"),

1 0