March 2018 - tor-commits - lists.torproject.org

[collector/release] Describe file protocol for Tor web server logs.
by karsten＠torproject.org 19 Mar '18

19 Mar '18

commit 190d90a800394cbe7e494b486e6ac7eac7377783 Author: iwakeh <iwakeh(a)torproject.org> Date: Mon Feb 26 15:24:50 2018 +0000 Describe file protocol for Tor web server logs. Part of task-20234. --- src/main/resources/docs/PROTOCOL | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/src/main/resources/docs/PROTOCOL b/src/main/resources/docs/PROTOCOL index 0949f3e..c84c286 100644 --- a/src/main/resources/docs/PROTOCOL +++ b/src/main/resources/docs/PROTOCOL @@ -235,6 +235,11 @@ are derived from the valid-after dates of the referencing microdesc consensus. +4.4 'webstats' below 'recent' + + 'webstats' contains compressed log files named according to + the 'Tor web server logs' specification, section 4.3 [0]. + 5.0 The Tar-ball's Directory Structure and the Internal Structure The 'out' directory is the internal storage that is used for @@ -381,4 +386,12 @@ and below the subdirectories of 'micro' and 'consensus-microdescs'. +5.4 'webstats' below 'out' + + 'webstats' contains compressed log files structured and named according + to the 'Tor web server logs' specification, section 4.3 [0]. + +A.0 Appendix +A.1 References +[0] https://metrics.torproject.org/web-server-logs.html#n-re-assembling-log-fil…

1 0

[collector/release] Add bastet to directory authorities to download votes for.
by karsten＠torproject.org 19 Mar '18

19 Mar '18

commit 727d4e54ef450204bb9b23404ae2cc9157fea12c Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Mar 7 14:46:50 2018 +0100 Add bastet to directory authorities to download votes for. We did download bastet's votes in the past after reading the consensus. But there could have been situations when there was no consensus that we did not explicitly ask for bastet's vote. Found per chance while setting up a CollecTor instance with a webstats module. --- CHANGELOG.md | 7 +++++++ src/main/resources/collector.properties | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eb14839..d4c3bd6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,10 @@ +# Changes in version 1.?.? - 2018-??-?? + + * Minor changes + - Add bastet to default list of directory authority fingerprints to + download votes for. + + # Changes in version 1.5.0 - 2018-02-26 * Major changes diff --git a/src/main/resources/collector.properties b/src/main/resources/collector.properties index 30dda2a..2aa556d 100644 --- a/src/main/resources/collector.properties +++ b/src/main/resources/collector.properties @@ -99,7 +99,7 @@ DirectoryAuthoritiesAddresses = 128.31.0.39:9131,86.59.21.38,194.109.206.212,131 # ## Comma separated list of directory authority fingerprints to download ## votes -DirectoryAuthoritiesFingerprintsForVotes = 0232AF901C31A04EE9848595AF9BB7620D4C5B2E,14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4,23D15D965BC35114467363C165C4F724B64B4F66,49015F787433103580E3B66A1707A00E60F2D15B,D586D18309DED4CD6D57C18FDB97EFA96D330566,E8A9C45EDE6D711294FADF8E7951F4DE6CA56B58,ED03BB616EB2F60BEC80151114BB25CEF515B226,EFCBE720AB3A82B99F9E953CD5BF50F7EEFC7B97 +DirectoryAuthoritiesFingerprintsForVotes = 0232AF901C31A04EE9848595AF9BB7620D4C5B2E,14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4,23D15D965BC35114467363C165C4F724B64B4F66,27102BC123E7AF1D4741AE047E160C91ADC76B21,49015F787433103580E3B66A1707A00E60F2D15B,D586D18309DED4CD6D57C18FDB97EFA96D330566,E8A9C45EDE6D711294FADF8E7951F4DE6CA56B58,ED03BB616EB2F60BEC80151114BB25CEF515B226,EFCBE720AB3A82B99F9E953CD5BF50F7EEFC7B97 # ## Download all server descriptors from the directory authorities at most ## once a day (only if DownloadRelayDescriptors is true)

1 0

[collector/release] Bump version to 1.5.0-dev.
by karsten＠torproject.org 19 Mar '18

19 Mar '18

commit ef1dfb6d32d5bb42a95052cd44790b6c586f38c9 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Feb 26 16:24:49 2018 +0100 Bump version to 1.5.0-dev. --- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.xml b/build.xml index ff8302b..fb219fa 100644 --- a/build.xml +++ b/build.xml @@ -8,7 +8,7 @@ <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> - <property name="release.version" value="1.5.0" /> + <property name="release.version" value="1.5.0-dev" /> <property name="project-main-class" value="org.torproject.collector.Main" /> <property name="name" value="collector"/> <property name="metricslibversion" value="2.2.0" />

1 0

[collector/release] Add a failing test.
by karsten＠torproject.org 19 Mar '18

19 Mar '18

commit 2e6fa506b3c9634837e6da4c8ad3dbf477936232 Author: iwakeh <iwakeh(a)torproject.org> Date: Wed Mar 7 13:46:51 2018 +0000 Add a failing test. Making a static method easier accessible for tests. --- .../collector/webstats/SanitizeWeblogs.java | 2 +- .../collector/webstats/SanitizeWeblogsTest.java | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java b/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java index 5a270dd..8ef9dde 100644 --- a/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java +++ b/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java @@ -187,7 +187,7 @@ public class SanitizeWeblogs extends CollecTorMain { } } - private static byte[] bytesFor(String line, long times) { + public static byte[] bytesFor(String line, long times) { return Stream.of(line).limit(times) .collect(Collectors.joining("\n", "", "\n")).getBytes(); } diff --git a/src/test/java/org/torproject/collector/webstats/SanitizeWeblogsTest.java b/src/test/java/org/torproject/collector/webstats/SanitizeWeblogsTest.java new file mode 100644 index 0000000..9ad4996 --- /dev/null +++ b/src/test/java/org/torproject/collector/webstats/SanitizeWeblogsTest.java @@ -0,0 +1,20 @@ +/* Copyright 2017--2018 The Tor Project + * See LICENSE for licensing information */ + +package org.torproject.collector.webstats; + +import static org.junit.Assert.assertEquals; + +import org.junit.Test; + +public class SanitizeWeblogsTest { + + @Test + public void bytesForTest() { + String lines = "line\nline\nline\nline\nline\n" + + "line\nline\nline\nline\nline\n"; + assertEquals(lines, new String(SanitizeWeblogs.bytesFor("line", 10))); + } + +} +

1 0

[collector/release] Avoid calling bytesFor() unnecessarily.
by karsten＠torproject.org 19 Mar '18

19 Mar '18

commit 8c1bdd9a8403a0a19149b6388bbc940ee632eaae Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Sat Mar 17 10:45:55 2018 +0100 Avoid calling bytesFor() unnecessarily. Found while looking into #25522. --- src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java b/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java index df178b4..fd301b0 100644 --- a/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java +++ b/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java @@ -169,9 +169,12 @@ public class SanitizeWeblogs extends CollecTorMain { OutputStream os = FileType.XZ.outputStream(baos)) { for (Map.Entry<String, Long> entry : lines.entrySet()) { long count = entry.getValue(); - byte[] batch = bytesFor(entry.getKey(), BATCH); + byte[] batch = null; while (count > 0) { if (count > BATCH) { + if (null == batch) { + batch = bytesFor(entry.getKey(), BATCH); + } os.write(batch); count -= BATCH; } else {

1 0

[collector/release] Prepare for 1.5.1 release.
by karsten＠torproject.org 19 Mar '18

19 Mar '18

commit 60ac1936169e35d598f5bcbdad616dabd656729d Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Mar 19 15:07:21 2018 +0100 Prepare for 1.5.1 release. --- CHANGELOG.md | 5 ++++- build.xml | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d4c3bd6..94f7a96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,7 @@ -# Changes in version 1.?.? - 2018-??-?? +# Changes in version 1.5.1 - 2018-03-19 + + * Medium changes + - Add the correct number of lines to sanitized webstats files. * Minor changes - Add bastet to default list of directory authority fingerprints to diff --git a/build.xml b/build.xml index fb219fa..d7372bc 100644 --- a/build.xml +++ b/build.xml @@ -8,7 +8,7 @@ <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> - <property name="release.version" value="1.5.0-dev" /> + <property name="release.version" value="1.5.1" /> <property name="project-main-class" value="org.torproject.collector.Main" /> <property name="name" value="collector"/> <property name="metricslibversion" value="2.2.0" />

1 0

[collector/release] Add lines according to their count.
by karsten＠torproject.org 19 Mar '18

19 Mar '18

commit 2c00f28ab7eaba0310d2d96eda7f81f2a301a4a5 Author: iwakeh <iwakeh(a)torproject.org> Date: Wed Mar 7 13:46:54 2018 +0000 Add lines according to their count. Making the test from the previous commit pass and fixing task-25522. --- src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java b/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java index 8ef9dde..df178b4 100644 --- a/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java +++ b/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java @@ -47,6 +47,7 @@ import java.util.SortedSet; import java.util.StringJoiner; import java.util.TreeMap; import java.util.TreeSet; +import java.util.function.UnaryOperator; import java.util.stream.Collectors; import java.util.stream.Stream; @@ -188,7 +189,7 @@ public class SanitizeWeblogs extends CollecTorMain { } public static byte[] bytesFor(String line, long times) { - return Stream.of(line).limit(times) + return Stream.iterate(line, UnaryOperator.identity()).limit(times) .collect(Collectors.joining("\n", "", "\n")).getBytes(); }

1 0

[collector/master] Prepare for 1.5.1 release.
by karsten＠torproject.org 19 Mar '18

19 Mar '18

commit 60ac1936169e35d598f5bcbdad616dabd656729d Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Mar 19 15:07:21 2018 +0100 Prepare for 1.5.1 release. --- CHANGELOG.md | 5 ++++- build.xml | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d4c3bd6..94f7a96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,7 @@ -# Changes in version 1.?.? - 2018-??-?? +# Changes in version 1.5.1 - 2018-03-19 + + * Medium changes + - Add the correct number of lines to sanitized webstats files. * Minor changes - Add bastet to default list of directory authority fingerprints to diff --git a/build.xml b/build.xml index fb219fa..d7372bc 100644 --- a/build.xml +++ b/build.xml @@ -8,7 +8,7 @@ <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> - <property name="release.version" value="1.5.0-dev" /> + <property name="release.version" value="1.5.1" /> <property name="project-main-class" value="org.torproject.collector.Main" /> <property name="name" value="collector"/> <property name="metricslibversion" value="2.2.0" />

1 0

[metrics-web/master] Update Onionoo specification to version 5.1.
by karsten＠torproject.org 19 Mar '18

19 Mar '18

commit a34a1c8804863877489d59cee27a935bea3e2196 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Mar 13 17:40:52 2018 +0100 Update Onionoo specification to version 5.1. Also schedule version 6.0 for release in one month. --- src/main/resources/web/jsps/onionoo.jsp | 45 ++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 17 deletions(-) diff --git a/src/main/resources/web/jsps/onionoo.jsp b/src/main/resources/web/jsps/onionoo.jsp index 0f29507..28b16c6 100644 --- a/src/main/resources/web/jsps/onionoo.jsp +++ b/src/main/resources/web/jsps/onionoo.jsp @@ -295,6 +295,16 @@ Extended the "version" parameter to bridges, added a Removed the $ from fingerprints in fields "effective_family", "alleged_family", and "indirect_family" on December 20, 2017. <a href="#versions_5_0" class="anchor">#</a></li> +<li><a id="versions_5_1"></a>5.1 +Always added a relay's own fingerprint to its "effective_family" and made +nickname fields "n" in summary documents and "nickname" in details documents +required fields on March 14, 2018. +<a href="#versions_5_1" class="anchor">#</a></li> +<li><a id="versions_6_0"></a>6.0 +(scheduled, but not deployed yet!): Include all exit addresses in +"exit_addresses", regardless of whether they're used as OR addresses or not, to +be deployed after April 14, 2018. +<a href="#versions_6_0" class="anchor">#</a></li> </ul> @@ -873,13 +883,13 @@ Relay summary objects contain the following key-value pairs: <li> <a id="summary_relay_n"></a> -n +n updated <code class="typeof">string</code> -optional +required <a href="#summary_relay_n" class="anchor">#</a> Relay nickname consisting of 1–19 alphanumerical characters. -Omitted if the relay nickname is "Unnamed". +Turned into required field on March 14, 2018. </li> @@ -936,13 +946,13 @@ Bridge summary objects contain the following key-value pairs: <li> <a id="summary_bridge_n"></a> -n +n updated <code class="typeof">string</code> -optional +required <a href="#summary_bridge_n" class="anchor">#</a> Bridge nickname consisting of 1–19 alphanumerical characters. -Omitted if the bridge nickname is "Unnamed". +Turned into required field on March 14, 2018. </li> @@ -1001,13 +1011,13 @@ Relay details objects contain the following key-value pairs: <li> <a id="details_relay_nickname"></a> -nickname +nickname updated <code class="typeof">string</code> -optional +required <a href="#details_relay_nickname" class="anchor">#</a> Relay nickname consisting of 1–19 alphanumerical characters. -Omitted if the relay nickname is "Unnamed". +Turned into required field on March 14, 2018. </li> @@ -1481,7 +1491,7 @@ the relay did not report which version it runs. <li> <a id="details_relay_effective_family"></a> -effective_family +effective_family updated <code class="typeof">array of strings</code> optional <a href="#details_relay_effective_family" class="anchor">#</a> @@ -1490,8 +1500,10 @@ Array of fingerprints of relays that are in an effective, mutual family relationship with this relay. These relays are part of this relay's family and they consider this relay to be part of their family. -Omitted if empty or if descriptor containing this information cannot be -found. +Always contains the relay's own fingerprint. +Omitted if the descriptor containing this information cannot be found. +Updated to always include the relay's own fingerprint on +March 14, 2018. </li> @@ -1646,14 +1658,13 @@ Bridge details objects contain the following key-value pairs: <li> <a id="details_bridge_nickname"></a> -nickname +nickname updated <code class="typeof">string</code> -optional +required <a href="#details_bridge_nickname" class="anchor">#</a> -Bridge nickname consisting of 1–19 -alphanumerical characters. -Omitted if the bridge nickname is "Unnamed". +Bridge nickname consisting of 1–19 alphanumerical characters. +Turned into required field on March 14, 2018. </li>

1 0

[metrics-web/master] Fix specified full path of sanitized webstats.
by karsten＠torproject.org 19 Mar '18

19 Mar '18

commit acc26196bcac4dc5ce908b1667fc22c677dd97f1 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Mar 19 14:50:01 2018 +0100 Fix specified full path of sanitized webstats. Also fix path in convert script. Fixes #25525. --- src/main/resources/spec/convert.sh | 2 +- src/main/resources/spec/web-server-logs.xml | 2 +- src/main/resources/web/jsps/web-server-logs.jsp | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/main/resources/spec/convert.sh b/src/main/resources/spec/convert.sh index 0287416..95287e2 100755 --- a/src/main/resources/spec/convert.sh +++ b/src/main/resources/spec/convert.sh @@ -1,6 +1,6 @@ #!/bin/bash for specfile in "bridge-descriptors" "web-server-logs"; do saxon-xslt $specfile.xml rfc2629.xslt xml2rfc-topblock=no | \ - tidy -q | awk -f convert.awk > ../web/WEB-INF/$specfile.jsp + tidy -q | awk -f convert.awk > ../web/jsps/$specfile.jsp done diff --git a/src/main/resources/spec/web-server-logs.xml b/src/main/resources/spec/web-server-logs.xml index 13cfad7..c180f8c 100644 --- a/src/main/resources/spec/web-server-logs.xml +++ b/src/main/resources/spec/web-server-logs.xml @@ -190,7 +190,7 @@ mod_log_config module</eref>.</t> <t>Sanitized log files may additionally be sorted into directories by virtual host and date as in: <list> - <t><virtual-host>/YYYY/MM/<virtual-host>_<physical-host>_access.log_YYYYMMDD[.xz]</t> + <t><virtual-host>/YYYY/MM/DD/<virtual-host>_<physical-host>_access.log_YYYYMMDD[.xz]</t> </list> The virtual hostnames, like 'metrics.torproject.org' or 'dist.torproject.org', are more familiar to the public and were therefore diff --git a/src/main/resources/web/jsps/web-server-logs.jsp b/src/main/resources/web/jsps/web-server-logs.jsp index 5e9cc79..8832b2a 100644 --- a/src/main/resources/web/jsps/web-server-logs.jsp +++ b/src/main/resources/web/jsps/web-server-logs.jsp @@ -277,7 +277,7 @@ of the filename. by virtual host and date as in: <ul class="empty"> <li> -<virtual-host>/YYYY/MM/<virtual-host>_<physical-host>_access.log_YYYYMMDD[.xz]</li> +<virtual-host>/YYYY/MM/DD/<virtual-host>_<physical-host>_access.log_YYYYMMDD[.xz]</li> </ul> The virtual hostnames, like 'metrics.torproject.org' or 'dist.torproject.org', are more familiar to the public and were

1 0