February 2020 - tor-commits - lists.torproject.org

[torspec/master] Prop 312: Clarify OS hostname API behaviour
by teor＠torproject.org 11 Feb '20

11 Feb '20

commit e759c0ec4e6433e6d8cb010dda72b92b7477ae5e Author: teor <teor(a)torproject.org> Date: Tue Feb 11 13:38:17 2020 +1000 Prop 312: Clarify OS hostname API behaviour Part of 33049. --- proposals/312-relay-auto-ipv6-addr.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/312-relay-auto-ipv6-addr.txt b/proposals/312-relay-auto-ipv6-addr.txt index 659e5a1..98c86c1 100644 --- a/proposals/312-relay-auto-ipv6-addr.txt +++ b/proposals/312-relay-auto-ipv6-addr.txt @@ -414,7 +414,8 @@ Ticket: #33073 and IPv6 addresses for: * the Address torrc option (see section 3.2.1), and * the local hostname. - However, OS APIs typically only return a single hostname. + However, OS APIs typically only return a single hostname. (Rather than a + separate hostname for IPv4 and IPv6.) For security reasons, directory authorities only use addresses that are explicitly configured in their torrc. Since hostname lookups may use DNS,

1 0

[tor-browser-build/master] Release preparations for 9.5a5
by sysrqb＠torproject.org 11 Feb '20

11 Feb '20

commit d3059ceb6288874b682076a5bf4aa1dff76511cd Author: Matthew Finkel <sysrqb(a)torproject.org> Date: Mon Feb 10 01:16:23 2020 +0000 Release preparations for 9.5a5 Versions bump and Changelog update --- projects/firefox-langpacks/config | 2 +- projects/firefox-locale-bundle/config | 2 +- projects/firefox/config | 2 +- .../tor-browser/Bundle-Data/Docs/ChangeLog.txt | 29 ++++++++++++++++++++++ projects/tor-browser/config | 4 +-- projects/tor-launcher/config | 2 +- projects/tor/config | 2 +- rbm.conf | 6 ++--- 8 files changed, 39 insertions(+), 10 deletions(-) diff --git a/projects/firefox-langpacks/config b/projects/firefox-langpacks/config index 34d6fad..17a707a 100644 --- a/projects/firefox-langpacks/config +++ b/projects/firefox-langpacks/config @@ -4,7 +4,7 @@ filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/buil var: ff_version: '[% pc("firefox", "var/firefox_version") %]' - ff_build: build1 + ff_build: build2 input_filename: 'dl-langpack-[% c("var/ff_arch") %]-[% c("version") %]' targets: diff --git a/projects/firefox-locale-bundle/config b/projects/firefox-locale-bundle/config index 2f4997f..2ea6db6 100644 --- a/projects/firefox-locale-bundle/config +++ b/projects/firefox-locale-bundle/config @@ -5,4 +5,4 @@ filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.gz' var: use_container: 0 ff_version: '[% pc("firefox", "var/firefox_version") %]' - ff_build: build1 + ff_build: build2 diff --git a/projects/firefox/config b/projects/firefox/config index f780084..845f9c6 100644 --- a/projects/firefox/config +++ b/projects/firefox/config @@ -8,7 +8,7 @@ git_submodule: 1 gpg_keyring: torbutton.gpg var: - firefox_platform_version: 68.4.1 + firefox_platform_version: 68.5.0 firefox_version: '[% c("var/firefox_platform_version") %]esr' torbrowser_branch: 9.5 branding_directory: '[% IF c("var/android") %]mobile/android[% ELSE %]browser[% END %]/branding/alpha' diff --git a/projects/tor-browser/Bundle-Data/Docs/ChangeLog.txt b/projects/tor-browser/Bundle-Data/Docs/ChangeLog.txt index ad3cc9d..16acbbf 100644 --- a/projects/tor-browser/Bundle-Data/Docs/ChangeLog.txt +++ b/projects/tor-browser/Bundle-Data/Docs/ChangeLog.txt @@ -1,3 +1,32 @@ +Tor Browser 9.5a5 -- February 12 2020 + * All Platforms + * Update Firefox to 68.5.0esr + * Bump NoScript to 11.0.13 + * Translations update + * Bug 30237: Control port module improvements for v3 client authentication + * Bug 32891: Add new default bridges + * Bug 31395: Remove inline script in aboutTor.xhtml + * Bug 27268: Preferences clean-up in Torbutton code + * Bug 32470: Backport fix for bug 1590538 + * Bug 32414: Make Services.search.addEngine obey FPI + * Bug 32948: Make referer behavior consistent regardless of private browing mode status + * Bug 461204: Improve the random number generator for the boundaries in multipart/form-data + * Android + * Bug 30767: Custom obfs4 bridge does not work on Tor Browser for Android + * Windows + OS X + Linux + * Update Tor to 0.4.3.1-alpha + * Update Tor Launcher to 0.2.21.1 + * Translations update + * Bug 30237: Add v3 onion services client authentication prompt + * Bug 32870: Update version of pion-webrtc + * Bug 32767: Remove Disconnect search + * Bug 30237: Add v3 onion services client authentication prompt + * Build System + * Linux + * Bug 32976: Build and bundle geckodriver + * OS X + * Bug 33200: Fix permissions on bookmarks.html + Tor Browser 9.5a4 -- January 10 2020 * All Platforms * Update Firefox to 68.4.1esr diff --git a/projects/tor-browser/config b/projects/tor-browser/config index d3e8e61..7fd61a9 100644 --- a/projects/tor-browser/config +++ b/projects/tor-browser/config @@ -74,9 +74,9 @@ input_files: enable: '[% c("var/snowflake") %]' - filename: Bundle-Data enable: '[% ! c("var/android") %]' - - URL: https://addons.cdn.mozilla.net/user-media/addons/722/noscript_security_suit… + - URL: https://addons.cdn.mozilla.net/user-media/addons/722/noscript_security_suit… name: noscript - sha256sum: 1b0daa12e1b8fbc8e30e7bdb5689b12774aaa9f883cb3e9ad57943a883ce65b5 + sha256sum: ced718d9d9e719547db392ad85d24a7ad5c6b54cc0f93f9c57f2fcb44bb4de56 - filename: 'RelativeLink/start-tor-browser.desktop' enable: '[% c("var/linux") %]' - filename: 'RelativeLink/execdesktop' diff --git a/projects/tor-launcher/config b/projects/tor-launcher/config index 874ad15..fec0612 100644 --- a/projects/tor-launcher/config +++ b/projects/tor-launcher/config @@ -1,5 +1,5 @@ # vim: filetype=yaml sw=2 -version: 0.2.21 +version: 0.2.21.1 git_url: https://git.torproject.org/tor-launcher.git git_hash: '[% c("version") %]' gpg_keyring: torbutton.gpg diff --git a/projects/tor/config b/projects/tor/config index c1c41a6..9b5ba3d 100644 --- a/projects/tor/config +++ b/projects/tor/config @@ -1,6 +1,6 @@ # vim: filetype=yaml sw=2 filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %]' -version: 0.4.3.1-alpha +version: 0.4.3.2-alpha git_hash: 'tor-[% c("version") %]' git_url: https://git.torproject.org/tor.git git_submodule: 1 diff --git a/rbm.conf b/rbm.conf index 77c6e4a..32b0575 100644 --- a/rbm.conf +++ b/rbm.conf @@ -24,10 +24,10 @@ buildconf: git_signtag_opt: '-s' var: - torbrowser_version: '9.5a4' - torbrowser_build: 'build2' + torbrowser_version: '9.5a5' + torbrowser_build: 'build1' torbrowser_incremental_from: - - 9.5a3 + - 9.5a4 project_name: tor-browser multi_lingual: 0 build_mar: 1

1 0

[tor-browser-build/master] Bug 30767: Support custom PT bridges on Android
by sysrqb＠torproject.org 11 Feb '20

11 Feb '20

commit 3ce57ded27baa63dfff074e9a10a62365dc3bc25 Author: Matthew Finkel <sysrqb(a)torproject.org> Date: Fri Feb 7 02:18:21 2020 +0000 Bug 30767: Support custom PT bridges on Android --- ...ustom-obfs4-bridge-does-not-work-on-Tor-B.patch | 146 +++++++++++++++++++++ projects/tor-onion-proxy-library/build | 1 + projects/tor-onion-proxy-library/config | 1 + 3 files changed, 148 insertions(+) diff --git a/projects/tor-onion-proxy-library/0001-Bug-30767-Custom-obfs4-bridge-does-not-work-on-Tor-B.patch b/projects/tor-onion-proxy-library/0001-Bug-30767-Custom-obfs4-bridge-does-not-work-on-Tor-B.patch new file mode 100644 index 0000000..2d1bc51 --- /dev/null +++ b/projects/tor-onion-proxy-library/0001-Bug-30767-Custom-obfs4-bridge-does-not-work-on-Tor-B.patch @@ -0,0 +1,146 @@ +From 0a482a749fd770827d3d6c71debb112a07e7fae3 Mon Sep 17 00:00:00 2001 +From: sisbell <shane.isbell(a)gmail.com> +Date: Thu, 14 Nov 2019 13:43:09 -0800 +Subject: [PATCH] Bug 30767: Custom obfs4 bridge does not work on Tor Browser + for Android + +TOPL fixes #134 +--- + .../thali/toronionproxy/TorConfigBuilder.java | 63 ++++++++----------- + 1 file changed, 26 insertions(+), 37 deletions(-) + +diff --git a/universal/src/main/java/com/msopentech/thali/toronionproxy/TorConfigBuilder.java b/universal/src/main/java/com/msopentech/thali/toronionproxy/TorConfigBuilder.java +index e324d9f..da9a6a2 100644 +--- a/universal/src/main/java/com/msopentech/thali/toronionproxy/TorConfigBuilder.java ++++ b/universal/src/main/java/com/msopentech/thali/toronionproxy/TorConfigBuilder.java +@@ -99,25 +99,32 @@ public final class TorConfigBuilder { + return this; + } + ++ @SettingsConfig ++ public TorConfigBuilder bridgesFromSettings() { ++ try { ++ addBridgesFromResources(); ++ } catch (IOException e) { ++ e.printStackTrace(); ++ } ++ return this; ++ } ++ + public TorConfigBuilder configurePluggableTransportsFromSettings(File pluggableTransportClient) throws IOException { +- List<String> supportedBridges = settings.getListOfSupportedBridges(); +- if (pluggableTransportClient == null || !settings.hasBridges() || supportedBridges.size() < 1) { ++ if (pluggableTransportClient == null) { + return this; + } + +- if (!pluggableTransportClient.exists() || !pluggableTransportClient.canExecute()) { ++ if (!pluggableTransportClient.exists()) { + throw new IOException("Bridge binary does not exist: " + pluggableTransportClient + .getCanonicalPath()); + } + +- if (supportedBridges.contains("obfs3") || supportedBridges.contains("obfs4")) { +- transportPluginObfs(pluggableTransportClient.getCanonicalPath()); +- } +- if (supportedBridges.contains("meek")) { +- transportPluginMeek(pluggableTransportClient.getCanonicalPath()); ++ if (!pluggableTransportClient.canExecute()) { ++ throw new IOException("Bridge binary is not executable: " + pluggableTransportClient ++ .getCanonicalPath()); + } +- String type = supportedBridges.contains("meek") ? "meek_lite" : "obfs4"; +- addBridgesFromResources(type, 2); ++ ++ transportPlugin(pluggableTransportClient.getCanonicalPath()); + return this; + } + +@@ -471,14 +478,8 @@ public final class TorConfigBuilder { + return transPort(settings.transPort()); + } + +- public TorConfigBuilder transportPluginMeek(String clientPath) { +- buffer.append("ClientTransportPlugin meek_lite exec ").append(clientPath).append('\n'); +- return this; +- } +- +- public TorConfigBuilder transportPluginObfs(String clientPath) { +- buffer.append("ClientTransportPlugin obfs3 exec ").append(clientPath).append('\n'); +- buffer.append("ClientTransportPlugin obfs4 exec ").append(clientPath).append('\n'); ++ public TorConfigBuilder transportPlugin(String clientPath) { ++ buffer.append("ClientTransportPlugin meek_lite,obfs3,obfs4 exec ").append(clientPath).append('\n'); + return this; + } + +@@ -489,7 +490,7 @@ public final class TorConfigBuilder { + + @SettingsConfig + public TorConfigBuilder useBridgesFromSettings() { +- return !settings.hasBridges() ? dontUseBridges() : this; ++ return settings.hasBridges() ? useBridges() : this; + } + + public TorConfigBuilder virtualAddressNetwork(String address) { +@@ -518,12 +519,12 @@ public final class TorConfigBuilder { + * </code> + * + */ +- TorConfigBuilder addBridgesFromResources(String type, int maxBridges) throws IOException { ++ TorConfigBuilder addBridgesFromResources() throws IOException { + if(settings.hasBridges()) { + InputStream bridgesStream = context.getInstaller().openBridgesStream(); + int formatType = bridgesStream.read(); +- if(formatType == 0) { +- addBridges(bridgesStream, type, maxBridges); ++ if (formatType == 0) { ++ addBridges(bridgesStream); + } else { + addCustomBridges(bridgesStream); + } +@@ -534,23 +535,14 @@ public final class TorConfigBuilder { + /** + * Add bridges from bridges.txt file. + */ +- private void addBridges(InputStream input, String bridgeType, int maxBridges) { +- if (input == null || isNullOrEmpty(bridgeType) || maxBridges < 1) { ++ private void addBridges(InputStream input) { ++ if (input == null) { + return; + } +- boolean hasAddedBridge = false; + List<Bridge> bridges = readBridgesFromStream(input); +- Collections.shuffle(bridges, new Random(System.nanoTime())); +- int bridgeCount = 0; + for (Bridge b : bridges) { +- if (b.type.equals(bridgeType)) { +- bridge(b.type, b.config); +- hasAddedBridge = true; +- if (++bridgeCount > maxBridges) +- break; +- } ++ bridge(b.type, b.config); + } +- if(hasAddedBridge) useBridges(); + } + + /** +@@ -560,15 +552,12 @@ public final class TorConfigBuilder { + if (input == null) { + return; + } +- boolean hasAddedBridge = false; + List<Bridge> bridges = readCustomBridgesFromStream(input); + for (Bridge b : bridges) { + if (b.type.equals("custom")) { + bridgeCustom(b.config); +- hasAddedBridge = true; + } + } +- if(hasAddedBridge) useBridges(); + } + + private static List<Bridge> readBridgesFromStream(InputStream input) { +-- +2.20.1 + diff --git a/projects/tor-onion-proxy-library/build b/projects/tor-onion-proxy-library/build index 0f2d976..9a8b31e 100644 --- a/projects/tor-onion-proxy-library/build +++ b/projects/tor-onion-proxy-library/build @@ -21,6 +21,7 @@ cd /var/tmp/build/[% project %]-[% c('version') %] # build will pull down the correct android tool versions patch -p1 < $rootdir/gradle.patch patch -p1 < $rootdir/canceldormant.patch +patch -p1 < $rootdir/0001-Bug-30767-Custom-obfs4-bridge-does-not-work-on-Tor-B.patch # Extract obfs4proxy from TorBrowser/Tor/PluggableTransports/obfs4proxy tar --strip-components=4 -xf $rootdir/[% c('input_files_by_name/obfs4') %] diff --git a/projects/tor-onion-proxy-library/config b/projects/tor-onion-proxy-library/config index ff3432c..d76949d 100644 --- a/projects/tor-onion-proxy-library/config +++ b/projects/tor-onion-proxy-library/config @@ -32,3 +32,4 @@ input_files: exec: '[% INCLUDE "fetch-gradle-dependencies" %]' - filename: gradle.patch - filename: canceldormant.patch + - filename: 0001-Bug-30767-Custom-obfs4-bridge-does-not-work-on-Tor-B.patch

1 0

[tor-browser-build/master] Bug 33200: Fix permissions on bookmarks.html
by sysrqb＠torproject.org 11 Feb '20

11 Feb '20

commit 241e805dbbefcb0860bf32a38b63fcd7edf71a68 Author: Nicolas Vigier <boklm(a)torproject.org> Date: Sun Feb 9 23:50:01 2020 +0100 Bug 33200: Fix permissions on bookmarks.html Fix a build reproducibility issue caused by different permissions on bookmarks.html. --- projects/tor-browser/build | 1 + 1 file changed, 1 insertion(+) diff --git a/projects/tor-browser/build b/projects/tor-browser/build index 2633ddc..b9732e8 100644 --- a/projects/tor-browser/build +++ b/projects/tor-browser/build @@ -170,6 +170,7 @@ cp defaults/preferences/000-tor-browser.js $rootdir mkdir -p chrome/en-US/locale/browser cp -p $rootdir/bookmarks.html chrome/en-US/locale/browser/ [% c("var/touch") %] chrome/en-US/locale/browser/bookmarks.html + chmod 600 chrome/en-US/locale/browser/bookmarks.html zip -Xm omni.ja chrome/en-US/locale/browser/bookmarks.html rm -rf chrome [% END %]

1 0

[torspec/master] Prop 311 & 312: Some changes affect IPv6 stats
by teor＠torproject.org 11 Feb '20

11 Feb '20

commit 01105c25434a7ead121d7f37adbb3a5afcd4f8a1 Author: teor <teor(a)torproject.org> Date: Mon Feb 10 12:45:49 2020 +1000 Prop 311 & 312: Some changes affect IPv6 stats When a change increases an IPv6 statistic, mention the impact in proposals 311 and 312. Related to proposal 313, which covers IPv6 statistics. Part of 33159. --- proposals/311-relay-ipv6-reachability.txt | 6 ++++++ proposals/312-relay-auto-ipv6-addr.txt | 5 +++++ 2 files changed, 11 insertions(+) diff --git a/proposals/311-relay-ipv6-reachability.txt b/proposals/311-relay-ipv6-reachability.txt index 85f089a..978dc9b 100644 --- a/proposals/311-relay-ipv6-reachability.txt +++ b/proposals/311-relay-ipv6-reachability.txt @@ -294,6 +294,12 @@ Ticket: #24404 The testing relay will confirm that test circuits can extend to both its IPv4 and IPv6 ORPorts. + Checking IPv6 ORPort reachability will create extra IPv6 connections on the + tor network. (See [Proposal 313: Relay IPv6 Statistics].) It won't directly + create much extra traffic, because reachability circuits don't send many + cells. But some client circuits may use the IPv6 connections created by + relay reachability self-tests. + 4.2.1. Selecting the Final Extending Relay IPv6 ORPort reachability checks require an IPv6 extend-capable relay as diff --git a/proposals/312-relay-auto-ipv6-addr.txt b/proposals/312-relay-auto-ipv6-addr.txt index 08fbcde..8914cb7 100644 --- a/proposals/312-relay-auto-ipv6-addr.txt +++ b/proposals/312-relay-auto-ipv6-addr.txt @@ -487,6 +487,11 @@ Ticket: #33073 IPv6 address, tor should use that address for reachability checks. If the reachability checks succeed, tor should use that address in its descriptor. + Doing relay directory fetches over IPv6 will create extra IPv6 connections + and IPv6 bandwidth on the tor network. (See + [Proposal 313: Relay IPv6 Statistics].) In addition, some client circuits + may use the IPv6 connections created by relay directory fetches. + 3.2.6. Disabling IPv6 Address Resolution Relays (and bridges) that have a reachable IPv6 address, but that address

1 0

[torspec/master] Prop 311: Add Optional IPv6 Bandwidth Self-Tests
by teor＠torproject.org 11 Feb '20

11 Feb '20

commit 58000de4faea96761d9e87dcd7cb1fcef63e961e Author: teor <teor(a)torproject.org> Date: Mon Feb 10 12:46:12 2020 +1000 Prop 311: Add Optional IPv6 Bandwidth Self-Tests Add an optional change that makes relays do bandwidth self-tests over IPv6. This change increases the connection and bandwidth IPv6 statistics. Related to proposal 313, which covers IPv6 statistics. Related to 33159. --- proposals/311-relay-ipv6-reachability.txt | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/proposals/311-relay-ipv6-reachability.txt b/proposals/311-relay-ipv6-reachability.txt index 978dc9b..e4c9414 100644 --- a/proposals/311-relay-ipv6-reachability.txt +++ b/proposals/311-relay-ipv6-reachability.txt @@ -573,6 +573,27 @@ Ticket: #24404 network makes it more likely that an existing connection can be used for the final hop of a relay IPv6 ORPort reachability check. +4.4.5. Relay Bandwidth Self-Tests Over IPv4 and IPv6 + + In this proposal, we only make relays (and bridges) use IPv6 for their + reachability self-tests. + + We propose this optional change, to improve the accuracy of relay (and + bridge) bandwidth self-tests: + + Relays (and bridges) perform bandwidth self-tests over IPv4 and IPv6. + + If we implement good abstractions for relay self-tests, then this change + will not need much extra code. + + If we implement IPv6 extends for all relay circuits (see section 4.4.4), + then this change will effectively be redundant. + + Doing relay bandwidth self-tests over IPv6 will create extra IPv6 + connections and IPv6 bandwidth on the tor network. (See + [Proposal 313: Relay IPv6 Statistics].) In addition, some client circuits + may use the IPv6 connections created by relay bandwidth self-tests. + 4.5. Alternate Reachability Designs We briefly mention some potential reachability designs, and the reasons that

1 0

[torspec/master] Prop 312: Clarify IPv6 Directory Failures
by teor＠torproject.org 11 Feb '20

11 Feb '20

commit aa53246772db5c0246ac71fb5c97acfac5854318 Author: teor <teor(a)torproject.org> Date: Mon Feb 10 12:49:43 2020 +1000 Prop 312: Clarify IPv6 Directory Failures Part of 33073. --- proposals/312-relay-auto-ipv6-addr.txt | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/312-relay-auto-ipv6-addr.txt b/proposals/312-relay-auto-ipv6-addr.txt index 8914cb7..c384d4c 100644 --- a/proposals/312-relay-auto-ipv6-addr.txt +++ b/proposals/312-relay-auto-ipv6-addr.txt @@ -465,7 +465,9 @@ Ticket: #33073 * explicitly configured with an IPv6 address, or * a publicly routable, reachable IPv6 address is discovered in an earlier step, - tor should start issuing IPv6 directory failure logs at warning level. + tor should start issuing IPv6 directory failure logs at warning level. Tor + may also record these directory failures as remote relay failures. (Rather + than ignoring them, as described in the previous paragraph.) (Alternately, tor could stop doing IPv6 directory requests entirely. But we prefer designs where all relays behave in a similar way, regardless of their

1 0

[torspec/master] Prop 311 & 312: Remove TODOs on Prop 313 link
by teor＠torproject.org 11 Feb '20

11 Feb '20

commit 638a9276d8cb622278efbe6ececf8985bef6de65 Author: teor <teor(a)torproject.org> Date: Mon Feb 10 12:50:25 2020 +1000 Prop 311 & 312: Remove TODOs on Prop 313 link Part of 33159. --- proposals/311-relay-ipv6-reachability.txt | 1 - proposals/312-relay-auto-ipv6-addr.txt | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/311-relay-ipv6-reachability.txt b/proposals/311-relay-ipv6-reachability.txt index e4c9414..9f536d3 100644 --- a/proposals/311-relay-ipv6-reachability.txt +++ b/proposals/311-relay-ipv6-reachability.txt @@ -770,7 +770,6 @@ References: [Proposal 313: Relay IPv6 Statistics]: https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-sta… - (TODO) [Relay Search]: https://metrics.torproject.org/rs.html diff --git a/proposals/312-relay-auto-ipv6-addr.txt b/proposals/312-relay-auto-ipv6-addr.txt index c384d4c..5228c6c 100644 --- a/proposals/312-relay-auto-ipv6-addr.txt +++ b/proposals/312-relay-auto-ipv6-addr.txt @@ -1518,7 +1518,7 @@ References: https://gitweb.torproject.org/torspec.git/tree/proposals/311-relay-ipv6-rea… [Proposal 313: Relay IPv6 Statistics]: - https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-sta… (TODO) + https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-sta… [RFC 4941: Privacy Extensions for IPv6]: https://tools.ietf.org/html/rfc4941

1 0

[torspec/master] Prop 313: Relay IPv6 Statistics - Initial Draft
by teor＠torproject.org 11 Feb '20

11 Feb '20

commit 27de9d175b363f2ea744692d705319be06aed3f0 Author: teor <teor(a)torproject.org> Date: Mon Feb 10 16:11:14 2020 +1000 Prop 313: Relay IPv6 Statistics - Initial Draft Related to tickets: 33159 (proposal), 33051 and 33052 (implementation). --- proposals/313-relay-ipv6-stats.txt | 399 +++++++++++++++++++++++++++++++++++++ 1 file changed, 399 insertions(+) diff --git a/proposals/313-relay-ipv6-stats.txt b/proposals/313-relay-ipv6-stats.txt new file mode 100644 index 0000000..2452918 --- /dev/null +++ b/proposals/313-relay-ipv6-stats.txt @@ -0,0 +1,399 @@ +Filename: 313-relay-ipv6-stats.txt +Title: Relay IPv6 Statistics +Author: teor +Created: 10-February-2020 +Status: Draft +Ticket: #33159 + +0. Abstract + + We propose that Tor relays (and bridges) should log the number of relays in + the consensus that support IPv6 extends, and IPv6 client connections. + + We also propose that Tor relays (and bridges) should collect statistics on + IPv6 connections and consumed bandwidth. Like tor's existing connection + and consumed bandwidth statistics, these new IPv6 statistics will be + published in each relay's extra-info descriptor. + +1. Introduction + + Tor relays (and bridges) can accept IPv6 client connections via their + ORPort. But current versions of tor need to have an explicitly configured + IPv6 address (see [Proposal 312: Relay Auto IPv6 Address]), and they don't + perform IPv6 reachability self-checks (see + [Proposal 311: Relay IPv6 Reachability]). + + As we implement these new IPv6 features in tor, we want to monitor their + impact on the IPv6 connections and bandwidth in the tor network. + + Tor developers also need to know how many relays support these new IPv6 + features, so they can test tor's IPv6 reachability checks. (In particular, + see section 4.3.1 in [Proposal 311: Relay IPv6 Reachability]: Refusing to + Publish the Descriptor.) + +2. Scope + + This proposal modifies Tor's behaviour as follows: + + Relays, bridges, and directory authorities log the number of relays that + support IPv6 clients, and IPv6 relay reachability checks. They also log the + corresponding consensus weight fractions. + + As an optional change, tor clients may also log this information. + + Relays, bridges, and directory authorities collect statistics on: + * IPv6 connections, and + * IPv6 consumed bandwidth. + The design of these statistics will be based on tor's existing connection + and consumed bandwidth statistics. + + Tor's existing consumed bandwidth statistics truncate their totals to the + nearest kilobyte. The existing connection statistics do not perform any + binning. + + We do not proposed to add any extra noise or binning to these statistics. + Instead, we expect to leave these changes until we have a consistent + privacy-preserving statistics framwework for tor. As an example of this + kind of framework, see + [Proposal 288: Privacy-Preserving Stats with Privcount (Shamir version)]. + + We avoid: + * splitting connection statistics into clients and relays, and + * collecting circuit statistics. + These statistics are more sensitive, so we want to implement + privacy-preserving statistics, before we consider adding them. + + Throughout this proposal, "relays" includes directory authorities, except + where they are specifically excluded. "relays" does not include bridges, + except where they are specifically included. (The first mention of "relays" + in each section should specifically exclude or include these other roles.) + + Tor clients do not collect any statistics for public reporting. Therefore, + clients are out of scope in this proposal. (Except for some optional changes + to client logs, where they log the number of IPv6 relays in the consensus). + + When this proposal describes Tor's current behaviour, it covers all + supported Tor versions (0.3.5.7 to 0.4.2.5), as of January 2020, except + where another version is specifically mentioned. + +3. Logging IPv6 Relays in the Consensus + + We propose that relays (and bridges) log: + * the number of relays, and + * the consensus weight fraction of relays, + in the consensus that: + * have an IPv6 ORPort, + * support IPv6 reachability checks, and + * support IPv6 clients. + + In order to test these changes, and provide easy access to these + statistics, we propose implementing a script that: + * downloads a consensus, and + * calculates and reports these statistics. + + As well as the statistics listed above, this script should also report the + following relay statistic: + * support IPv6 reachability checks and IPv6 clients. + + The following consensus weight fractions should divide by the total + consensus weight: + * have an IPv6 ORPort (all relays have an IPv4 ORPort), and + * support IPv6 reachability checks (all relays support IPv4 reachability). + + The following consensus weight fractions should divide by the + "usable Guard" consensus weight: + * support IPv6 clients, and + * support IPv6 reachability checks and IPv6 clients. + + "Usable Guards" have the Guard flag, but do not have the Exit flag. If the + Guard also has the BadExit flag, the Exit flag should be ignored. + + We propose that these logs happen whenever tor: + * receives a consensus from a directory server, or + * loads a live, valid, cached consensus from disk. + + As an optional change, tor clients may also log this information. Some of + this information is not directly relevant to clients, but these logs may + help developers (and users). + +4. Collecting IPv6 Consumed Bandwidth Statistics + + We propose that relays (and bridges) collect IPv6 consumed bandwidth + statistics. + + To minimise development and testing effort, we propose re-using the existing + "bw_array" code in rephist.c. + + In particular, tor currently counts these bandwidth statistics: + * read, + * write, + * dir_read, and + * dir_write. + + We propose adding the following bandwidth statistics: + * ipv6_read, and + * ipv6_write. + (The IPv4 statistics can be calculated by subtracting the IPv6 statistics + from the existing total consumed bandwidth statistics.) + + We propose adding a new BandwidthStatistics torrc option and consensus + parameter, which activates reporting of all these statistics. Currently, + the existing statistics are controlled by ExtraInfoStatistics, but we + propose using the new BandwidthStatistics option for them as well. + + The default value of this option should be "auto", which checks the + consensus parameter. If there is no consensus parameter, the default should + be 1. (The existing bandwidth statistics are reported by default.) + + TODO: Should we collect IPv6 bandwidth statistics on bridges? + On bridges, should bandwidth statistics be on or off by default? + + If we do want to collect bridge statistics, and we think it's safe, + modify proposals 311 and 312 to allow bridge statistics. + +5. Collecting IPv6 Connection Statistics + + We propose that relays (and bridges) collect IPv6 connection statistics. + + To minimise development and testing effort, we propose re-using the existing + "bidi" code in rephist.c. (This code may require some refactoring, because + the "bidi" totals are globals, rather than a struct.) + + In particular, tor currently counts these connection statistics: + * below threshold, + * mostly read, + * mostly written, and + * both read and written. + + We propose adding IPv6 variants of all these statistics. (The IPv4 + statistics can be calculated by subtracting the IPv6 statistics from the + existing total connection statistics.) + + We propose using the existing ConnDirectionStatistics torrc option, and + adding a consensus parameter with the same name. This option will control + the new and existing connection statistics. + + The default value of this option should be "auto", which checks the + consensus parameter. If there is no consensus parameter, the default should + be 0. (The existing connection direction statistics are reported by + default.) + + TODO: Do enough relays report ConnDirectionStatistics, for accurate IPv6 + connection statistics? + * at least 25% of relays have IPv6 + * at the end of the project, we expect at least 33% of relays to have + deployed tor 0.4.4-stable + + If not, we should turn on ConnDirectionStatistics by default. (Or set the + consensus parameter for a few days, to collect these statistics.) + +6. Directory Protocol Specification Changes + + We propose adding IPv6 variants of the consumed bandwidth and connection + direction statistics to the tor directory protocol. + + We propose the following additions to the [Tor Directory Protocol] + specification, in section 2.1.2. Each addition should be inserted below the + existing consumed bandwidth and connection direction specifications. + + "ipv6-read-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] + "ipv6-write-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] + + Declare how much bandwidth the OR has used recently, on IPv6 + connections. See "read-history" and "write-history" for more details. + (The migration notes do not apply to IPv6.) + + "ipv6-conn-bi-direct" YYYY-MM-DD HH:MM:SS (NSEC s) BELOW,READ,WRITE,BOTH NL + [At most once] + + Number of IPv6 connections, that are used uni-directionally or + bi-directionally. See "conn-bi-direct" for more details. + + We also propose the following replacement, in the same section: + + "dirreq-read-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] + "dirreq-write-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] + + Declare how much bandwidth the OR has spent on answering directory + requests. See "read-history" and "write-history" for more details. + (The migration notes do not apply to dirreq.) + + This replacement is optional, but it may avoid the 3 *read-history + definitions getting out of sync. + +7. Optional Changes + + We propose some optional changes to help relay operators, tor developers, + and tor's network health. We also expect that these changes will drive IPv6 + relay adoption. + + Some of these changes may be more appropriate as future work, or along with + other proposed features. + +7.1. Log IPv6 Statistics in Tor's Heartbeat Logs + + We propose this optional change, so relay operators can see their own IPv6 + statistics: + + We propose that tor logs its IPv6 consumed bandwidth and connection + statistics in its regular "heartbeat" logs. + + These heartbeat statistics should be collected over the lifetime of the tor + process, rather than using the state file, like the statistics in sections + 4 and 5. + + Tor's existing heartbeat logs already show its consumed bandwidth and + connections (in the link protocol counts). + + We may also want to show IPv6 consumed bandwidth and connections as a + propotion of the total consumed bandwidth and connections. + + These statistics only show a relay's local bandwidth usage, so they can't + be used for reporting. + +7.2. Show IPv6 Statistics on Consensus Health + + The [Consensus Health] website displays a wide rage of tor statistics, + based on the most recent consensus. + + We propose this optional change, to: + * help relay operators diagnose issues with IPv6 on their relays, and + * to drive IPv6 adoption on tor relays. + + Consensus Health adds an IPv6 section, with the following relay statistics: + * have an IPv6 ORPort, + * support IPv6 reachability checks, + * support IPv6 clients, and + * support IPv6 reachability checks and IPv6 clients. + + The definitions of these statistics are in section 3. + + These changes can be tested using the script proposed in section 3. + +7.3. Add an IPv6 Reachability Pseudo-Flag on Relay Search + + The [Relay Search] website displays tor relay information, based on the + current consensus and relay descriptors. + + We propose this optional change, to: + * help relay operators diagnose issues with IPv6 on their relays, and + * drive IPv6 adoption on tor relays. + + Relay Search adds a pseudo-flag for relay IPv6 reachability support. + + This pseudo-flag should be given to relays that have: + * a reachable IPv6 ORPort (in the consensus), and + * support tor subprotocol version "Relay=3" (or later). + See [Proposal 311: Relay IPv6 Reachability] for details. + + TODO: Is this a useful change? + Are there better ways of driving IPv6 adoption? + +7.4. Add IPv6 Connections and Consumed Bandwidth Graphs to Tor Metrics + + The [Tor Metrics: Traffic] website displays connection and bandwidth + information for the tor network, based on relay extra-info descriptors. + + We propose these optional changes, to: + * help tor developers improve IPv6 support on the tor network, + * help diagnose issues with IPv6 on the tor network, and + * drive IPv6 adoption on tor relays. + + Tor Metrics adds the following information to the graphs on the Traffic + page: + + Consumed Bandwidth by IP version + * added to the existing [Tor Metrics: Advertised bandwidth by IP version] + page + * as a stacked graph, like + [Tor Metrics: Advertised and consumed bandwidth by relay flags] + + Fraction of connections used uni-/bidirectionally by IP version + * added to the existing + [Tor Metrics: Fraction of connections used uni-/bidirectionally] page + * as a stacked graph, like + [Tor Metrics: Advertised and consumed bandwidth by relay flags] + +8. Test Plan + + We provide a quick summary of our testing plans. + +8.1. Testing IPv6 Relay Consensus Count Logging + + We propose to test these changes using chutney networks. However, chutney + creates a limited number of relays, so we also need to test these changes + on the public tor network. + + Therefore, we propose to test these changes on the public network with a + small number of relays and bridges. + + We can use the script and the tor logs to cross-check these calculations. + The Tor Metrics team may also independently check these calculations. + + Once these changes are merged, they will be monitored by tor developers, as + more volunteer relay operators deploy the relevant tor versions. (And as the + number of IPv6 relays in the consensus increases.) + +8.2. Testing IPv6 Extra-Info Statistics + + We propose to test the connection and consumed bandwidth statistics using + chutney networks. However, chutney runs for a short amount of time, and + creates a limited amount of traffic, so we also need to test these changes + on the public tor network. + + In particular, we have struggled to test statistics using chutney, because + tor's hard-coded statistics period is 24 hours. (And most chutney networks + run for under 1 minute.) + + Therefore, we propose to test these changes on the public network with a + small number of relays and bridges. + + During 2020, the Tor Metrics team will analyse these statistics on the + public tor network, and provide IPv6 progress reports. We expect that we may + discover some bugs during the first analysis. + + Once these changes are merged, they will be monitored by tor developers, as + more volunteer relay operators deploy the relevant tor versions. (And as the + number of IPv6 relays in the consensus increases.) + +References: + +[Consensus Health]: + https://consensus-health.torproject.org/ + +[Proposal 288: Privacy-Preserving Stats with Privcount (Shamir version)]: + https://gitweb.torproject.org/torspec.git/tree/proposals/288-privcount-with… + +[Proposal 311: Relay IPv6 Reachability]: + https://gitweb.torproject.org/torspec.git/tree/proposals/311-relay-ipv6-rea… + +[Proposal 312: Relay Auto IPv6 Address]: + https://gitweb.torproject.org/torspec.git/tree/proposals/312-relay-auto-ipv… + +[Relay Search]: + https://metrics.torproject.org/rs.html + +[Tor Directory Protocol]: + (version 3) https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt + +[Tor Manual Page]: + https://2019.www.torproject.org/docs/tor-manual.html.en + +[Tor Metrics: Advertised and consumed bandwidth by relay flags]: + https://metrics.torproject.org/bandwidth-flags.html + +[Tor Metrics: Advertised bandwidth by IP version]: + https://metrics.torproject.org/advbw-ipv6.html + +[Tor Metrics: Fraction of connections used uni-/bidirectionally]: + https://metrics.torproject.org/connbidirect.html + +[Tor Metrics: Traffic]: + https://metrics.torproject.org/bandwidth-flags.html + +[Tor Specification]: + https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt

1 0

[torspec/master] Prop 313: Only Script, No Tor Logs
by teor＠torproject.org 11 Feb '20

11 Feb '20

commit 91356f5db02b6a62afa3061278872b8d607db7ea Author: teor <teor(a)torproject.org> Date: Tue Feb 11 08:32:25 2020 +1000 Prop 313: Only Script, No Tor Logs Remove the tor logs from the proposal, only write a script. (Some of these calculations are medium-term, and are most relevent during Sponsor 55.) Part of 33159. --- proposals/313-relay-ipv6-stats.txt | 87 +++++++++++++++++++------------------- 1 file changed, 44 insertions(+), 43 deletions(-) diff --git a/proposals/313-relay-ipv6-stats.txt b/proposals/313-relay-ipv6-stats.txt index b521e9b..e615afe 100644 --- a/proposals/313-relay-ipv6-stats.txt +++ b/proposals/313-relay-ipv6-stats.txt @@ -7,13 +7,19 @@ Ticket: #33159 0. Abstract - We propose that Tor relays (and bridges) should log the number of relays in - the consensus that support IPv6 extends, and IPv6 client connections. - - We also propose that Tor relays (and bridges) should collect statistics on - IPv6 connections and consumed bandwidth. Like tor's existing connection - and consumed bandwidth statistics, these new IPv6 statistics will be - published in each relay's extra-info descriptor. + We propose that: + * tor relays should collect statistics on IPv6 connections, and + * tor relays and bridges should collect statistics on consumed bandwidth. + Like tor's existing connection and consumed bandwidth statistics, these new + IPv6 statistics will be published in each relay's extra-info descriptor. + + We also plan to write a script that shows the number of relays in the + consensus that support: + * IPv6 extends, and + * IPv6 client connections. + This script will be used for medium-term monitoring, during the deployment + of tor's IPv6 changes in 2020. (See [Proposal 311: Relay IPv6 Reachability] + and [Proposal 312: Relay Auto IPv6 Address].) 1. Introduction @@ -35,12 +41,6 @@ Ticket: #33159 This proposal modifies Tor's behaviour as follows: - Relays, bridges, and directory authorities log the number of relays that - support IPv6 clients, and IPv6 relay reachability checks. They also log the - corresponding consensus weight fractions. - - As an optional change, tor clients may also log this information. - Relays, bridges, and directory authorities collect statistics on: * IPv6 connections, and * IPv6 consumed bandwidth. @@ -69,31 +69,31 @@ Ticket: #33159 in each section should specifically exclude or include these other roles.) Tor clients do not collect any statistics for public reporting. Therefore, - clients are out of scope in this proposal. (Except for some optional changes - to client logs, where they log the number of IPv6 relays in the consensus). + clients are out of scope in this proposal. When this proposal describes Tor's current behaviour, it covers all supported Tor versions (0.3.5.7 to 0.4.2.5), as of January 2020, except where another version is specifically mentioned. -3. Logging IPv6 Relays in the Consensus + This proposal also includes a medium-term monitoring script, which + calculates the number of relays in the consensus that support IPv6 extends, + and IPv6 client connections. + +3. Monitoring IPv6 Relays in the Consensus - We propose that relays (and bridges) log: + We propose writing a script that calculates: * the number of relays, and * the consensus weight fraction of relays, in the consensus that: * have an IPv6 ORPort, - * support IPv6 reachability checks, and - * support IPv6 clients. - - In order to test these changes, and provide easy access to these - statistics, we propose implementing a script that: - * downloads a consensus, and - * calculates and reports these statistics. + * support IPv6 reachability checks, + * support IPv6 clients, and + * support IPv6 reachability checks, and IPv6 clients. - As well as the statistics listed above, this script should also report the - following relay statistic: - * support IPv6 reachability checks and IPv6 clients. + In order to provide easy access to these statistics, we propose + that the script should: + * download a consensus (or read an existing consensus), and + * calculate and report these statistics. The following consensus weight fractions should divide by the total consensus weight: @@ -108,13 +108,12 @@ Ticket: #33159 "Usable Guards" have the Guard flag, but do not have the Exit flag. If the Guard also has the BadExit flag, the Exit flag should be ignored. - We propose that these logs happen whenever tor: - * receives a consensus from a directory server, or - * loads a live, valid, cached consensus from disk. + Note that this definition of "Usable Guards" is only valid when the + consensus contains many more guards than exits. That is, Wgd must be 0 in + the consensus. (See the [Tor Directory Protocol] for more details.) - As an optional change, tor clients may also log this information. Some of - this information is not directly relevant to clients, but these logs may - help developers (and users). + Therefore, the script should check that Wgd is 0. If it is not, the script + should log a warning about the accuracy of the "Usable Guard" statistics. 4. Collecting IPv6 Consumed Bandwidth Statistics @@ -322,21 +321,23 @@ Ticket: #33159 We provide a quick summary of our testing plans. -8.1. Testing IPv6 Relay Consensus Count Logging +8.1. Testing IPv6 Relay Consensus Calculations - We propose to test these changes using chutney networks. However, chutney - creates a limited number of relays, so we also need to test these changes - on the public tor network. + We propose to test the IPv6 Relay consensus script using chutney networks. + However, chutney creates a limited number of relays, so we also need to + test these changes on consensuses from the public tor network. - Therefore, we propose to test these changes on the public network with a - small number of relays and bridges. + Some of these calculations are similar to the calculations that tor will do, + to find out if IPv6 reachability checks are reliable. So we may be able to + check the script against tor's reachability logs. (See section 4.3.1 in + [Proposal 311: Relay IPv6 Reachability]: Refusing to Publish the + Descriptor.) - We can use the script and the tor logs to cross-check these calculations. The Tor Metrics team may also independently check these calculations. - Once these changes are merged, they will be monitored by tor developers, as - more volunteer relay operators deploy the relevant tor versions. (And as the - number of IPv6 relays in the consensus increases.) + Once the script is completed, its output will be monitored by tor + developers, as more volunteer relay operators deploy the relevant tor + versions. (And as the number of IPv6 relays in the consensus increases.) 8.2. Testing IPv6 Extra-Info Statistics

1 0