February 2019 - tor-commits - lists.torproject.org

[sbws/master] docs: change scaling section level
by juga＠torproject.org 23 Feb '19

23 Feb '19

commit c05d2567005952d9b627fda7943b9c14672c29c5 Author: juga0 <juga(a)riseup.net> Date: Thu Feb 7 12:28:58 2019 +0000 docs: change scaling section level --- docs/source/specification.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/specification.rst b/docs/source/specification.rst index afda2d4..d48f4e2 100644 --- a/docs/source/specification.rst +++ b/docs/source/specification.rst @@ -203,8 +203,8 @@ a percententage (currently 60%) than the number of relays in the consensus, additional Header Lines MUST be added (see XXX) to the Bandwith File and the relays SHOULD NOT be included. -Scaling bandwidth measurements -::::::::::::::::::::::::::::::::: +Scaling the bandwidth measurements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Consensus bandwidth obtained by new implementations MUST be comparable to the consensus bandwidth, therefore they MUST implement torflow_scaling_.

1 0

[sbws/master] docs: replace explanation on how to store the measurements
by juga＠torproject.org 23 Feb '19

23 Feb '19

commit dea4bb3fbf2519270faa4eb914c5c56654a820ae Author: juga0 <juga(a)riseup.net> Date: Thu Feb 7 12:16:07 2019 +0000 docs: replace explanation on how to store the measurements --- docs/source/specification.rst | 30 ++++++++---------------------- 1 file changed, 8 insertions(+), 22 deletions(-) diff --git a/docs/source/specification.rst b/docs/source/specification.rst index e2b687b..06f1ca1 100644 --- a/docs/source/specification.rst +++ b/docs/source/specification.rst @@ -164,30 +164,16 @@ Selecting the data to download Source code: :func:`sbws.core.scanner._should_keep_result` -Simple result storage -~~~~~~~~~~~~~~~~~~~~~ +Writing the measurements to the filesystem +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Internally, sbws has a hierarchy of ``Result`` classes for easy managing of -different types of result (success, error-because-of-circuit-error, -error-because-[...] etc.). These results get converted into JSON strings and -stored -- **one per line** -- in text files in a data directory. +For every measured relay, the measurement result is put in a queue. +There's an independent thread getting measurements from the queue every second. +Every new measurement is appended to a file as a json line +(but the file itself is not json!). +The file is named with the current date. Every day a new file is created. -The text files are simply named after the date. For example: -``2018-03-20.txt``. - -The sbws scanner only appends to these files, and it automatically starts a new -file when the system's clock ticks past midnight UTC. - -To avoid any weird timezone-related issues, consumers of sbws scanner data (such -as the generate and stats scripts) SHOULD read more files than strictly -necessary. For example, if the validity period is 5 days, they should read 6 -days of files. Because all results have a Unix timestamp, consumers of sbws -data can easily determine which results are just outside the validity period as -they are reading them in. - - -Simple result processing -~~~~~~~~~~~~~~~~~~~~~~~~ +Source code: :func:`sbws.lib.resultdump.ResultDump.enter` Every hour the directory authorities vote to come to a consensus about the state of the Tor network. The bandwidth authorities need to use the results

1 0

[sbws/master] docs: explain how the generator works
by juga＠torproject.org 23 Feb '19

23 Feb '19

commit f1885b082361e4bc68db1301bb3a59d6b9eb9472 Author: juga0 <juga(a)riseup.net> Date: Thu Feb 7 12:26:07 2019 +0000 docs: explain how the generator works --- docs/source/specification.rst | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/source/specification.rst b/docs/source/specification.rst index 06f1ca1..874adbc 100644 --- a/docs/source/specification.rst +++ b/docs/source/specification.rst @@ -175,13 +175,16 @@ The file is named with the current date. Every day a new file is created. Source code: :func:`sbws.lib.resultdump.ResultDump.enter` -Every hour the directory authorities vote to come to a consensus about the -state of the Tor network. The bandwidth authorities need to use the results -that have been gathered to inform their vote about relays' bandwidths. To do -this they use sbws generate. +Running the generator +----------------------- -The relays' bandwidth measurements (``Results``) to be added to the Bandwidth -File MUST be first selected and MUST be then then scaled. +Every hour, the generator: +#. Aggregate all the measurements (not older than 6 six days) for every relay. +#. Filter the measurements +#. Scale the measurements +#. Write the bandwidth file + +Source code: :func:`sbws.lib.v3bwfile.V3BWFile.from_results` Selecting bandwidth measurements :::::::::::::::::::::::::::::::::::

1 0

[sbws/master] docs: change section title to filtering
by juga＠torproject.org 23 Feb '19

23 Feb '19

commit d311ca9fe884d9d936acc6188786409cdb189107 Author: juga0 <juga(a)riseup.net> Date: Thu Feb 7 12:27:52 2019 +0000 docs: change section title to filtering --- docs/source/specification.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/specification.rst b/docs/source/specification.rst index 874adbc..afda2d4 100644 --- a/docs/source/specification.rst +++ b/docs/source/specification.rst @@ -186,15 +186,15 @@ Every hour, the generator: Source code: :func:`sbws.lib.v3bwfile.V3BWFile.from_results` -Selecting bandwidth measurements -::::::::::::::::::::::::::::::::::: +Filtering the bandwidth measurements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each relay bandwidth measurements are selected in the following way: -1. At least two bandwidth measurements (``Result`` s) MUST have been obtained +#. At least two bandwidth measurements (``Result`` s) MUST have been obtained within an arbitrary number of seconds (currently one day). If they are not, the relay MUST NOT be included in the Bandwith File. -2. The measurements than are are older than an arbitrary number of senconds +#. The measurements than are are older than an arbitrary number of senconds in the past MUST be discarded. Currently this number is the same as ``data_period`` (5 days).

1 0

[sbws/master] docs: explain how is selected the data to download
by juga＠torproject.org 23 Feb '19

23 Feb '19

commit 81e2d40eca3a0588bb12e8174bab1306a8c473b2 Author: juga0 <juga(a)riseup.net> Date: Thu Feb 7 11:02:41 2019 +0000 docs: explain how is selected the data to download --- docs/source/specification.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/source/specification.rst b/docs/source/specification.rst index 56bd8b1..e2b687b 100644 --- a/docs/source/specification.rst +++ b/docs/source/specification.rst @@ -150,6 +150,20 @@ Selecting a second relay Source code: :func:`sbws.core.scanner.measure_relay` +Selecting the data to download +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +#. While the downloaded data is smaller than 1GB or the number of download + is minor than 5: +#. Randomly, select a 16MiB range. +#. If it takes less than 5 seconds, select a bigger range and don't keep any + information. +#. If it takes more than 10 seconds, select an smaller range and don't keep any + information. +#. Store the number of bytes downloaded and the time it took. + +Source code: :func:`sbws.core.scanner._should_keep_result` + Simple result storage ~~~~~~~~~~~~~~~~~~~~~

1 0

[meek/webextension] Change -profile to --profile in a comment.
by dcf＠torproject.org 23 Feb '19

23 Feb '19

commit 9a822c9e99e0bf23c542427de4eae3493ebccbc3 Author: David Fifield <david(a)bamsoftware.com> Date: Sat Feb 23 01:38:49 2019 -0700 Change -profile to --profile in a comment. --- meek-client-torbrowser/meek-client-torbrowser.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meek-client-torbrowser/meek-client-torbrowser.go b/meek-client-torbrowser/meek-client-torbrowser.go index ab1ed9c..48b055c 100644 --- a/meek-client-torbrowser/meek-client-torbrowser.go +++ b/meek-client-torbrowser/meek-client-torbrowser.go @@ -337,7 +337,7 @@ func main() { // Unset environment variables that Firefox sets after a restart (as // caused by, for example, an update or the installation of an add-on). - // XRE_PROFILE_PATH, in particular, overrides the -profile option that + // XRE_PROFILE_PATH, in particular, overrides the --profile option that // runFirefox sets, causing Firefox to run with profile.default instead // of profile.meek-http-helper, which conflicts with the profile.default // that is already running. See https://bugs.torproject.org/13247,

1 0

[meek/master] Minor refactoring of meek-client-torbrowser.
by dcf＠torproject.org 23 Feb '19

23 Feb '19

commit a0aa256c3bf692b31117acb6d885a0b012661014 Author: David Fifield <david(a)bamsoftware.com> Date: Fri Feb 22 16:42:41 2019 -0700 Minor refactoring of meek-client-torbrowser. --- meek-client-torbrowser/meek-client-torbrowser.go | 65 ++++++++++++------------ 1 file changed, 32 insertions(+), 33 deletions(-) diff --git a/meek-client-torbrowser/meek-client-torbrowser.go b/meek-client-torbrowser/meek-client-torbrowser.go index 2daa108..f3ffc68 100644 --- a/meek-client-torbrowser/meek-client-torbrowser.go +++ b/meek-client-torbrowser/meek-client-torbrowser.go @@ -203,6 +203,30 @@ func isBrowserProfileUpToDate(templatePath string, profilePath string) bool { // the path to the Firefox profile to use. Otherwise, the profile is chosen in a // platform-specific way (see linux.go, mac.go, windows.go). func runFirefox() (cmd *exec.Cmd, stdout io.Reader, err error) { + // Unset environment variables that Firefox sets after a restart (as + // caused by, for example, an update or the installation of an add-on). + // XRE_PROFILE_PATH, in particular, overrides the -profile option that + // runFirefox sets, causing Firefox to run with profile.default instead + // of profile.meek-http-helper, which conflicts with the profile.default + // that is already running. See https://bugs.torproject.org/13247, + // particularly #comment:17 and #comment:18. The environment variable + // names come from + // https://hg.mozilla.org/mozilla-central/file/cfde3603b020/toolkit/xre/nsAppR… + for _, varname := range []string{ + "XRE_PROFILE_PATH", + "XRE_PROFILE_LOCAL_PATH", + "XRE_PROFILE_NAME", + "XRE_START_OFFLINE", + "NO_EM_RESTART", + "XUL_APP_FILE", + "XRE_BINARY_PATH", + } { + err = os.Unsetenv(varname) + if err != nil { + return + } + } + // Mac OS X needs absolute paths for firefox and for the profile. var absFirefoxPath string absFirefoxPath, err = filepath.Abs(firefoxPath) @@ -327,29 +351,14 @@ func main() { sigChan := make(chan os.Signal, 1) signal.Notify(sigChan, syscall.SIGTERM) - // Unset environment variables that Firefox sets after a restart (as - // caused by, for example, an update or the installation of an add-on). - // XRE_PROFILE_PATH, in particular, overrides the -profile option that - // runFirefox sets, causing Firefox to run with profile.default instead - // of profile.meek-http-helper, which conflicts with the profile.default - // that is already running. See https://bugs.torproject.org/13247, - // particularly #comment:17 and #comment:18. The environment variable - // names come from - // https://hg.mozilla.org/mozilla-central/file/cfde3603b020/toolkit/xre/nsAppR… - var firefoxRestartEnvVars = []string{ - "XRE_PROFILE_PATH", - "XRE_PROFILE_LOCAL_PATH", - "XRE_PROFILE_NAME", - "XRE_START_OFFLINE", - "NO_EM_RESTART", - "XUL_APP_FILE", - "XRE_BINARY_PATH", - } - for _, varname := range firefoxRestartEnvVars { - err := os.Unsetenv(varname) - if err != nil { - log.Fatal(err) - } + if os.Getenv("TOR_PT_EXIT_ON_STDIN_CLOSE") == "1" { + // This environment variable means we should treat EOF on stdin + // just like SIGTERM: https://bugs.torproject.org/15435. + go func() { + io.Copy(ioutil.Discard, os.Stdin) + log.Print("synthesizing SIGTERM because of stdin close") + sigChan <- syscall.SIGTERM + }() } // Start firefox. @@ -375,16 +384,6 @@ func main() { } defer logKill(meekClientCmd.Process) - if os.Getenv("TOR_PT_EXIT_ON_STDIN_CLOSE") == "1" { - // This environment variable means we should treat EOF on stdin - // just like SIGTERM: https://bugs.torproject.org/15435. - go func() { - io.Copy(ioutil.Discard, os.Stdin) - log.Printf("synthesizing SIGTERM because of stdin close") - sigChan <- syscall.SIGTERM - }() - } - sig := <-sigChan log.Printf("sig %s", sig) err = logSignal(meekClientCmd.Process, sig)

1 0

[meek/webextension] Fix the overlong message checks.
by dcf＠torproject.org 23 Feb '19

23 Feb '19

commit de03366fbe1f23cbb21d41aec8f4913f189ecb8b Author: David Fifield <david(a)bamsoftware.com> Date: Fri Feb 22 18:58:50 2019 -0700 Fix the overlong message checks. It'll never exceed math.MaxUint32 if you convert it to uint32 first... I didn't notice what nonsense this was until I saw the commit email. len returns int, which is specified to be either 32 or 64 bits, so it will never be truncated when coverting to uint64. https://golang.org/ref/spec#Numeric_types --- webextension/native/main.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/webextension/native/main.go b/webextension/native/main.go index 10a2b90..3f4ec3a 100644 --- a/webextension/native/main.go +++ b/webextension/native/main.go @@ -139,7 +139,7 @@ func writeResponseSpec(w io.Writer, spec *responseSpec) error { } length := len(encodedSpec) - if uint32(length) > math.MaxUint32 { + if uint64(length) > math.MaxUint32 { return fmt.Errorf("response spec is too long to represent: %d", length) } err = binary.Write(w, binary.BigEndian, uint32(length)) @@ -178,7 +178,7 @@ func recvWebExtensionMessage(r io.Reader) ([]byte, error) { // https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Nati… func sendWebExtensionMessage(w io.Writer, message []byte) error { length := len(message) - if uint32(length) > math.MaxUint32 { + if uint64(length) > math.MaxUint32 { return fmt.Errorf("WebExtension message is too long to represent: %d", length) } err := binary.Write(w, NativeEndian, uint32(length))

1 0

[meek/webextension] Set network.proxy.socks_remote_dns=false inside the WebExtension.
by dcf＠torproject.org 23 Feb '19

23 Feb '19

commit e0c466d67711eaf1b264791d93f53360432b9cbd Author: David Fifield <david(a)bamsoftware.com> Date: Wed Feb 20 19:26:48 2019 -0700 Set network.proxy.socks_remote_dns=false inside the WebExtension. This is to work around a Firefox patch that otherwise prevents DNS requests from being made when proxyType=="direct". Compare to https://gitweb.torproject.org/pluggable-transports/meek.git/commit/?id=32ca… --- webextension/background.js | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/webextension/background.js b/webextension/background.js index 00500a0..7ad06fe 100644 --- a/webextension/background.js +++ b/webextension/background.js @@ -277,6 +277,22 @@ browser.webRequest.onBeforeRequest.addListener( ["blocking"] ); +// Allow unproxied DNS, working around a Tor Browser patch: https://bugs.torproject.org/11183#comment:6. +// We manually override the proxy for every request, and in makeProxyInfo we set +// proxyDNS:true wherever necessary, so name resolution uses the proxy despite +// this pref. +// +// In Tor Browser, the pref changes here are only temporary. The +// meek-http-helper profile has a user.js file that sets a default blackhole +// proxy, as a safety feature in case something goes wrong running the headless +// browser. +// +// We only care to set proxyDNS here, but must additionally set proxyType until +// Firefox 63 because of a bug. +// https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/… +// https://bugzilla.mozilla.org/show_bug.cgi?id=1487121 +browser.proxy.settings.set({value: {proxyType: "system", proxyDNS: false}}); + // Connect to our native process. let port = browser.runtime.connectNative("meek.http.helper");

1 0

[meek/webextension] Use JavaScript compatible with Firefox 60.
by dcf＠torproject.org 23 Feb '19

23 Feb '19

commit 82e5eb9eec252d68d66d21718d7260b80fb261fc Author: David Fifield <david(a)bamsoftware.com> Date: Wed Feb 20 18:18:17 2019 -0700 Use JavaScript compatible with Firefox 60. Object.fromEntries is not available until 63: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Ob… --- webextension/background.js | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/webextension/background.js b/webextension/background.js index b43e614..00500a0 100644 --- a/webextension/background.js +++ b/webextension/background.js @@ -182,7 +182,10 @@ async function roundtrip(request) { let headers = Object.entries(request.header != null ? request.header : {}) .map(x => ({name: x[0], value: x[1]})); // Remove all browser headers that conflict with requested headers. - let overrides = Object.fromEntries(headers.map(x => [x.name.toLowerCase(), true])); + let overrides = {}; + for (let name of Object.keys(headers)) { + overrides[name.toLowerCase()] = true; + } // Also remove some unnecessary or potentially tracking-enabling headers. for (let name of ["Accept", "Accept-Language", "Cookie", "Origin", "User-Agent"]) { overrides[name.toLowerCase()] = true;

1 0