March 2019 - tor-commits - lists.torproject.org

[tor/maint-0.3.5] Merge remote-tracking branch 'tor-github/pr/727' into maint-0.3.5
by teor＠torproject.org 19 Mar '19

19 Mar '19

commit 17e3eea68538f81a1375009fa6881ad89e11d5db Merge: 5d41e2223 cc23afddd Author: teor <teor(a)torproject.org> Date: Tue Mar 19 15:43:05 2019 +1000 Merge remote-tracking branch 'tor-github/pr/727' into maint-0.3.5 changes/bug29530_035 | 5 +++++ src/test/test_address.c | 6 +++--- 2 files changed, 8 insertions(+), 3 deletions(-)

1 0

[chutney/master] test-network: wait for bootstrap rather than waiting for a set time
by teor＠torproject.org 19 Mar '19

19 Mar '19

commit a113d6a46f63d70f84afa3adc4f3baab2ed414f0 Author: teor <teor(a)torproject.org> Date: Wed Mar 13 13:24:29 2019 +1000 test-network: wait for bootstrap rather than waiting for a set time Chutney uses the new wait_for_bootstrap feature to start verifying after bootstrap is complete. If bootstrap fails, chutney verifies anyway. We'll fail on bootstrap once we implement 20473. Closes 28203. --- README | 23 ++++++++++++++--------- tools/test-network-impl.sh | 5 ++++- 2 files changed, 18 insertions(+), 10 deletions(-) diff --git a/README b/README index 28f90a0..d2bcfad 100644 --- a/README +++ b/README @@ -150,18 +150,23 @@ Multiple Tests: Waiting for the network: - The tools/test-network.sh script waits CHUTNEY_START_TIME seconds - (default: 40) before calling chutney verify, because that's the minimum + The tools/test-network.sh script waits for all the tor nodes to bootstrap + before calling chutney verify. If CHUTNEY_START_TIME is negative, the + script leaves the network running, and exits immediately (without waiting + for bootstrap, or verifying). + + The wait_for_bootstrap command waits for up to CHUTNEY_START_TIME seconds, + checking whether the logged bootstrapped status for every node is 100%. If + it is, great: it succeeds. If not, it dumps the bootstrap statuses and + exits. test-network.sh does not exit immediately if a tor node fails to + bootstrap. Instead, it attempts to verify. We'll add an option to fail on + tor bootstrap failure in #20473. + + The default CHUTNEY_START_TIME is 40 seconds, because that's the minimum amount of time it takes to bootstrap a consensus containing relays. (It takes 20 seconds for the authorities to create the first consensus, then 20 seconds for relays to bootstrap, submit their descriptors, and be - included in the next consensus.) If CHUTNEY_START_TIME is negative, the - script leaves the network running, and exits immediately (without verifying). - - chutney also has an experimental wait_for_bootstrap command. It waits for - up to CHUTNEY_START_TIME seconds, checking whether the logged bootstrapped - status for every node is 100%. If it is, great: it succeeds. If not, it - dumps the bootstrap statuses and exits. + included in the next consensus.) Commands like "chutney verify" start immediately, and keep trying for CHUTNEY_BOOTSTRAP_TIME seconds (default: 60). If it hasn't been diff --git a/tools/test-network-impl.sh b/tools/test-network-impl.sh index 2de1bfa..ca086df 100755 --- a/tools/test-network-impl.sh +++ b/tools/test-network-impl.sh @@ -20,7 +20,10 @@ CHUTNEY="$CHUTNEY_PATH/chutney" if [ "$CHUTNEY_START_TIME" -ge 0 ]; then $ECHO "Waiting $CHUTNEY_START_TIME seconds for a consensus containing relays to be generated..." - sleep "$CHUTNEY_START_TIME" + # We log bootstrap status for debugging purposes. + # We'll fix bootstrap errors in #20473. + "$CHUTNEY" wait_for_bootstrap "$CHUTNEY_NETWORK" \ + || $ECHO "Tor bootstrap failed, ignoring for now." else $ECHO "Chutney network launched and running. To stop the network, use:" $ECHO "$CHUTNEY stop $CHUTNEY_NETWORK"

1 0

[chutney/master] wait_for_bootstrap: When logging un-bootstrapped nodes, also log crashes
by teor＠torproject.org 19 Mar '19

19 Mar '19

commit ee396986d85df7cd736159d9031857573499877e Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Mar 13 10:17:58 2019 -0400 wait_for_bootstrap: When logging un-bootstrapped nodes, also log crashes Previously, if you said wait_for_bootstrap, it would tell you if a node wasn't bootstrapped, but wouldn't help you notice whether it was crashed or running. --- lib/chutney/TorNet.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index e22afea..7269a00 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -1199,6 +1199,7 @@ class Network(object): print("Bootstrap failed. Node status:") for c in controllers: + c.check(listRunning=False, listNonRunning=True) print(c.getLastBootstrapStatus()) return False

1 0

[chutney/master] wait_for_bootstrap: Fix the regex for bootstrap messages
by teor＠torproject.org 19 Mar '19

19 Mar '19

commit bd76a94b69c46e566c6d0517dd7b829f04191c6a Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Mar 13 10:14:29 2019 -0400 wait_for_bootstrap: Fix the regex for bootstrap messages The regex was almost right, but it required a space after the percent symbol. In reality, this space should only be there when there is a keyword following. Part of 22132. --- lib/chutney/TorNet.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index 4415907..e22afea 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -837,7 +837,7 @@ class LocalNodeController(NodeController): percent,keyword,message=-100,"no_message","No bootstrap messages yet." with open(logfname, 'r') as f: for line in f: - m = re.search(r'Bootstrapped (\d+)% ($[^$]*\))?: (.*)', line) + m = re.search(r'Bootstrapped (\d+)%( $[^$]*\))?: (.*)', line) if m: percent, keyword, message = m.groups() percent = int(percent)

1 0

[chutney/master] wait_for_bootstrap: log node names with bootstrap statuses
by teor＠torproject.org 19 Mar '19

19 Mar '19

commit 18af751e0b86c00368230905208f6b4b6646e4a7 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Mar 13 10:24:15 2019 -0400 wait_for_bootstrap: log node names with bootstrap statuses --- lib/chutney/TorNet.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index 7269a00..3e8b94e 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -647,6 +647,10 @@ class LocalNodeController(NodeController): NodeController.__init__(self, env) self._env = env + def getNick(self): + """Return the nickname for this node.""" + return self._env['nick'] + def getPid(self): """Assuming that this node has its pidfile in ${dir}/pid, return the pid of the running process, or None if there is no pid in the @@ -1200,7 +1204,7 @@ class Network(object): print("Bootstrap failed. Node status:") for c in controllers: c.check(listRunning=False, listNonRunning=True) - print(c.getLastBootstrapStatus()) + print("{}: {}".format(c.getNick(), c.getLastBootstrapStatus())) return False

1 0

[chutney/master] README: updates after recent features
by teor＠torproject.org 19 Mar '19

19 Mar '19

commit 84127eb6eb5b090e666f938d744cc823fc0b5f69 Author: teor <teor(a)torproject.org> Date: Wed Mar 13 12:47:42 2019 +1000 README: updates after recent features Closes 29762. --- README | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/README b/README index 23c6ed2..28f90a0 100644 --- a/README +++ b/README @@ -10,7 +10,8 @@ It is supposed to be a good tool for: Right now it only sorta does these things. You will need: - - Python 2.7 or later (Python 3 support is an ongoing work), and + - Python 2.7, or a supported Python 3, + - (we support Python versions that are still getting updates), - Tor binaries. Chutney checks for Tor binaries in this order: @@ -33,6 +34,7 @@ Automated Setup, Verification, and Shutdown: (The script tries hard to find tor.) ./tools/test-network.sh --chutney-path <chutney-directory> (The script is pretty good at finding chutney.) + ./tools/test-network.sh --allow-failures <N> test-network.sh looks for some tor binaries (either in a nearby build directory or in your $PATH), configures a comprehensive tor test network, @@ -42,6 +44,14 @@ itself. Relative paths are supported. You can modify its configuration using command-line arguments, or use the chutney environmental variables documented below: +Verification Options: + # repeats bootstrap and verify + --allow-failures CHUTNEY_ALLOW_FAILURES=N + # repeats verify + --rounds CHUTNEY_ROUNDS=N + # makes multiple connections within verify + --connections CHUTNEY_CONNECTIONS=N + Timing Options: --start-time CHUTNEY_START_TIME=N --bootstrap-time CHUTNEY_BOOTSTRAP_TIME=N @@ -49,9 +59,6 @@ Timing Options: Traffic Options: --data CHUTNEY_DATA_BYTES=N - # connections are simultaneous, rounds are sequential - --connections CHUTNEY_CONNECTIONS=N - --rounds CHUTNEY_ROUNDS=N --hs-multi-client CHUTNEY_HS_MULTI_CLIENT=N Address/DNS Options: @@ -83,6 +90,7 @@ Standard Actions: ./chutney configure networks/basic ./chutney start networks/basic ./chutney status networks/basic + ./chutney wait_for_bootstrap networks/basic ./chutney verify networks/basic ./chutney hup networks/basic ./chutney stop networks/basic @@ -127,16 +135,34 @@ Bandwidth File Tests: date +%s > /tmp/bwfile ./tools/test-network.sh --flavour bwfile +Multiple Tests: + + Chutney can allow a certain number of failed tests. You can either set + CHUTNEY_ALLOW_FAILURES or use an --allow-failures command-line option to + control this. Chutney will then reattempt the test, from bootstrap + through shutdown, until either it succeeds, or until it has failed + $CHUTNEY_ALLOW_FAILURES+1 times. The default value is zero, so the default + behavior will not change. + + You can also use CHUTNEY_ROUNDS=N to run multiple verification rounds, or + CHUTNEY_CONNECTIONS=N to make multiple connections within each verification + round. Any round or connection failure will fail the current test. + Waiting for the network: The tools/test-network.sh script waits CHUTNEY_START_TIME seconds - (default: 20) before calling chutney verify, because that's the minimum + (default: 40) before calling chutney verify, because that's the minimum amount of time it takes to bootstrap a consensus containing relays. - (It takes 5-10 seconds for the authorities to create the first consensus, - then 10 seconds for relays to bootstrap, submit their descriptors, and be + (It takes 20 seconds for the authorities to create the first consensus, + then 20 seconds for relays to bootstrap, submit their descriptors, and be included in the next consensus.) If CHUTNEY_START_TIME is negative, the script leaves the network running, and exits immediately (without verifying). + chutney also has an experimental wait_for_bootstrap command. It waits for + up to CHUTNEY_START_TIME seconds, checking whether the logged bootstrapped + status for every node is 100%. If it is, great: it succeeds. If not, it + dumps the bootstrap statuses and exits. + Commands like "chutney verify" start immediately, and keep trying for CHUTNEY_BOOTSTRAP_TIME seconds (default: 60). If it hasn't been successful after that time, it fails. If CHUTNEY_BOOTSTRAP_TIME is negative, @@ -191,10 +217,12 @@ Using DNS: Use --dns-conf resolv.conf (relative paths work). The configuration files: + networks/basic holds the configuration for the network you're configuring above. It refers to some torrc template files in torrc_templates/. The working files: + chutney sticks its working files, including all data directories, log files, etc, in ./net/. Each tor instance gets a subdirectory of net/nodes. @@ -202,6 +230,7 @@ The working files: environment variable. Test scripts: + The test scripts are stored in the "scripts/chutney_tests" directory. These Python files must define a "run_test(network)" function. Files starting with an underscore ("_") are ignored.

1 0

[chutney/master] wait_for_bootstrap: another regex tweak
by teor＠torproject.org 19 Mar '19

19 Mar '19

commit 287c61a4c055f76b00d52c370c5a30948bd936e5 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Mar 13 10:25:47 2019 -0400 wait_for_bootstrap: another regex tweak This tweak makes us capture _only_ the status keyword, rather than the keyword plus its surrounding parentheses and space. --- lib/chutney/TorNet.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index 3e8b94e..4aec528 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -841,7 +841,8 @@ class LocalNodeController(NodeController): percent,keyword,message=-100,"no_message","No bootstrap messages yet." with open(logfname, 'r') as f: for line in f: - m = re.search(r'Bootstrapped (\d+)%( $[^$]*\))?: (.*)', line) + m = re.search(r'Bootstrapped (\d+)%(?: $([^$]*)\))?: (.*)', + line) if m: percent, keyword, message = m.groups() percent = int(percent)

1 0

[chutney/master] Increase CHUTNEY_START_TIME to 60
by teor＠torproject.org 19 Mar '19

19 Mar '19

commit d8355b434e8e256606f75fdd8796bdbf4f8d0e8f Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Mar 14 13:45:15 2019 -0400 Increase CHUTNEY_START_TIME to 60 --- README | 2 +- lib/chutney/TorNet.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README b/README index 23c6ed2..f7ec527 100644 --- a/README +++ b/README @@ -130,7 +130,7 @@ Bandwidth File Tests: Waiting for the network: The tools/test-network.sh script waits CHUTNEY_START_TIME seconds - (default: 20) before calling chutney verify, because that's the minimum + (default: 60) before calling chutney verify, because that's the minimum amount of time it takes to bootstrap a consensus containing relays. (It takes 5-10 seconds for the authorities to create the first consensus, then 10 seconds for relays to bootstrap, submit their descriptors, and be diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index e48c1b7..ca099ba 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -1186,7 +1186,7 @@ class Network(object): def wait_for_bootstrap(self): print("Waiting for nodes to bootstrap...") - limit = getenv_int("CHUTNEY_START_TIME", 20) + limit = getenv_int("CHUTNEY_START_TIME", 60) delay = 0.5 controllers = [n.getController() for n in self._nodes] elapsed = 0.0

1 0

[chutney/master] fixup! Increase CHUTNEY_START_TIME to 60
by teor＠torproject.org 19 Mar '19

19 Mar '19

commit 3f349e991760a53072376a4b81794e612eea360c Author: teor <teor(a)torproject.org> Date: Mon Mar 18 12:27:57 2019 +1000 fixup! Increase CHUTNEY_START_TIME to 60 --- tools/test-network-impl.sh | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tools/test-network-impl.sh b/tools/test-network-impl.sh index 2de1bfa..055c15b 100755 --- a/tools/test-network-impl.sh +++ b/tools/test-network-impl.sh @@ -8,11 +8,13 @@ if ! "$CHUTNEY_PATH/tools/bootstrap-network.sh" "$NETWORK_FLAVOUR"; then exit 1 fi -# chutney starts verifying after 20 seconds, keeps on trying for 60 seconds, -# and then stops immediately (by default) +# chutney starts verifying after CHUTNEY_START_TIME seconds, +# keeps on trying for CHUTNEY_BOOTSTRAP_TIME seconds, +# and then stops after CHUTNEY_STOP_TIME seconds. # Even the fastest chutney networks take 5-10 seconds for their first consensus # and then 10 seconds after that for relays to bootstrap and upload descriptors -export CHUTNEY_START_TIME=${CHUTNEY_START_TIME:-40} +# But chutney defaults to running a bit more slowly, so it is more reliable. +export CHUTNEY_START_TIME=${CHUTNEY_START_TIME:-60} export CHUTNEY_BOOTSTRAP_TIME=${CHUTNEY_BOOTSTRAP_TIME:-60} export CHUTNEY_STOP_TIME=${CHUTNEY_STOP_TIME:-0}

1 0

[chutney/master] Merge remote-tracking branch 'tor-github/pr/11'
by teor＠torproject.org 19 Mar '19

19 Mar '19

commit 3f1002cab00c3587ce8d015612b6ed396cfd7446 Merge: 3f349e9 a113d6a Author: teor <teor(a)torproject.org> Date: Tue Mar 19 15:08:52 2019 +1000 Merge remote-tracking branch 'tor-github/pr/11' Merge READMEs from both sets of changes. README | 46 +++++++++++++++++++++++++++++++++++----------- tools/test-network-impl.sh | 5 ++++- 2 files changed, 39 insertions(+), 12 deletions(-) diff --cc README index f7ec527,d2bcfad..93309cb --- a/README +++ b/README @@@ -127,15 -135,38 +135,28 @@@ Bandwidth File Tests date +%s > /tmp/bwfile ./tools/test-network.sh --flavour bwfile + Multiple Tests: + + Chutney can allow a certain number of failed tests. You can either set + CHUTNEY_ALLOW_FAILURES or use an --allow-failures command-line option to + control this. Chutney will then reattempt the test, from bootstrap + through shutdown, until either it succeeds, or until it has failed + $CHUTNEY_ALLOW_FAILURES+1 times. The default value is zero, so the default + behavior will not change. + + You can also use CHUTNEY_ROUNDS=N to run multiple verification rounds, or + CHUTNEY_CONNECTIONS=N to make multiple connections within each verification + round. Any round or connection failure will fail the current test. + Waiting for the network: - The tools/test-network.sh script waits CHUTNEY_START_TIME seconds - (default: 60) before calling chutney verify, because that's the minimum - The tools/test-network.sh script waits for all the tor nodes to bootstrap - before calling chutney verify. If CHUTNEY_START_TIME is negative, the - script leaves the network running, and exits immediately (without waiting - for bootstrap, or verifying). - - The wait_for_bootstrap command waits for up to CHUTNEY_START_TIME seconds, - checking whether the logged bootstrapped status for every node is 100%. If - it is, great: it succeeds. If not, it dumps the bootstrap statuses and - exits. test-network.sh does not exit immediately if a tor node fails to - bootstrap. Instead, it attempts to verify. We'll add an option to fail on - tor bootstrap failure in #20473. - - The default CHUTNEY_START_TIME is 40 seconds, because that's the minimum -- amount of time it takes to bootstrap a consensus containing relays. - (It takes 5-10 seconds for the authorities to create the first consensus, - then 10 seconds for relays to bootstrap, submit their descriptors, and be - included in the next consensus.) If CHUTNEY_START_TIME is negative, the - script leaves the network running, and exits immediately (without verifying). - (It takes 20 seconds for the authorities to create the first consensus, - then 20 seconds for relays to bootstrap, submit their descriptors, and be - included in the next consensus.) ++ The tools/test-network.sh script uses the chutney wait_for_bootstrap ++ command to wait for up to CHUTNEY_START_TIME seconds, checking whether the ++ logged bootstrapped status for every node is 100%. It it is, great: it ++ succeeds. If not, it dumps the bootstrap statuses and exits. ++ test-network.sh does not exit immediately if a tor node fails to bootstrap. ++ Instead, it attempts to verify. We'll add an option to fail on tor ++ bootstrap failure in #20473. Commands like "chutney verify" start immediately, and keep trying for CHUTNEY_BOOTSTRAP_TIME seconds (default: 60). If it hasn't been

1 0