March 2020 - tor-commits - lists.torproject.org

[chutney/master] README: Document how a tor network bootstraps
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit 59c9a9764aa41e1988c878b386405bf639e890f1 Author: teor <teor(a)torproject.org> Date: Wed Feb 19 15:17:39 2020 +1000 README: Document how a tor network bootstraps Part of 33232. --- README | 39 +++++++++++++++++++++++++++++++++------ 1 file changed, 33 insertions(+), 6 deletions(-) diff --git a/README b/README index 9712232..2e850ff 100644 --- a/README +++ b/README @@ -157,15 +157,40 @@ Multiple Tests: CHUTNEY_CONNECTIONS=N to make multiple connections within each verification round. Any round or connection failure will fail the current test. -Waiting for the network: +Bootstrapping the network: + + Chutney expects a tor network to bootstrap in these stages: + 1. All directory authorities (DirAuths) bootstrap to 100%. + 2. The DirAuths produce the first consensus. + + 3. Relays bootstrap to 100%. + 4. Relays with "AssumeReachable 1" publish their descriptors to the + DirAuths. + 5. The DirAuths produce a consensus containing enough nodes for + relay reachability self-tests (usually 3 nodes). + + 6. Relays perform ORPort reachability self-tests. + If the consensus contains at least 1 exit, relays also perform DirPort + reachability self-tests. + 7. Relays publish their descriptors to the DirAuths. + 8. The DirAuths produce a consensus containing all authorities (including + the bridge authority, if present) and all relays. + + 9. Clients bootstrap to 100%. + (Clients start bootstrapping as soon as the first consensus is + available.) + 10. Onion Services publish their descriptors to Onion Service directories + (otherwise known as hidden service directories, or HSDirs). The tools/test-network.sh script uses the chutney wait_for_bootstrap command to wait for up to CHUTNEY_START_TIME seconds (default: 120), checking - whether the logged bootstrapped status for every node is 100%. It it is, - great: it succeeds. If not, it dumps the bootstrap statuses and exits. - test-network.sh does not exit immediately if a tor node fails to bootstrap. - Instead, it attempts to verify. We'll add an option to fail on tor - bootstrap failure in #20473. + whether the logged bootstrapped status for every node is 100% (step 9). If + all the nodes have bootstrapped, wait_for_bootstrap succeeds. If some nodes + have not bootstrapped, wait_for_bootstrap dumps the bootstrap statuses, and + exits with a failure. But test-network.sh does not exit immediately if a tor + node fails to bootstrap. Instead, it attempts to verify. + +Verifying the network: Commands like "chutney verify" start immediately, and keep trying for CHUTNEY_BOOTSTRAP_TIME seconds (default: 60). If it hasn't been @@ -173,6 +198,8 @@ Waiting for the network: the script leaves the network running, and exits after CHUTNEY_START_TIME (without verifying). +Shutting down the network: + The tools/test-network.sh script waits CHUTNEY_STOP_TIME seconds after verifying, then exits (default: immediately). If CHUTNEY_STOP_TIME is negative, the script leaves the network running, and exits after verifying.

1 0

[chutney/master] test-network: Require the network to bootstrap
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit bf3197b0434ec28c319794fae31fdb5be6491107 Author: teor <teor(a)torproject.org> Date: Wed Feb 19 16:17:19 2020 +1000 test-network: Require the network to bootstrap Before launching "chutney verify", test-network checks that all the nodes have bootstrapped. Part of 33232. --- README | 3 +-- tools/test-network-impl.sh | 14 ++++++++++---- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/README b/README index 2e850ff..0d14d64 100644 --- a/README +++ b/README @@ -187,8 +187,7 @@ Bootstrapping the network: whether the logged bootstrapped status for every node is 100% (step 9). If all the nodes have bootstrapped, wait_for_bootstrap succeeds. If some nodes have not bootstrapped, wait_for_bootstrap dumps the bootstrap statuses, and - exits with a failure. But test-network.sh does not exit immediately if a tor - node fails to bootstrap. Instead, it attempts to verify. + exits with a failure. Verifying the network: diff --git a/tools/test-network-impl.sh b/tools/test-network-impl.sh index a272f30..dc8d28c 100755 --- a/tools/test-network-impl.sh +++ b/tools/test-network-impl.sh @@ -29,10 +29,16 @@ CHUTNEY="$CHUTNEY_PATH/chutney" if [ "$CHUTNEY_START_TIME" -ge 0 ]; then $ECHO "Waiting $CHUTNEY_START_TIME seconds for a consensus containing relays to be generated..." - # We log bootstrap status for debugging purposes. - # We'll fix bootstrap errors in #20473. - "$CHUTNEY" wait_for_bootstrap "$CHUTNEY_NETWORK" \ - || $ECHO "Tor bootstrap failed, ignoring for now." + # We require the network to bootstrap, before we verify + if ! "$CHUTNEY" wait_for_bootstrap "$CHUTNEY_NETWORK"; then + "$DIAGNOSTICS" + CHUTNEY_WARNINGS_IGNORE_EXPECTED=false \ + CHUTNEY_WARNINGS_SUMMARY=false \ + "$WARNING_COMMAND" + "$WARNINGS" + $ECHO "chutney boostrap failed (in wait_for_bootstrap)" + exit 1 + fi else $ECHO "Chutney network launched and running. To stop the network, use:" $ECHO "$CHUTNEY stop $CHUTNEY_NETWORK"

1 0

[chutney/master] TorNet: Use wallclock elapsed time for bootstrap
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit ea611d7633d4aa87c725e9cb7133501c338df8ff Author: teor <teor(a)torproject.org> Date: Wed Feb 19 15:37:27 2020 +1000 TorNet: Use wallclock elapsed time for bootstrap The previous code counted the number of sleeps. Using wallclock time is more accurate, and the code is simpler. Part of 33232. --- lib/chutney/TorNet.py | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index 6dbda36..0917fbf 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -1448,12 +1448,13 @@ class Network(object): print("Sending SIGHUP to nodes") return all([n.getController().hup() for n in self._nodes]) + CHECK_NETWORK_STATUS_DELAY = 0.5 + def wait_for_bootstrap(self): print("Waiting for nodes to bootstrap...") - limit = getenv_int("CHUTNEY_START_TIME", 60) - delay = 0.5 + start = time.time() + limit = start + getenv_int("CHUTNEY_START_TIME", 60) controllers = [n.getController() for n in self._nodes] - elapsed = 0.0 most_recent_status = [ None ] * len(controllers) while True: all_bootstrapped = True @@ -1464,13 +1465,15 @@ class Network(object): if pct != 100: all_bootstrapped = False + now = time.time() if all_bootstrapped: - print("Everything bootstrapped after %s sec"%elapsed) + elapsed = now - start + print("Everything bootstrapped after {} sec" + .format(int(elapsed))) return True - if elapsed >= limit: + if now >= limit: break - time.sleep(delay) - elapsed += delay + time.sleep(Network.CHECK_NETWORK_STATUS_DELAY) print("Bootstrap failed. Node status:") for c, status in zip(controllers,most_recent_status):

1 0

[chutney/master] TorNet: Print incremental bootstrap progress
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit f366fe1361e353e329a037ed6ba7d2e4b291ae19 Author: teor <teor(a)torproject.org> Date: Wed Feb 19 15:53:07 2020 +1000 TorNet: Print incremental bootstrap progress wait_for_bootstrap now prints bootstrap progress every 20 seconds. Part of 33232. --- lib/chutney/TorNet.py | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index 3fa94ac..fb6486e 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -1448,18 +1448,29 @@ class Network(object): print("Sending SIGHUP to nodes") return all([n.getController().hup() for n in self._nodes]) - def print_bootstrap_status(self, controllers, most_recent_status): - print("Bootstrap failed. Node status:") + def print_bootstrap_status(self, controllers, most_recent_status, + elapsed=None, + msg="Bootstrap in progress"): + header = "Node status:" + elapsed_msg = "" + if elapsed: + elapsed_msg = " for {} seconds".format(int(elapsed)) + if msg: + header = "{}{}. {}".format(msg, elapsed_msg, header) + print(header) for c, status in zip(controllers,most_recent_status): c.check(listRunning=False, listNonRunning=True) print("{}: {}".format(c.getNick(), status)) + print() CHECK_NETWORK_STATUS_DELAY = 0.5 + PRINT_NETWORK_STATUS_DELAY = 20.0 def wait_for_bootstrap(self): - print("Waiting for nodes to bootstrap...") + print("Waiting for nodes to bootstrap...\n") start = time.time() limit = start + getenv_int("CHUTNEY_START_TIME", 60) + next_print_status = start + Network.PRINT_NETWORK_STATUS_DELAY controllers = [n.getController() for n in self._nodes] most_recent_status = [ None ] * len(controllers) while True: @@ -1472,16 +1483,22 @@ class Network(object): all_bootstrapped = False now = time.time() + elapsed = now - start if all_bootstrapped: - elapsed = now - start print("Everything bootstrapped after {} sec" .format(int(elapsed))) return True if now >= limit: break + if now >= next_print_status: + self.print_bootstrap_status(controllers, most_recent_status, + elapsed=elapsed) + next_print_status = now + Network.PRINT_NETWORK_STATUS_DELAY time.sleep(Network.CHECK_NETWORK_STATUS_DELAY) - self.print_bootstrap_status(controllers, most_recent_status) + self.print_bootstrap_status(controllers, most_recent_status, + elapsed=elapsed, + msg="Bootstrap failed") return False def stop(self):

1 0

[chutney/master] TorNet: Improve template variable docs
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit 5040534d38b08034d0bd20995f2651bd50ca8f18 Author: teor <teor(a)torproject.org> Date: Mon Mar 9 15:54:00 2020 +1000 TorNet: Improve template variable docs Part of 33379. --- lib/chutney/TorNet.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index fb6486e..d5b90a6 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -563,9 +563,9 @@ class LocalNodeBuilder(NodeBuilder): # Environment members used: # torrc -- which torrc file to use # torrc_template_path -- path to search for torrc files and include files - # authority -- bool -- are we an authority? + # authority -- bool -- are we an authority? (includes bridge authorities) # bridgeauthority -- bool -- are we a bridge authority? - # relay -- bool -- are we a relay? + # relay -- bool -- are we a relay? (includes exits and bridges) # bridge -- bool -- are we a bridge? # hs -- bool -- are we a hidden service? # nodenum -- int -- set by chutney -- which unique node index is this? @@ -1182,6 +1182,10 @@ class TorEnviron(chutney.Templating.Environ): process exits, the child tor processes will exit dns_conf: the path to a DNS config file for Tor Exits. If this file is empty or unreadable, Tor will try 127.0.0.1:53. + authority: are we an authority? (includes bridge authorities) + bridgeauthority: are we a bridge authority? + relay: are we a relay? (includes exits and bridges) + bridge: are we a bridge? """ def __init__(self, parent=None, **kwargs):

1 0

[chutney/master] README: Document "wait for relays in the consensus"
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit f40871d780fcc4d095ea955261465a3e67f28f52 Author: teor <teor(a)torproject.org> Date: Wed Feb 19 20:57:50 2020 +1000 README: Document "wait for relays in the consensus" Part of 33379. --- README | 47 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 33 insertions(+), 14 deletions(-) diff --git a/README b/README index 0d14d64..0f4cf4c 100644 --- a/README +++ b/README @@ -162,32 +162,51 @@ Bootstrapping the network: Chutney expects a tor network to bootstrap in these stages: 1. All directory authorities (DirAuths) bootstrap to 100%. 2. The DirAuths produce the first consensus. - - 3. Relays bootstrap to 100%. - 4. Relays with "AssumeReachable 1" publish their descriptors to the + Usually, this consensus only contains authorities. + 3. The DirAuths produce a bootstrap consensus. + This consensus has enough relays for: + * clients and relays to bootstrap, and + * relays to perform reachability self-tests. + Usually, this consensus needs at least 3 nodes. + This consensus is usually the first or second consensus. + + 4. Relays bootstrap to 100%. + 5. Relays with "AssumeReachable 1" publish their descriptors to the DirAuths. - 5. The DirAuths produce a consensus containing enough nodes for - relay reachability self-tests (usually 3 nodes). 6. Relays perform ORPort reachability self-tests. If the consensus contains at least 1 exit, relays also perform DirPort reachability self-tests. 7. Relays publish their descriptors to the DirAuths. - 8. The DirAuths produce a consensus containing all authorities (including - the bridge authority, if present) and all relays. + 8. The DirAuths produce a full consensus. A full consensus contains: + * the authorities + * any bridge authorities, if present, and + * all relays (including exits). + Bridges, clients, and onion services are not included in the consensus. 9. Clients bootstrap to 100%. - (Clients start bootstrapping as soon as the first consensus is - available.) + (Clients can bootstrap as soon as the consensus contains enough nodes, + so step 9 depends on step 3, not step 8.) 10. Onion Services publish their descriptors to Onion Service directories (otherwise known as hidden service directories, or HSDirs). The tools/test-network.sh script uses the chutney wait_for_bootstrap - command to wait for up to CHUTNEY_START_TIME seconds (default: 120), checking - whether the logged bootstrapped status for every node is 100% (step 9). If - all the nodes have bootstrapped, wait_for_bootstrap succeeds. If some nodes - have not bootstrapped, wait_for_bootstrap dumps the bootstrap statuses, and - exits with a failure. + command to wait for the network to bootstrap. wait_for_bootstrap waits up to + CHUTNEY_START_TIME seconds (default: 120), checking whether the logged + bootstrapped status for every node is 100% (step 9). Once all the nodes have + bootstrapped, wait_for_bootstrap starts waiting for a full consensus. + + When waiting for a full consensus, wait_for_bootstrap checks if each relay + is in the consensus (step 8). Once all the relays are in the consensus, + wait_for_bootstrap succeeds. + + If the time limit has elapsed, and some nodes have not bootstrapped, or + there are some nodes missing from the consensus, wait_for_bootstrap dumps the + bootstrap statuses, and exits with a failure. + + (wait_for_bootstrap does not check if all the onion services have published + their descriptors to the HSDirs. But we would like to implement that feature + some time.) Verifying the network:

1 0

[chutney/master] warnings: Use $CHUTNEY_WARNINGS_SKIP
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit 25dd564a422ddf79314c96beee2cbc22665866e0 Author: teor <teor(a)torproject.org> Date: Mon Mar 9 15:52:53 2020 +1000 warnings: Use $CHUTNEY_WARNINGS_SKIP When $CHUTNEY_WARNINGS_SKIP is true, skip all warnings. --- tools/warnings.sh | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tools/warnings.sh b/tools/warnings.sh index 850bd49..174393e 100755 --- a/tools/warnings.sh +++ b/tools/warnings.sh @@ -21,6 +21,13 @@ # # CHUTNEY_WARNINGS_SUMMARY: # set to "true" to merge warnings from all instances +# +# CHUTNEY_WARNINGS_SKIP: +# set to "true" to skip all warnings + +if [ "$CHUTNEY_WARNINGS_SKIP" = true ]; then + exit 0 +fi if [ ! -d "$CHUTNEY_PATH" ] || [ ! -x "$CHUTNEY_PATH/chutney" ]; then # looks like a broken path: use the path to this tool instead

1 0

[chutney/master] test-network: Fix the wording of a status message
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit 75aea5d138d8473ab1e8f508fd626eff89150b00 Author: teor <teor(a)torproject.org> Date: Mon Mar 9 15:52:18 2020 +1000 test-network: Fix the wording of a status message --- tools/test-network-impl.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/test-network-impl.sh b/tools/test-network-impl.sh index dc8d28c..4fa4482 100755 --- a/tools/test-network-impl.sh +++ b/tools/test-network-impl.sh @@ -28,7 +28,7 @@ export CHUTNEY_STOP_TIME=${CHUTNEY_STOP_TIME:-0} CHUTNEY="$CHUTNEY_PATH/chutney" if [ "$CHUTNEY_START_TIME" -ge 0 ]; then - $ECHO "Waiting $CHUTNEY_START_TIME seconds for a consensus containing relays to be generated..." + $ECHO "Waiting up to $CHUTNEY_START_TIME seconds for a consensus containing relays to be generated..." # We require the network to bootstrap, before we verify if ! "$CHUTNEY" wait_for_bootstrap "$CHUTNEY_NETWORK"; then "$DIAGNOSTICS"

1 0

[chutney/master] TorNet: Add status code constants
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit d62288806414b45c584f5d8fd4b8b29e3dcae618 Author: teor <teor(a)torproject.org> Date: Mon Mar 9 15:57:39 2020 +1000 TorNet: Add status code constants And update some old code to use them. Part of 33379. --- lib/chutney/TorNet.py | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index d5b90a6..415b284 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -1037,6 +1037,14 @@ class LocalNodeController(NodeController): logname = "notice.log" return os.path.join(datadir, logname) + INTERNAL_ERROR_CODE = -400 + NOT_YET_IMPLEMENTED_CODE = -300 + MISSING_FILE_CODE = -200 + NO_RECORDS_CODE = -100 + INCOMPLETE_RECORDS_CODE = -50 + NO_PROGRESS_CODE = 0 + SUCCESS_CODE = 100 + def getLastBootstrapStatus(self): """Look through the logs and return the last bootstrap message received as a 3-tuple of percentage complete, keyword @@ -1044,8 +1052,11 @@ class LocalNodeController(NodeController): """ logfname = self.getLogfile() if not os.path.exists(logfname): - return (-200, "no_logfile", "There is no logfile yet.") - percent,keyword,message=-100,"no_message","No bootstrap messages yet." + return (LocalNodeController.MISSING_FILE_CODE, + "no_logfile", "There is no logfile yet.") + percent = LocalNodeController.NO_RECORDS_CODE + keyword = "no_message" + message = "No bootstrap messages yet." with open(logfname, 'r') as f: for line in f: m = re.search(r'Bootstrapped (\d+)%(?: $([^$]*)\))?: (.*)',

1 0

[chutney/master] TorNet: Tweak the bootstrap status timings
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit dd33dc1f1ef7101fab446b99513b17bd18cf1c13 Author: teor <teor(a)torproject.org> Date: Mon Mar 9 15:59:17 2020 +1000 TorNet: Tweak the bootstrap status timings Part of 33379. --- lib/chutney/TorNet.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index 415b284..e76d9f6 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -1478,8 +1478,8 @@ class Network(object): print("{}: {}".format(c.getNick(), status)) print() - CHECK_NETWORK_STATUS_DELAY = 0.5 - PRINT_NETWORK_STATUS_DELAY = 20.0 + CHECK_NETWORK_STATUS_DELAY = 1.0 + PRINT_NETWORK_STATUS_DELAY = 10.0 def wait_for_bootstrap(self): print("Waiting for nodes to bootstrap...\n")

1 0