October 2014 - tor-commits - lists.torproject.org

[chutney/master] Enable Debugging of chutney-launched tor nodes
by nickm＠torproject.org 08 Oct '14

08 Oct '14

commit 1836b7244e53031217667e4d43159f520a496ef6 Author: teor <teor2345(a)gmail.com> Date: Thu Oct 2 00:30:43 2014 +1000 Enable Debugging of chutney-launched tor nodes Set DisableDebuggerAttachment to 0, because if you're running a test network, you might actually want to use a debugger on tor. --- torrc_templates/common.i | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/torrc_templates/common.i b/torrc_templates/common.i index 2b5e4fa..42eebc4 100644 --- a/torrc_templates/common.i +++ b/torrc_templates/common.i @@ -9,5 +9,5 @@ Log notice file ${dir}/notice.log Log info file ${dir}/info.log ProtocolWarnings 1 SafeLogging 0 +DisableDebuggerAttachment 0 ${authorities} -

1 0

[chutney/master] Check RunAsDaemon before using wait()
by nickm＠torproject.org 08 Oct '14

08 Oct '14

commit f34ac80509272296c25de03c71763c995b1f7eb6 Author: teor <teor2345(a)gmail.com> Date: Thu Oct 2 01:58:00 2014 +1000 Check RunAsDaemon before using wait() Avoid hangs by checking for RunAsDaemon 1 in the torrc before using wait() on a launched tor process. Iff it's absent, use poll(). A configurable poll() time is provided in env['poll_launch_time'], and a default in env['poll_launch_time_default'] is used when the torrc is unexpectedly missing RunAsDaemon 1. (TODO: We may only need one of these?) --- lib/chutney/TorNet.py | 83 ++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 75 insertions(+), 8 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index e4ddc7a..87f1abe 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -556,12 +556,35 @@ class LocalNodeController(NodeController): sys.exit(0) else: raise - # XXXX this requires that RunAsDaemon is set. - p.wait() - if p.returncode != 0: - print "Couldn't launch %s (%s): %s" % (self._env['nick'], - " ".join(cmdline), - p.returncode) + if self.waitOnLaunch(): + # this requires that RunAsDaemon is set + p.wait() + else: + # this does not require RunAsDaemon to be set, but is slower. + # + # poll() only catches failures before the call itself + # so let's sleep a little first + # this does, of course, slow down process launch + # which can require an adjustment to the voting interval + # + # avoid writing a newline or space when polling + # so output comes out neatly + sys.stdout.write('.') + sys.stdout.flush() + time.sleep(self._env['poll_launch_time']) + p.poll() + if p.returncode != None and p.returncode != 0: + if self._env['poll_launch_time'] is None: + print "Couldn't launch %s (%s): %s" % (self._env['nick'], + " ".join(cmdline), + p.returncode) + else: + print ("Couldn't poll %s (%s) " + "after waiting %s seconds for launch" + ": %s") % (self._env['nick'], + " ".join(cmdline), + self._env['poll_launch_time'], + p.returncode) return False return True @@ -581,6 +604,33 @@ class LocalNodeController(NodeController): self._env['nick']) os.remove(lf) + def waitOnLaunch(self): + """Check whether we can wait() for the tor process to launch""" + # TODO: is this the best place for this code? + # RunAsDaemon default is 0 + runAsDaemon = False + with open(self._getTorrcFname(), 'r') as f: + for line in f.readlines(): + stline = line.strip() + # if the line isn't all whitespace or blank + if len(stline) > 0: + splline = stline.split() + # if the line has at least two tokens on it + if (len(splline) > 0 + and splline[0].lower() == "RunAsDaemon".lower() + and splline[1] == "1"): + # use the RunAsDaemon value from the torrc + # TODO: multiple values? + runAsDaemon = True + if runAsDaemon: + # we must use wait() instead of poll() + self._env['poll_launch_time'] = None + return True; + else: + # we must use poll() instead of wait() + if self._env['poll_launch_time'] is None: + self._env['poll_launch_time'] = self._env['poll_launch_time_default'] + return False; DEFAULTS = { 'authority': False, @@ -605,6 +655,12 @@ DEFAULTS = { 'authorities': "AlternateDirAuthority bleargh bad torrc file!", 'bridges': "Bridge bleargh bad torrc file!", 'core': True, + # poll_launch_time: None means wait on launch (requires RunAsDaemon), + # otherwise, poll after that many seconds (can be fractional/decimal) + 'poll_launch_time': None, + # Used when poll_launch_time is None, but RunAsDaemon is not set + # Set low so that we don't interfere with the voting interval + 'poll_launch_time_default': 0.1, } @@ -729,8 +785,19 @@ class Network(object): self.start() def start(self): - print "Starting nodes" - return all([n.getController().start() for n in self._nodes]) + if self._dfltEnv['poll_launch_time'] is not None: + # format polling correctly - avoid printing a newline + sys.stdout.write("Starting nodes") + sys.stdout.flush() + else: + print "Starting nodes" + rv = all([n.getController().start() for n in self._nodes]) + # now print a newline unconditionally - this stops poll()ing + # output from being squashed together, at the cost of a blank + # line in wait()ing output + print "" + return rv + def hup(self): print "Sending SIGHUP to nodes"

1 0

[chutney/master] Force authorities to vote Exit & Guard for all nodes
by nickm＠torproject.org 08 Oct '14

08 Oct '14

commit c18823e44b8ee6de3c1b7723fbc650a96810c082 Author: teor <teor2345(a)gmail.com> Date: Thu Oct 2 02:07:19 2014 +1000 Force authorities to vote Exit & Guard for all nodes Use the new tor TestingDirAuthVoteExit option, and the TestingDirAuthVoteGuard option, to make all authorities vote all relays (and authorities) as exits and guards. This ensures that tor's make test-network will work. But it's a sledgehammer approach. Requires either the new tor TestingDirAuthVoteExit option, or the issue-13161-check-torrc-options patch for chutney. --- torrc_templates/authority.tmpl | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/torrc_templates/authority.tmpl b/torrc_templates/authority.tmpl index 7bf99af..eda123c 100644 --- a/torrc_templates/authority.tmpl +++ b/torrc_templates/authority.tmpl @@ -7,3 +7,13 @@ TestingV3AuthInitialVotingInterval 300 TestingV3AuthInitialVoteDelay 2 TestingV3AuthInitialDistDelay 2 TestingV3AuthVotingStartOffset 0 + +# Work around situations where the Exit and Guard flags aren't being set +# These flags are set eventually, but it takes ~30 minutes +# We could be more precise here, but it's easiest just to vote everything +# Clients are sensible enough to filter out Exits without any exit ports, +# and Guards without ORPorts +# If your tor doesn't recognise TestingDirAuthVoteExit, update your chutney +# to a version that includes the issue-13161-check-torrc-options features +TestingDirAuthVoteExit * +TestingDirAuthVoteGuard *

1 0

[chutney/master] Merge branch 'issue-13161-typos' into issue-13161-check-torrc-options
by nickm＠torproject.org 08 Oct '14

08 Oct '14

commit df1d0a17c2e920167a03f268c379700d9f559332 Merge: f34ac80 6d5cb08 Author: teor <teor2345(a)gmail.com> Date: Thu Oct 2 02:25:43 2014 +1000 Merge branch 'issue-13161-typos' into issue-13161-check-torrc-options Fix a single typo in lib/chutney/TorNet.py lib/chutney/TorNet.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --cc lib/chutney/TorNet.py index 87f1abe,32dc173..f18503a --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@@ -223,68 -220,10 +223,68 @@@ class LocalNodeBuilder(NodeBuilder) torrc_template = self._getTorrcTemplate() output = torrc_template.format(self._env) if checkOnly: - # XXXX Is it time-cosuming to format? If so, cache here. + # XXXX Is it time-consuming to format? If so, cache here. return + # now filter the options we're about to write, commenting out + # the options that the current tor binary doesn't support + tor = self._env['tor'] + # find the options the current tor binary supports, and cache them + if tor not in _TORRC_OPTIONS: + # Note: some versions of tor (e.g. 0.2.4.23) require + # --list-torrc-options to be the first argument + cmdline = [ + tor, + "--list-torrc-options", + "--hush"] + try: + opts = subprocess.check_output(cmdline, + bufsize=-1, + universal_newlines=True) + except OSError as e: + # only catch file not found error + if e.errno == errno.ENOENT: + print ("Cannot find tor binary %r. Use " + "CHUTNEY_TOR environment variable to set the " + "path, or put the binary into $PATH.") % tor + sys.exit(0) + else: + raise + # check we received a list of options, and nothing else + assert re.match(r'(^\w+$)+', opts, flags=re.MULTILINE) + torrc_opts = opts.split() + # cache the options for this tor binary's path + _TORRC_OPTIONS[tor] = torrc_opts + else: + torrc_opts = _TORRC_OPTIONS[tor] + # check if each option is supported before writing it + # TODO: what about unsupported values? + # e.g. tor 0.2.4.23 doesn't support TestingV3AuthInitialVoteDelay 2 + # but later version do. I say throw this one to the user. with open(fn_out, 'w') as f: - f.write(output) + # we need to do case-insensitive option comparison + # even if this is a static whitelist, + # so we convert to lowercase as close to the loop as possible + lower_opts = [opt.lower() for opt in torrc_opts] + # keep ends when splitting lines, so we can write them out + # using writelines() without messing around with "\n"s + for line in output.splitlines(True): + # check if the first word on the line is a supported option, + # preserving empty lines and comment lines + sline = line.strip() + if (len(sline) == 0 + or sline[0] == '#' + or sline.split()[0].lower() in lower_opts): + f.writelines([line]) + else: + # well, this could get spammy + # TODO: warn once per option per tor binary + # TODO: print tor version? + print ("The tor binary at %r does not support the " + "option in the torrc line:\n" + "%r") % (tor, line.strip()) + # we could decide to skip these lines entirely + # TODO: write tor version? + f.writelines(["# " + tor + " unsupported: " + line]) def _getTorrcTemplate(self): """Return the template used to write the torrc for this node."""

1 0

[tor/master] Add TestingDirAuthVoteExit option (like TestingDirAuthVoteGuard)
by nickm＠torproject.org 08 Oct '14

08 Oct '14

commit 27f30040f63d13234d07e58cf3d0d2a3cbd93ce5 Author: teor <teor2345(a)gmail.com> Date: Wed Oct 1 17:44:21 2014 +1000 Add TestingDirAuthVoteExit option (like TestingDirAuthVoteGuard) Add the TestingDirAuthVoteExit option, a list of nodes to vote Exit for, regardless of their uptime, bandwidth, or exit policy. TestingTorNetwork must be set for this option to have any effect. Works around an issue where authorities would take up to 35 minutes to give nodes the Exit flag in a test network, despite short consensus intervals. Partially implements ticket 13161. --- changes/feature13161-TestingDirAuthVoteExit | 7 +++++++ doc/tor.1.txt | 9 +++++++++ src/or/config.c | 1 + src/or/dirserv.c | 15 +++++++++++---- src/or/or.h | 4 ++++ 5 files changed, 32 insertions(+), 4 deletions(-) diff --git a/changes/feature13161-TestingDirAuthVoteExit b/changes/feature13161-TestingDirAuthVoteExit new file mode 100644 index 0000000..d6c8f41 --- /dev/null +++ b/changes/feature13161-TestingDirAuthVoteExit @@ -0,0 +1,7 @@ + o Minor features (testing): + - Add the TestingDirAuthVoteExit option, a list of nodes to vote + Exit for regardless of their uptime, bandwidth, or exit policy. + TestingTorNetwork must be set for this option to have any effect. + Works around an issue where authorities would take up to 35 minutes + to give nodes the Exit flag in a test network, despite short + consensus intervals. Partially implements ticket 13161. diff --git a/doc/tor.1.txt b/doc/tor.1.txt index ff17819..c0b8c11 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -2197,6 +2197,15 @@ The following options are used for running a testing Tor network. Try this often to download a v3 authority certificate before giving up. Changing this requires that **TestingTorNetwork** is set. (Default: 8) +[[TestingDirAuthVoteExit]] **TestingDirAuthVoteExit** __node__,__node__,__...__:: + A list of identity fingerprints, nicknames, country codes and + address patterns of nodes to vote Exit for regardless of their + uptime, bandwidth, or exit policy. See the **ExcludeNodes** + option for more information on how to specify nodes. + + + In order for this option to have any effect, **TestingTorNetwork** + has to be set. + [[TestingDirAuthVoteGuard]] **TestingDirAuthVoteGuard** __node__,__node__,__...__:: A list of identity fingerprints, nicknames, country codes and address patterns of nodes to vote Guard for regardless of their diff --git a/src/or/config.c b/src/or/config.c index 921d032..3b37a12 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -439,6 +439,7 @@ static config_var_t option_vars_[] = { V(TestingDescriptorMaxDownloadTries, UINT, "8"), V(TestingMicrodescMaxDownloadTries, UINT, "8"), V(TestingCertMaxDownloadTries, UINT, "8"), + V(TestingDirAuthVoteExit, ROUTERSET, NULL), V(TestingDirAuthVoteGuard, ROUTERSET, NULL), VAR("___UsingTestNetworkDefaults", BOOL, UsingTestNetworkDefaults_, "0"), diff --git a/src/or/dirserv.c b/src/or/dirserv.c index 374cfa6..c8f47e6 100644 --- a/src/or/dirserv.c +++ b/src/or/dirserv.c @@ -2169,12 +2169,19 @@ set_routerstatus_from_routerinfo(routerstatus_t *rs, rs->ipv6_orport = ri->ipv6_orport; } - /* Iff we are in a testing network, use TestingDirAuthVoteGuard to + /* Iff we are in a testing network, use TestingDirAuthVoteExit to + give out Exit flags, and TestingDirAuthVoteGuard to give out Guard flags. */ - if (options->TestingTorNetwork && - routerset_contains_routerstatus(options->TestingDirAuthVoteGuard, + if (options->TestingTorNetwork) { + if (routerset_contains_routerstatus(options->TestingDirAuthVoteExit, + rs, 0)) { + rs->is_exit = 1; + } + + if (routerset_contains_routerstatus(options->TestingDirAuthVoteGuard, rs, 0)) { - rs->is_possible_guard = 1; + rs->is_possible_guard = 1; + } } } diff --git a/src/or/or.h b/src/or/or.h index 54cee46..4130ea6 100644 --- a/src/or/or.h +++ b/src/or/or.h @@ -4056,6 +4056,10 @@ typedef struct { /** Minimum value for the Fast flag threshold on testing networks. */ uint64_t TestingMinFastFlagThreshold; + /** Relays in a testing network which should be voted Exit + * regardless of exit policy. */ + routerset_t *TestingDirAuthVoteExit; + /** Relays in a testing network which should be voted Guard * regardless of uptime and bandwidth. */ routerset_t *TestingDirAuthVoteGuard;

1 0

[tor/master] test-network.sh: Use "/bin/echo -n" rather than builtin echo
by nickm＠torproject.org 08 Oct '14

08 Oct '14

commit 7c0215f8cafd2b7e1d251439b97ad97ec46e6211 Author: teor <teor2345(a)gmail.com> Date: Wed Oct 1 17:56:53 2014 +1000 test-network.sh: Use "/bin/echo -n" rather than builtin echo The default shell on OS X is bash, which has a builtin echo. When called in "sh" mode, this echo does not accept "-n". This patch uses "/bin/echo -n" instead. Partially fixes issue 13161. --- changes/bug13161-test-network-echo-n | 3 +++ src/test/test-network.sh | 6 ++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/changes/bug13161-test-network-echo-n b/changes/bug13161-test-network-echo-n new file mode 100644 index 0000000..501ebdd --- /dev/null +++ b/changes/bug13161-test-network-echo-n @@ -0,0 +1,3 @@ + o Minor bugfixes: + - Stop using "echo -n", as some shells' built-in echo doesn't support + "-n". Instead, use "/bin/echo -n". Partially fixes bug 13161. diff --git a/src/test/test-network.sh b/src/test/test-network.sh index 7b59864..e1bed5c 100755 --- a/src/test/test-network.sh +++ b/src/test/test-network.sh @@ -1,5 +1,7 @@ #! /bin/sh +ECHO_N="/bin/echo -n" + until [ -z $1 ] do case $1 in @@ -40,8 +42,8 @@ PATH="$TOR_DIR/src/or:$TOR_DIR/src/tools:$PATH" # Sleep some, waiting for the network to bootstrap. # TODO: Add chutney command 'bootstrap-status' and use that instead. BOOTSTRAP_TIME=18 -echo -n "$myname: sleeping for $BOOTSTRAP_TIME seconds" +$ECHO_N "$myname: sleeping for $BOOTSTRAP_TIME seconds" n=$BOOTSTRAP_TIME; while [ $n -gt 0 ]; do - sleep 1; n=$(expr $n - 1); echo -n . + sleep 1; n=$(expr $n - 1); $ECHO_N . done; echo "" ./chutney verify $CHUTNEY_NETWORK

1 0

[tor/master] Add test-network delay option
by nickm＠torproject.org 08 Oct '14

08 Oct '14

commit bae73343902072c24469d42deffe90b22b7fff6e Author: teor <teor2345(a)gmail.com> Date: Wed Oct 1 18:05:04 2014 +1000 Add test-network delay option Add a --delay option to test-network.sh, which configures the delay before the chutney network tests for data transmission. The default remains at 18 seconds if the argument isn't specified. Apparently we should be using bootstrap status for this (eventually). Partially implements ticket 13161. --- changes/feature13161-test-network-delay-option | 4 ++++ src/test/test-network.sh | 6 +++++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/changes/feature13161-test-network-delay-option b/changes/feature13161-test-network-delay-option new file mode 100644 index 0000000..1cf2e71 --- /dev/null +++ b/changes/feature13161-test-network-delay-option @@ -0,0 +1,4 @@ + o Minor features (testing): + - Add a --delay option to test-network.sh, which configures the delay + before the chutney network tests for data transmission. + Partially implements ticket 13161. diff --git a/src/test/test-network.sh b/src/test/test-network.sh index e1bed5c..4fe4e63 100755 --- a/src/test/test-network.sh +++ b/src/test/test-network.sh @@ -17,6 +17,10 @@ do export NETWORK_FLAVOUR="$2" shift ;; + --delay|--sleep|--bootstrap-time|--time) + export BOOTSTRAP_TIME="$2" + shift + ;; *) echo "Sorry, I don't know what to do with '$1'." exit 2 @@ -41,7 +45,7 @@ PATH="$TOR_DIR/src/or:$TOR_DIR/src/tools:$PATH" # Sleep some, waiting for the network to bootstrap. # TODO: Add chutney command 'bootstrap-status' and use that instead. -BOOTSTRAP_TIME=18 +BOOTSTRAP_TIME=${BOOTSTRAP_TIME:-18} $ECHO_N "$myname: sleeping for $BOOTSTRAP_TIME seconds" n=$BOOTSTRAP_TIME; while [ $n -gt 0 ]; do sleep 1; n=$(expr $n - 1); $ECHO_N .

1 0

[tor/master] Merge remote-tracking branches 'teor/issue-13161-test-network' and 'teor/issue-13161-TestingDirAuthVoteExit'
by nickm＠torproject.org 08 Oct '14

08 Oct '14

commit cc5571e1f1e4d61d584fe20959b686d798bb7691 Merge: 40375fb bae7334 27f3004 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 8 15:46:29 2014 -0400 Merge remote-tracking branches 'teor/issue-13161-test-network' and 'teor/issue-13161-TestingDirAuthVoteExit' changes/bug13161-test-network-echo-n | 3 +++ changes/feature13161-TestingDirAuthVoteExit | 7 +++++++ changes/feature13161-test-network-delay-option | 4 ++++ doc/tor.1.txt | 9 +++++++++ src/or/config.c | 1 + src/or/dirserv.c | 15 +++++++++++---- src/or/or.h | 4 ++++ src/test/test-network.sh | 12 +++++++++--- 8 files changed, 48 insertions(+), 7 deletions(-) diff --cc src/test/test-network.sh index fb33842,4fe4e63,7b59864..d28fbde --- a/src/test/test-network.sh +++ b/src/test/test-network.sh @@@@ -39,14 -45,9 -39,9 +45,14 @@@@ PATH="$TOR_DIR/src/or:$TOR_DIR/src/tool # Sleep some, waiting for the network to bootstrap. # TODO: Add chutney command 'bootstrap-status' and use that instead. - -BOOTSTRAP_TIME=18 - -echo -n "$myname: sleeping for $BOOTSTRAP_TIME seconds" + +BOOTSTRAP_TIME=${BOOTSTRAP_TIME:-18} + +$ECHO_N "$myname: sleeping for $BOOTSTRAP_TIME seconds" n=$BOOTSTRAP_TIME; while [ $n -gt 0 ]; do - - sleep 1; n=$(expr $n - 1); echo -n . + + sleep 1; n=$(expr $n - 1); $ECHO_N . done; echo "" ./chutney verify $CHUTNEY_NETWORK ++VERIFY_EXIT_STATUS=$? ++# work around a bug/feature in make -j2 (or more) ++# where make hangs if any child processes are still alive ++./chutney stop $CHUTNEY_NETWORK ++exit $VERIFY_EXIT_STATUS

1 0

[tor/master] Stop an apparent test-network hang when used with make -j2
by nickm＠torproject.org 08 Oct '14

08 Oct '14

commit 9a2d4b66479695da0a26a7168d6ddcfd6838151f Author: teor <teor2345(a)gmail.com> Date: Sat Oct 4 13:18:56 2014 +1000 Stop an apparent test-network hang when used with make -j2 If (GNU) Make 3.81 is running processes in parallel using -j2 (or more), it waits until all descendent processes have exited before it returns to the shell. When a command like "make -j2 test-network" is run, this means that test-network.sh apparently hangs until it either make is forcibly terminated, or all the chutney-launched tor processes have exited. A workaround is to use make without -j, or make -j1 if there is an existing alias to "make -jn" in the shell. We resolve this bug in tor by using "chutney stop" after "chutney verify" in test-network.sh. --- changes/bug13331-make-j2-test-network-hang | 3 +++ src/test/test-network.sh | 3 +++ 2 files changed, 6 insertions(+) diff --git a/changes/bug13331-make-j2-test-network-hang b/changes/bug13331-make-j2-test-network-hang new file mode 100644 index 0000000..85c0ad8 --- /dev/null +++ b/changes/bug13331-make-j2-test-network-hang @@ -0,0 +1,3 @@ + o Minor bugfixes: + - Stop an apparent test-network hang when used with make -j2. + Fixes bug 13331. diff --git a/src/test/test-network.sh b/src/test/test-network.sh index 7b59864..d7fdb35 100755 --- a/src/test/test-network.sh +++ b/src/test/test-network.sh @@ -45,3 +45,6 @@ n=$BOOTSTRAP_TIME; while [ $n -gt 0 ]; do sleep 1; n=$(expr $n - 1); echo -n . done; echo "" ./chutney verify $CHUTNEY_NETWORK +# work around a bug/feature in make -j2 (or more) +# where make hangs if any child processes are still alive +./chutney stop $CHUTNEY_NETWORK

1 0

[tor/master] Merge remote-tracking branch 'teor/test-network-hang-on-make-j2'
by nickm＠torproject.org 08 Oct '14

08 Oct '14

commit 40375fbce595b6279d5af5ecf7e6e46f43901558 Merge: 22a0708 c0794d2 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 8 15:42:20 2014 -0400 Merge remote-tracking branch 'teor/test-network-hang-on-make-j2' changes/bug13331-make-j2-test-network-hang | 3 +++ src/test/test-network.sh | 5 +++++ 2 files changed, 8 insertions(+)

1 0