June 2019 - tor-commits - lists.torproject.org

[chutney/master] test scripts: If a network is not supported, exit with 77
by teor＠torproject.org 11 Jun '19

11 Jun '19

commit d8c700e178e171cb49c37a76c574c97ced5295e1 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu May 9 13:07:57 2019 -0400 test scripts: If a network is not supported, exit with 77 This will help automake's test drivers realize that it has been skipped. --- tools/bootstrap-network.sh | 5 +++++ tools/test-network-impl.sh | 4 ++++ tools/test-network.sh | 3 +++ 3 files changed, 12 insertions(+) diff --git a/tools/bootstrap-network.sh b/tools/bootstrap-network.sh index f767e7f..34db0f3 100755 --- a/tools/bootstrap-network.sh +++ b/tools/bootstrap-network.sh @@ -53,6 +53,11 @@ export CHUTNEY_NETWORK="$CHUTNEY_PATH/networks/$NETWORK_FLAVOUR" "$CHUTNEY" stop "$CHUTNEY_NETWORK" +if ! "$CHUTNEY" supported "$CHUTNEY_NETWORK"; then + echo "%myname: network not supported." + exit 77 +fi + echo "$myname: bootstrapping network: $NETWORK_FLAVOUR" "$CHUTNEY" configure "$CHUTNEY_NETWORK" diff --git a/tools/test-network-impl.sh b/tools/test-network-impl.sh index a63fe97..73b942b 100755 --- a/tools/test-network-impl.sh +++ b/tools/test-network-impl.sh @@ -1,6 +1,10 @@ #!/bin/sh if ! "$CHUTNEY_PATH/tools/bootstrap-network.sh" "$NETWORK_FLAVOUR"; then + if test "$?" = 77; then + $ECHO "SKIP: $NETWORK_FLAVOR not supported." + exit 77 + fi CHUTNEY_WARNINGS_IGNORE_EXPECTED=false CHUTNEY_WARNINGS_SUMMARY=false \ "$WARNING_COMMAND" "$WARNINGS" diff --git a/tools/test-network.sh b/tools/test-network.sh index f28acea..4571499 100755 --- a/tools/test-network.sh +++ b/tools/test-network.sh @@ -351,6 +351,9 @@ while [ "$n_attempts" -lt "$max_attempts" ]; do $ECHO "==== Chutney succeeded after $n_attempts attempt(s)." exit 0 fi + if test "$?" = 77; then + exit 77 + fi done $ECHO "Chutney failed $n_attempts times; we may have a problem here."

1 0

[chutney/master] Add a "supported" command that checks whether a network can work.
by teor＠torproject.org 11 Jun '19

11 Jun '19

commit 43d034bb1b00171f3d0607c7fcd7fd7c120f76e9 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu May 9 13:06:25 2019 -0400 Add a "supported" command that checks whether a network can work. Right now a network is unsupported if it requires IPV6 and we don't have it, if the directory authorities don't actually have dirauth support, or if one of the binaries is missing. --- lib/chutney/TorNet.py | 139 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 129 insertions(+), 10 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index a8be5bc..2f1b37b 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -23,6 +23,7 @@ import importlib from chutney.Debug import debug_flag, debug +import chutney.Host import chutney.Templating import chutney.Traffic import chutney.Util @@ -38,6 +39,9 @@ torrc_option_warn_count = 0 # Get verbose tracebacks, so we can diagnose better. cgitb.enable(format="plain") +class MissingBinaryException(Exception): + pass + def getenv_int(envvar, default): """ Return the value of the environment variable 'envar' as an integer, @@ -132,11 +136,14 @@ def _warnMissingTor(tor_path, cmdline, tor_name="tor"): "containing {}.") .format(tor_name, tor_path, " ".join(cmdline), tor_name)) -def run_tor(cmdline): +def run_tor(cmdline, exit_on_missing=True): """Run the tor command line cmdline, which must start with the path or name of a tor binary. Returns the combined stdout and stderr of the process. + + If exit_on_missing is true, warn and exit if the tor binary is missing. + Otherwise, raise a MissingBinaryException. """ if not debug_flag: cmdline.append("--quiet") @@ -149,20 +156,26 @@ def run_tor(cmdline): except OSError as e: # only catch file not found error if e.errno == errno.ENOENT: - _warnMissingTor(cmdline[0], cmdline) - sys.exit(1) + if exit_on_missing: + _warnMissingTor(cmdline[0], cmdline) + sys.exit(1) + else: + raise MissingBinaryException() else: raise except subprocess.CalledProcessError as e: # only catch file not found error if e.returncode == 127: - _warnMissingTor(cmdline[0], cmdline) - sys.exit(1) + if exit_on_missing: + _warnMissingTor(cmdline[0], cmdline) + sys.exit(1) + else: + raise MissingBinaryException() else: raise return stdouterr -def launch_process(cmdline, tor_name="tor", stdin=None): +def launch_process(cmdline, tor_name="tor", stdin=None, exit_on_missing=True): """Launch the command line cmdline, which must start with the path or name of a binary. Use tor_name as the canonical name of the binary. Pass stdin to the Popen constructor. @@ -183,8 +196,11 @@ def launch_process(cmdline, tor_name="tor", stdin=None): except OSError as e: # only catch file not found error if e.errno == errno.ENOENT: - _warnMissingTor(cmdline[0], cmdline, tor_name=tor_name) - sys.exit(1) + if exit_on_missing: + _warnMissingTor(cmdline[0], cmdline, tor_name=tor_name) + sys.exit(1) + else: + raise MissingBinaryException() else: raise return p @@ -206,6 +222,25 @@ def run_tor_gencert(cmdline, passphrase): return stdouterr @chutney.Util.memoized +def tor_exists(tor): + """Return true iff this tor binary exists.""" + try: + run_tor([tor, "--quiet", "--version"], exit_on_missing=False) + return True + except MissingBinaryException: + return False + +(a)chutney.Util.memoized +def tor_gencert_exists(gencert): + """Return true iff this tor-gencert binary exists.""" + try: + p = launch_process([gencert, "--help"], exit_on_missing=False) + p.wait() + return True + except MissingBinaryException: + return False + +(a)chutney.Util.memoized def get_tor_version(tor): """Return the version of the tor binary. Versions are cached for each unique tor path. @@ -240,6 +275,41 @@ def get_torrc_options(tor): return torrc_opts +(a)chutney.Util.memoized +def get_tor_modules(tor): + """Check the list of compile-time modules advertised by the given + 'tor' binary, and return a map from module name to a boolean + describing whether it is supported. + + Unlisted modules are ones that Tor did not treat as compile-time + optional modules. + """ + cmdline = [ + tor, + "--list-modules", + "--quiet" + ] + try: + mods = run_tor(cmdline) + except subprocess.CalledProcessError as e: + # Tor doesn't support --list-modules; act as if it said nothing. + mods = "" + + supported = {} + for line in mods.split("\n"): + m = re.match(r'^(\S+): (yes|no)', line) + if not m: + continue + supported[m.group(1)] = (m.group(2) == "yes") + + return supported + +def tor_has_module(tor, modname, default=True): + """Return true iff the given tor binary supports a given compile-time + module. If the module is not listed, return 'default'. + """ + return get_tor_modules(tor).get(modname, default) + class Node(object): """A Node represents a Tor node or a set of Tor nodes. It's created @@ -358,6 +428,11 @@ class NodeBuilder(_NodeCommon): """Called on each nodes after all nodes configure.""" + def isSupported(self, net): + """Return true if this node appears to have everything it needs; + false otherwise.""" + + class NodeController(_NodeCommon): """Abstract base class. A NodeController is responsible for running a @@ -501,6 +576,21 @@ class LocalNodeBuilder(NodeBuilder): # self.net.addNode(self) pass + def isSupported(self, net): + """Return true if this node appears to have everything it needs; + false otherwise.""" + + if not tor_exists(self._env['tor']): + print("No binary found for %r"%self._env['tor']) + return False + + if self._env['authority']: + if not tor_has_module(self._env['tor'], "dirauth"): + print("No dirauth support in %r"%self._env['tor']) + return False + if not tor_gencert_exists(self._env['tor-gencert']): + print("No binary found for tor-gencert %r"%self._env['tor-gencrrt']) + def _makeDataDir(self): """Create the data directory (with keys subdirectory) for this node. """ @@ -1044,14 +1134,17 @@ class TorEnviron(chutney.Templating.Environ): dns_conf = TorEnviron.OFFLINE_DNS_RESOLV_CONF return "ServerDNSResolvConfFile %s" % (dns_conf) +KNOWN_REQUIREMENTS = { + "IPV6": chutney.Host.is_ipv6_supported +} class Network(object): - """A network of Tor nodes, plus functions to manipulate them """ def __init__(self, defaultEnviron): self._nodes = [] + self._requirements = [] self._dfltEnv = defaultEnviron self._nextnodenum = 0 @@ -1060,6 +1153,12 @@ class Network(object): self._nextnodenum += 1 self._nodes.append(n) + def _addRequirement(self, requirement): + requirement = requirement.upper() + if requirement not in KNOWN_REQUIREMENTS: + raise RuntimemeError(("Unrecognized requirement %r"%requirement)) + self._requirements.append(requirement) + def move_aside_nodes_dir(self): """Move aside the nodes directory, if it exists and is not a link. Used for backwards-compatibility only: nodes is created as a link to @@ -1120,6 +1219,22 @@ class Network(object): for n in self._nodes: n.getBuilder().checkConfig(self) + def supported(self): + """Check whether this network is supported by the set of binaries + and host information we have. + """ + missing_any = False + for r in self._requirements: + if not KNOWN_REQUIREMENTS[r](): + print(("Can't run this network: %s is missing.")) + missing_any = True + for n in self._nodes: + if not n.getBuilder().isSupported(self): + missing_any = False + + if missing_any: + sys.exit(1) + def configure(self): self.create_new_nodes_dir() network = self @@ -1236,6 +1351,10 @@ class Network(object): sys.stdout.flush() +def Require(feature): + network = _THE_NETWORK + network._addRequirement(feature) + def ConfigureNodes(nodelist): network = _THE_NETWORK @@ -1244,7 +1363,6 @@ def ConfigureNodes(nodelist): if n._env['bridgeauthority']: network._dfltEnv['hasbridgeauth'] = True - def getTests(): tests = [] chutney_path = get_absolute_chutney_path() @@ -1275,6 +1393,7 @@ def exit_on_error(err_msg): def runConfigFile(verb, data): _GLOBALS = dict(_BASE_ENVIRON=_BASE_ENVIRON, Node=Node, + Require=Require, ConfigureNodes=ConfigureNodes, _THE_NETWORK=_THE_NETWORK, torrc_option_warn_count=0,

1 0

[chutney/master] Preliminary: add a memoization decorator
by teor＠torproject.org 11 Jun '19

11 Jun '19

commit de96081be3a40378819e14419e7e7f91845cc8c9 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu May 9 11:41:35 2019 -0400 Preliminary: add a memoization decorator We memoize enough things in enough places that we should provide an actual decorator for this. --- lib/chutney/TorNet.py | 65 +++++++++++++++++++-------------------------------- lib/chutney/Util.py | 13 +++++++++++ 2 files changed, 37 insertions(+), 41 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index 16f4345..a8be5bc 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -25,6 +25,7 @@ from chutney.Debug import debug_flag, debug import chutney.Templating import chutney.Traffic +import chutney.Util _BASE_ENVIRON = None _TOR_VERSIONS = None @@ -204,49 +205,40 @@ def run_tor_gencert(cmdline, passphrase): assert empty_stderr is None return stdouterr +(a)chutney.Util.memoized def get_tor_version(tor): """Return the version of the tor binary. Versions are cached for each unique tor path. """ - # find the version of the current tor binary, and cache it - if tor not in _TOR_VERSIONS: - cmdline = [ - tor, - "--version", - ] - tor_version = run_tor(cmdline) - # clean it up a bit - tor_version = tor_version.strip() - tor_version = tor_version.replace("version ", "") - tor_version = tor_version.replace(").", ")") - # check we received a tor version, and nothing else - assert re.match(r'^[-+.() A-Za-z0-9]+$', tor_version) - # cache the version for this tor binary's path - _TOR_VERSIONS[tor] = tor_version - else: - tor_version = _TOR_VERSIONS[tor] + cmdline = [ + tor, + "--version", + ] + tor_version = run_tor(cmdline) + # clean it up a bit + tor_version = tor_version.strip() + tor_version = tor_version.replace("version ", "") + tor_version = tor_version.replace(").", ")") + # check we received a tor version, and nothing else + assert re.match(r'^[-+.() A-Za-z0-9]+$', tor_version) + return tor_version +(a)chutney.Util.memoized def get_torrc_options(tor): """Return the torrc options supported by the tor binary. Options are cached for each unique tor path. """ - # find the options the current tor binary supports, and cache them - if tor not in _TORRC_OPTIONS: - cmdline = [ - tor, - "--list-torrc-options", - ] - opts = run_tor(cmdline) - # check we received a list of options, and nothing else - assert re.match(r'(^\w+$)+', opts, flags=re.MULTILINE) - torrc_opts = opts.split() - # cache the options for this tor binary's path - _TORRC_OPTIONS[tor] = torrc_opts - else: - torrc_opts = _TORRC_OPTIONS[tor] - return torrc_opts + cmdline = [ + tor, + "--list-torrc-options", + ] + opts = run_tor(cmdline) + # check we received a list of options, and nothing else + assert re.match(r'(^\w+$)+', opts, flags=re.MULTILINE) + torrc_opts = opts.split() + return torrc_opts class Node(object): @@ -1319,17 +1311,8 @@ def parseArgs(): def main(): global _BASE_ENVIRON - global _TOR_VERSIONS - global _TORRC_OPTIONS global _THE_NETWORK _BASE_ENVIRON = TorEnviron(chutney.Templating.Environ(**DEFAULTS)) - # _TOR_VERSIONS gets initialised on demand as a map of - # "/path/to/tor" => "Tor version ..." - _TOR_VERSIONS = dict() - # _TORRC_OPTIONS gets initialised on demand as a map of - # "/path/to/tor" => ["SupportedOption1", "SupportedOption2", ...] - # Or it can be pre-populated as a static whitelist of options - _TORRC_OPTIONS = dict() _THE_NETWORK = Network(_BASE_ENVIRON) args = parseArgs() diff --git a/lib/chutney/Util.py b/lib/chutney/Util.py new file mode 100644 index 0000000..f8b838e --- /dev/null +++ b/lib/chutney/Util.py @@ -0,0 +1,13 @@ + + +def memoized(fn): + """Decorator: memoize a function.""" + memory = {} + def memoized_fn(*args, **kwargs): + key = (args, tuple(sorted(kwargs.items()))) + try: + result = memory[key] + except KeyError: + result = memory[key] = fn(*args, **kwargs) + return result + return memoized_fn

1 0

[chutney/master] Merge remote-tracking branch 'tor-github/pr/32'
by teor＠torproject.org 11 Jun '19

11 Jun '19

1 0

[chutney/master] Travis: show the correct macOS tor versions
by teor＠torproject.org 11 Jun '19

11 Jun '19

commit 65ca1288d123565b248d6849131f98d02f2c3e9b Author: teor <teor(a)torproject.org> Date: Fri May 24 15:13:52 2019 +1000 Travis: show the correct macOS tor versions Closes ticket 30820. --- .travis.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/.travis.yml b/.travis.yml index e97703f..50719b8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -26,22 +26,22 @@ matrix: ## The latest tor version in homebrew is on this page: ## https://formulae.brew.sh/formula/tor ## The Travis version might be slightly older. - - env: NETWORK_FLAVOUR="bridges-min" - - env: NETWORK_FLAVOUR="hs-v2-min" - - env: NETWORK_FLAVOUR="hs-v3-min" - - env: NETWORK_FLAVOUR="single-onion-v23" - - env: NETWORK_FLAVOUR="bridges+ipv6-min" + - env: TOR="master-nightly" NETWORK_FLAVOUR="bridges-min" + - env: TOR="master-nightly" NETWORK_FLAVOUR="hs-v2-min" + - env: TOR="master-nightly" NETWORK_FLAVOUR="hs-v3-min" + - env: TOR="master-nightly" NETWORK_FLAVOUR="single-onion-v23" + - env: TOR="stable-release" NETWORK_FLAVOUR="bridges+ipv6-min" os: osx language: c python: ## The IPv6 exit test doesn't actually require IPv6, see #30182. ## But we'll keep this test, because it does test IPv6 exit config. - - env: NETWORK_FLAVOUR="ipv6-exit-min" - - env: NETWORK_FLAVOUR="hs-v23-ipv6-md" + - env: TOR="master-nightly" NETWORK_FLAVOUR="ipv6-exit-min" + - env: TOR="stable-release" NETWORK_FLAVOUR="hs-v23-ipv6-md" os: osx language: c python: - - env: NETWORK_FLAVOUR="single-onion-ipv6-md" + - env: TOR="stable-release" NETWORK_FLAVOUR="single-onion-ipv6-md" os: osx language: c python:

1 0

[chutney/master] Travis: add a note about v2 onion service IPv6 tests
by teor＠torproject.org 11 Jun '19

11 Jun '19

commit 6a44c9f4ef5f3991776535fabae0c1aa0850f4f7 Author: teor <teor(a)torproject.org> Date: Mon Jun 10 20:26:35 2019 +1000 Travis: add a note about v2 onion service IPv6 tests See #23588 for v3 onion service IPv6 in tor. See #30279 for v3 onion service IPv6 in chutney CI. --- .travis.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.travis.yml b/.travis.yml index 50719b8..065915d 100644 --- a/.travis.yml +++ b/.travis.yml @@ -41,6 +41,8 @@ matrix: os: osx language: c python: + ## v2 onion services only, see #30279 for v3 onion services + ## needs #23588 in macOS homebrew, once 0.4.1-stable is released - env: TOR="stable-release" NETWORK_FLAVOUR="single-onion-ipv6-md" os: osx language: c

1 0

[translation/https_everywhere_completed] Update translations for https_everywhere_completed
by translation＠torproject.org 11 Jun '19

11 Jun '19

commit ec828d8f372b8e98b8777cb7c40aaba69078d42c Author: Translation commit bot <translation(a)torproject.org> Date: Tue Jun 11 01:16:06 2019 +0000 Update translations for https_everywhere_completed --- pt_BR/https-everywhere.dtd | 1 + 1 file changed, 1 insertion(+) diff --git a/pt_BR/https-everywhere.dtd b/pt_BR/https-everywhere.dtd index 8717fe8ce..2ccd61752 100644 --- a/pt_BR/https-everywhere.dtd +++ b/pt_BR/https-everywhere.dtd @@ -35,6 +35,7 @@ <!ENTITY https-everywhere.options.storedRulesetsVersion "Versão do conjunto de regras armazenadas:"> <!ENTITY https-everywhere.options.updatesLastChecked "Atualizações verificadas pela última vez:"> <!ENTITY https-everywhere.options.updatesLastCheckedNever "nunca"> +<!ENTITY https-everywhere.options.debuggingRulesets "Depurando conjuntos de regras (avançado)"> <!ENTITY https-everywhere.prefs.export_settings "Exportar Configurações"> <!ENTITY https-everywhere.prefs.reset_defaults "Restaurar as configurações padrão">

1 0

[translation/https_everywhere] Update translations for https_everywhere
by translation＠torproject.org 11 Jun '19

11 Jun '19

commit b7586597be6a8e3cc1cac92930a5f40a2ac8ae0c Author: Translation commit bot <translation(a)torproject.org> Date: Tue Jun 11 01:15:56 2019 +0000 Update translations for https_everywhere --- pt_BR/https-everywhere.dtd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pt_BR/https-everywhere.dtd b/pt_BR/https-everywhere.dtd index a56f37b1e..2ccd61752 100644 --- a/pt_BR/https-everywhere.dtd +++ b/pt_BR/https-everywhere.dtd @@ -35,7 +35,7 @@ <!ENTITY https-everywhere.options.storedRulesetsVersion "Versão do conjunto de regras armazenadas:"> <!ENTITY https-everywhere.options.updatesLastChecked "Atualizações verificadas pela última vez:"> <!ENTITY https-everywhere.options.updatesLastCheckedNever "nunca"> -<!ENTITY https-everywhere.options.debuggingRulesets "Debugging Rulesets (advanced)"> +<!ENTITY https-everywhere.options.debuggingRulesets "Depurando conjuntos de regras (avançado)"> <!ENTITY https-everywhere.prefs.export_settings "Exportar Configurações"> <!ENTITY https-everywhere.prefs.reset_defaults "Restaurar as configurações padrão">

1 0

[tor/maint-0.4.1] Merge remote-tracking branch 'tor-github/pr/1094' into maint-0.4.0
by nickm＠torproject.org 10 Jun '19

10 Jun '19

commit c8447f95ec202bff9d99941dcdf0dca91e0edc3f Merge: 430dd2da6 d761c3ca6 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Jun 10 18:44:20 2019 -0400 Merge remote-tracking branch 'tor-github/pr/1094' into maint-0.4.0 .travis.yml | 4 ++++ changes/bug30744 | 3 +++ 2 files changed, 7 insertions(+) diff --cc .travis.yml index 30e6d9a6b,a1a48baff..7d8877c2a --- a/.travis.yml +++ b/.travis.yml @@@ -70,7 -70,17 +70,11 @@@ matrix ## Careful! We use global envs, which makes it hard to exclude or ## allow failures by env: ## https://docs.travis-ci.com/user/customizing-the-build#matching-jobs-with-al… + allow_failures: + ## test-stem sometimes hangs on Travis + - env: TEST_STEM="yes" + exclude: - ## Clang doesn't work in containerized builds, see below. - - compiler: clang - sudo: false - ## Non-containerized gcc are slow and redundant. - - compiler: gcc - sudo: required ## gcc on OSX is less useful, because the default compiler is clang. - compiler: gcc os: osx

1 0

[tor/maint-0.4.1] Merge branch 'maint-0.4.0' into maint-0.4.1
by nickm＠torproject.org 10 Jun '19

10 Jun '19

commit 5b07ff40112390518d95caa90447cd5faf938cf7 Merge: 6a72cc259 c8447f95e Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Jun 10 18:44:31 2019 -0400 Merge branch 'maint-0.4.0' into maint-0.4.1 .travis.yml | 4 ++++ changes/bug30744 | 3 +++ 2 files changed, 7 insertions(+)

1 0