February 2017 - tor-commits - lists.torproject.org

[chutney/master] Handle CHUTNEY_DATA_DIR correctly, even if it's relative
by teor＠torproject.org 27 Feb '17

27 Feb '17

commit bc7a8eab200cc38d919e23c48938672de90ecabb Author: teor <teor2345(a)gmail.com> Date: Mon Feb 27 23:34:34 2017 +1100 Handle CHUTNEY_DATA_DIR correctly, even if it's relative And handle it consistently across chutney and scripts. Closes 21568. --- README | 4 +++- tools/bootstrap-network.sh | 13 ++++++++++++- tools/hsaddress.sh | 12 +++++++++++- tools/test-network.sh | 24 +++++++++++++++++------- tools/warnings.sh | 12 +++++++++++- 5 files changed, 54 insertions(+), 11 deletions(-) diff --git a/README b/README index f126145..b2ce936 100644 --- a/README +++ b/README @@ -55,9 +55,11 @@ Warning Options: CHUTNEY_WARNINGS_SUMMARY=false --no-warnings CHUTNEY_WARNINGS_SKIP=true -Other Options: +Expert Options: --coverage USE_COVERAGE_BINARY=true + --net-dir CHUTNEY_DATA_DIR --dry-run NETWORK_DRY_RUN=true + (These are advanced options: in the past, they have had long-standing bugs.) Standard Actions: ./chutney configure networks/basic diff --git a/tools/bootstrap-network.sh b/tools/bootstrap-network.sh index 496bc95..7fb2eed 100755 --- a/tools/bootstrap-network.sh +++ b/tools/bootstrap-network.sh @@ -13,6 +13,7 @@ # (default: 'basic') # +# Get a working chutney path if [ ! -d "$CHUTNEY_PATH" -o ! -x "$CHUTNEY_PATH/chutney" ]; then # looks like a broken path: use the path to this tool instead TOOLS_PATH=`dirname "$0"` @@ -23,6 +24,16 @@ if [ -d "$PWD/$CHUTNEY_PATH" -a -x "$PWD/$CHUTNEY_PATH/chutney" ]; then export CHUTNEY_PATH="$PWD/$CHUTNEY_PATH" fi +# Get a working net path +if [ ! -d "$CHUTNEY_DATA_DIR" ]; then + # looks like a broken path: use the chutney path as a base + export CHUTNEY_DATA_DIR="$CHUTNEY_PATH/net" +fi +if [ -d "$PWD/$CHUTNEY_DATA_DIR" ]; then + # looks like a relative path: make chutney path absolute + export CHUTNEY_DATA_DIR="$PWD/$CHUTNEY_DATA_DIR" +fi + VOTING_OFFSET=6 CHUTNEY="$CHUTNEY_PATH/chutney" myname=$(basename "$0") @@ -48,7 +59,7 @@ echo "$myname: bootstrapping network: $flavour" # files like this. offset=$(expr $ $(date +%s) + $VOTING_OFFSET $ % 300) CONFOPT="TestingV3AuthVotingStartOffset" -for file in "$CHUTNEY_PATH"/net/nodes/*a/torrc ; do +for file in "$CHUTNEY_DATA_DIR"/nodes/*a/torrc ; do sed -i.bak -e "s/^${CONFOPT}.*$/${CONFOPT} $offset/1" $file done diff --git a/tools/hsaddress.sh b/tools/hsaddress.sh index c5d3a6f..517c6b5 100755 --- a/tools/hsaddress.sh +++ b/tools/hsaddress.sh @@ -17,8 +17,18 @@ if [ -d "$PWD/$CHUTNEY_PATH" -a -x "$PWD/$CHUTNEY_PATH/chutney" ]; then export CHUTNEY_PATH="$PWD/$CHUTNEY_PATH" fi +# Get a working net path +if [ ! -d "$CHUTNEY_DATA_DIR" ]; then + # looks like a broken path: use the chutney path as a base + export CHUTNEY_DATA_DIR="$CHUTNEY_PATH/net" +fi +if [ -d "$PWD/$CHUTNEY_DATA_DIR" ]; then + # looks like a relative path: make chutney path absolute + export CHUTNEY_DATA_DIR="$PWD/$CHUTNEY_DATA_DIR" +fi + NAME=$(basename "$0") -DEST="$CHUTNEY_PATH/net/nodes" +DEST="$CHUTNEY_DATA_DIR/nodes" TARGET=hidden_service/hostname function usage() { diff --git a/tools/test-network.sh b/tools/test-network.sh index 1308c08..6924c20 100755 --- a/tools/test-network.sh +++ b/tools/test-network.sh @@ -91,13 +91,7 @@ do export CHUTNEY_LISTEN_ADDRESS_V6="$2" shift ;; - --coverage) - export USE_COVERAGE_BINARY=true - ;; - --dry-run) - # process arguments, but don't call any other scripts - export NETWORK_DRY_RUN=true - ;; + # Warning Options # we summarise unexpected warnings by default # this shows all warnings per-node --all-warnings) @@ -108,6 +102,22 @@ do --no-warnings) export CHUTNEY_WARNINGS_SKIP=true ;; + # Expert options + # Code Coverage Binary + --coverage) + export USE_COVERAGE_BINARY=true + ;; + # Do Nothing (but process arguments and set environmental variables) + --dry-run) + # process arguments, but don't call any other scripts + export NETWORK_DRY_RUN=true + ;; + # The net directory, usually chutney/net + --net-dir) + export CHUTNEY_DATA_DIR="$2" + shift + ;; + # Oops *) echo "$myname: Sorry, I don't know what to do with '$1'." echo "$UPDATE_YOUR_CHUTNEY" diff --git a/tools/warnings.sh b/tools/warnings.sh index 761375e..02912cb 100755 --- a/tools/warnings.sh +++ b/tools/warnings.sh @@ -21,6 +21,16 @@ if [ -d "$PWD/$CHUTNEY_PATH" -a -x "$PWD/$CHUTNEY_PATH/chutney" ]; then export CHUTNEY_PATH="$PWD/$CHUTNEY_PATH" fi +# Get a working net path +if [ ! -d "$CHUTNEY_DATA_DIR" ]; then + # looks like a broken path: use the chutney path as a base + export CHUTNEY_DATA_DIR="$CHUTNEY_PATH/net" +fi +if [ -d "$PWD/$CHUTNEY_DATA_DIR" ]; then + # looks like a relative path: make chutney path absolute + export CHUTNEY_DATA_DIR="$PWD/$CHUTNEY_DATA_DIR" +fi + function show_warnings() { # Work out the file and filter settings if [ "$CHUTNEY_WARNINGS_SUMMARY" = true ]; then @@ -75,7 +85,7 @@ if [ -t 1 ]; then fi CHUTNEY="$CHUTNEY_PATH/chutney" NAME=$(basename "$0") -DEST="$CHUTNEY_PATH/net/nodes" +DEST="$CHUTNEY_DATA_DIR/nodes" LOG_FILE=info.log # ignore warnings we expect to get every time chutney runs CHUTNEY_WARNINGS_IGNORE_EXPECTED=${CHUTNEY_WARNINGS_IGNORE_EXPECTED:-0}

1 0

[chutney/master] Make a unique directory each chutney run, and symlink to a standard name
by teor＠torproject.org 27 Feb '17

27 Feb '17

commit f2fe67e98d07ab26b62c2204b9087a40ecf70cb6 Author: teor <teor2345(a)gmail.com> Date: Mon Feb 27 23:36:33 2017 +1100 Make a unique directory each chutney run, and symlink to a standard name Instead of moving the directory to a unique path at the start of the next run, which makes it impossible to find if running several networks in succession, like we do in 'make test-network-all'. Closes 21567. --- lib/chutney/TorNet.py | 79 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 69 insertions(+), 10 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index 33b7cde..6e3b77a 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -68,6 +68,26 @@ def get_absolute_net_path(): # ok, it's relative to the current directory, whatever that is return os.path.abspath(relative_net_path) +def get_absolute_nodes_path(): + # there's no way to customise this: we really don't need more options + return os.path.join(get_absolute_net_path(), 'nodes') + +def get_new_absolute_nodes_path(now=time.time()): + # automatically chosen to prevent path collisions, and result in an ordered + # series of directory path names + # should only be called by 'chutney configure', all other chutney commands + # should use get_absolute_nodes_path() + nodesdir = get_absolute_nodes_path() + newdir = newdirbase = "%s.%d" % (nodesdir, now) + # if the time is the same, fall back to a simple integer count + # (this is very unlikely to happen unless the clock changes: it's not + # possible to run multiple chutney networks at the same time) + i = 0 + while os.path.exists(newdir): + i += 1 + newdir = "%s.%d" % (newdirbase, i) + return newdir + class Node(object): """A Node represents a Tor node or a set of Tor nodes. It's created @@ -838,29 +858,68 @@ class Network(object): self._nextnodenum += 1 self._nodes.append(n) - def move_aside_nodes(self): - net_base_dir = get_absolute_net_path() - nodesdir = os.path.join(net_base_dir, 'nodes') + def move_aside_nodes_dir(self): + """Move aside the nodes directory, if it exists and is not a link. + Used for backwards-compatibility only: nodes is created as a link to + a new directory with a unique name in the current implementation. + """ + nodesdir = get_absolute_nodes_path() + # only move the directory if it exists if not os.path.exists(nodesdir): return + # and if it's not a link + if os.path.islink(nodesdir): + return - newdir = newdirbase = "%s.%d" % (nodesdir, time.time()) - i = 0 - while os.path.exists(newdir): - i += 1 - newdir = "%s.%d" % (newdirbase, i) + # subtract 1 second to avoid collisions and get the correct ordering + newdir = get_new_absolute_nodes_path(time.time() - 1) print("NOTE: renaming %r to %r" % (nodesdir, newdir)) os.rename(nodesdir, newdir) + def create_new_nodes_dir(self): + """Create a new directory with a unique name, and symlink it to nodes + """ + # for backwards compatibility, move aside the old nodes directory + # (if it's not a link) + self.move_aside_nodes_dir() + + # the unique directory we'll create + newnodesdir = get_new_absolute_nodes_path() + # the canonical name we'll link it to + nodeslink = get_absolute_nodes_path() + + # this path should be unique and should not exist + if os.path.exists(newnodesdir): + raise RuntimeError( + 'get_new_absolute_nodes_path returned a path that exists') + + # if this path exists, it must be a link + if os.path.exists(nodeslink) and not os.path.islink(nodeslink): + raise RuntimeError( + 'get_absolute_nodes_path returned a path that exists and is not a link') + + # create the new, uniquely named directory, and link it to nodes + print("NOTE: creating %r, linking to %r" % (newnodesdir, nodeslink)) + # this gets created with mode 0700, that's probably ok + mkdir_p(newnodesdir) + try: + os.unlink(nodeslink) + except OSError as e: + # it's ok if the link doesn't exist, we're just about to make it + if e.errno == errno.ENOENT: + pass + else: + raise + os.symlink(newnodesdir, nodeslink) + def _checkConfig(self): for n in self._nodes: n.getBuilder().checkConfig(self) def configure(self): - # shutil.rmtree(os.path.join(os.getcwd(),'net','nodes'),ignore_errors=True) - self.move_aside_nodes() + self.create_new_nodes_dir() network = self altauthlines = [] bridgelines = []

1 0

[chutney/master] Make chutney accept relative paths, as well as the scripts
by teor＠torproject.org 27 Feb '17

27 Feb '17

commit cdd9119df445161162617d36b03cc6c20bcd8f53 Author: teor <teor2345(a)gmail.com> Date: Mon Feb 27 21:08:49 2017 +1100 Make chutney accept relative paths, as well as the scripts Closes 21467 (finally!) --- lib/chutney/TorNet.py | 28 ++++++++++++++++++++++++---- tools/bootstrap-network.sh | 3 --- tools/test-network.sh | 27 +++++++++++---------------- 3 files changed, 35 insertions(+), 23 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index c9668f8..33b7cde 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -47,6 +47,26 @@ def mkdir_p(d, mode=448): return raise +def get_absolute_chutney_path(): + # use the current directory as the default + # (./chutney already sets CHUTNEY_PATH using the path to the script) + # use tools/test-network.sh if you want chutney to try really hard to find + # itself + relative_chutney_path = os.environ.get('CHUTNEY_PATH', os.getcwd()) + return os.path.abspath(relative_chutney_path) + +def get_absolute_net_path(): + # use the chutney path as the default + absolute_chutney_path = get_absolute_chutney_path() + relative_net_path = os.environ.get('CHUTNEY_DATA_DIR', 'net') + # but what is it relative to? + # let's check if it's in CHUTNEY_PATH first, to preserve + # backwards-compatible behaviour + chutney_net_path = os.path.join(absolute_chutney_path, relative_net_path) + if os.path.isdir(chutney_net_path): + return chutney_net_path + # ok, it's relative to the current directory, whatever that is + return os.path.abspath(relative_net_path) class Node(object): @@ -672,7 +692,7 @@ DEFAULTS = { 'hs_directory': 'hidden_service', 'hs-hostname': None, 'connlimit': 60, - 'net_base_dir': os.environ.get('CHUTNEY_DATA_DIR', 'net'), + 'net_base_dir': get_absolute_net_path(), 'tor': os.environ.get('CHUTNEY_TOR', 'tor'), 'tor-gencert': os.environ.get('CHUTNEY_TOR_GENCERT', None), 'auth_cert_lifetime': 12, @@ -681,7 +701,7 @@ DEFAULTS = { # so we default to ipv6_addr None 'ipv6_addr': os.environ.get('CHUTNEY_LISTEN_ADDRESS_V6', None), 'dirserver_flags': 'no-v2', - 'chutney_dir': '.', + 'chutney_dir': get_absolute_chutney_path(), 'torrc_fname': '${dir}/torrc', 'orport_base': 5000, 'dirport_base': 7000, @@ -819,7 +839,7 @@ class Network(object): self._nodes.append(n) def move_aside_nodes(self): - net_base_dir = os.environ.get('CHUTNEY_DATA_DIR', 'net') + net_base_dir = get_absolute_net_path() nodesdir = os.path.join(net_base_dir, 'nodes') if not os.path.exists(nodesdir): @@ -927,7 +947,7 @@ def ConfigureNodes(nodelist): def getTests(): tests = [] - chutney_path = os.environ.get('CHUTNEY_PATH', '') + chutney_path = get_absolute_chutney_path() if len(chutney_path) > 0 and chutney_path[-1] != '/': chutney_path += "/" for x in os.listdir(chutney_path + "scripts/chutney_tests/"): diff --git a/tools/bootstrap-network.sh b/tools/bootstrap-network.sh index bbc77f7..496bc95 100755 --- a/tools/bootstrap-network.sh +++ b/tools/bootstrap-network.sh @@ -38,9 +38,6 @@ export CHUTNEY_NETWORK="$CHUTNEY_PATH/networks/$NETWORK_FLAVOUR" [ -e "$CHUTNEY_NETWORK" ] || \ { echo "$myname: missing network file: $CHUTNEY_NETWORK"; exit 1; } -# Chutney must be launched at $CHUTNEY_PATH, at least until #21521 is fixed -cd "$CHUTNEY_PATH" - "$CHUTNEY" stop "$CHUTNEY_NETWORK" echo "$myname: bootstrapping network: $flavour" diff --git a/tools/test-network.sh b/tools/test-network.sh index 008e410..1308c08 100755 --- a/tools/test-network.sh +++ b/tools/test-network.sh @@ -193,26 +193,24 @@ if [ -d "$PWD/$CHUTNEY_PATH" -a -x "$PWD/$CHUTNEY_PATH/chutney" ]; then export CHUTNEY_PATH="$PWD/$CHUTNEY_PATH" fi -# For picking up the right tor binaries. +# For picking up the right tor binaries +# Choose coverage binaries, if selected +tor_name=tor +tor_gencert_name=tor-gencert +if [ "$USE_COVERAGE_BINARY" = true ]; then + tor_name=tor-cov +fi # If $TOR_DIR isn't set, chutney looks for tor binaries by name or path # using $CHUTNEY_TOR and $CHUTNEY_TOR_GENCERT, and then falls back to # looking for tor and tor-gencert in $PATH if [ -d "$TOR_DIR" ]; then - tor_name=tor - tor_gencert_name=tor-gencert - if [ "$USE_COVERAGE_BINARY" = true ]; then - tor_name=tor-cov - fi + # TOR_DIR is absolute, so these are absolute paths export CHUTNEY_TOR="${TOR_DIR}/src/or/${tor_name}" export CHUTNEY_TOR_GENCERT="${TOR_DIR}/src/tools/${tor_gencert_name}" else - # make binary paths absolute - if [ -x "$PWD/$CHUTNEY_TOR" ]; then - export CHUTNEY_TOR="$PWD/$CHUTNEY_TOR" - fi - if [ -x "$PWD/$CHUTNEY_TOR_GENCERT" ]; then - export CHUTNEY_TOR_GENCERT="$PWD/$CHUTNEY_TOR_GENCERT" - fi + # these are binary names, they will be searched for in $PATH + export CHUTNEY_TOR="${tor_name}" + export CHUTNEY_TOR_GENCERT="${tor_gencert_name}" fi # Set the variables for the chutney network flavour @@ -226,9 +224,6 @@ if [ "$NETWORK_DRY_RUN" = true ]; then return 2>/dev/null || exit fi -# Chutney must be launched at $CHUTNEY_PATH, at least until #21521 is fixed -cd "$CHUTNEY_PATH" - "$CHUTNEY_PATH/tools/bootstrap-network.sh" "$NETWORK_FLAVOUR" || exit 2 # chutney starts verifying after 20 seconds, keeps on trying for 60 seconds,

1 0

[chutney/master] When we fail to parse an option, remind the user to update chutney
by teor＠torproject.org 27 Feb '17

27 Feb '17

commit 6e9658c906bd641d068f680ea707da0bde213b98 Author: teor <teor2345(a)gmail.com> Date: Mon Feb 27 19:56:01 2017 +1100 When we fail to parse an option, remind the user to update chutney Partial fix for 20091. --- tools/test-network.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/test-network.sh b/tools/test-network.sh index 2b60ec7..008e410 100755 --- a/tools/test-network.sh +++ b/tools/test-network.sh @@ -9,6 +9,9 @@ myname=$(basename "$0") export CHUTNEY_WARNINGS_IGNORE_EXPECTED=${CHUTNEY_WARNINGS_IGNORE_EXPECTED:-true} export CHUTNEY_WARNINGS_SUMMARY=${CHUTNEY_WARNINGS_SUMMARY:-true} +# what we say when we fail +UPDATE_YOUR_CHUTNEY="Please update your chutney uisng 'git pull'." + until [ -z "$1" ] do case "$1" in @@ -107,6 +110,7 @@ do ;; *) echo "$myname: Sorry, I don't know what to do with '$1'." + echo "$UPDATE_YOUR_CHUTNEY" # continue processing arguments during a dry run if [ "$NETWORK_DRY_RUN" != true ]; then exit 2

1 0

[chutney/master] Make warnings summary mode even quieter
by teor＠torproject.org 27 Feb '17

27 Feb '17

commit 9936947b6dee8035c29ca596db310a5e3c0142b6 Author: teor <teor2345(a)gmail.com> Date: Mon Feb 27 16:56:02 2017 +1100 Make warnings summary mode even quieter Follow-up to #21371. --- tools/warnings.sh | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/tools/warnings.sh b/tools/warnings.sh index dfe5de7..761375e 100755 --- a/tools/warnings.sh +++ b/tools/warnings.sh @@ -22,27 +22,43 @@ if [ -d "$PWD/$CHUTNEY_PATH" -a -x "$PWD/$CHUTNEY_PATH/chutney" ]; then fi function show_warnings() { + # Work out the file and filter settings if [ "$CHUTNEY_WARNINGS_SUMMARY" = true ]; then - echo "${GREEN}All `basename $1`:${NC}" FILE="$1/*/$LOG_FILE" else - echo "${GREEN}Node `basename $1`:${NC}" FILE="$1/$LOG_FILE" fi if [ "$CHUTNEY_WARNINGS_IGNORE_EXPECTED" = true -a \ -e "$IGNORE_FILE" ]; then CAT="grep -v -f" - echo " ${GREEN}(Ignoring expected warnings, run chutney/tools/warnings.sh to see all warnings)${NC}" else CAT=cat IGNORE_FILE= fi - # Label errs as "Warning:", they're infrequent enough it doesn't matter - $CAT $IGNORE_FILE $FILE | \ - sed -n -E 's/^.*\[(warn|err)\]//p' | sort | uniq -c | \ - sed -e 's/^\s*//' -e "s/ *$[0-9][0-9]*$ *$.*$/ ${YELLOW}Warning:${NC} \2${YELLOW} Number: \1${NC}/" + # Silence any messages if we are in summary mode, and there are no warnings + # must be kept in sync with the filter commands below + if [ `$CAT $IGNORE_FILE $FILE | $SED_E "$FILTER" | wc -c` -eq 0 -a \ + "$CHUTNEY_WARNINGS_SUMMARY" = true ]; then + ECHO=true + else + ECHO=echo + fi + # Give context to the warnings we're about to display + if [ "$CHUTNEY_WARNINGS_SUMMARY" = true ]; then + $ECHO "${GREEN}All `basename $1`:${NC}" + else + $ECHO "${GREEN}Node `basename $1`:${NC}" + fi + if [ "$CHUTNEY_WARNINGS_IGNORE_EXPECTED" = true -a \ + -e "$IGNORE_FILE" ]; then + $ECHO " ${GREEN}(Ignoring expected warnings, run chutney/tools/warnings.sh to see all warnings)${NC}" + fi + # Display the warnings, after filtering and counting occurrences + # must be kept in sync with the filter commands above + $CAT $IGNORE_FILE $FILE | $SED_E "$FILTER" | sort | uniq -c | \ + sed -e 's/^\s*//' -e "s/ *$[0-9][0-9]*$ *$.*$/${YELLOW}Warning:${NC} \2${YELLOW} Number: \1${NC}/" if [ "$CHUTNEY_WARNINGS_SUMMARY" != true ]; then - echo "" + $ECHO "" fi } @@ -67,6 +83,9 @@ CHUTNEY_WARNINGS_IGNORE_EXPECTED=${CHUTNEY_WARNINGS_IGNORE_EXPECTED:-0} IGNORE_FILE="$CHUTNEY_PATH/tools/ignore.warnings" # merge all log files into one before counting entries CHUTNEY_WARNINGS_SUMMARY=${CHUTNEY_WARNINGS_SUMMARY:-0} +SED_E='sed -n -E' +# Label errs as "Warning:", they're infrequent enough it doesn't matter +FILTER='s/^.*\[(warn|err)\]//p' [ -d "$DEST" ] || { echo "$NAME: no logs available"; exit 1; } if [ $# -eq 0 ];

1 0

[chutney/master] Use bash rather than the default shell
by teor＠torproject.org 27 Feb '17

27 Feb '17

commit 0d57f921570789f05a6912fb88ad018a137fbb64 Author: teor <teor2345(a)gmail.com> Date: Mon Feb 27 16:02:13 2017 +1100 Use bash rather than the default shell Closes #21563, related to tor's #21562. --- chutney | 2 +- tools/bootstrap-network.sh | 2 +- tools/hsaddress.sh | 2 +- tools/test-network.sh | 5 +++-- tools/warnings.sh | 2 +- 5 files changed, 7 insertions(+), 6 deletions(-) diff --git a/chutney b/chutney index 5fbc5d2..0d4b391 100755 --- a/chutney +++ b/chutney @@ -1,4 +1,4 @@ -#!/usr/bin/env sh +#!/usr/bin/env bash set -o errexit set -o nounset diff --git a/tools/bootstrap-network.sh b/tools/bootstrap-network.sh index ed1ddf3..bbc77f7 100755 --- a/tools/bootstrap-network.sh +++ b/tools/bootstrap-network.sh @@ -1,4 +1,4 @@ -#! /bin/sh +#!/usr/bin/env bash # # 1. potentially stop running network # 2. bootstrap a network from scratch as quickly as possible diff --git a/tools/hsaddress.sh b/tools/hsaddress.sh index 5c8a828..c5d3a6f 100755 --- a/tools/hsaddress.sh +++ b/tools/hsaddress.sh @@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash # # Usage: # tools/hsaddress.sh [hs_node] diff --git a/tools/test-network.sh b/tools/test-network.sh index a8d8bcd..2b60ec7 100755 --- a/tools/test-network.sh +++ b/tools/test-network.sh @@ -1,4 +1,4 @@ -#! /bin/sh +#!/usr/bin/env bash ECHO_N="/bin/echo -n" @@ -218,7 +218,8 @@ export CHUTNEY_NETWORK="$CHUTNEY_PATH/networks/$NETWORK_FLAVOUR" # And finish up if we're doing a dry run if [ "$NETWORK_DRY_RUN" = true ]; then # we can't exit here, it breaks argument processing - return + # this only works in bash: return semantics are shell-specific + return 2>/dev/null || exit fi # Chutney must be launched at $CHUTNEY_PATH, at least until #21521 is fixed diff --git a/tools/warnings.sh b/tools/warnings.sh index bba2dcc..dfe5de7 100755 --- a/tools/warnings.sh +++ b/tools/warnings.sh @@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash # # Usage: # tools/warnings.sh [node]

1 0

[chutney/master] Check that relative paths actually have what we need in them
by teor＠torproject.org 27 Feb '17

27 Feb '17

commit 729ea601c5ce2858bbf15f29b0f18d1beb4b4fb0 Author: teor <teor2345(a)gmail.com> Date: Mon Feb 27 15:19:39 2017 +1100 Check that relative paths actually have what we need in them If they don't, use the script path. Finally fixes #21521, I hope! --- tools/bootstrap-network.sh | 11 +++++++---- tools/hsaddress.sh | 11 +++++++---- tools/test-network.sh | 5 +++-- tools/warnings.sh | 11 +++++++---- 4 files changed, 24 insertions(+), 14 deletions(-) diff --git a/tools/bootstrap-network.sh b/tools/bootstrap-network.sh index 438de62..ed1ddf3 100755 --- a/tools/bootstrap-network.sh +++ b/tools/bootstrap-network.sh @@ -13,11 +13,14 @@ # (default: 'basic') # -# make chutney path absolute -if [ -d "$PWD/$CHUTNEY_PATH" ]; then +if [ ! -d "$CHUTNEY_PATH" -o ! -x "$CHUTNEY_PATH/chutney" ]; then + # looks like a broken path: use the path to this tool instead + TOOLS_PATH=`dirname "$0"` + export CHUTNEY_PATH=`dirname "$TOOLS_PATH"` +fi +if [ -d "$PWD/$CHUTNEY_PATH" -a -x "$PWD/$CHUTNEY_PATH/chutney" ]; then + # looks like a relative path: make chutney path absolute export CHUTNEY_PATH="$PWD/$CHUTNEY_PATH" -elif [ ! -d "$CHUTNEY_PATH" ]; then - export CHUTNEY_PATH="$PWD" fi VOTING_OFFSET=6 diff --git a/tools/hsaddress.sh b/tools/hsaddress.sh index de6f023..5c8a828 100755 --- a/tools/hsaddress.sh +++ b/tools/hsaddress.sh @@ -7,11 +7,14 @@ # Examples: tools/hsaddress.sh # tools/hsaddress.sh 025h -# make chutney path absolute -if [ -d "$PWD/$CHUTNEY_PATH" ]; then +if [ ! -d "$CHUTNEY_PATH" -o ! -x "$CHUTNEY_PATH/chutney" ]; then + # looks like a broken path: use the path to this tool instead + TOOLS_PATH=`dirname "$0"` + export CHUTNEY_PATH=`dirname "$TOOLS_PATH"` +fi +if [ -d "$PWD/$CHUTNEY_PATH" -a -x "$PWD/$CHUTNEY_PATH/chutney" ]; then + # looks like a relative path: make chutney path absolute export CHUTNEY_PATH="$PWD/$CHUTNEY_PATH" -elif [ ! -d "$CHUTNEY_PATH" ]; then - export CHUTNEY_PATH="$PWD" fi NAME=$(basename "$0") diff --git a/tools/test-network.sh b/tools/test-network.sh index 547858f..a8d8bcd 100755 --- a/tools/test-network.sh +++ b/tools/test-network.sh @@ -146,7 +146,8 @@ if [ ! -d "$TOR_DIR" ]; then fi # make TOR_DIR absolute -if [ -d "$PWD/$TOR_DIR" ]; then +if [ -d "$PWD/$TOR_DIR" -a -d "$PWD/$TOR_DIR/src/or" -a \ + -d "$PWD/$TOR_DIR/src/tools" ]; then export TOR_DIR="$PWD/$TOR_DIR" fi @@ -184,7 +185,7 @@ CHUTNEY_PATH=\`pwd\`/chutney" fi # make chutney path absolute -if [ -d "$PWD/$CHUTNEY_PATH" ]; then +if [ -d "$PWD/$CHUTNEY_PATH" -a -x "$PWD/$CHUTNEY_PATH/chutney" ]; then export CHUTNEY_PATH="$PWD/$CHUTNEY_PATH" fi diff --git a/tools/warnings.sh b/tools/warnings.sh index 52187bd..bba2dcc 100755 --- a/tools/warnings.sh +++ b/tools/warnings.sh @@ -11,11 +11,14 @@ # CHUTNEY_WARNINGS_IGNORE_EXPECTED: set to "true" to filter expected warnings # CHUTNEY_WARNINGS_SUMMARY: set to "true" to merge warnings from all instances -# make chutney path absolute -if [ -d "$PWD/$CHUTNEY_PATH" ]; then +if [ ! -d "$CHUTNEY_PATH" -o ! -x "$CHUTNEY_PATH/chutney" ]; then + # looks like a broken path: use the path to this tool instead + TOOLS_PATH=`dirname "$0"` + export CHUTNEY_PATH=`dirname "$TOOLS_PATH"` +fi +if [ -d "$PWD/$CHUTNEY_PATH" -a -x "$PWD/$CHUTNEY_PATH/chutney" ]; then + # looks like a relative path: make chutney path absolute export CHUTNEY_PATH="$PWD/$CHUTNEY_PATH" -elif [ ! -d "$CHUTNEY_PATH" ]; then - export CHUTNEY_PATH="$PWD" fi function show_warnings() {

1 0

[chutney/master] Change the default mode when creating directories
by teor＠torproject.org 27 Feb '17

27 Feb '17

commit 501dff31d2b0ac60087e4559fa137bc5578cd853 Author: cypherpunks <cypherpunks(a)torproject.org> Date: Fri Feb 24 10:13:52 2017 +0000 Change the default mode when creating directories Every call to `mkdir_p` uses mode 448 which is a sensible default. Using this mode as the global default prevents future problems, removes duplicate comments and simplifies existing calls. Closes #21552. --- lib/chutney/TorNet.py | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index ced5727..c9668f8 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -32,12 +32,12 @@ _THE_NETWORK = None cgitb.enable(format="plain") -def mkdir_p(d, mode=511): +def mkdir_p(d, mode=448): """Create directory 'd' and all of its parents as needed. Unlike os.makedirs, does not give an error if d already exists. - 511 is the decimal representation of the octal number 0777. Since - python2 only supports 0777 and python3 only supports 0o777, we can use + 448 is the decimal representation of the octal number 0700. Since + python2 only supports 0700 and python3 only supports 0o700, we can use neither. """ try: @@ -335,13 +335,9 @@ class LocalNodeBuilder(NodeBuilder): def _makeDataDir(self): """Create the data directory (with keys subdirectory) for this node. - - 448 is the decimal representation of the octal number 0700. Since - python2 only supports 0700 and python3 only supports 0o700, we can - use neither. """ datadir = self._env['dir'] - mkdir_p(os.path.join(datadir, 'keys'), 448) + mkdir_p(os.path.join(datadir, 'keys')) def _makeHiddenServiceDir(self): """Create the hidden service subdirectory for this node. @@ -349,13 +345,9 @@ class LocalNodeBuilder(NodeBuilder): The directory name is stored under the 'hs_directory' environment key. It is combined with the 'dir' data directory key to yield the path to the hidden service directory. - - 448 is the decimal representation of the octal number 0700. Since - python2 only supports 0700 and python3 only supports 0o700, we can - use neither. """ datadir = self._env['dir'] - mkdir_p(os.path.join(datadir, self._env['hs_directory']), 448) + mkdir_p(os.path.join(datadir, self._env['hs_directory'])) def _genAuthorityKey(self): """Generate an authority identity and signing key for this authority,

1 0

[stem/master] Replace pycrypto in hidden_service_descriptor
by atagar＠torproject.org 27 Feb '17

27 Feb '17

commit 5ab96033f09f7c3c711910ac6fe12c6dbf00f612 Author: Patrick O'Doherty <p(a)trickod.com> Date: Sat Feb 25 17:47:49 2017 -0800 Replace pycrypto in hidden_service_descriptor Deprecate all use of pycrypto in hidden_service_descriptor module and replace with the cryptography library. --- stem/descriptor/hidden_service_descriptor.py | 29 +++++++++++++--------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/stem/descriptor/hidden_service_descriptor.py b/stem/descriptor/hidden_service_descriptor.py index 97cd19e..30754b8 100644 --- a/stem/descriptor/hidden_service_descriptor.py +++ b/stem/descriptor/hidden_service_descriptor.py @@ -296,9 +296,8 @@ class HiddenServiceDescriptor(Descriptor): @staticmethod def _decrypt_basic_auth(content, authentication_cookie): - from Crypto.Cipher import AES - from Crypto.Util import Counter - from Crypto.Util.number import bytes_to_long + from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes + from cryptography.hazmat.backends import default_backend try: client_blocks = int(binascii.hexlify(content[1:2]), 16) @@ -322,15 +321,15 @@ class HiddenServiceDescriptor(Descriptor): # try decrypting the session key - counter = Counter.new(128, initial_value = 0) - cipher = AES.new(authentication_cookie, AES.MODE_CTR, counter = counter) - session_key = cipher.decrypt(encrypted_session_key) + cipher = Cipher(algorithms.AES(authentication_cookie), modes.CTR('\x00' * len(iv)), default_backend()) + decryptor = cipher.decryptor() + session_key = decryptor.update(encrypted_session_key) + decryptor.finalize() # attempt to decrypt the intro points with the session key - counter = Counter.new(128, initial_value = bytes_to_long(iv)) - cipher = AES.new(session_key, AES.MODE_CTR, counter = counter) - decrypted = cipher.decrypt(encrypted) + cipher = Cipher(algorithms.AES(session_key), modes.CTR(iv), default_backend()) + decryptor = cipher.decryptor() + decrypted = decryptor.update(encrypted) + decryptor.finalize() # check if the decryption looks correct @@ -341,17 +340,15 @@ class HiddenServiceDescriptor(Descriptor): @staticmethod def _decrypt_stealth_auth(content, authentication_cookie): - from Crypto.Cipher import AES - from Crypto.Util import Counter - from Crypto.Util.number import bytes_to_long + from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes + from cryptography.hazmat.backends import default_backend # byte 1 = authentication type, 2-17 = input vector, 18 on = encrypted content - iv, encrypted = content[1:17], content[17:] - counter = Counter.new(128, initial_value = bytes_to_long(iv)) - cipher = AES.new(authentication_cookie, AES.MODE_CTR, counter = counter) + cipher = Cipher(algorithms.AES(authentication_cookie), modes.CTR(iv), default_backend()) + decryptor = cipher.decryptor() - return cipher.decrypt(encrypted) + return decryptor.update(encrypted) + decryptor.finalize() @staticmethod def _parse_introduction_points(content):

1 0

[stem/master] Migrate from pycrypto to cryptography module
by atagar＠torproject.org 27 Feb '17

27 Feb '17

commit 67252eb7e9ed54a2ed53bc20cc1dcac4dc82b15a Merge: c5cef00 1551fc5 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Feb 26 16:47:19 2017 -0800 Migrate from pycrypto to cryptography module Stem had an optional dependency on pycrypto to validate descriptor signatures but seems the module is no longer maintained... https://github.com/dlitz/pycrypto/issues/173 Moving to the cryptography module (https://cryptography.io/en/latest/) Thanks Patrick! docs/change_log.rst | 1 + docs/faq.rst | 22 +++++++++-- requirements.txt | 2 +- run_tests.py | 2 +- stem/descriptor/__init__.py | 20 +++++----- stem/descriptor/hidden_service_descriptor.py | 36 +++++++++--------- stem/descriptor/server_descriptor.py | 5 +++ stem/prereq.py | 18 +++++---- test/mocking.py | 46 ++++++++--------------- test/settings.cfg | 14 +++---- test/unit/descriptor/hidden_service_descriptor.py | 4 +- test/util.py | 4 +- 12 files changed, 92 insertions(+), 82 deletions(-)

1 0