April 2019 - tor-commits - lists.torproject.org

[chutney/master] bootstrap-network: Actually use the first argument as the network flavour
by teor＠torproject.org 12 Apr '19

12 Apr '19

commit f6d409540a751008b788138daa037dd3289e2f12 Author: teor <teor(a)torproject.org> Date: Mon Apr 8 08:31:59 2019 +1000 bootstrap-network: Actually use the first argument as the network flavour Before these bug 30058 fixes, bootstrap-network.sh: * Logged the first argument as the network flavour * default: basic (when there are no arguments) * Passed $NETWORK_FLAVOUR to chutney as the network flavour Now, bootstrap-network.sh: * Uses the first argument as the network flavour * default: $NETWORK_FLAVOUR (when there are no arguments) * default: bridges+hs-v2 (when $NETWORK_FLAVOUR is not set) * This default is the same as test-network.sh's default * Logs the network flavour * Passes the network flavour to chutney Fixes bug 30058; bugfix on commit 822b30c8 in February 2017. --- tools/bootstrap-network.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tools/bootstrap-network.sh b/tools/bootstrap-network.sh index 0aadb70..75d1034 100755 --- a/tools/bootstrap-network.sh +++ b/tools/bootstrap-network.sh @@ -41,8 +41,10 @@ myname=$(basename "$0") { echo "$myname: missing chutney directory: $CHUTNEY_PATH"; exit 1; } [ -x "$CHUTNEY" ] || \ { echo "$myname: missing chutney: $CHUTNEY"; exit 1; } -flavour=basic; [ -n "$1" ] && { flavour=$1; shift; } +# Set the variables for the chutney network flavour +export NETWORK_FLAVOUR=${NETWORK_FLAVOUR:-"bridges+hs-v2"} +[ -n "$1" ] && { NETWORK_FLAVOUR=$1; shift; } export CHUTNEY_NETWORK="$CHUTNEY_PATH/networks/$NETWORK_FLAVOUR" [ -e "$CHUTNEY_NETWORK" ] || \

1 0

[chutney/master] tests: Stop requiring python2 for module tests
by teor＠torproject.org 12 Apr '19

12 Apr '19

commit 191e280f636a724d5c80b0ff74e571711b8e94fb Author: teor <teor(a)torproject.org> Date: Mon Apr 8 10:06:12 2019 +1000 tests: Stop requiring python2 for module tests And make the modules executable, so we can run them in unit tests. Part of 30063. --- lib/chutney/Debug.py | 2 +- lib/chutney/Templating.py | 2 +- lib/chutney/TorNet.py | 2 +- lib/chutney/Traffic.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/chutney/Debug.py b/lib/chutney/Debug.py old mode 100644 new mode 100755 index 556ac5c..108735f --- a/lib/chutney/Debug.py +++ b/lib/chutney/Debug.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python # # Copyright 2011 Nick Mathewson, Michael Stone # Copyright 2013 The Tor Project diff --git a/lib/chutney/Templating.py b/lib/chutney/Templating.py old mode 100644 new mode 100755 index de3de90..1e97cbe --- a/lib/chutney/Templating.py +++ b/lib/chutney/Templating.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python # # Copyright 2011 Nick Mathewson, Michael Stone # diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index b0061b6..16f4345 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python # # Copyright 2011 Nick Mathewson, Michael Stone # Copyright 2013 The Tor Project diff --git a/lib/chutney/Traffic.py b/lib/chutney/Traffic.py old mode 100644 new mode 100755 index 766766a..7577d0f --- a/lib/chutney/Traffic.py +++ b/lib/chutney/Traffic.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python # # Copyright 2013 The Tor Project #

1 0

[chutney/master] TODO: we have done some of the TODOs
by teor＠torproject.org 12 Apr '19

12 Apr '19

commit 9bb251f8035832920776332f8b74ed6c97007729 Author: teor <teor(a)torproject.org> Date: Mon Apr 8 09:04:16 2019 +1000 TODO: we have done some of the TODOs But we forgot to take them off our list. Part of bug 30059. --- TODO | 7 ------- 1 file changed, 7 deletions(-) diff --git a/TODO b/TODO index 4490c05..b1ea8ef 100644 --- a/TODO +++ b/TODO @@ -19,13 +19,6 @@ - Pipe stdout to a file, not to nowhere with "quiet" -- Log stuff. - - Have launcher check for configuration - Resolve XXXX comments - -- Document everything - -- Start writing passive "tests" like "did it bootstrap?" or "any - warnings in the logs?", etc

1 0

[chutney/master] README: fix grammar
by teor＠torproject.org 12 Apr '19

12 Apr '19

commit de76537ce4700895e517dc3a068f4dead77375cd Author: teor <teor(a)torproject.org> Date: Sat Apr 6 13:15:20 2019 +1000 README: fix grammar --- README | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README b/README index 0cc5a1e..59b97cd 100644 --- a/README +++ b/README @@ -11,7 +11,7 @@ Right now it only sorta does these things. You will need: - Python 2.7, or a supported Python 3, - - (we support Python versions that are still getting updates), + - (we support Python versions that are still getting updates), and - Tor binaries. Chutney checks for Tor binaries in this order:

1 0

[chutney/master] README: specify the default CHUTNEY_START_TIME
by teor＠torproject.org 12 Apr '19

12 Apr '19

commit aceff9ee30898ec0135002a9d5d7c9a680be91c8 Author: teor <teor(a)torproject.org> Date: Sat Apr 6 12:06:05 2019 +1000 README: specify the default CHUTNEY_START_TIME --- README | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README b/README index 93309cb..9fdee93 100644 --- a/README +++ b/README @@ -151,9 +151,9 @@ Multiple Tests: Waiting for the network: The tools/test-network.sh script uses the chutney wait_for_bootstrap - command to wait for up to CHUTNEY_START_TIME seconds, checking whether the - logged bootstrapped status for every node is 100%. It it is, great: it - succeeds. If not, it dumps the bootstrap statuses and exits. + command to wait for up to CHUTNEY_START_TIME seconds (default: 60), checking + whether the logged bootstrapped status for every node is 100%. It it is, + great: it succeeds. If not, it dumps the bootstrap statuses and exits. test-network.sh does not exit immediately if a tor node fails to bootstrap. Instead, it attempts to verify. We'll add an option to fail on tor bootstrap failure in #20473.

1 0

[chutney/master] README: rewrite the bandwidth tests section
by teor＠torproject.org 12 Apr '19

12 Apr '19

commit 8ddd5efff908bcd5998a8149418af7f06e5a30cb Author: teor <teor(a)torproject.org> Date: Sat Apr 6 13:14:55 2019 +1000 README: rewrite the bandwidth tests section --- README | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/README b/README index 9fdee93..0cc5a1e 100644 --- a/README +++ b/README @@ -99,15 +99,18 @@ Bandwidth Tests: ./chutney configure networks/basic-min ./chutney start networks/basic-min ./chutney status networks/basic-min - CHUTNEY_DATA_BYTES=104857600 ./chutney verify networks/basic-min # Send 100MB of data per client connection - # verify produces performance figures for: - # Single Stream Bandwidth: the speed of the slowest stream, end-to-end - # Overall tor Bandwidth: the sum of the bandwidth across each tor instance - # This approximates the CPU-bound tor performance on the current machine, - # assuming everything is multithreaded and network performance is infinite. + CHUTNEY_DATA_BYTES=104857600 ./chutney verify networks/basic-min ./chutney stop networks/basic-min +If chutney sends at least 5 MB of data, and it takes at least one second, +verify produces performance figures for: + - Single Stream Bandwidth: the speed of the slowest stream, end-to-end + - Overall tor Bandwidth: the sum of the bandwidth across each tor instance +The overall bandwidth approximates the CPU-bound tor performance on the +current machine, assuming tor, chutney, and the OS are multithreaded, and +network performance is infinite. + Connection Tests: ./chutney configure networks/basic-025 ./chutney start networks/basic-025

1 0

[chutney/master] tests: Stop requiring a tor (or SOCKS) proxy for the Traffic.py tests
by teor＠torproject.org 12 Apr '19

12 Apr '19

commit 25162cb860eb0dab828aae9718e758628b0b2d83 Author: teor <teor(a)torproject.org> Date: Mon Apr 8 10:08:00 2019 +1000 tests: Stop requiring a tor (or SOCKS) proxy for the Traffic.py tests Part of 30063. --- lib/chutney/Traffic.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/chutney/Traffic.py b/lib/chutney/Traffic.py index 7577d0f..cb0af2f 100755 --- a/lib/chutney/Traffic.py +++ b/lib/chutney/Traffic.py @@ -431,11 +431,11 @@ class TrafficTester(): def main(): """Test the TrafficTester by sending and receiving some data.""" DATA = "a foo is a bar" * 1000 - proxy = ('localhost', 9008) bind_to = ('localhost', int(sys.argv[1])) tt = TrafficTester(bind_to, DATA) - tt.add(Source(tt, bind_to, DATA, proxy)) + # Don't use a proxy for self-testing, so that we avoid tor entirely + tt.add(Source(tt, bind_to, DATA)) success = tt.run() if success:

1 0

[chutney/master] tests: Add unit tests to Debug.py
by teor＠torproject.org 12 Apr '19

12 Apr '19

commit 7a3db9879d0f9a6116456910207c4e2a19bf5169 Author: teor <teor(a)torproject.org> Date: Mon Apr 8 10:08:36 2019 +1000 tests: Add unit tests to Debug.py Part of 30063. --- lib/chutney/Debug.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/lib/chutney/Debug.py b/lib/chutney/Debug.py index 108735f..85c9eee 100755 --- a/lib/chutney/Debug.py +++ b/lib/chutney/Debug.py @@ -11,6 +11,7 @@ from __future__ import print_function import cgitb import os +import sys # Get verbose tracebacks, so we can diagnose better. cgitb.enable(format="plain") @@ -24,3 +25,18 @@ def debug(s): "Print a debug message on stdout if debug_flag is True." if debug_flag: print("DEBUG: %s" % s) + + +def main(): + global debug_flag + debug("This message should appear if $CHUTNEY_DEBUG is true.") + debug_flag = True + debug("This message should always appear.") + debug_flag = False + debug("This message should never appear.") + # We don't test tracebacks, because it's hard to know what to expect + # (and they make python exit with a non-zero exit status) + return 0 + +if __name__ == '__main__': + sys.exit(main())

1 0

[chutney/master] Travis: Run unit-tests.sh before running the tor integration test
by teor＠torproject.org 12 Apr '19

12 Apr '19

commit 03bcfa025f0eb4a33e7735883411328b62d834ba Author: teor <teor(a)torproject.org> Date: Mon Apr 8 10:32:26 2019 +1000 Travis: Run unit-tests.sh before running the tor integration test Closes 30063. --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index 8a140dd..812ca82 100644 --- a/.travis.yml +++ b/.travis.yml @@ -163,6 +163,7 @@ install: - tor --version script: + - tests/unit-tests.sh - tools/test-network.sh --allow-failures 1 after_failure:

1 0

[chutney/master] test: Add tests/unit-tests.sh
by teor＠torproject.org 12 Apr '19

12 Apr '19

commit 09fbfaefbb6f9ad59bf3947e3e9c6ec0d2fe3989 Author: teor <teor(a)torproject.org> Date: Mon Apr 8 10:31:45 2019 +1000 test: Add tests/unit-tests.sh Test each chutney module separately, using its own unit tests. Part of 30063. --- tests/unit-tests.sh | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/tests/unit-tests.sh b/tests/unit-tests.sh new file mode 100755 index 0000000..91a1841 --- /dev/null +++ b/tests/unit-tests.sh @@ -0,0 +1,78 @@ +#!/bin/sh + +# Exit on errors +set -e +# Verbose mode +set -v + + +# Output is prefixed with the name of the script +myname=$(basename "$0") + +echo "$myname: finding chutney directory" +TEST_DIR=$(dirname "$0") +CHUTNEY_DIR=$(dirname "$TEST_DIR") + +echo "$myname: changing to chutney directory" +cd "$CHUTNEY_DIR" + + +echo "$myname: running Debug.py tests" + +LOG_FILE=$(mktemp) +export LOG_FILE +test -n "$LOG_FILE" + +unset CHUTNEY_DEBUG +export CHUTNEY_DEBUG +lib/chutney/Debug.py | tee "$LOG_FILE" +LOG_FILE_LINES=$(wc -l "$LOG_FILE" | tr -s " " | cut -d " " -f2) +test "$LOG_FILE_LINES" -eq 1 + +LOG_FILE=$(mktemp) +export LOG_FILE +test -n "$LOG_FILE" + +export CHUTNEY_DEBUG=1 +lib/chutney/Debug.py | tee "$LOG_FILE" +LOG_FILE_LINES=$(wc -l "$LOG_FILE" | tr -s " " | cut -d " " -f2) +test "$LOG_FILE_LINES" -eq 2 + +unset CHUTNEY_DEBUG +export CHUTNEY_DEBUG + + +echo "$myname: running Templating.py tests" + +LOG_FILE=$(mktemp) +export LOG_FILE +test -n "$LOG_FILE" + +lib/chutney/Templating.py torrc_templates/common.i | tee "$LOG_FILE" +grep -q owning_controller_process "$LOG_FILE" +grep -q connlimit "$LOG_FILE" +grep -q controlport "$LOG_FILE" +grep -q nick "$LOG_FILE" +grep -q authorities "$LOG_FILE" +grep -q dir "$LOG_FILE" + + +echo "$myname: running Traffic.py tests" + +LOG_FILE=$(mktemp) +export LOG_FILE +test -n "$LOG_FILE" + +# Choose an arbitrary port +PYTHONPATH=$PYTHONPATH:lib lib/chutney/Traffic.py 9999 | tee "$LOG_FILE" + +# Traffic.py produces output with a single newline. But we don't want to get +# too picky about the details: allow an extra line and a few extra chars. +LOG_FILE_LINES=$(wc -l "$LOG_FILE" | tr -s " " | cut -d " " -f2) +test "$LOG_FILE_LINES" -le 2 +LOG_FILE_CHARS=$(wc -c "$LOG_FILE" | tr -s " " | cut -d " " -f2) +test "$LOG_FILE_CHARS" -le 4 + + +# We don't test TorNet.py: it's integration tested with tor using the +# chutney/chutney script

1 0