March 2020 - tor-commits - lists.torproject.org

[tor/maint-0.4.1] Merge branch 'bug33195_035' into bug33195_041
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit c96cf1494357af5a2fd0309ef3d26aba52105edb Merge: 5298113da add387c50 Author: teor <teor(a)torproject.org> Date: Thu Feb 13 13:55:55 2020 +1000 Merge branch 'bug33195_035' into bug33195_041 Conflicts: * Keep TOR_TEST_RNG_SEED from maint-0.4.1 * Keep the ordering from bug33195_035 .travis.yml | 57 ++++++++++++++++++++++++++++++++--------------------- changes/ticket33194 | 4 ++++ changes/ticket33195 | 4 ++++ 3 files changed, 42 insertions(+), 23 deletions(-) diff --cc .travis.yml index 6cfc4e51f,6047c4470..14b0e12c7 --- a/.travis.yml +++ b/.travis.yml @@@ -61,10 -63,18 +63,18 @@@ matrix ## We run rust on Linux, because it's faster than rust on macOS ## We check rust offline - env: RUST_OPTIONS="--enable-rust" TOR_RUST_DEPENDENCIES=true - ## We check asciidoc with distcheck, to make sure we remove doc products - - env: DISTCHECK="yes" ASCIIDOC_OPTIONS="" SKIP_MAKE_CHECK="yes" - ## We check disable module dirauth - - env: MODULES_OPTIONS="--disable-module-dirauth" + + ## We check NSS + ## Use -std=gnu99 to turn off some newer features, and maybe turn on some + ## extra gcc warnings? + - env: NSS_OPTIONS="--enable-nss" C_DIALECT_OPTIONS="-std=gnu99" + + ## We include a single coverage build with the best options for coverage - - env: COVERAGE_OPTIONS="--enable-coverage" HARDENING_OPTIONS="" ++ - env: COVERAGE_OPTIONS="--enable-coverage" HARDENING_OPTIONS="" TOR_TEST_RNG_SEED="636f766572616765" + + ## We clone our stem repo and run `make test-stem` + - env: TEST_STEM="yes" SKIP_MAKE_CHECK="yes" + ## macOS builds are very slow, and we have a limited number of ## concurrent macOS jobs. We're not actively developing Rust, so it is ## the lowest priority.

1 0

[tor/maint-0.3.5] Travis: Remove a redundant distcheck job
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit fc3555100f26201213f4da19d2d2904edc76a229 Author: teor <teor(a)torproject.org> Date: Thu Feb 13 13:22:19 2020 +1000 Travis: Remove a redundant distcheck job Part of 33194. --- .travis.yml | 4 ---- changes/ticket33194 | 2 ++ 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/.travis.yml b/.travis.yml index 323ebf5b3..38519db62 100644 --- a/.travis.yml +++ b/.travis.yml @@ -53,16 +53,12 @@ matrix: - env: CHUTNEY="yes" CHUTNEY_ALLOW_FAILURES="2" SKIP_MAKE_CHECK="yes" compiler: clang ## We check asciidoc with distcheck, to make sure we remove doc products - ## We use Linux clang, because there are no other Linux clang jobs - env: DISTCHECK="yes" ASCIIDOC_OPTIONS="" SKIP_MAKE_CHECK="yes" - compiler: clang ## We include a single coverage build with the best options for coverage - env: COVERAGE_OPTIONS="--enable-coverage" HARDENING_OPTIONS="" ## We run rust on Linux, because it's faster than rust on macOS ## We check rust offline - env: RUST_OPTIONS="--enable-rust" TOR_RUST_DEPENDENCIES=true - ## We check asciidoc with distcheck, to make sure we remove doc products - - env: DISTCHECK="yes" ASCIIDOC_OPTIONS="" SKIP_MAKE_CHECK="yes" ## We check disable module dirauth - env: MODULES_OPTIONS="--disable-module-dirauth" ## macOS builds are very slow, and we have a limited number of diff --git a/changes/ticket33194 b/changes/ticket33194 new file mode 100644 index 000000000..fd9efda51 --- /dev/null +++ b/changes/ticket33194 @@ -0,0 +1,2 @@ + o Testing: + - Remove a redundant distcheck job. Closes ticket 33194.

1 0

[tor/maint-0.3.5] Travis: Require the macOS IPv6 chutney job
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit 03650e0f60ec37c758f90f3e02bf2ace05959a7e Author: teor <teor(a)torproject.org> Date: Thu Feb 13 13:13:40 2020 +1000 Travis: Require the macOS IPv6 chutney job The job was previously set to fast_finish / allow_failure, to speed up the build. Closes ticket 33195. --- .travis.yml | 6 ++++-- changes/ticket33195 | 4 ++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 28a45788b..323ebf5b3 100644 --- a/.travis.yml +++ b/.travis.yml @@ -91,8 +91,10 @@ matrix: - env: RUST_OPTIONS="--enable-rust --enable-cargo-online-mode" compiler: clang os: osx - - env: CHUTNEY="yes" CHUTNEY_ALLOW_FAILURES="2" SKIP_MAKE_CHECK="yes" - os: osx + ## Since we're actively developing IPv6, we want to require the IPv6 + ## chutney tests + #- env: CHUTNEY="yes" CHUTNEY_ALLOW_FAILURES="2" SKIP_MAKE_CHECK="yes" + # os: osx ## (Linux only) Use a recent Linux image (Ubuntu Bionic) dist: bionic diff --git a/changes/ticket33195 b/changes/ticket33195 new file mode 100644 index 000000000..11abd4816 --- /dev/null +++ b/changes/ticket33195 @@ -0,0 +1,4 @@ + o Testing: + - Stop allowing the Chutney IPv6 Travis job to fail. This job was + previously configured to fast_finish (which requires allow_failure), to + speed up the build. Closes ticket 33195.

1 0

[tor/maint-0.3.5] Travis: Sort jobs in order of speed
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit add387c5071a1ee7889e3789e7b4143bb8856d0a Author: teor <teor(a)torproject.org> Date: Thu Feb 13 13:37:02 2020 +1000 Travis: Sort jobs in order of speed Putting the slowest jobs first takes full advantage of Travis concurrency. Closes 33194. --- .travis.yml | 47 ++++++++++++++++++++++++++++++----------------- changes/ticket33194 | 2 ++ 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/.travis.yml b/.travis.yml index 38519db62..6047c4470 100644 --- a/.travis.yml +++ b/.travis.yml @@ -39,28 +39,42 @@ env: matrix: ## include creates builds with gcc, linux, unless we override those defaults include: - ## We run basic tests on macOS + ## We run chutney on macOS, because macOS Travis has IPv6 + - env: CHUTNEY="yes" CHUTNEY_ALLOW_FAILURES="2" SKIP_MAKE_CHECK="yes" + os: osx + + ## We also run basic tests on macOS - compiler: clang os: osx ## Turn off some newer features, turn on clang's -Wtypedef-redefinition env: C_DIALECT_OPTIONS="-std=gnu99" - ## We check NSS - ## Use -std=gnu99 to turn off some newer features, and maybe turn on some - ## extra gcc warnings? - - env: NSS_OPTIONS="--enable-nss" C_DIALECT_OPTIONS="-std=gnu99" + ## We run chutney on Linux, because it's faster than chutney on macOS ## Chutney is a fast job, clang is slower on Linux, so we do Chutney clang - env: CHUTNEY="yes" CHUTNEY_ALLOW_FAILURES="2" SKIP_MAKE_CHECK="yes" compiler: clang + ## We check asciidoc with distcheck, to make sure we remove doc products - env: DISTCHECK="yes" ASCIIDOC_OPTIONS="" SKIP_MAKE_CHECK="yes" - ## We include a single coverage build with the best options for coverage - - env: COVERAGE_OPTIONS="--enable-coverage" HARDENING_OPTIONS="" + + ## We check disable module dirauth + - env: MODULES_OPTIONS="--disable-module-dirauth" + ## We run rust on Linux, because it's faster than rust on macOS ## We check rust offline - env: RUST_OPTIONS="--enable-rust" TOR_RUST_DEPENDENCIES=true - ## We check disable module dirauth - - env: MODULES_OPTIONS="--disable-module-dirauth" + + ## We check NSS + ## Use -std=gnu99 to turn off some newer features, and maybe turn on some + ## extra gcc warnings? + - env: NSS_OPTIONS="--enable-nss" C_DIALECT_OPTIONS="-std=gnu99" + + ## We include a single coverage build with the best options for coverage + - env: COVERAGE_OPTIONS="--enable-coverage" HARDENING_OPTIONS="" + + ## We clone our stem repo and run `make test-stem` + - env: TEST_STEM="yes" SKIP_MAKE_CHECK="yes" + ## macOS builds are very slow, and we have a limited number of ## concurrent macOS jobs. We're not actively developing Rust, so it is ## the lowest priority. @@ -68,11 +82,6 @@ matrix: #- env: RUST_OPTIONS="--enable-rust --enable-cargo-online-mode" # compiler: clang # os: osx - ## We run chutney on macOS, because macOS Travis has IPv6 - - env: CHUTNEY="yes" CHUTNEY_ALLOW_FAILURES="2" SKIP_MAKE_CHECK="yes" - os: osx - ## We clone our stem repo and run `make test-stem` - - env: TEST_STEM="yes" SKIP_MAKE_CHECK="yes" ## Allow the build to report success (with non-required sub-builds ## continuing to run) if all required sub-builds have succeeded. @@ -84,9 +93,13 @@ matrix: ## macOS rust and chutney are very slow, so we let the build finish before ## they are done. We'd like to fast finish, but still eventually show ## any failures in the build status. But Travis doesn't have that ability. - - env: RUST_OPTIONS="--enable-rust --enable-cargo-online-mode" - compiler: clang - os: osx + + ## Since this job is disabled, there's not much point having an exception + ## for it + #- env: RUST_OPTIONS="--enable-rust --enable-cargo-online-mode" + # compiler: clang + # os: osx + ## Since we're actively developing IPv6, we want to require the IPv6 ## chutney tests #- env: CHUTNEY="yes" CHUTNEY_ALLOW_FAILURES="2" SKIP_MAKE_CHECK="yes" diff --git a/changes/ticket33194 b/changes/ticket33194 index fd9efda51..b87e55348 100644 --- a/changes/ticket33194 +++ b/changes/ticket33194 @@ -1,2 +1,4 @@ o Testing: - Remove a redundant distcheck job. Closes ticket 33194. + - Sort the Travis jobs in order of speed. Putting the slowest jobs first + takes full advantage of Travis job concurrency. Closes ticket 33194.

1 0

[tor/master] Merge remote-tracking branch 'tor-github/pr/1735' into maint-0.4.3
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit dbadfe75e60b9ad6ddfbb2907de97486534aa8c2 Merge: 3e42004e5 3e8e4af1a Author: teor <teor(a)torproject.org> Date: Fri Mar 13 13:00:58 2020 +1000 Merge remote-tracking branch 'tor-github/pr/1735' into maint-0.4.3 .travis.yml | 64 +++++++++++++++++++++++++++++++---------------------- changes/ticket33194 | 4 ++++ changes/ticket33195 | 4 ++++ 3 files changed, 46 insertions(+), 26 deletions(-)

1 0

[tor/master] Merge branch 'maint-0.4.3'
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit 3b4303e7f6d92fdc36632f1dd98b4557cffa089e Merge: 1d17541cf dbadfe75e Author: teor <teor(a)torproject.org> Date: Fri Mar 13 13:01:31 2020 +1000 Merge branch 'maint-0.4.3'

1 0

[torspec/master] dir-list: Fix a version typo
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit f81b1e6cc53b91abb3ae206807bc371fac1b7cbf Author: teor <teor(a)torproject.org> Date: Fri Mar 13 11:21:31 2020 +1000 dir-list: Fix a version typo --- dir-list-spec.txt | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/dir-list-spec.txt b/dir-list-spec.txt index f0dccbb..3557721 100644 --- a/dir-list-spec.txt +++ b/dir-list-spec.txt @@ -81,9 +81,8 @@ 1.0.0 - The legacy fallback directory list format 2.0.0 - Adds name and extrainfo structured comments, and section separator - comments to make the list easier to parse. - - 2.1.0 - Adds a source list comment to the header. + comments to make the list easier to parses. Also adds a source list + comment to the header. 3.0.0 - Modifies the format of the source list comment. @@ -221,11 +220,11 @@ transition to signed offers, it will be "descriptor,offer-list". Afterwards, it will be "descriptor". - In version 2.1.0, only one source name was allowed after "source=", + In version 2.0.0, only one source name was allowed after "source=", and the deprecated "whitelist" source name was used instead of "offer-list". - This line was added in version 2.1.0 of this specification. The format + This line was added in version 2.0.0 of this specification. The format of this line was modified in version 3.0.0 of this specification. "/*" SP+ key_value SP+ "*/" SP* NL

1 0

[tor/master] Merge branch 'pr1764_squashed'
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit 1d17541cfa357adf54104ab47a114dbfdc5d5f4b Merge: 8bc4822ba 78c9ff705 Author: teor <teor(a)torproject.org> Date: Fri Mar 13 11:14:20 2020 +1000 Merge branch 'pr1764_squashed' changes/ticket33451 | 3 + scripts/git/git-install-tools.sh | 189 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 192 insertions(+)

1 0

[tor/master] Add a script to install the scripts in scripts/git appropriately.
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit 78c9ff70529421de2301e4186a0912859548c552 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Feb 25 14:59:41 2020 -0500 Add a script to install the scripts in scripts/git appropriately. To run this, say something like TOR_DEVTOOL_DIR=~/bin scripts/git/git-install-tools.sh all To see what it would do, give it the -n flag. To get help, give it the -h flag. --- changes/ticket33451 | 3 + scripts/git/git-install-tools.sh | 189 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 192 insertions(+) diff --git a/changes/ticket33451 b/changes/ticket33451 new file mode 100644 index 000000000..74dd6d1ad --- /dev/null +++ b/changes/ticket33451 @@ -0,0 +1,3 @@ + o Minor features (developer tools): + - Add a script ("git-install-tools.sh") to install git hooks and helper + scripts. Closes ticket 33451. diff --git a/scripts/git/git-install-tools.sh b/scripts/git/git-install-tools.sh new file mode 100755 index 000000000..ef8623a01 --- /dev/null +++ b/scripts/git/git-install-tools.sh @@ -0,0 +1,189 @@ +#!/usr/bin/env bash + +SCRIPT_NAME=$(basename "$0") +SCRIPTS_DIR=$(dirname "$0") + +TOOL_NAMES=(push-all pull-all merge-forward list-tor-branches) + +function usage() +{ + echo "$SCRIPT_NAME [-h] [-n] [-v] [-f] <all|hooks|tools|aliases>" + echo + echo " flags:" + echo " -h: show this help text" + echo " -n: dry-run" + echo " -v: verbose mode" + echo " -f: force-install even if \$TOR_DEVTOOLS_DIR looks fishy" + echo + echo " modes:" + echo " hooks: install git hooks in this repository." + echo " tools: install scripts in \$TOR_DEVTOOLS_DIR" + echo " aliases: set up global git aliases for git tools in \$TOR_DEVTOOLS_DIR" + echo " all: all of the above." +} + +INSTALL_HOOKS=0 +INSTALL_TOOLS=0 +INSTALL_ALIASES=0 + +DRY_RUN=0 +VERBOSE=0 +FORCE=0 + +while getopts "hnfv" opt; do + case "$opt" in + h) usage + exit 0 + ;; + n) DRY_RUN=1 + ;; + v) VERBOSE=1 + ;; + f) FORCE=1 + ;; + *) echo + usage + exit 1 + ;; + esac +done + +for item in "${@:$OPTIND}"; do + case "$item" in + hooks) INSTALL_HOOKS=1 + ;; + tools) INSTALL_TOOLS=1 + ;; + aliases) INSTALL_ALIASES=1 + ;; + all) INSTALL_HOOKS=1 + INSTALL_TOOLS=1 + INSTALL_ALIASES=1 + ;; + *) echo "Unrecognized mode '$item'" + usage + exit 1 + ;; + esac +done + +if [[ $VERBOSE = 1 ]]; then + function note() + { + echo "$@" + } +else + function note() + { + true + } +fi + +function fail() +{ + echo "$@" 1>&2 + exit 1 +} + +if [[ $INSTALL_HOOKS = 0 && $INSTALL_TOOLS = 0 && $INSTALL_ALIASES = 0 ]]; then + echo "Nothing to do. Try $SCRIPT_NAME -h for a list of commands." + exit 0 +fi + +if [[ $INSTALL_TOOLS = 1 || $INSTALL_ALIASES = 1 ]]; then + if [[ -z "$TOR_DEVTOOLS_DIR" ]] ; then + fail "\$TOR_DEVTOOLS_DIR was not set." + fi + note "Checking whether \$TOR_DEVTOOLS_DIR ($TOR_DEVTOOLS_DIR) is a git repo..." + GITDIR=$(cd "$TOR_DEVTOOLS_DIR" && git rev-parse --git-dir 2>/dev/null) + note "GITDIR is $GITDIR" + if [[ -n "$GITDIR" ]] ; then + cat <<EOF +You have asked me to install to \$TOR_DEVTOOLS_DIR ($TOR_DEVTOOLS_DIR). +That is inside a git repository, so you might not want to install there: +depending on what you pull or push, you might find yourself giving somebody +else write access to your scripts. I think you should just use ~/bin or +something. +EOF + + echo + if [[ "$FORCE" = 1 ]] ; then + echo "I will install anyway, since you said '-f'." + else + echo "I will not install. You can tell me -f if you are really sure." + exit 1 + fi + else + note "It was not." + fi +fi + +if [[ ! -d "$SCRIPTS_DIR" || ! -e "$SCRIPTS_DIR/git-push-all.sh" ]]; then + fail "Couldn't find scripts in '$SCRIPTS_DIR'" +fi + +if [[ $DRY_RUN = 1 ]]; then + echo "** DRY RUN **" + RUN="echo >>" +else + RUN= +fi + +set -e + +# ====================================================================== +if [[ $INSTALL_HOOKS = 1 ]]; then + HOOKS_DIR=$(git rev-parse --git-path hooks) + + note "Looking for hooks directory" + + if [[ -z "$HOOKS_DIR" || ! -d "$HOOKS_DIR" ]]; then + fail "Couldn't find git hooks directory." + fi + + note "Found hooks directory in $HOOKS_DIR" + + note "Installing hooks" + for fn in "$SCRIPTS_DIR"/*.git-hook; do + name=$(basename "$fn") + $RUN install --backup "$fn" "${HOOKS_DIR}/${name%.git-hook}" + done +fi + + +# ====================================================================== +if [[ $INSTALL_TOOLS = 1 ]]; then + note "Installing tools." + note "Looking for \$TOR_DEVTOOLS_DIR ($TOR_DEVTOOLS_DIR)" + + if [[ ! -d "$TOR_DEVTOOLS_DIR" ]]; then + note "Creating directory" + $RUN mkdir -p "$TOR_DEVTOOLS_DIR" + fi + + note "Copying scripts" + for tool in "${TOOL_NAMES[@]}"; do + $RUN install --backup "${SCRIPTS_DIR}/git-${tool}.sh" "${TOR_DEVTOOLS_DIR}/" + done +fi + +# ====================================================================== +if [[ $INSTALL_ALIASES = 1 ]]; then + note "Installing aliases." + note "Looking for \$TOR_DEVTOOLS_DIR ($TOR_DEVTOOLS_DIR)" + + note "Checking for ${TOR_DEVTOOLS_DIR}/git-push-all.sh" + if [[ ! -x "${TOR_DEVTOOLS_DIR}/git-push-all.sh" ]]; then + if [[ $DRY_RUN = 0 ]]; then + fail "Could not find scripts in \$TOR_DEVTOOLS_DIR" + fi + fi + + note "Setting aliases" + for tool in "${TOOL_NAMES[@]}"; do + $RUN git config --global "alias.$tool" \!"${TOR_DEVTOOLS_DIR}/git-${tool}.sh" + done + +fi + +note Done.

1 0

[chutney/master] TorNet: Split out print_bootstrap_status()
by teor＠torproject.org 13 Mar '20

13 Mar '20

commit cd1ad673299d27f891e34a94c6324459a5e1324e Author: teor <teor(a)torproject.org> Date: Wed Feb 19 15:52:07 2020 +1000 TorNet: Split out print_bootstrap_status() Split print_bootstrap_status() out into its own function. Part of 33232. --- lib/chutney/TorNet.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/lib/chutney/TorNet.py b/lib/chutney/TorNet.py index 0917fbf..3fa94ac 100644 --- a/lib/chutney/TorNet.py +++ b/lib/chutney/TorNet.py @@ -1448,6 +1448,12 @@ class Network(object): print("Sending SIGHUP to nodes") return all([n.getController().hup() for n in self._nodes]) + def print_bootstrap_status(self, controllers, most_recent_status): + print("Bootstrap failed. Node status:") + for c, status in zip(controllers,most_recent_status): + c.check(listRunning=False, listNonRunning=True) + print("{}: {}".format(c.getNick(), status)) + CHECK_NETWORK_STATUS_DELAY = 0.5 def wait_for_bootstrap(self): @@ -1475,11 +1481,7 @@ class Network(object): break time.sleep(Network.CHECK_NETWORK_STATUS_DELAY) - print("Bootstrap failed. Node status:") - for c, status in zip(controllers,most_recent_status): - c.check(listRunning=False, listNonRunning=True) - print("{}: {}".format(c.getNick(), status)) - + self.print_bootstrap_status(controllers, most_recent_status) return False def stop(self):

1 0