January 2020 - tor-commits - lists.torproject.org

[fallback-scripts/master] test: Stop duplicating the list of relay fingerprints
by teor＠torproject.org 20 Jan '20

20 Jan '20

commit d118388763151fd1e295c853277696f9eff8f881 Author: teor <teor(a)torproject.org> Date: Mon Dec 2 14:03:30 2019 +1000 test: Stop duplicating the list of relay fingerprints Part of 28863. --- test.sh | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/test.sh b/test.sh index 0dc177a..7a7b23d 100755 --- a/test.sh +++ b/test.sh @@ -13,10 +13,23 @@ if command -v shellcheck; then find . -name "*.sh" -exec shellcheck "$EXCLUSIONS" {} + fi + +# List of relay fingerprints to test with generate and lookup ## moria1, Serge, no caches extra info, no dir port, doesn't exist +TEST_RELAY_LIST=" +9695DFC35FFEB861329B9F1AB04C46397020CE31 +BA44A889E64B93FAA2B114E02C2A279A8555C533 +001524DD403D729F08F7E5D77813EF12756CFA8D +5AFAC3D00E97D6733112CC9CA2A788691FA87125 +AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +" + ## TODO: validate output from all 3 commands using grep, grep, and stem? -$PYTHON generateFallbackDirLine.py 9695DFC35FFEB861329B9F1AB04C46397020CE31 BA44A889E64B93FAA2B114E02C2A279A8555C533 001524DD403D729F08F7E5D77813EF12756CFA8D 5AFAC3D00E97D6733112CC9CA2A788691FA87125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA -$PYTHON lookupFallbackDirContact.py 9695DFC35FFEB861329B9F1AB04C46397020CE31 BA44A889E64B93FAA2B114E02C2A279A8555C533 001524DD403D729F08F7E5D77813EF12756CFA8D 5AFAC3D00E97D6733112CC9CA2A788691FA87125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +# We need each relay fingerprint as a separate argument +# shellcheck disable=SC2086 +$PYTHON generateFallbackDirLine.py $TEST_RELAY_LIST +# shellcheck disable=SC2086 +$PYTHON lookupFallbackDirContact.py $TEST_RELAY_LIST ## If we get the top 200 relays from Onionoo, we generate a list with about ## 10 fallbacks

1 0

[fallback-scripts/master] test: Add shellcheck tests
by teor＠torproject.org 20 Jan '20

20 Jan '20

commit 348c36e75c168a20c92ad3e6c17e56d9f9e2b249 Author: teor <teor(a)torproject.org> Date: Mon Dec 2 13:34:12 2019 +1000 test: Add shellcheck tests Part of 28863. --- .travis.yml | 11 +++++++++++ test.sh | 7 +++++++ 2 files changed, 18 insertions(+) diff --git a/.travis.yml b/.travis.yml index b7bfc94..a219026 100644 --- a/.travis.yml +++ b/.travis.yml @@ -38,6 +38,7 @@ matrix: packages: ## macOS needs pip(2) from homebrew python - python@2 + - shellcheck ## See ticket #30928 for more information. This key should be ## removed at some point to speed up builds. update: true @@ -97,6 +98,15 @@ dist: bionic ## Recent is Xcode 11.2 on macOS 10.14 as of October 2019 osx_image: xcode11.2 +## Download our dependencies +## We don't install shellcheck on macOS, because brew update is slow +addons: + ## (Linux only) + apt: + packages: + ## Optional build dependencies + - shellcheck + before_install: ## Set pipefail: we use pipes - set -o pipefail || echo "pipefail failed" @@ -105,6 +115,7 @@ install: ## List installed package versions - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then dpkg-query --show; fi - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew list --versions; fi + - if command -v shellcheck ; then shellcheck --version; fi - $PYTHON --version ## Are we running python 2? - if $PYTHON --version 2>&1 | cut -d" " -f2 | grep -q '2[.]'; then PY2=1; else PY2= ; fi diff --git a/test.sh b/test.sh index a6bd19a..0dc177a 100755 --- a/test.sh +++ b/test.sh @@ -6,6 +6,13 @@ set -e # Use the default python spelling, if the user hasn't specified one PYTHON=${PYTHON:-python} +# Test our shell script code portability and quality (including this script) +# SC1117 was disabled after 0.5, because it was too pedantic +EXCLUSIONS="--exclude=SC1117" +if command -v shellcheck; then + find . -name "*.sh" -exec shellcheck "$EXCLUSIONS" {} + +fi + ## moria1, Serge, no caches extra info, no dir port, doesn't exist ## TODO: validate output from all 3 commands using grep, grep, and stem? $PYTHON generateFallbackDirLine.py 9695DFC35FFEB861329B9F1AB04C46397020CE31 BA44A889E64B93FAA2B114E02C2A279A8555C533 001524DD403D729F08F7E5D77813EF12756CFA8D 5AFAC3D00E97D6733112CC9CA2A788691FA87125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

1 0

[fallback-scripts/master] generate: Handle new exceptions added by stem
by teor＠torproject.org 20 Jan '20

20 Jan '20

commit 22ac1c66a920cb0b6b4181df4a32a3e842ecb250 Author: teor <teor(a)torproject.org> Date: Mon Dec 2 13:56:09 2019 +1000 generate: Handle new exceptions added by stem But maintain compatibility with older stem versions. Part of 28863. --- generateFallbackDirLine.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/generateFallbackDirLine.py b/generateFallbackDirLine.py index b856c93..2177e00 100755 --- a/generateFallbackDirLine.py +++ b/generateFallbackDirLine.py @@ -9,9 +9,18 @@ import sys import urllib2 +import stem import stem.descriptor.remote import stem.util.tor_tools +## Stem version compatibility +try: + stem_DownloadFailed = stem.DownloadFailed +except AttributeError: + class DummyException(BaseException): + pass + stem_DownloadFailed = DummyException + if len(sys.argv) <= 1: print('Usage: %s fingerprint ...' % sys.argv[0]) sys.exit(1) @@ -29,6 +38,9 @@ for fingerprint in sys.argv[1:]: continue else: raise + except stem_DownloadFailed as exc: + print('# %s not found by stem' % fingerprint) + continue if not desc.dir_port: print("# %s needs a DirPort" % fingerprint)

1 0

[fallback-scripts/master] test: Move tests from .travis.yml into test.sh
by teor＠torproject.org 20 Jan '20

20 Jan '20

commit 2e69c92783f24d781066df2b297190ee95214e91 Author: teor <teor(a)torproject.org> Date: Mon Dec 2 13:31:52 2019 +1000 test: Move tests from .travis.yml into test.sh This change makes it easier to run the travis tests locally. Part of 28863. --- .travis.yml | 11 ++--------- test.sh | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+), 9 deletions(-) diff --git a/.travis.yml b/.travis.yml index d26ed93..b7bfc94 100644 --- a/.travis.yml +++ b/.travis.yml @@ -117,15 +117,8 @@ install: script: ## Run some fallback-scripts tests - ## moria1, Serge, no caches extra info, no dir port, doesn't exist - ## TODO: validate output from all 3 commands using grep, grep, and stem? - - $PYTHON generateFallbackDirLine.py 9695DFC35FFEB861329B9F1AB04C46397020CE31 BA44A889E64B93FAA2B114E02C2A279A8555C533 001524DD403D729F08F7E5D77813EF12756CFA8D 5AFAC3D00E97D6733112CC9CA2A788691FA87125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA - - $PYTHON lookupFallbackDirContact.py 9695DFC35FFEB861329B9F1AB04C46397020CE31 BA44A889E64B93FAA2B114E02C2A279A8555C533 001524DD403D729F08F7E5D77813EF12756CFA8D 5AFAC3D00E97D6733112CC9CA2A788691FA87125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA - ## If we get the top 200 relays from Onionoo, we generate a list with about - ## 10 fallbacks - - export TOR_FB_ONIONOO_LIMIT=200 - ## Hide info-level logs - - $PYTHON updateFallbackDirs.py 2>&1 | tee -a fallback.log | grep -v "INFO::" + ## Outputs fallback.log with info-level logs + - ./test.sh after_failure: ## Show all the output, including info-level logs diff --git a/test.sh b/test.sh new file mode 100755 index 0000000..a6bd19a --- /dev/null +++ b/test.sh @@ -0,0 +1,18 @@ +#!/bin/sh + +set -v +set -e + +# Use the default python spelling, if the user hasn't specified one +PYTHON=${PYTHON:-python} + +## moria1, Serge, no caches extra info, no dir port, doesn't exist +## TODO: validate output from all 3 commands using grep, grep, and stem? +$PYTHON generateFallbackDirLine.py 9695DFC35FFEB861329B9F1AB04C46397020CE31 BA44A889E64B93FAA2B114E02C2A279A8555C533 001524DD403D729F08F7E5D77813EF12756CFA8D 5AFAC3D00E97D6733112CC9CA2A788691FA87125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +$PYTHON lookupFallbackDirContact.py 9695DFC35FFEB861329B9F1AB04C46397020CE31 BA44A889E64B93FAA2B114E02C2A279A8555C533 001524DD403D729F08F7E5D77813EF12756CFA8D 5AFAC3D00E97D6733112CC9CA2A788691FA87125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA + +## If we get the top 200 relays from Onionoo, we generate a list with about +## 10 fallbacks +export TOR_FB_ONIONOO_LIMIT=200 +## Hide info-level logs +$PYTHON updateFallbackDirs.py 2>&1 | tee -a fallback.log | grep -v "INFO::"

1 0

[fallback-scripts/master] test: Use single # for comments
by teor＠torproject.org 20 Jan '20

20 Jan '20

commit 5b38e8989b1d55c9319cecb1703e5b7e2781085d Author: teor <teor(a)torproject.org> Date: Mon Dec 2 14:05:17 2019 +1000 test: Use single # for comments And move a comment. Part of 28863. --- test.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/test.sh b/test.sh index 7a7b23d..6f2d5e2 100755 --- a/test.sh +++ b/test.sh @@ -13,9 +13,10 @@ if command -v shellcheck; then find . -name "*.sh" -exec shellcheck "$EXCLUSIONS" {} + fi +# TODO: validate output from all 3 commands using grep, grep, and stem? # List of relay fingerprints to test with generate and lookup -## moria1, Serge, no caches extra info, no dir port, doesn't exist +# moria1, Serge, no caches extra info, no dir port, doesn't exist TEST_RELAY_LIST=" 9695DFC35FFEB861329B9F1AB04C46397020CE31 BA44A889E64B93FAA2B114E02C2A279A8555C533 @@ -24,15 +25,14 @@ BA44A889E64B93FAA2B114E02C2A279A8555C533 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA " -## TODO: validate output from all 3 commands using grep, grep, and stem? # We need each relay fingerprint as a separate argument # shellcheck disable=SC2086 $PYTHON generateFallbackDirLine.py $TEST_RELAY_LIST # shellcheck disable=SC2086 $PYTHON lookupFallbackDirContact.py $TEST_RELAY_LIST -## If we get the top 200 relays from Onionoo, we generate a list with about -## 10 fallbacks +# If we get the top 200 relays from Onionoo, we generate a list with about +# 10 fallbacks export TOR_FB_ONIONOO_LIMIT=200 -## Hide info-level logs +# Hide info-level logs $PYTHON updateFallbackDirs.py 2>&1 | tee -a fallback.log | grep -v "INFO::"

1 0

[fallback-scripts/master] test: Add more fingerprints of each type
by teor＠torproject.org 20 Jan '20

20 Jan '20

commit 965fb032481ad7437ead68d8aa414eb12a59e59d Author: teor <teor(a)torproject.org> Date: Mon Dec 2 14:25:52 2019 +1000 test: Add more fingerprints of each type Some relays are unreliable, but we still want the tests to check the different types of relays. Part of 28863. --- test.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/test.sh b/test.sh index 6f2d5e2..bffebc1 100755 --- a/test.sh +++ b/test.sh @@ -16,12 +16,14 @@ fi # TODO: validate output from all 3 commands using grep, grep, and stem? # List of relay fingerprints to test with generate and lookup -# moria1, Serge, no caches extra info, no dir port, doesn't exist +# moria1, Serge, no caches extra info x 2, no dir port x 2, doesn't exist TEST_RELAY_LIST=" 9695DFC35FFEB861329B9F1AB04C46397020CE31 BA44A889E64B93FAA2B114E02C2A279A8555C533 -001524DD403D729F08F7E5D77813EF12756CFA8D +025B66CEBC070FCB0519D206CF0CF4965C20C96E +0338F9F55111FE8E3570E7DE117EF3AF999CC1D7 5AFAC3D00E97D6733112CC9CA2A788691FA87125 +5DB9AE27A44EB7B476CC04A66C67A71C97A001E6 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA "

1 0

[fallback-scripts/master] lookup: Explicitly say "no contact"
by teor＠torproject.org 20 Jan '20

20 Jan '20

commit 218778081da63af9b4dbe7cfddb59cab86259b07 Author: teor <teor(a)torproject.org> Date: Mon Dec 2 14:26:48 2019 +1000 lookup: Explicitly say "no contact" Rather than just printing "None". Part of 28863. --- lookupFallbackDirContact.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lookupFallbackDirContact.py b/lookupFallbackDirContact.py index 14c53d1..88bad62 100755 --- a/lookupFallbackDirContact.py +++ b/lookupFallbackDirContact.py @@ -21,8 +21,9 @@ descriptor_list_fingerprints = [] for d in descriptor_list: assert d.fingerprint in sys.argv[1:] descriptor_list_fingerprints.append(d.fingerprint) - print "{} {}".format(d.fingerprint, d.contact) + contact = d.contact if d.contact else "(no contact)" + print "{} {}".format(d.fingerprint, contact) for fingerprint in sys.argv[1:]: if fingerprint not in descriptor_list_fingerprints: - print "{} not found in current descriptors".format(fingerprint) + print "{} # not found in current descriptors".format(fingerprint)

1 0

[fallback-scripts/master] generate: py3: Update using six.moves
by teor＠torproject.org 20 Jan '20

20 Jan '20

commit 3830e30575f4b19664cf90c0abbaec1d28f3be77 Author: teor <teor(a)torproject.org> Date: Mon Dec 2 14:54:32 2019 +1000 generate: py3: Update using six.moves Part of 28863. --- generateFallbackDirLine.py | 5 +++-- requirements.txt | 1 + 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/generateFallbackDirLine.py b/generateFallbackDirLine.py index 2177e00..f5e7424 100755 --- a/generateFallbackDirLine.py +++ b/generateFallbackDirLine.py @@ -7,7 +7,8 @@ # generateFallbackDirLine.py fingerprint ... import sys -import urllib2 + +from six.moves import urllib import stem import stem.descriptor.remote @@ -32,7 +33,7 @@ for fingerprint in sys.argv[1:]: try: desc = stem.descriptor.remote.get_server_descriptors(fingerprint).run()[0] - except urllib2.HTTPError as exc: + except urllib.error.HTTPError as exc: if exc.code == 404: print('# %s not found in recent descriptors' % fingerprint) continue diff --git a/requirements.txt b/requirements.txt index 31ee04a..dc71795 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,3 @@ python-dateutil +six stem

1 0

[fallback-scripts/master] lookup: py3: Use py3's print function
by teor＠torproject.org 20 Jan '20

20 Jan '20

commit 108d97403fb1194952aab2dd359150f9dbd3f82d Author: teor <teor(a)torproject.org> Date: Mon Dec 2 14:55:05 2019 +1000 lookup: py3: Use py3's print function Part of 28863. --- lookupFallbackDirContact.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lookupFallbackDirContact.py b/lookupFallbackDirContact.py index 88bad62..db17bd7 100755 --- a/lookupFallbackDirContact.py +++ b/lookupFallbackDirContact.py @@ -11,7 +11,7 @@ import sys import stem.descriptor.remote as remote if len(sys.argv) <= 1: - print "Usage: {} fingerprint ...".format(sys.argv[0]) + print("Usage: {} fingerprint ...".format(sys.argv[0])) sys.exit(-1) # we need descriptors, because the consensus does not have contact infos @@ -22,8 +22,8 @@ for d in descriptor_list: assert d.fingerprint in sys.argv[1:] descriptor_list_fingerprints.append(d.fingerprint) contact = d.contact if d.contact else "(no contact)" - print "{} {}".format(d.fingerprint, contact) + print("{} {}".format(d.fingerprint, contact)) for fingerprint in sys.argv[1:]: if fingerprint not in descriptor_list_fingerprints: - print "{} # not found in current descriptors".format(fingerprint) + print("{} # not found in current descriptors".format(fingerprint))

1 0

[fallback-scripts/master] test: Handle the log file in Travis, not the test script
by teor＠torproject.org 20 Jan '20

20 Jan '20

commit 34df66d2b648b61a714b7ba5450eb15f7a7d6259 Author: teor <teor(a)torproject.org> Date: Mon Dec 2 14:45:20 2019 +1000 test: Handle the log file in Travis, not the test script Part of 28863. --- .travis.yml | 4 ++-- test.sh | 3 +-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/.travis.yml b/.travis.yml index a219026..2d0bcf4 100644 --- a/.travis.yml +++ b/.travis.yml @@ -128,8 +128,8 @@ install: script: ## Run some fallback-scripts tests - ## Outputs fallback.log with info-level logs - - ./test.sh + ## Hide info-level logs, unless there is an error + - ./test.sh 2>&1 | tee fallback.log | grep -v "INFO::" after_failure: ## Show all the output, including info-level logs diff --git a/test.sh b/test.sh index bffebc1..eddc624 100755 --- a/test.sh +++ b/test.sh @@ -36,5 +36,4 @@ $PYTHON lookupFallbackDirContact.py $TEST_RELAY_LIST # If we get the top 200 relays from Onionoo, we generate a list with about # 10 fallbacks export TOR_FB_ONIONOO_LIMIT=200 -# Hide info-level logs -$PYTHON updateFallbackDirs.py 2>&1 | tee -a fallback.log | grep -v "INFO::" +$PYTHON updateFallbackDirs.py

1 0