March 2021 - tor-commits - lists.torproject.org

[sbws/master] Merge remote-tracking branch 'gitlab/merge-requests/71' into maint-1.1
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 279874f4f4ef90f50b87580cd347b6b2ce62c390 Merge: cde8950 fc3d3b9 Author: Georg Koppen <gk(a)torproject.org> Date: Tue Mar 16 13:07:48 2021 +0000 Merge remote-tracking branch 'gitlab/merge-requests/71' into maint-1.1 docs/source/torflow_aggr.rst | 25 ++++++++++++++++++ sbws/globals.py | 8 ++++++ sbws/lib/scaling.py | 40 +++++++++++++++++++++++++---- sbws/lib/v3bwfile.py | 31 ++++++++++++++-------- sbws/util/stem.py | 22 +++++++++++++++- tests/unit/lib/test_scaling.py | 58 +++++++++++++++++++++++++++++++++++++++++- 6 files changed, 166 insertions(+), 18 deletions(-)

1 0

[sbws/master] Merge branch 'maint-1.1'
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit fd139b89de20b6d38941f137325e05069995793b Merge: 63d71cc 279874f Author: juga0 <juga(a)riseup.net> Date: Tue Mar 16 14:18:41 2021 +0000 Merge branch 'maint-1.1' .gitlab-ci.yml | 71 +++++++++++++++++++++++++------------- docs/source/torflow_aggr.rst | 25 ++++++++++++++ sbws/config.default.ini | 8 ++--- sbws/core/scanner.py | 8 +++-- sbws/globals.py | 8 +++++ sbws/lib/destination.py | 38 +++++++++++--------- sbws/lib/relaylist.py | 6 +++- sbws/lib/resultdump.py | 9 +++-- sbws/lib/scaling.py | 40 ++++++++++++++++++--- sbws/lib/v3bwfile.py | 31 +++++++++++------ sbws/util/stem.py | 34 ++++++++++++++---- setup.cfg | 3 ++ tests/integration/run.sh | 5 ++- tests/integration/start_chutney.sh | 2 ++ tests/unit/lib/test_resultdump.py | 9 ++--- tests/unit/lib/test_scaling.py | 58 ++++++++++++++++++++++++++++++- tox.ini | 4 +-- 17 files changed, 278 insertions(+), 81 deletions(-)

1 0

[sbws/maint-1.1] fix: CI: Update Python versions
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 193f5da1b0024aefb03edee547d770f0584761f3 Author: juga0 <juga(a)riseup.net> Date: Tue Mar 2 12:45:43 2021 +0000 fix: CI: Update Python versions Closes #40055. --- .gitlab-ci.yml | 55 +++++++++++++++++++++++++++++++++++-------------------- tox.ini | 4 ++-- 2 files changed, 37 insertions(+), 22 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 10dfbc2..098781f 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -5,9 +5,16 @@ # 0.4.6 stable by Jun 15, 2021 # 0.4.5 (LTS) EOL Feb 15, 2023 # 0.3.5 (LTS) EOL Feb 1, 2022 +# Python releases: +# 3.10 stable by 2021-10-04: https://www.python.org/dev/peps/pep-0619/ +# Python stable releases: https://www.python.org/downloads/ +# 3.9 EOL 2025-10 PEP 596 +# 3.8 EOL 2024-10 PEP 569, newest major release +# 3.7 EOL 2023-06-27 PEP 537, included in Debian buster +# 3.6 EOL 2021-12-23 PEP 494 variables: - BASE_IMAGE: python:3.7 + BASE_IMAGE: python:3.8 RELEASE: tor-nightly-master-buster # Without version, the default available in the Debian repository will be # installed. @@ -38,47 +45,55 @@ before_script: after_script: - tox -e stats -python36tor035: +python36: variables: BASE_IMAGE: python:3.6 + image: $BASE_IMAGE + script: + - tox -e py36 + - tox -e integration + +python37tor035: + variables: + BASE_IMAGE: python:3.7 RELEASE: tor-nightly-0.3.5.x-buster TOR: tor=0.3.5* image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python36tor045: +python37tor045: variables: - BASE_IMAGE: python:3.6 + BASE_IMAGE: python:3.7 RELEASE: tor-nightly-0.4.5.x-buster TOR: tor=0.4.5* image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python36tormaster: +python37tormaster: variables: - BASE_IMAGE: python:3.6 + BASE_IMAGE: python:3.7 RELEASE: tor-nightly-master-buster TOR: tor=0.4.6* image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python36torstable: +python37torstable: variables: - BASE_IMAGE: python:3.6 + BASE_IMAGE: python:3.7 RELEASE: buster TOR: tor image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python37: +python38: # This will overwrite the default before_script, so need to repeat the # commands before_script: @@ -100,24 +115,24 @@ python37: script: - tox -e inst - tox -e setup - - tox -e py37 + - tox -e py38 - tox -e integration - tox -e lint - tox -e doc -python38: +python39: variables: - BASE_IMAGE: python:3.8 + BASE_IMAGE: python:3.9 image: $BASE_IMAGE script: - - tox -e py38 + - tox -e py39 - tox -e integration -python39: +python310: variables: - BASE_IMAGE: python:3.9-rc-buster + BASE_IMAGE: python:3.10-rc-buster image: $BASE_IMAGE script: - - tox -e py39 + - tox -e py310 - tox -e integration allow_failure: true diff --git a/tox.ini b/tox.ini index acd31aa..bb66836 100644 --- a/tox.ini +++ b/tox.ini @@ -1,13 +1,13 @@ [tox] skip_missing_interpreters = True -envlist = py{36, 37, 38, 39}, inst, setup, integration, lint, stats, doc +envlist = py{36, 37, 38, 39, 310}, inst, setup, integration, lint, stats, doc [travis] python = 3.6: py36, inst, setup, unit, integration, lint, doc 3.7: py37, inst, setup, unit, integration, lint, doc 3.8: py38, inst, setup, unit, integration, lint, doc - 3.9-dev: py39, inst, setup, unit, integration, lint, doc + 3.9: py39, inst, setup, unit, integration, lint, doc nightly: pynightly, inst, setup, unit, integration, lint, doc ; [testenv]

1 0

[sbws/maint-1.1] fix: CI: Exit from integration script
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit a49cf7029bf5b3a6414ca89fd31a243bec319ef4 Author: juga0 <juga(a)riseup.net> Date: Wed Mar 3 12:38:18 2021 +0000 fix: CI: Exit from integration script when any of the commands fail. --- tests/integration/run.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/integration/run.sh b/tests/integration/run.sh index e870b50..70ee5f2 100755 --- a/tests/integration/run.sh +++ b/tests/integration/run.sh @@ -1,5 +1,8 @@ #!/bin/bash -set -x +# Instead of exiting inmmediately when any of this commands fail, +# the scanner, generate and coverage lines could continue and store there was +# an error on that command. It's just simpler with `-e`. +set -ex tests/integration/start_chutney.sh python3 scripts/tools/sbws-http-server.py --port 28888 &>/dev/null &

1 0

[sbws/maint-1.1] fix: relalist: Use the consensus timestamp
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit aa6b79d343dfe43fa6bb447a9d9c37d41c37e589 Author: juga0 <juga(a)riseup.net> Date: Fri Mar 5 11:51:59 2021 +0000 fix: relalist: Use the consensus timestamp to the relay consensus timestamps list, so that it can be tested it was in a concrete consensus. --- sbws/lib/relaylist.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/sbws/lib/relaylist.py b/sbws/lib/relaylist.py index 4f33799..2a0bd8d 100644 --- a/sbws/lib/relaylist.py +++ b/sbws/lib/relaylist.py @@ -71,7 +71,11 @@ class Relay: self.relay_in_recent_consensus = timestamps.DateTimeSeq( [], MAX_RECENT_CONSENSUS_COUNT ) - self.update_relay_in_recent_consensus() + # Use the same timestamp as the consensus, so that it can be tested + # that the relay was in a consensus using this timestamp. + # Note that this doesn't change the number of consensus the relay was + # in. + self.update_relay_in_recent_consensus(timestamp) # The number of times that a relay is "prioritized" to be measured. # It is incremented in ``RelayPrioritizer.best_priority`` self.relay_recent_priority_list = timestamps.DateTimeSeq(

1 0

[sbws/maint-1.1] fix: CI: Update tor versions
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 842ed52f34992d416930a4beace9b2d8016af574 Author: juga0 <juga(a)riseup.net> Date: Tue Mar 2 12:40:52 2021 +0000 fix: CI: Update tor versions --- .gitlab-ci.yml | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 9b4fe40..10dfbc2 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,3 +1,11 @@ +# core/tor releases: +# https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/CoreTorRele… +# As of 2021.03.02 all dirauths are running with version greater or equal to +# 0.4.5. +# 0.4.6 stable by Jun 15, 2021 +# 0.4.5 (LTS) EOL Feb 15, 2023 +# 0.3.5 (LTS) EOL Feb 1, 2022 + variables: BASE_IMAGE: python:3.7 RELEASE: tor-nightly-master-buster @@ -40,21 +48,21 @@ python36tor035: - tox -e py36 - tox -e integration -python36tor041: +python36tor045: variables: BASE_IMAGE: python:3.6 - RELEASE: tor-nightly-0.4.1.x-buster - TOR: tor=0.4.1* + RELEASE: tor-nightly-0.4.5.x-buster + TOR: tor=0.4.5* image: $BASE_IMAGE script: - tox -e py36 - tox -e integration -python36tor042: +python36tormaster: variables: BASE_IMAGE: python:3.6 - RELEASE: tor-nightly-0.4.2.x-buster - TOR: tor=0.4.2* + RELEASE: tor-nightly-master-buster + TOR: tor=0.4.6* image: $BASE_IMAGE script: - tox -e py36

1 0

[sbws/maint-1.1] fix: CI: Temporal workaround for #40072
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 4e7ad5d78d58c9068ece2239cacc2ddd6d13d7b1 Author: juga0 <juga(a)riseup.net> Date: Fri Mar 5 13:45:39 2021 +0000 fix: CI: Temporal workaround for #40072 --- tests/integration/start_chutney.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/integration/start_chutney.sh b/tests/integration/start_chutney.sh index 04f9641..4dbcb68 100755 --- a/tests/integration/start_chutney.sh +++ b/tests/integration/start_chutney.sh @@ -23,4 +23,6 @@ git pull ./chutney status networks/bwscanner ./chutney wait_for_bootstrap networks/bwscanner +# temporal workaround for https://gitlab.torproject.org/tpo/core/chutney/-/issues/40016 +sleep 60 cd $CURRENT_DIR

1 0

[sbws/maint-1.1] minor: Change info logs to debug or warning
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 3b1f7b79bfab9262567e487f0429635781f6fcab Author: juga0 <juga(a)riseup.net> Date: Tue Mar 2 17:21:15 2021 +0000 minor: Change info logs to debug or warning when they contain sensitive information, eg. Web server or are too verbose for the debug level. Also add log to indicate when the main loop is actually started. --- sbws/core/scanner.py | 3 ++- sbws/lib/destination.py | 34 ++++++++++++++++++++-------------- sbws/lib/resultdump.py | 9 ++++++--- 3 files changed, 28 insertions(+), 18 deletions(-) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index f9ab1ce..c7dde75 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -377,7 +377,7 @@ def measure_relay(args, conf, destinations, cb, rl, relay): # always fail when there's only one Web server. if not is_usable and \ relay.is_exit_not_bad_allowing_port(dest.port): - log.info( + log.debug( "Exit %s (%s) that can't exit all ips, with exit policy %s, failed" " to connect to %s via circuit %s (%s). Reason: %s. Trying again " "with it as entry.", relay.fingerprint, relay.nickname, @@ -575,6 +575,7 @@ def main_loop(args, conf, controller, relay_list, circuit_builder, result_dump, measured. """ + log.info("Started the main loop to measure the relays.") hbeat = Heartbeat(conf.getpath('paths', 'state_fname')) # Set the time to wait for a thread to finish as the half of an HTTP diff --git a/sbws/lib/destination.py b/sbws/lib/destination.py index 5330dbf..f007a7a 100644 --- a/sbws/lib/destination.py +++ b/sbws/lib/destination.py @@ -207,12 +207,12 @@ class Destination: self._delta_seconds_retry *= factor or self._factor if self._delta_seconds_retry > self._max_seconds_between_retries: self._delta_seconds_retry = self._max_seconds_between_retries - log.info("Incremented the time to try destination %s past the " + log.debug("Incremented the time to try destination %s past the " "limit, capping it at %s hours.", self.url, self._delta_seconds_retry / 60 / 60) else: - log.info("Incremented the time to try destination %s to %s hours.", - self.url, self._delta_seconds_retry / 60 / 60) + log.debug("Incremented the time to try destination %s to %s " + "hours.", self.url, self._delta_seconds_retry / 60 / 60) def _get_last_try_in_seconds_ago(self): """ @@ -250,23 +250,29 @@ class Destination: # Failed the last X consecutive times if self._are_last_attempts_failures(): - # The log here will appear in all the the queued - # relays and threads. - log.warning("The last %s times the destination %s failed. " + # The log here will appear in all the the queued relays and + # threads. + log.debug("The last %s times the destination %s failed. " + "It last ran %s seconds ago. " + "Disabled for %s seconds.", + self._max_num_failures, self.url, + self._get_last_try_in_seconds_ago(), + self._delta_seconds_retry) + log.warning("The last %s times a destination failed. " "It last ran %s seconds ago. " - "Disabled for %s seconds.", - self._max_num_failures, self.url, + "Disabled for %s seconds." + "Please, add more destinations or increment the " + "number of maximum number of consecutive failures " + "in the configuration.", + self._max_num_failures, self._get_last_try_in_seconds_ago(), self._delta_seconds_retry) - log.warning("Please, add more destinations or increment the " - "number of maximum number of consecutive failures " - "in the configuration.") # It was not used for a while and the last time it was used # was long ago, then try again if self._is_last_try_old_enough(): - log.info("The destination %s was not tried for %s seconds, " - "it is going to by tried again.", self.url, - self._get_last_try_in_seconds_ago()) + log.debug("The destination %s was not tried for %s seconds, " + "it is going to by tried again.", self.url, + self._get_last_try_in_seconds_ago()) # Set the next time to retry higher, in case this attempt fails self._increment_time_to_retry() return True diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py index b287242..cc61bcc 100644 --- a/sbws/lib/resultdump.py +++ b/sbws/lib/resultdump.py @@ -799,14 +799,17 @@ class ResultDump: msg += ". As exit." if as_exit else ". As entry." # When the error is that there are not more functional destinations. if result.type == "error-destination": - log.info("Shutting down because there are not functional " - "destinations.") + log.warning("Shutting down because there are not functional " + "destinations.") # NOTE: Because this is executed in a thread, stop_threads can not # be call from here, it has to be call from the main thread. # Instead set the singleton end event, that will call stop_threads # from the main process. settings.end_event.set() - log.info(msg) + # log every success/error measurement at debug level, to don't log the + # the Web server at info level and because there's already a + # heartbeat msg to indicate progress. + log.debug(msg) def enter(self): """Main loop for the ResultDump thread.

1 0

[sbws/maint-1.1] fix: scanner: Log times kept
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit dc60a3c0215e28d8d32e9d0ffd4aa8872a8f70c3 Author: juga0 <juga(a)riseup.net> Date: Wed Mar 3 09:42:38 2021 +0000 fix: scanner: Log times kept not only the times that are not kept. Closes #40060 --- sbws/core/scanner.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index e446391..2fba10f 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -466,14 +466,17 @@ def _should_keep_result(did_request_maximum, result_time, download_times): # In the normal case, we didn't ask for the maximum allowed amount. So we # should only allow ourselves to keep results that are between the min and # max allowed time + msg = "Keeping measurement time {:.2f}".format(result_time) if not did_request_maximum and \ result_time >= download_times['min'] and \ result_time < download_times['max']: + log.debug(msg) return True # If we did request the maximum amount, we should keep the result as long # as it took less than the maximum amount of time if did_request_maximum and \ result_time < download_times['max']: + log.debug(msg) return True # In all other cases, return false log.debug('Not keeping result time %f.%s', result_time,

1 0

[sbws/maint-1.1] major: Change default log level to info
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit fa70b0a1edd01062aa82724ff4371116d00225fa Author: juga0 <juga(a)riseup.net> Date: Wed Mar 3 07:57:15 2021 +0000 major: Change default log level to info also change formatting to show thread. --- sbws/config.default.ini | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sbws/config.default.ini b/sbws/config.default.ini index c47e4fd..4080189 100644 --- a/sbws/config.default.ini +++ b/sbws/config.default.ini @@ -119,13 +119,13 @@ to_file_max_bytes = 10485760 to_file_num_backups = 50 # Level to log at. Debug, info, warning, error, critical. # `level` must be set to the lower of all the handler levels. -level = debug -to_file_level = debug +level = info +to_file_level = info to_stdout_level = info to_syslog_level = info # Format string to use when logging -format = %(asctime)s %(module)s[%(process)s]: <%(levelname)s> %(message)s +format = %(asctime)s %(module)s[%(process)s]: <%(levelname)s> (%(threadName)s) %(message)s to_stdout_format = ${format} to_syslog_format = %(module)s[%(process)s]: <%(levelname)s> %(message)s # verbose formatter useful for debugging -to_file_format = %(asctime)s %(levelname)s %(threadName)s %(filename)s:%(lineno)s - %(funcName)s - %(message)s +to_file_format = %(asctime)s %(levelname)s (%(threadName)s) %(filename)s:%(lineno)s - %(funcName)s - %(message)s

1 0