March 2021 - tor-commits - lists.torproject.org

[status-site/main] Merge branch 'main' into 'main'
by anarcat＠torproject.org 16 Mar '21

16 Mar '21

commit 5b203daa9140afacb82c943b89a75922e7d5899c Merge: 02df8c5 db32be6 Author: anarcat <anarcat(a)torproject.org> Date: Tue Mar 16 18:46:20 2021 +0000 Merge branch 'main' into 'main' Update categories, service titles and descriptions Adds a third category for websites; updates category and service names as per internal conventions; and adds descriptions for each service. See merge request tpo/tpa/status-site!3 config.yml | 63 ++++++++++++++++--------------- content/issues/2021-01-19-status-page.md | 18 --------- content/issues/2021-01-25-gmail-bounce.md | 2 +- content/issues/2021-01-28-dir-auth.md | 4 +- 4 files changed, 36 insertions(+), 51 deletions(-)

1 0

[status-site/main] Update categories, service titles and descriptions
by anarcat＠torproject.org 16 Mar '21

16 Mar '21

commit db32be6ea810a99bae31c4c3323d8e8d1930937c Author: Duncan <duncan(a)torproject.org> Date: Tue Mar 16 18:46:19 2021 +0000 Update categories, service titles and descriptions Closes: #3 --- config.yml | 63 ++++++++++++++++--------------- content/issues/2021-01-19-status-page.md | 18 --------- content/issues/2021-01-25-gmail-bounce.md | 2 +- content/issues/2021-01-28-dir-auth.md | 4 +- 4 files changed, 36 insertions(+), 51 deletions(-) diff --git a/config.yml b/config.yml index df39c7a..ddb2571 100644 --- a/config.yml +++ b/config.yml @@ -121,9 +121,12 @@ params: # For help, see the wiki: # https://github.com/cstate/cstate/wiki/Customization categories: - - name: Tor network - - name: TPA - description: internal Tor services + - name: Tor Network + description: Core services that are integral to the Tor Network. + - name: Tor Project websites + description: Official websites and services hosted and maintained by the Tor Project. + - name: Internal systems + description: Tools and services that help us collaborate with one another, managed by the Tor Project System Administration Team (TPA). closed: true - name: Uncategorized untitled: true @@ -134,33 +137,33 @@ params: # For help, see the wiki: # https://github.com/cstate/cstate/wiki/Customization systems: - - name: v3 services - category: Tor network - - name: v2 services - category: Tor network - - name: directory authorities - description: whatever it is that those things do - category: Tor network - - name: check.tpo - description: the check.torproject.org website - category: Tor network + - name: v3 Onion Services + description: Third generation onion services – running private services (e.g. websites) that are only accessible through the Tor network. + category: Tor Network + - name: v2 Onion Services + description: Second generation onion services – running private services (e.g. websites) that are only accessible through the Tor network. + category: Tor Network + - name: Directory Authorities + description: Special-purpose relays that maintain a list of currently-running relays and periodically publish a consensus together. + category: Tor Network + - name: torproject.org + description: Our official website and its associated portals (e.g. community.torproject.org) + category: Tor Project websites + - name: Tor Check + description: The check.torproject.org website, used to confirm you're connected to the Tor Network. + category: Tor Project websites - name: GitLab - description: gitlab.torproject.org - category: TPA + description: Our code-review, bug tracking, and collaboration platform of choice (gitlab.torproject.org) + category: Internal systems + - name: Gitolite + description: An access control layer on top of git, used to provide read/write access. + category: Internal systems - name: Email - description: general email delivery - category: TPA - - name: lists - description: mailing lists (mailman) - category: TPA - - name: gitolite - description: read/write access to git - category: TPA - - name: websites - description: the many websites under torproject.org - category: TPA - - name: other - category: Uncategorized + description: General email delivery from within the Tor Project. + category: Internal systems + - name: Mailing lists + description: Our public and private mailing lists (lists.torproject.org) powered by Mailman, the GNU Mailing List Manager. + category: Internal systems # What date format to use? # @@ -239,8 +242,8 @@ params: # Default: We continuously monitor the status of our services and if there are any interruptions, a note will be posted here. description: | We continuously monitor the status of our services and if there - are any interruptions, a note will be posted here. If you need to - do a modification on this page, [follow the + are any interruptions an update will be posted here. If you need to + modify this page, [follow the documentation](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/s… ([mirror](https://web.archive.org/web/2/https://gitlab.torproject.org/tpo/tpa…) # Tabs on homepage diff --git a/content/issues/2021-01-19-status-page.md b/content/issues/2021-01-19-status-page.md deleted file mode 100644 index 7fe22c4..0000000 --- a/content/issues/2021-01-19-status-page.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Status page deployment -date: 2021-01-15 18:29:00 -resolved: true -resolvedWhen: 2021-01-20 19:52:56 -# Possible severity levels: down, disrupted, notice -severity: notice -affected: - - other -section: issue ---- - -We're in the process of deploying [cstate](https://github.com/cstate/cstate) for a status page, this -page might need a few more updates to be actually relevant. Follow -[ticket 40138](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40138) in GitLab for more information. - -Update: the site is mostly operational, even if updates are a little -clunky. Considered fixed. diff --git a/content/issues/2021-01-25-gmail-bounce.md b/content/issues/2021-01-25-gmail-bounce.md index dba9281..19950c2 100644 --- a/content/issues/2021-01-25-gmail-bounce.md +++ b/content/issues/2021-01-25-gmail-bounce.md @@ -7,7 +7,7 @@ resolvedWhen: 2021-01-25 21:40:00 severity: notice affected: - Email - - lists + - Mailing lists section: issue --- diff --git a/content/issues/2021-01-28-dir-auth.md b/content/issues/2021-01-28-dir-auth.md index 1db8571..fe5b7d3 100644 --- a/content/issues/2021-01-28-dir-auth.md +++ b/content/issues/2021-01-28-dir-auth.md @@ -6,8 +6,8 @@ resolvedWhen: 2021-01-29 05:30:00 # Possible severity levels: down, disrupted, notice severity: disrupted affected: - - directory authorities - - v3 services + - Directory Authorities + - v3 Onion Services section: issue ---

1 0

[sbws/master] fix: CI: Update Python versions
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 193f5da1b0024aefb03edee547d770f0584761f3 Author: juga0 <juga(a)riseup.net> Date: Tue Mar 2 12:45:43 2021 +0000 fix: CI: Update Python versions Closes #40055. --- .gitlab-ci.yml | 55 +++++++++++++++++++++++++++++++++++-------------------- tox.ini | 4 ++-- 2 files changed, 37 insertions(+), 22 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 10dfbc2..098781f 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -5,9 +5,16 @@ # 0.4.6 stable by Jun 15, 2021 # 0.4.5 (LTS) EOL Feb 15, 2023 # 0.3.5 (LTS) EOL Feb 1, 2022 +# Python releases: +# 3.10 stable by 2021-10-04: https://www.python.org/dev/peps/pep-0619/ +# Python stable releases: https://www.python.org/downloads/ +# 3.9 EOL 2025-10 PEP 596 +# 3.8 EOL 2024-10 PEP 569, newest major release +# 3.7 EOL 2023-06-27 PEP 537, included in Debian buster +# 3.6 EOL 2021-12-23 PEP 494 variables: - BASE_IMAGE: python:3.7 + BASE_IMAGE: python:3.8 RELEASE: tor-nightly-master-buster # Without version, the default available in the Debian repository will be # installed. @@ -38,47 +45,55 @@ before_script: after_script: - tox -e stats -python36tor035: +python36: variables: BASE_IMAGE: python:3.6 + image: $BASE_IMAGE + script: + - tox -e py36 + - tox -e integration + +python37tor035: + variables: + BASE_IMAGE: python:3.7 RELEASE: tor-nightly-0.3.5.x-buster TOR: tor=0.3.5* image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python36tor045: +python37tor045: variables: - BASE_IMAGE: python:3.6 + BASE_IMAGE: python:3.7 RELEASE: tor-nightly-0.4.5.x-buster TOR: tor=0.4.5* image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python36tormaster: +python37tormaster: variables: - BASE_IMAGE: python:3.6 + BASE_IMAGE: python:3.7 RELEASE: tor-nightly-master-buster TOR: tor=0.4.6* image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python36torstable: +python37torstable: variables: - BASE_IMAGE: python:3.6 + BASE_IMAGE: python:3.7 RELEASE: buster TOR: tor image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python37: +python38: # This will overwrite the default before_script, so need to repeat the # commands before_script: @@ -100,24 +115,24 @@ python37: script: - tox -e inst - tox -e setup - - tox -e py37 + - tox -e py38 - tox -e integration - tox -e lint - tox -e doc -python38: +python39: variables: - BASE_IMAGE: python:3.8 + BASE_IMAGE: python:3.9 image: $BASE_IMAGE script: - - tox -e py38 + - tox -e py39 - tox -e integration -python39: +python310: variables: - BASE_IMAGE: python:3.9-rc-buster + BASE_IMAGE: python:3.10-rc-buster image: $BASE_IMAGE script: - - tox -e py39 + - tox -e py310 - tox -e integration allow_failure: true diff --git a/tox.ini b/tox.ini index acd31aa..bb66836 100644 --- a/tox.ini +++ b/tox.ini @@ -1,13 +1,13 @@ [tox] skip_missing_interpreters = True -envlist = py{36, 37, 38, 39}, inst, setup, integration, lint, stats, doc +envlist = py{36, 37, 38, 39, 310}, inst, setup, integration, lint, stats, doc [travis] python = 3.6: py36, inst, setup, unit, integration, lint, doc 3.7: py37, inst, setup, unit, integration, lint, doc 3.8: py38, inst, setup, unit, integration, lint, doc - 3.9-dev: py39, inst, setup, unit, integration, lint, doc + 3.9: py39, inst, setup, unit, integration, lint, doc nightly: pynightly, inst, setup, unit, integration, lint, doc ; [testenv]

1 0

[sbws/master] fix: CI: Update tor versions
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 842ed52f34992d416930a4beace9b2d8016af574 Author: juga0 <juga(a)riseup.net> Date: Tue Mar 2 12:40:52 2021 +0000 fix: CI: Update tor versions --- .gitlab-ci.yml | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 9b4fe40..10dfbc2 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,3 +1,11 @@ +# core/tor releases: +# https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/CoreTorRele… +# As of 2021.03.02 all dirauths are running with version greater or equal to +# 0.4.5. +# 0.4.6 stable by Jun 15, 2021 +# 0.4.5 (LTS) EOL Feb 15, 2023 +# 0.3.5 (LTS) EOL Feb 1, 2022 + variables: BASE_IMAGE: python:3.7 RELEASE: tor-nightly-master-buster @@ -40,21 +48,21 @@ python36tor035: - tox -e py36 - tox -e integration -python36tor041: +python36tor045: variables: BASE_IMAGE: python:3.6 - RELEASE: tor-nightly-0.4.1.x-buster - TOR: tor=0.4.1* + RELEASE: tor-nightly-0.4.5.x-buster + TOR: tor=0.4.5* image: $BASE_IMAGE script: - tox -e py36 - tox -e integration -python36tor042: +python36tormaster: variables: BASE_IMAGE: python:3.6 - RELEASE: tor-nightly-0.4.2.x-buster - TOR: tor=0.4.2* + RELEASE: tor-nightly-master-buster + TOR: tor=0.4.6* image: $BASE_IMAGE script: - tox -e py36

1 0

[sbws/master] fix: relalist: Use the consensus timestamp
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit aa6b79d343dfe43fa6bb447a9d9c37d41c37e589 Author: juga0 <juga(a)riseup.net> Date: Fri Mar 5 11:51:59 2021 +0000 fix: relalist: Use the consensus timestamp to the relay consensus timestamps list, so that it can be tested it was in a concrete consensus. --- sbws/lib/relaylist.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/sbws/lib/relaylist.py b/sbws/lib/relaylist.py index 4f33799..2a0bd8d 100644 --- a/sbws/lib/relaylist.py +++ b/sbws/lib/relaylist.py @@ -71,7 +71,11 @@ class Relay: self.relay_in_recent_consensus = timestamps.DateTimeSeq( [], MAX_RECENT_CONSENSUS_COUNT ) - self.update_relay_in_recent_consensus() + # Use the same timestamp as the consensus, so that it can be tested + # that the relay was in a consensus using this timestamp. + # Note that this doesn't change the number of consensus the relay was + # in. + self.update_relay_in_recent_consensus(timestamp) # The number of times that a relay is "prioritized" to be measured. # It is incremented in ``RelayPrioritizer.best_priority`` self.relay_recent_priority_list = timestamps.DateTimeSeq(

1 0

[sbws/master] fix: CI: Exit from integration script
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit a49cf7029bf5b3a6414ca89fd31a243bec319ef4 Author: juga0 <juga(a)riseup.net> Date: Wed Mar 3 12:38:18 2021 +0000 fix: CI: Exit from integration script when any of the commands fail. --- tests/integration/run.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/integration/run.sh b/tests/integration/run.sh index e870b50..70ee5f2 100755 --- a/tests/integration/run.sh +++ b/tests/integration/run.sh @@ -1,5 +1,8 @@ #!/bin/bash -set -x +# Instead of exiting inmmediately when any of this commands fail, +# the scanner, generate and coverage lines could continue and store there was +# an error on that command. It's just simpler with `-e`. +set -ex tests/integration/start_chutney.sh python3 scripts/tools/sbws-http-server.py --port 28888 &>/dev/null &

1 0

[sbws/master] fix: CI: Temporal workaround for #40072
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 4e7ad5d78d58c9068ece2239cacc2ddd6d13d7b1 Author: juga0 <juga(a)riseup.net> Date: Fri Mar 5 13:45:39 2021 +0000 fix: CI: Temporal workaround for #40072 --- tests/integration/start_chutney.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/integration/start_chutney.sh b/tests/integration/start_chutney.sh index 04f9641..4dbcb68 100755 --- a/tests/integration/start_chutney.sh +++ b/tests/integration/start_chutney.sh @@ -23,4 +23,6 @@ git pull ./chutney status networks/bwscanner ./chutney wait_for_bootstrap networks/bwscanner +# temporal workaround for https://gitlab.torproject.org/tpo/core/chutney/-/issues/40016 +sleep 60 cd $CURRENT_DIR

1 0

[sbws/master] minor: Change log warning to info or debug
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 290030aedd6d6ade7e53c22dacab149e3e5b1b8f Author: juga0 <juga(a)riseup.net> Date: Tue Mar 2 17:33:12 2021 +0000 minor: Change log warning to info or debug when it contains sensitive information. --- sbws/core/scanner.py | 2 +- sbws/util/stem.py | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index c7dde75..e446391 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -388,7 +388,7 @@ def measure_relay(args, conf, destinations, cb, rl, relay): circ_fps, nicknames, exit_policy = r circ_id, reason = cb.build_circuit(circ_fps) if not circ_id: - log.warning( + log.info( "Exit %s (%s) that can't exit all ips, failed to create " " circuit as entry: %s (%s).", relay.fingerprint, relay.nickname, circ_fps, nicknames) diff --git a/sbws/util/stem.py b/sbws/util/stem.py index 9fea31f..38d0570 100644 --- a/sbws/util/stem.py +++ b/sbws/util/stem.py @@ -29,12 +29,12 @@ def attach_stream_to_circuit_listener(controller, circ_id): circuit_str(controller, circ_id)) try: controller.attach_stream(st.id, circ_id) - except (UnsatisfiableRequest, InvalidRequest) as e: - log.warning('Couldn\'t attach stream to circ %s: %s', - circ_id, e) - except OperationFailed as e: - log.exception("Error attaching stream %s to circ %s: %s", - st.id, circ_id, e) + # So far we never saw this error. + except ( + UnsatisfiableRequest, InvalidRequest, OperationFailed + ) as e: + log.debug('Error attaching stream %s to circ %s: %s', + st.id, circ_id, e) else: pass return closure_stream_event_listener

1 0

[sbws/master] major: Change default log level to info
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit fa70b0a1edd01062aa82724ff4371116d00225fa Author: juga0 <juga(a)riseup.net> Date: Wed Mar 3 07:57:15 2021 +0000 major: Change default log level to info also change formatting to show thread. --- sbws/config.default.ini | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sbws/config.default.ini b/sbws/config.default.ini index c47e4fd..4080189 100644 --- a/sbws/config.default.ini +++ b/sbws/config.default.ini @@ -119,13 +119,13 @@ to_file_max_bytes = 10485760 to_file_num_backups = 50 # Level to log at. Debug, info, warning, error, critical. # `level` must be set to the lower of all the handler levels. -level = debug -to_file_level = debug +level = info +to_file_level = info to_stdout_level = info to_syslog_level = info # Format string to use when logging -format = %(asctime)s %(module)s[%(process)s]: <%(levelname)s> %(message)s +format = %(asctime)s %(module)s[%(process)s]: <%(levelname)s> (%(threadName)s) %(message)s to_stdout_format = ${format} to_syslog_format = %(module)s[%(process)s]: <%(levelname)s> %(message)s # verbose formatter useful for debugging -to_file_format = %(asctime)s %(levelname)s %(threadName)s %(filename)s:%(lineno)s - %(funcName)s - %(message)s +to_file_format = %(asctime)s %(levelname)s (%(threadName)s) %(filename)s:%(lineno)s - %(funcName)s - %(message)s

1 0

[sbws/master] fix: scanner: Log times kept
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit dc60a3c0215e28d8d32e9d0ffd4aa8872a8f70c3 Author: juga0 <juga(a)riseup.net> Date: Wed Mar 3 09:42:38 2021 +0000 fix: scanner: Log times kept not only the times that are not kept. Closes #40060 --- sbws/core/scanner.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index e446391..2fba10f 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -466,14 +466,17 @@ def _should_keep_result(did_request_maximum, result_time, download_times): # In the normal case, we didn't ask for the maximum allowed amount. So we # should only allow ourselves to keep results that are between the min and # max allowed time + msg = "Keeping measurement time {:.2f}".format(result_time) if not did_request_maximum and \ result_time >= download_times['min'] and \ result_time < download_times['max']: + log.debug(msg) return True # If we did request the maximum amount, we should keep the result as long # as it took less than the maximum amount of time if did_request_maximum and \ result_time < download_times['max']: + log.debug(msg) return True # In all other cases, return false log.debug('Not keeping result time %f.%s', result_time,

1 0