March 2021 - tor-commits - lists.torproject.org

[status-site/main] Update categories, service titles and descriptions
by anarcat＠torproject.org 16 Mar '21

16 Mar '21

commit db32be6ea810a99bae31c4c3323d8e8d1930937c Author: Duncan <duncan(a)torproject.org> Date: Tue Mar 16 18:46:19 2021 +0000 Update categories, service titles and descriptions Closes: #3 --- config.yml | 63 ++++++++++++++++--------------- content/issues/2021-01-19-status-page.md | 18 --------- content/issues/2021-01-25-gmail-bounce.md | 2 +- content/issues/2021-01-28-dir-auth.md | 4 +- 4 files changed, 36 insertions(+), 51 deletions(-) diff --git a/config.yml b/config.yml index df39c7a..ddb2571 100644 --- a/config.yml +++ b/config.yml @@ -121,9 +121,12 @@ params: # For help, see the wiki: # https://github.com/cstate/cstate/wiki/Customization categories: - - name: Tor network - - name: TPA - description: internal Tor services + - name: Tor Network + description: Core services that are integral to the Tor Network. + - name: Tor Project websites + description: Official websites and services hosted and maintained by the Tor Project. + - name: Internal systems + description: Tools and services that help us collaborate with one another, managed by the Tor Project System Administration Team (TPA). closed: true - name: Uncategorized untitled: true @@ -134,33 +137,33 @@ params: # For help, see the wiki: # https://github.com/cstate/cstate/wiki/Customization systems: - - name: v3 services - category: Tor network - - name: v2 services - category: Tor network - - name: directory authorities - description: whatever it is that those things do - category: Tor network - - name: check.tpo - description: the check.torproject.org website - category: Tor network + - name: v3 Onion Services + description: Third generation onion services – running private services (e.g. websites) that are only accessible through the Tor network. + category: Tor Network + - name: v2 Onion Services + description: Second generation onion services – running private services (e.g. websites) that are only accessible through the Tor network. + category: Tor Network + - name: Directory Authorities + description: Special-purpose relays that maintain a list of currently-running relays and periodically publish a consensus together. + category: Tor Network + - name: torproject.org + description: Our official website and its associated portals (e.g. community.torproject.org) + category: Tor Project websites + - name: Tor Check + description: The check.torproject.org website, used to confirm you're connected to the Tor Network. + category: Tor Project websites - name: GitLab - description: gitlab.torproject.org - category: TPA + description: Our code-review, bug tracking, and collaboration platform of choice (gitlab.torproject.org) + category: Internal systems + - name: Gitolite + description: An access control layer on top of git, used to provide read/write access. + category: Internal systems - name: Email - description: general email delivery - category: TPA - - name: lists - description: mailing lists (mailman) - category: TPA - - name: gitolite - description: read/write access to git - category: TPA - - name: websites - description: the many websites under torproject.org - category: TPA - - name: other - category: Uncategorized + description: General email delivery from within the Tor Project. + category: Internal systems + - name: Mailing lists + description: Our public and private mailing lists (lists.torproject.org) powered by Mailman, the GNU Mailing List Manager. + category: Internal systems # What date format to use? # @@ -239,8 +242,8 @@ params: # Default: We continuously monitor the status of our services and if there are any interruptions, a note will be posted here. description: | We continuously monitor the status of our services and if there - are any interruptions, a note will be posted here. If you need to - do a modification on this page, [follow the + are any interruptions an update will be posted here. If you need to + modify this page, [follow the documentation](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/s… ([mirror](https://web.archive.org/web/2/https://gitlab.torproject.org/tpo/tpa…) # Tabs on homepage diff --git a/content/issues/2021-01-19-status-page.md b/content/issues/2021-01-19-status-page.md deleted file mode 100644 index 7fe22c4..0000000 --- a/content/issues/2021-01-19-status-page.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Status page deployment -date: 2021-01-15 18:29:00 -resolved: true -resolvedWhen: 2021-01-20 19:52:56 -# Possible severity levels: down, disrupted, notice -severity: notice -affected: - - other -section: issue ---- - -We're in the process of deploying [cstate](https://github.com/cstate/cstate) for a status page, this -page might need a few more updates to be actually relevant. Follow -[ticket 40138](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40138) in GitLab for more information. - -Update: the site is mostly operational, even if updates are a little -clunky. Considered fixed. diff --git a/content/issues/2021-01-25-gmail-bounce.md b/content/issues/2021-01-25-gmail-bounce.md index dba9281..19950c2 100644 --- a/content/issues/2021-01-25-gmail-bounce.md +++ b/content/issues/2021-01-25-gmail-bounce.md @@ -7,7 +7,7 @@ resolvedWhen: 2021-01-25 21:40:00 severity: notice affected: - Email - - lists + - Mailing lists section: issue --- diff --git a/content/issues/2021-01-28-dir-auth.md b/content/issues/2021-01-28-dir-auth.md index 1db8571..fe5b7d3 100644 --- a/content/issues/2021-01-28-dir-auth.md +++ b/content/issues/2021-01-28-dir-auth.md @@ -6,8 +6,8 @@ resolvedWhen: 2021-01-29 05:30:00 # Possible severity levels: down, disrupted, notice severity: disrupted affected: - - directory authorities - - v3 services + - Directory Authorities + - v3 Onion Services section: issue ---

1 0

[sbws/master] fix: CI: Update Python versions
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 193f5da1b0024aefb03edee547d770f0584761f3 Author: juga0 <juga(a)riseup.net> Date: Tue Mar 2 12:45:43 2021 +0000 fix: CI: Update Python versions Closes #40055. --- .gitlab-ci.yml | 55 +++++++++++++++++++++++++++++++++++-------------------- tox.ini | 4 ++-- 2 files changed, 37 insertions(+), 22 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 10dfbc2..098781f 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -5,9 +5,16 @@ # 0.4.6 stable by Jun 15, 2021 # 0.4.5 (LTS) EOL Feb 15, 2023 # 0.3.5 (LTS) EOL Feb 1, 2022 +# Python releases: +# 3.10 stable by 2021-10-04: https://www.python.org/dev/peps/pep-0619/ +# Python stable releases: https://www.python.org/downloads/ +# 3.9 EOL 2025-10 PEP 596 +# 3.8 EOL 2024-10 PEP 569, newest major release +# 3.7 EOL 2023-06-27 PEP 537, included in Debian buster +# 3.6 EOL 2021-12-23 PEP 494 variables: - BASE_IMAGE: python:3.7 + BASE_IMAGE: python:3.8 RELEASE: tor-nightly-master-buster # Without version, the default available in the Debian repository will be # installed. @@ -38,47 +45,55 @@ before_script: after_script: - tox -e stats -python36tor035: +python36: variables: BASE_IMAGE: python:3.6 + image: $BASE_IMAGE + script: + - tox -e py36 + - tox -e integration + +python37tor035: + variables: + BASE_IMAGE: python:3.7 RELEASE: tor-nightly-0.3.5.x-buster TOR: tor=0.3.5* image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python36tor045: +python37tor045: variables: - BASE_IMAGE: python:3.6 + BASE_IMAGE: python:3.7 RELEASE: tor-nightly-0.4.5.x-buster TOR: tor=0.4.5* image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python36tormaster: +python37tormaster: variables: - BASE_IMAGE: python:3.6 + BASE_IMAGE: python:3.7 RELEASE: tor-nightly-master-buster TOR: tor=0.4.6* image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python36torstable: +python37torstable: variables: - BASE_IMAGE: python:3.6 + BASE_IMAGE: python:3.7 RELEASE: buster TOR: tor image: $BASE_IMAGE script: - - tox -e py36 + - tox -e py37 - tox -e integration -python37: +python38: # This will overwrite the default before_script, so need to repeat the # commands before_script: @@ -100,24 +115,24 @@ python37: script: - tox -e inst - tox -e setup - - tox -e py37 + - tox -e py38 - tox -e integration - tox -e lint - tox -e doc -python38: +python39: variables: - BASE_IMAGE: python:3.8 + BASE_IMAGE: python:3.9 image: $BASE_IMAGE script: - - tox -e py38 + - tox -e py39 - tox -e integration -python39: +python310: variables: - BASE_IMAGE: python:3.9-rc-buster + BASE_IMAGE: python:3.10-rc-buster image: $BASE_IMAGE script: - - tox -e py39 + - tox -e py310 - tox -e integration allow_failure: true diff --git a/tox.ini b/tox.ini index acd31aa..bb66836 100644 --- a/tox.ini +++ b/tox.ini @@ -1,13 +1,13 @@ [tox] skip_missing_interpreters = True -envlist = py{36, 37, 38, 39}, inst, setup, integration, lint, stats, doc +envlist = py{36, 37, 38, 39, 310}, inst, setup, integration, lint, stats, doc [travis] python = 3.6: py36, inst, setup, unit, integration, lint, doc 3.7: py37, inst, setup, unit, integration, lint, doc 3.8: py38, inst, setup, unit, integration, lint, doc - 3.9-dev: py39, inst, setup, unit, integration, lint, doc + 3.9: py39, inst, setup, unit, integration, lint, doc nightly: pynightly, inst, setup, unit, integration, lint, doc ; [testenv]

1 0

[sbws/master] fix: CI: Update tor versions
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 842ed52f34992d416930a4beace9b2d8016af574 Author: juga0 <juga(a)riseup.net> Date: Tue Mar 2 12:40:52 2021 +0000 fix: CI: Update tor versions --- .gitlab-ci.yml | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 9b4fe40..10dfbc2 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,3 +1,11 @@ +# core/tor releases: +# https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/CoreTorRele… +# As of 2021.03.02 all dirauths are running with version greater or equal to +# 0.4.5. +# 0.4.6 stable by Jun 15, 2021 +# 0.4.5 (LTS) EOL Feb 15, 2023 +# 0.3.5 (LTS) EOL Feb 1, 2022 + variables: BASE_IMAGE: python:3.7 RELEASE: tor-nightly-master-buster @@ -40,21 +48,21 @@ python36tor035: - tox -e py36 - tox -e integration -python36tor041: +python36tor045: variables: BASE_IMAGE: python:3.6 - RELEASE: tor-nightly-0.4.1.x-buster - TOR: tor=0.4.1* + RELEASE: tor-nightly-0.4.5.x-buster + TOR: tor=0.4.5* image: $BASE_IMAGE script: - tox -e py36 - tox -e integration -python36tor042: +python36tormaster: variables: BASE_IMAGE: python:3.6 - RELEASE: tor-nightly-0.4.2.x-buster - TOR: tor=0.4.2* + RELEASE: tor-nightly-master-buster + TOR: tor=0.4.6* image: $BASE_IMAGE script: - tox -e py36

1 0

[sbws/master] fix: relalist: Use the consensus timestamp
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit aa6b79d343dfe43fa6bb447a9d9c37d41c37e589 Author: juga0 <juga(a)riseup.net> Date: Fri Mar 5 11:51:59 2021 +0000 fix: relalist: Use the consensus timestamp to the relay consensus timestamps list, so that it can be tested it was in a concrete consensus. --- sbws/lib/relaylist.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/sbws/lib/relaylist.py b/sbws/lib/relaylist.py index 4f33799..2a0bd8d 100644 --- a/sbws/lib/relaylist.py +++ b/sbws/lib/relaylist.py @@ -71,7 +71,11 @@ class Relay: self.relay_in_recent_consensus = timestamps.DateTimeSeq( [], MAX_RECENT_CONSENSUS_COUNT ) - self.update_relay_in_recent_consensus() + # Use the same timestamp as the consensus, so that it can be tested + # that the relay was in a consensus using this timestamp. + # Note that this doesn't change the number of consensus the relay was + # in. + self.update_relay_in_recent_consensus(timestamp) # The number of times that a relay is "prioritized" to be measured. # It is incremented in ``RelayPrioritizer.best_priority`` self.relay_recent_priority_list = timestamps.DateTimeSeq(

1 0

[sbws/master] fix: CI: Exit from integration script
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit a49cf7029bf5b3a6414ca89fd31a243bec319ef4 Author: juga0 <juga(a)riseup.net> Date: Wed Mar 3 12:38:18 2021 +0000 fix: CI: Exit from integration script when any of the commands fail. --- tests/integration/run.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/integration/run.sh b/tests/integration/run.sh index e870b50..70ee5f2 100755 --- a/tests/integration/run.sh +++ b/tests/integration/run.sh @@ -1,5 +1,8 @@ #!/bin/bash -set -x +# Instead of exiting inmmediately when any of this commands fail, +# the scanner, generate and coverage lines could continue and store there was +# an error on that command. It's just simpler with `-e`. +set -ex tests/integration/start_chutney.sh python3 scripts/tools/sbws-http-server.py --port 28888 &>/dev/null &

1 0

[sbws/master] fix: CI: Temporal workaround for #40072
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 4e7ad5d78d58c9068ece2239cacc2ddd6d13d7b1 Author: juga0 <juga(a)riseup.net> Date: Fri Mar 5 13:45:39 2021 +0000 fix: CI: Temporal workaround for #40072 --- tests/integration/start_chutney.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/integration/start_chutney.sh b/tests/integration/start_chutney.sh index 04f9641..4dbcb68 100755 --- a/tests/integration/start_chutney.sh +++ b/tests/integration/start_chutney.sh @@ -23,4 +23,6 @@ git pull ./chutney status networks/bwscanner ./chutney wait_for_bootstrap networks/bwscanner +# temporal workaround for https://gitlab.torproject.org/tpo/core/chutney/-/issues/40016 +sleep 60 cd $CURRENT_DIR

1 0

[sbws/master] minor: Change log warning to info or debug
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 290030aedd6d6ade7e53c22dacab149e3e5b1b8f Author: juga0 <juga(a)riseup.net> Date: Tue Mar 2 17:33:12 2021 +0000 minor: Change log warning to info or debug when it contains sensitive information. --- sbws/core/scanner.py | 2 +- sbws/util/stem.py | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index c7dde75..e446391 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -388,7 +388,7 @@ def measure_relay(args, conf, destinations, cb, rl, relay): circ_fps, nicknames, exit_policy = r circ_id, reason = cb.build_circuit(circ_fps) if not circ_id: - log.warning( + log.info( "Exit %s (%s) that can't exit all ips, failed to create " " circuit as entry: %s (%s).", relay.fingerprint, relay.nickname, circ_fps, nicknames) diff --git a/sbws/util/stem.py b/sbws/util/stem.py index 9fea31f..38d0570 100644 --- a/sbws/util/stem.py +++ b/sbws/util/stem.py @@ -29,12 +29,12 @@ def attach_stream_to_circuit_listener(controller, circ_id): circuit_str(controller, circ_id)) try: controller.attach_stream(st.id, circ_id) - except (UnsatisfiableRequest, InvalidRequest) as e: - log.warning('Couldn\'t attach stream to circ %s: %s', - circ_id, e) - except OperationFailed as e: - log.exception("Error attaching stream %s to circ %s: %s", - st.id, circ_id, e) + # So far we never saw this error. + except ( + UnsatisfiableRequest, InvalidRequest, OperationFailed + ) as e: + log.debug('Error attaching stream %s to circ %s: %s', + st.id, circ_id, e) else: pass return closure_stream_event_listener

1 0

[sbws/master] major: Change default log level to info
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit fa70b0a1edd01062aa82724ff4371116d00225fa Author: juga0 <juga(a)riseup.net> Date: Wed Mar 3 07:57:15 2021 +0000 major: Change default log level to info also change formatting to show thread. --- sbws/config.default.ini | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sbws/config.default.ini b/sbws/config.default.ini index c47e4fd..4080189 100644 --- a/sbws/config.default.ini +++ b/sbws/config.default.ini @@ -119,13 +119,13 @@ to_file_max_bytes = 10485760 to_file_num_backups = 50 # Level to log at. Debug, info, warning, error, critical. # `level` must be set to the lower of all the handler levels. -level = debug -to_file_level = debug +level = info +to_file_level = info to_stdout_level = info to_syslog_level = info # Format string to use when logging -format = %(asctime)s %(module)s[%(process)s]: <%(levelname)s> %(message)s +format = %(asctime)s %(module)s[%(process)s]: <%(levelname)s> (%(threadName)s) %(message)s to_stdout_format = ${format} to_syslog_format = %(module)s[%(process)s]: <%(levelname)s> %(message)s # verbose formatter useful for debugging -to_file_format = %(asctime)s %(levelname)s %(threadName)s %(filename)s:%(lineno)s - %(funcName)s - %(message)s +to_file_format = %(asctime)s %(levelname)s (%(threadName)s) %(filename)s:%(lineno)s - %(funcName)s - %(message)s

1 0

[sbws/master] fix: scanner: Log times kept
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit dc60a3c0215e28d8d32e9d0ffd4aa8872a8f70c3 Author: juga0 <juga(a)riseup.net> Date: Wed Mar 3 09:42:38 2021 +0000 fix: scanner: Log times kept not only the times that are not kept. Closes #40060 --- sbws/core/scanner.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index e446391..2fba10f 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -466,14 +466,17 @@ def _should_keep_result(did_request_maximum, result_time, download_times): # In the normal case, we didn't ask for the maximum allowed amount. So we # should only allow ourselves to keep results that are between the min and # max allowed time + msg = "Keeping measurement time {:.2f}".format(result_time) if not did_request_maximum and \ result_time >= download_times['min'] and \ result_time < download_times['max']: + log.debug(msg) return True # If we did request the maximum amount, we should keep the result as long # as it took less than the maximum amount of time if did_request_maximum and \ result_time < download_times['max']: + log.debug(msg) return True # In all other cases, return false log.debug('Not keeping result time %f.%s', result_time,

1 0

[sbws/master] minor: Change info logs to debug or warning
by juga＠torproject.org 16 Mar '21

16 Mar '21

commit 3b1f7b79bfab9262567e487f0429635781f6fcab Author: juga0 <juga(a)riseup.net> Date: Tue Mar 2 17:21:15 2021 +0000 minor: Change info logs to debug or warning when they contain sensitive information, eg. Web server or are too verbose for the debug level. Also add log to indicate when the main loop is actually started. --- sbws/core/scanner.py | 3 ++- sbws/lib/destination.py | 34 ++++++++++++++++++++-------------- sbws/lib/resultdump.py | 9 ++++++--- 3 files changed, 28 insertions(+), 18 deletions(-) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index f9ab1ce..c7dde75 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -377,7 +377,7 @@ def measure_relay(args, conf, destinations, cb, rl, relay): # always fail when there's only one Web server. if not is_usable and \ relay.is_exit_not_bad_allowing_port(dest.port): - log.info( + log.debug( "Exit %s (%s) that can't exit all ips, with exit policy %s, failed" " to connect to %s via circuit %s (%s). Reason: %s. Trying again " "with it as entry.", relay.fingerprint, relay.nickname, @@ -575,6 +575,7 @@ def main_loop(args, conf, controller, relay_list, circuit_builder, result_dump, measured. """ + log.info("Started the main loop to measure the relays.") hbeat = Heartbeat(conf.getpath('paths', 'state_fname')) # Set the time to wait for a thread to finish as the half of an HTTP diff --git a/sbws/lib/destination.py b/sbws/lib/destination.py index 5330dbf..f007a7a 100644 --- a/sbws/lib/destination.py +++ b/sbws/lib/destination.py @@ -207,12 +207,12 @@ class Destination: self._delta_seconds_retry *= factor or self._factor if self._delta_seconds_retry > self._max_seconds_between_retries: self._delta_seconds_retry = self._max_seconds_between_retries - log.info("Incremented the time to try destination %s past the " + log.debug("Incremented the time to try destination %s past the " "limit, capping it at %s hours.", self.url, self._delta_seconds_retry / 60 / 60) else: - log.info("Incremented the time to try destination %s to %s hours.", - self.url, self._delta_seconds_retry / 60 / 60) + log.debug("Incremented the time to try destination %s to %s " + "hours.", self.url, self._delta_seconds_retry / 60 / 60) def _get_last_try_in_seconds_ago(self): """ @@ -250,23 +250,29 @@ class Destination: # Failed the last X consecutive times if self._are_last_attempts_failures(): - # The log here will appear in all the the queued - # relays and threads. - log.warning("The last %s times the destination %s failed. " + # The log here will appear in all the the queued relays and + # threads. + log.debug("The last %s times the destination %s failed. " + "It last ran %s seconds ago. " + "Disabled for %s seconds.", + self._max_num_failures, self.url, + self._get_last_try_in_seconds_ago(), + self._delta_seconds_retry) + log.warning("The last %s times a destination failed. " "It last ran %s seconds ago. " - "Disabled for %s seconds.", - self._max_num_failures, self.url, + "Disabled for %s seconds." + "Please, add more destinations or increment the " + "number of maximum number of consecutive failures " + "in the configuration.", + self._max_num_failures, self._get_last_try_in_seconds_ago(), self._delta_seconds_retry) - log.warning("Please, add more destinations or increment the " - "number of maximum number of consecutive failures " - "in the configuration.") # It was not used for a while and the last time it was used # was long ago, then try again if self._is_last_try_old_enough(): - log.info("The destination %s was not tried for %s seconds, " - "it is going to by tried again.", self.url, - self._get_last_try_in_seconds_ago()) + log.debug("The destination %s was not tried for %s seconds, " + "it is going to by tried again.", self.url, + self._get_last_try_in_seconds_ago()) # Set the next time to retry higher, in case this attempt fails self._increment_time_to_retry() return True diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py index b287242..cc61bcc 100644 --- a/sbws/lib/resultdump.py +++ b/sbws/lib/resultdump.py @@ -799,14 +799,17 @@ class ResultDump: msg += ". As exit." if as_exit else ". As entry." # When the error is that there are not more functional destinations. if result.type == "error-destination": - log.info("Shutting down because there are not functional " - "destinations.") + log.warning("Shutting down because there are not functional " + "destinations.") # NOTE: Because this is executed in a thread, stop_threads can not # be call from here, it has to be call from the main thread. # Instead set the singleton end event, that will call stop_threads # from the main process. settings.end_event.set() - log.info(msg) + # log every success/error measurement at debug level, to don't log the + # the Web server at info level and because there's already a + # heartbeat msg to indicate progress. + log.debug(msg) def enter(self): """Main loop for the ResultDump thread.

1 0