tor-commits February 2021

tor-commits@lists.torproject.org

19 participants
1772 discussions

[sbws/master] fix: resultdump: Log if relay was measured as exit
by juga＠torproject.org 19 Feb '21

19 Feb '21

commit beaf6de889bc75d53a6b0b90d12ab85aa0db56a0 Author: juga0 <juga(a)riseup.net> Date: Wed Feb 3 13:15:45 2021 +0000 fix: resultdump: Log if relay was measured as exit or entry. Closes #40048 --- sbws/lib/resultdump.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py index 542837e..6673e9a 100644 --- a/sbws/lib/resultdump.py +++ b/sbws/lib/resultdump.py @@ -792,6 +792,10 @@ class ResultDump: "destination {}: {}".format( result.fingerprint, result.nickname, result.circ, result.dest_url, result.msg) + # The result doesn't store the exit policies, so it can't be logged + # whether it was an exit. + as_exit = result.circ[1] == result.fingerprint + msg += ". As exit." if as_exit else ". As entry." # When the error is that there are not more functional destinations. if result.type == "error-destination": log.info("Shutting down because there are not functional "

1 0

[sbws/master] fix: doc: Add bwauthealth tool
by juga＠torproject.org 19 Feb '21

19 Feb '21

commit 3ddc9f1e4648bcab2ed59059e6f6bdf29f3ce982 Author: juga0 <juga(a)riseup.net> Date: Sun Sep 27 10:40:58 2020 +0000 fix: doc: Add bwauthealth tool --- docs/source/monitoring_bandwidth.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/source/monitoring_bandwidth.rst b/docs/source/monitoring_bandwidth.rst index dedf90d..09326c4 100644 --- a/docs/source/monitoring_bandwidth.rst +++ b/docs/source/monitoring_bandwidth.rst @@ -24,6 +24,13 @@ It should be approximately equal for all bwauths. https://metrics.torproject.org/totalcw.html + +Not measured relays and descriptors and consensus updates +--------------------------------------------------------- + +Run the tool https://gitlab.torproject.org/juga/bwauthealth. + + Total bandwidth ---------------

1 0

[sbws/master] fix: doc: Add the bwauths timeline wiki
by juga＠torproject.org 19 Feb '21

19 Feb '21

commit 81d2bea0008a4ef2fc6ded2ee0bd56a6aa7cbdd7 Author: juga0 <juga(a)riseup.net> Date: Sun Sep 27 10:43:53 2020 +0000 fix: doc: Add the bwauths timeline wiki Closes #40013. --- docs/source/monitoring_bandwidth.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/source/monitoring_bandwidth.rst b/docs/source/monitoring_bandwidth.rst index 09326c4..d9ebdf2 100644 --- a/docs/source/monitoring_bandwidth.rst +++ b/docs/source/monitoring_bandwidth.rst @@ -1,6 +1,15 @@ Monitoring bandwidth changes in the Tor Network ================================================ +Bandwidth authorities timeline +------------------------------ + +Events that can affect the data generated by the bwauths: + +https://gitlab.torproject.org/tpo/network-health/sbws/-/wikis/bandwidth%20authorities%20timeline + +This page might be moved to a different location. + Bwauths number of measured relays ---------------------------------

1 0

[sbws/master] fix: doc: Add consensus health page
by juga＠torproject.org 19 Feb '21

19 Feb '21

commit 6e6a8f3ba534cbd93b830fe3ffd5ce40abe8e77d Author: juga0 <juga(a)riseup.net> Date: Sun Sep 27 10:32:12 2020 +0000 fix: doc: Add consensus health page about bwauths measured relays. --- docs/source/monitoring_bandwidth.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/source/monitoring_bandwidth.rst b/docs/source/monitoring_bandwidth.rst index 54d1219..dedf90d 100644 --- a/docs/source/monitoring_bandwidth.rst +++ b/docs/source/monitoring_bandwidth.rst @@ -1,6 +1,18 @@ Monitoring bandwidth changes in the Tor Network ================================================ +Bwauths number of measured relays +--------------------------------- + +It should be approximately equal for all bwauths. + +.. image:: images/20200927_consensushealth_bwauths.png + :alt: bwauths measured relays + +https://consensus-health.torproject.org/graphs.html#votedaboutgraphs + +http://tgnv2pssfumdedyw.onion/graphs.html#votedaboutgraphs + Total consensus weights across bandwidth authorities ----------------------------------------------------

1 0

[sbws/master] Merge branch 'maint-1.1' into master
by juga＠torproject.org 19 Feb '21

19 Feb '21

commit c8d1cc4f5d8b68688fd7dca48491531a8a9d4c30 Merge: 2e38dcb 81d2bea Author: Georg Koppen <gk(a)torproject.org> Date: Mon Oct 5 07:12:08 2020 +0000 Merge branch 'maint-1.1' into master docs/source/monitoring_bandwidth.rst | 49 ++++++++++++++++++++++++++++-------- 1 file changed, 39 insertions(+), 10 deletions(-)

1 0

[sbws/master] fix: doc: Move consensus weight to top
by juga＠torproject.org 19 Feb '21

19 Feb '21

commit 75af5c3575670121cb933135ab79ce49e7eb0f7b Author: juga0 <juga(a)riseup.net> Date: Sun Sep 27 10:08:15 2020 +0000 fix: doc: Move consensus weight to top and explain what to check. --- docs/source/monitoring_bandwidth.rst | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/source/monitoring_bandwidth.rst b/docs/source/monitoring_bandwidth.rst index 888d544..54d1219 100644 --- a/docs/source/monitoring_bandwidth.rst +++ b/docs/source/monitoring_bandwidth.rst @@ -1,6 +1,17 @@ Monitoring bandwidth changes in the Tor Network ================================================ +Total consensus weights across bandwidth authorities +---------------------------------------------------- + +It should be approximately equal for all bwauths. + +.. image:: images/totalcw.png + :alt: total consensus weight + + +https://metrics.torproject.org/totalcw.html + Total bandwidth --------------- @@ -23,13 +34,3 @@ Should not increase. https://metrics.torproject.org/torperf.html - - -Total consensus weights across bandwidth authorities ----------------------------------------------------- - -.. image:: images/totalcw.png - :alt: total consensus weight - - -https://metrics.torproject.org/totalcw.html

1 0

[sbws/master] fix: v3bwfile: Tor version added in bandwidth v1.4.0
by juga＠torproject.org 19 Feb '21

19 Feb '21

commit 50821110c908e5da9f1aab3b14883e39bb226614 Author: juga0 <juga(a)riseup.net> Date: Mon Dec 7 11:28:42 2020 +0000 fix: v3bwfile: Tor version added in bandwidth v1.4.0 since, by mistake, the bandwidth file version here was never updated to v1.5.0. This patch only changes the constants names, but logic remains the same. Related to torspec#35. --- sbws/lib/v3bwfile.py | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py index 45563b6..553265b 100644 --- a/sbws/lib/v3bwfile.py +++ b/sbws/lib/v3bwfile.py @@ -125,13 +125,9 @@ HEADER_KEYS_V1_4 = [ 'time_to_report_half_network', ] + HEADER_RECENT_MEASUREMENTS_EXCLUDED_KEYS -# KeyValues added in the Bandwidth File v1.5.0 -# XXX: Change SPEC_VERSION when all the v1.5.0 keys are added, before a new -# sbws release. # Tor version will be obtained from the state file, so it won't be pass as an # argument, but will be self-initialized. -HEADER_KEYS_V1_5_TO_INIT = ['tor_version'] -HEADER_KEYS_V1_5 = HEADER_KEYS_V1_5_TO_INIT +HEADER_KEYS_V1_4_TO_INIT = ['tor_version'] # KeyValues that are initialized from arguments, not self-initialized. HEADER_INIT_KEYS = ( @@ -139,7 +135,7 @@ HEADER_INIT_KEYS = ( + HEADER_KEYS_V1_3 + HEADER_KEYS_V1_2 + HEADER_KEYS_V1_4 - + HEADER_KEYS_V1_5_TO_INIT + + HEADER_KEYS_V1_4_TO_INIT ) HEADER_INT_KEYS = HEADER_KEYS_V1_2 + HEADER_KEYS_V1_4 @@ -150,7 +146,7 @@ HEADER_UNORDERED_KEYS = ( + HEADER_KEYS_V1_3 + HEADER_KEYS_V1_2 + HEADER_KEYS_V1_4 - + HEADER_KEYS_V1_5 + + HEADER_KEYS_V1_4_TO_INIT ) # List of all the KeyValues currently being used to generate the file HEADER_ALL_KEYS = HEADER_KEYS_V1_1_ORDERED + HEADER_UNORDERED_KEYS

1 0

[sbws/master] Merge remote-tracking branch 'gitlab/merge-requests/34' into maint-1.1
by juga＠torproject.org 19 Feb '21

19 Feb '21

commit 8fdd85b73315a5dd72de68c8a876ca9fe823fe7d Merge: 81d2bea f014084 Author: Georg Koppen <gk(a)torproject.org> Date: Mon Nov 30 10:02:29 2020 +0000 Merge remote-tracking branch 'gitlab/merge-requests/34' into maint-1.1 sbws/core/scanner.py | 3 ++- tests/integration/test_files.py | 7 +++++-- tests/unit/lib/test_heartbeat.py | 2 ++ tests/unit/lib/test_v3bwfile.py | 2 ++ 4 files changed, 11 insertions(+), 3 deletions(-)

1 0

[sbws/master] chg: circuitbuilder: Simplify building circuit
by juga＠torproject.org 19 Feb '21

19 Feb '21

commit b3ab8ebf9209f4e396f42e254afcca58fefc1e58 Author: juga0 <juga(a)riseup.net> Date: Tue Dec 8 15:43:23 2020 +0000 chg: circuitbuilder: Simplify building circuit Since sbws is only building 2 hop paths, there is no need to add random relays to the path, or convert back and forth between fingerprint and ``Relay`` objects. This will eliminate the circuit errors: - Tor seems to no longer think XXX is a relay - Can not build a circuit, no path. - Can not build a circuit with the current relays. If a relay is not longer running when attempting to build the circuit, it will probably fail with one of the other circuit errors: TIMEOUT, DESTROYED or CHANNEL_CLOSED. Closes: #40027 --- sbws/lib/circuitbuilder.py | 62 ++++++---------------------------------------- 1 file changed, 8 insertions(+), 54 deletions(-) diff --git a/sbws/lib/circuitbuilder.py b/sbws/lib/circuitbuilder.py index 45b807f..fb23737 100644 --- a/sbws/lib/circuitbuilder.py +++ b/sbws/lib/circuitbuilder.py @@ -91,62 +91,16 @@ class CircuitBuilder: # build the circuit, the relays are not just choosen as random as this class # does. class GapsCircuitBuilder(CircuitBuilder): - ''' The build_circuit member function takes a list. Falsey values in the - list will be replaced with relays chosen uniformally at random; Truthy - values will be assumed to be relays. ''' + """Same as ``CircuitBuilder`` but implements build_circuit.""" def __init__(self, *a, **kw): super().__init__(*a, **kw) - def _normalize_path(self, path): - ''' Change fingerprints/nicks to relay descriptor and change Falsey - values to None. Return the new path, or None if error ''' - new_path = [] - for fp in path: - if not fp: - new_path.append(None) - continue - relay = Relay(fp, self.controller) - if not relay.fingerprint: - log.debug('Tor seems to no longer think %s is a relay', fp) - return None - new_path.append(relay) - return new_path - - def _random_sample_relays(self, number, blacklist): - ''' Get <number> random relays from self.relays that are not in the - blacklist. Return None if it cannot be done because too many are - blacklisted. Otherwise return a list of relays. ''' - all_fps = [r.fingerprint for r in self.relays] - black_fps = [r.fingerprint for r in blacklist] - if len(black_fps) + number > len(all_fps): - return None - chosen_fps = [] - while len(chosen_fps) < number: - choice = self.rng.choice(all_fps) - if choice in black_fps: - continue - chosen_fps.append(choice) - black_fps.append(choice) - return [Relay(fp, self.controller) for fp in chosen_fps] - def build_circuit(self, path): - ''' <path> is a list of relays and Falsey values. Relays can be - specified by fingerprint or nickname, and fingerprint is highly - recommended. Falsey values (like None) will be replaced with relays - chosen uniformally at random. A relay will not be in a circuit twice. - ''' - if not valid_circuit_length(path): - return None, "Can not build a circuit, invalid path." - path = self._normalize_path(path) - if path is None: - return None, "Can not build a circuit, no path." - num_missing = len(['foo' for r in path if not r]) - insert_relays = self._random_sample_relays( - num_missing, [r for r in path if r is not None]) - if insert_relays is None: - path = ','.join([r.nickname if r else str(None) for r in path]) - return None, "Can not build a circuit with the current relays." - assert len(insert_relays) == num_missing - path = [r.fingerprint if r else insert_relays.pop().fingerprint - for r in path] + """Return parent class build circuit method. + + Since sbws is only building 2 hop paths, there is no need to add random + relays to the path, or convert back and forth between fingerprint and + ``Relay`` objects. + + """ return self._build_circuit_impl(path)

1 0

[sbws/master] chg: scanner: Stop storing recent_measurement_attempt
by juga＠torproject.org 19 Feb '21

19 Feb '21

commit f014084108eafd6478fbe5e2e0afc6855435c87f Author: juga0 <juga(a)riseup.net> Date: Mon Nov 23 20:02:26 2020 +0000 chg: scanner: Stop storing recent_measurement_attempt because it stores a timestamp for each attempt, which makes state.dat grow thousand of lines (json). Closes #40023, #40020 --- sbws/core/scanner.py | 3 ++- tests/integration/test_files.py | 7 +++++-- tests/unit/lib/test_heartbeat.py | 2 ++ tests/unit/lib/test_v3bwfile.py | 2 ++ 4 files changed, 11 insertions(+), 3 deletions(-) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index e5d5e90..8b6a6a4 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -523,7 +523,8 @@ def main_loop(args, conf, controller, relay_list, circuit_builder, result_dump, # Don't start measuring a relay if sbws is stopping. if settings.end_event.is_set(): break - relay_list.increment_recent_measurement_attempt() + # 40023, disable to decrease state.dat json lines + # relay_list.increment_recent_measurement_attempt() target.increment_relay_recent_measurement_attempt() num_relays += 1 # callback and callback_err must be non-blocking diff --git a/tests/integration/test_files.py b/tests/integration/test_files.py index aa7973d..c44340c 100644 --- a/tests/integration/test_files.py +++ b/tests/integration/test_files.py @@ -32,7 +32,9 @@ def test_state(conf): assert 1 == state.count("recent_consensus") assert 1 == state.count("recent_priority_list") assert 15 == state.count("recent_priority_relay") - assert 15 == state.count("recent_measurement_attempt") + # Because of 40023, it's 0. Change to 15 if we store it again at some point + # assert 15 == state.count("recent_measurement_attempt") + assert not state.count("recent_measurement_attempt") def test_v3bwfile(conf): @@ -42,7 +44,8 @@ def test_v3bwfile(conf): assert "1" == bwfile.header.recent_consensus_count assert "1" == bwfile.header.recent_priority_list_count assert "15" == bwfile.header.recent_priority_relay_count - assert "15" == bwfile.header.recent_measurement_attempt_count + # Because of 40023, there is not this header + # assert "15" == bwfile.header.recent_measurement_attempt_count for bwline in bwfile.bw_lines: assert 1 == bwline.relay_in_recent_consensus_count assert 1 == bwline.relay_recent_priority_list_count diff --git a/tests/unit/lib/test_heartbeat.py b/tests/unit/lib/test_heartbeat.py index b2ad634..8a2b8ea 100644 --- a/tests/unit/lib/test_heartbeat.py +++ b/tests/unit/lib/test_heartbeat.py @@ -1,10 +1,12 @@ """Unit tests for heartbeat""" import logging +import pytest from sbws.lib import heartbeat from sbws.util.state import State +(a)pytest.mark.skip(reason="increment_recent_measurement_attempt() disabled") def test_total_measured_percent(conf, caplog): state = State(conf["paths"]["state_fname"]) state["recent_priority_list"] = [1, 2, 3] diff --git a/tests/unit/lib/test_v3bwfile.py b/tests/unit/lib/test_v3bwfile.py index 07f59d7..d1448d1 100644 --- a/tests/unit/lib/test_v3bwfile.py +++ b/tests/unit/lib/test_v3bwfile.py @@ -4,6 +4,7 @@ import json import logging import math import os.path +import pytest from unittest import mock from sbws import __version__ as version @@ -588,6 +589,7 @@ def test_recent_consensus_count(root_data_path, datadir): assert "1" == header.recent_consensus_count +(a)pytest.mark.skip(reason="increment_recent_measurement_attempt() disabled") def test_recent_measurement_attempt_count(root_data_path, datadir): state_fpath = os.path.join(root_data_path, '.sbws/state.dat') assert 15 == V3BWHeader.recent_measurement_attempt_count_from_file(

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits February 2021