February 2019 - tor-commits - lists.torproject.org

[sbws/master] relayprioritizer: comment debug log, too many lines
by juga＠torproject.org 15 Feb '19

15 Feb '19

commit 7f743a8c76ca1e2514b6fe4df7517add77daf449 Author: juga0 <juga(a)riseup.net> Date: Tue Dec 18 09:10:32 2018 +0000 relayprioritizer: comment debug log, too many lines After several iterations, the number of relays that are prioritized is high, this log prints so many lines. --- sbws/lib/relayprioritizer.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/sbws/lib/relayprioritizer.py b/sbws/lib/relayprioritizer.py index 9c065a7..ee79a19 100644 --- a/sbws/lib/relayprioritizer.py +++ b/sbws/lib/relayprioritizer.py @@ -81,11 +81,11 @@ class RelayPrioritizer: # is no longer valid freshness = result.time - oldest_allowed if isinstance(result, ResultError) \ - and prioritize_result_error is True: - log.debug('Cutting freshness for a %s result by %d%% for ' - '%s', result.type.value, - result.freshness_reduction_factor * 100, - relay.nickname) + and prioritize_result_error is True: + # log.debug('Cutting freshness for a %s result by %d%% for' + # ' %s', result.type.value, + # result.freshness_reduction_factor * 100, + # relay.nickname) # result.freshness_reduction_factor are hard-coded values # on how much prioritize measurements that failed # depending on the type of error.

1 0

[sbws/master] relayprioritizer: add comment about future refactor
by juga＠torproject.org 15 Feb '19

15 Feb '19

commit 603f83adcaff021c8c5ba557a5e61d09c19ca79d Author: juga0 <juga(a)riseup.net> Date: Tue Dec 18 09:16:40 2018 +0000 relayprioritizer: add comment about future refactor --- sbws/lib/relayprioritizer.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbws/lib/relayprioritizer.py b/sbws/lib/relayprioritizer.py index 13ed491..e920bd1 100644 --- a/sbws/lib/relayprioritizer.py +++ b/sbws/lib/relayprioritizer.py @@ -97,6 +97,7 @@ class RelayPrioritizer: # or create constants. freshness *= max(1.0-result.freshness_reduction_factor, 0) priority += freshness + # In a future refactor, do not create a new attribute relay.priority = priority # Sort the relays by their priority, with the smallest (best) priority # relays at the front @@ -114,5 +115,7 @@ class RelayPrioritizer: for relay in relays[0:upper_limit]: log.debug('Returning next relay %s with priority %f', relay.nickname, relay.priority) + # In a future refactor, a new attribute should not be created, + # then no need to remove it. del(relay.priority) yield relay

1 0

[sbws/master] relayprioritizer: add comment about reading resultdump
by juga＠torproject.org 15 Feb '19

15 Feb '19

commit efe8245f333f11cf4e98eec6504d9e0478fe7656 Author: juga0 <juga(a)riseup.net> Date: Tue Dec 18 09:14:17 2018 +0000 relayprioritizer: add comment about reading resultdump --- sbws/lib/relayprioritizer.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbws/lib/relayprioritizer.py b/sbws/lib/relayprioritizer.py index ee79a19..13ed491 100644 --- a/sbws/lib/relayprioritizer.py +++ b/sbws/lib/relayprioritizer.py @@ -67,6 +67,10 @@ class RelayPrioritizer: relays = set(copy.deepcopy(self.relay_list.relays)) if not self.measure_authorities: relays = relays.difference(set(self.relay_list.authorities)) + # Since there will be new measurements every time this method is called + # again, update the list of results. + # In a future refactor with other data structure there should not be + # needed. rd = self.result_dump for relay in relays: results = rd.results_for_relay(relay)

1 0

[sbws/master] Merge branch 'bug28868'
by juga＠torproject.org 15 Feb '19

15 Feb '19

commit e23a25abced0b022f452249cf9c3fea97b09f9ed Merge: f173396 40a94b4 Author: juga0 <juga(a)riseup.net> Date: Fri Feb 15 17:50:50 2019 +0000 Merge branch 'bug28868' Solved conflicts: sbws/lib/relayprioritizer.py by applying theirs. docs/source/specification.rst | 51 +++-------------------- sbws/lib/relayprioritizer.py | 95 +++++++++++++++++++++++++++---------------- 2 files changed, 66 insertions(+), 80 deletions(-)

1 0

[sbws/master] docs: update documentation about relays' prioritization
by juga＠torproject.org 15 Feb '19

15 Feb '19

commit 40a94b4a672654b8413c227f9e81e7ec8c7eaf23 Author: juga0 <juga(a)riseup.net> Date: Tue Jan 22 10:05:36 2019 +0000 docs: update documentation about relays' prioritization By using the method docstring, we don't need to keep updated documentation about the same functionallity. Also, the documentation in the method is simpler. I found this explanation complex, long and confusing. This is the most complex part of the code. --- docs/source/specification.rst | 51 +++++-------------------------------------- 1 file changed, 5 insertions(+), 46 deletions(-) diff --git a/docs/source/specification.rst b/docs/source/specification.rst index f35f517..698842d 100644 --- a/docs/source/specification.rst +++ b/docs/source/specification.rst @@ -94,52 +94,11 @@ How it all works We now describe various core parts of sbws. -Simple relay prioritization -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This may be the most complex part of sbws. - -Sbws makes an effort to prioritize measurements of relays that don't have many -recent results. For example: relays that just joined the Tor network or relays -that haven't been online in the past few days. This goal is achieved using a -min-priority queue and the concept of *freshness*. - -Freshness is defined as the amount of time between when the measurement was -made and the time sbws will stop considering it valid. Thus, a measurement made -more recently will have more time until it is no longer valid (higher -freshness) and a measurement made a long time ago will have very little time -until it is no longer valid (lower freshness). - -Over time, sbws will make many measurements for a given relay. The sum of these -measurements' freshnesses is the relay's priority. As we are prioritizing like -a min-priority queue, a higher sum of freshnesses means *worse* priority while -a lower sum of freshnesses means *better* priority. - - Example: AlphaRelay33 joined the network yesterday and sbws has measured it - once so far. BetaRelay87 has been in the network for years and has been - getting measured regularly approximately once a day. BetaRelay87 has five - measurements that are still valid, with freshnesses 10, 100, 500, 1000, and - 1500. AlphaRelay33's one measurement has freshness 3000. Because the sum of - BetaRelay87's 5 measurements is greater than AlphaRelay33's one measurement, - AlphaRelay33 has *better* priority and will be measured next before - BetaRelay87. - - Example: AlphaRelay33 is still a brand new relay with its one measurement - in the last day with freshness 3000. CharlieRelay9 has been in the network - for a long time, but had technical issues last week and hasn't been online in - many days. When CharlieRelay9's operator finally gets him back online, he - still has one valid measurement with freshness 10. Because AlphaRelay33's - measurement is fresher, CharlieRelay9 has *better* priority and will get - measured first. - -Sometimes measurements fail. Hopefully they fail because of transient issues, -and with that hope in mind, it would be nice if a relay with a failed -measurement didn't have to wait a long time to have another chance at a -successful measurement. For this reason, when summing the freshnesses of -results for a given relay, sbws will artificially *reduce* the freshness for -measurements that were not successful. This makes the sum of freshnesses lower -for that relay, and therefore the priority *better* so it can be measured again -sooner. +In which order are relays measured? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. automethod:: sbws.lib.relayprioritizer.RelayPrioritizer.best_priority + :noindex: Simple result storage ~~~~~~~~~~~~~~~~~~~~~

1 0

[sbws/master] circuitbuilder: Stop building the circuit 3 times
by juga＠torproject.org 15 Feb '19

15 Feb '19

commit 4642c1093a3ed73ddea2e71f26458591967a25c2 Author: juga0 <juga(a)riseup.net> Date: Wed Feb 6 16:22:26 2019 +0000 circuitbuilder: Stop building the circuit 3 times It is observed that when the circuit can not be built the first time, consecutive attemps will also fail. The relay will be measured anyway in the next iteration. This will also speed up the scanner. There is no need to remove the 3 attemps in Destination. _perform_usability_test, since the method will be removed in #28897. Fixes bug #29295. Bugfix v0.1.0. --- sbws/lib/circuitbuilder.py | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/sbws/lib/circuitbuilder.py b/sbws/lib/circuitbuilder.py index 396f238..5eb8a02 100644 --- a/sbws/lib/circuitbuilder.py +++ b/sbws/lib/circuitbuilder.py @@ -74,19 +74,13 @@ class CircuitBuilder: timeout = self.circuit_timeout fp_path = '[' + ' -> '.join([p for p in path]) + ']' log.debug('Building %s', fp_path) - error = None - for _ in range(0, 3): - try: - circ_id = c.new_circuit( - path, await_build=True, timeout=timeout) - except (InvalidRequest, CircuitExtensionFailed, - ProtocolError, Timeout) as e: - log.debug(e) - error = str(e) - continue - else: - return circ_id, None - return None, error + try: + circ_id = c.new_circuit( + path, await_build=True, timeout=timeout) + except (InvalidRequest, CircuitExtensionFailed, + ProtocolError, Timeout) as e: + return None, str(e) + return circ_id, None def __del__(self): c = self.controller

1 0

[sbws/master] circuitbuilder: circuit path is not a number in the code
by juga＠torproject.org 15 Feb '19

15 Feb '19

commit 851401be4b84d2b7efe4ff26d3ddc7e61d5f86cb Author: juga0 <juga(a)riseup.net> Date: Wed Feb 6 17:11:20 2019 +0000 circuitbuilder: circuit path is not a number in the code There is no any case in our code where path is an integer. Remove asserts, check only the lenght of the circuit and check it in a more logical way. --- sbws/lib/circuitbuilder.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/sbws/lib/circuitbuilder.py b/sbws/lib/circuitbuilder.py index 5eb8a02..8efc8de 100644 --- a/sbws/lib/circuitbuilder.py +++ b/sbws/lib/circuitbuilder.py @@ -17,10 +17,7 @@ class PathLengthException(Exception): def valid_circuit_length(path): - assert isinstance(path, int) or isinstance(path, list) - if isinstance(path, int): - return path > 0 and path <= 8 - return len(path) > 0 and len(path) <= 8 + return 0 < len(path) <= 8 class CircuitBuilder:

1 0

[sbws/master] circuitbuilder: add comment about future refactor
by juga＠torproject.org 15 Feb '19

15 Feb '19

commit b6c67753771daebafb2521c5875997dc573ea430 Author: juga0 <juga(a)riseup.net> Date: Wed Feb 6 17:34:39 2019 +0000 circuitbuilder: add comment about future refactor --- sbws/lib/circuitbuilder.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbws/lib/circuitbuilder.py b/sbws/lib/circuitbuilder.py index 95e670a..c5bae5c 100644 --- a/sbws/lib/circuitbuilder.py +++ b/sbws/lib/circuitbuilder.py @@ -87,6 +87,9 @@ class CircuitBuilder: self.built_circuits.clear() +# In a future refactor, remove this class, since sbws chooses the relays to +# build the circuit, the relays are not just choosen as random as this class +# does. class GapsCircuitBuilder(CircuitBuilder): ''' The build_circuit member function takes a list. Falsey values in the list will be replaced with relays chosen uniformally at random; Truthy

1 0

[sbws/master] circuitbuilder: return reasons on circuit build errors
by juga＠torproject.org 15 Feb '19

15 Feb '19

commit a4aa5e7d5e7b72caf1a545b0b26e43ffe2946dfc Author: juga0 <juga(a)riseup.net> Date: Wed Feb 6 17:16:39 2019 +0000 circuitbuilder: return reasons on circuit build errors --- sbws/lib/circuitbuilder.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/sbws/lib/circuitbuilder.py b/sbws/lib/circuitbuilder.py index be10c2b..95e670a 100644 --- a/sbws/lib/circuitbuilder.py +++ b/sbws/lib/circuitbuilder.py @@ -136,16 +136,13 @@ class GapsCircuitBuilder(CircuitBuilder): return None, "Can not build a circuit, invalid path." path = self._normalize_path(path) if path is None: - return None + return None, "Can not build a circuit, no path." num_missing = len(['foo' for r in path if not r]) insert_relays = self._random_sample_relays( num_missing, [r for r in path if r is not None]) if insert_relays is None: path = ','.join([r.nickname if r else str(None) for r in path]) - log.warning( - 'Problem building a circuit to satisfy %s with available ' - 'relays in the network', path) - return None + return None, "Can not build a circuit with the current relays." assert len(insert_relays) == num_missing path = [r.fingerprint if r else insert_relays.pop().fingerprint for r in path]

1 0

[sbws/master] circuitbuilder: Stop raising exception on path invalid.
by juga＠torproject.org 15 Feb '19

15 Feb '19

commit 0773f09038b9422872302347df2ad52263f351e3 Author: juga0 <juga(a)riseup.net> Date: Wed Feb 6 17:15:19 2019 +0000 circuitbuilder: Stop raising exception on path invalid. Instead, return the reason why the circuit could not be built, as other errors do. --- sbws/lib/circuitbuilder.py | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/sbws/lib/circuitbuilder.py b/sbws/lib/circuitbuilder.py index 8efc8de..be10c2b 100644 --- a/sbws/lib/circuitbuilder.py +++ b/sbws/lib/circuitbuilder.py @@ -7,15 +7,6 @@ import logging log = logging.getLogger(__name__) -class PathLengthException(Exception): - def __init__(self, message=None, errors=None): - if message is not None: - super().__init__(message) - else: - super().__init__() - self.errors = errors - - def valid_circuit_length(path): return 0 < len(path) <= 8 @@ -66,7 +57,7 @@ class CircuitBuilder: was an error building the circuit. """ if not valid_circuit_length(path): - raise PathLengthException() + return None, "Can not build a circuit, invalid path." c = self.controller timeout = self.circuit_timeout fp_path = '[' + ' -> '.join([p for p in path]) + ']' @@ -142,7 +133,7 @@ class GapsCircuitBuilder(CircuitBuilder): chosen uniformally at random. A relay will not be in a circuit twice. ''' if not valid_circuit_length(path): - raise PathLengthException() + return None, "Can not build a circuit, invalid path." path = self._normalize_path(path) if path is None: return None

1 0