July 2020 - tor-commits - lists.torproject.org

[sbws/maint-1.1] fix: doc: Comment on Debian/Ubuntu releases
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 9ee5f6407b7bb495399c7eb9bb619b3e512f86cb Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 13:13:52 2020 +0000 fix: doc: Comment on Debian/Ubuntu releases because sometimes the package might not be in Debian stable or testing and we are not checking Ubuntu releases. --- INSTALL.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/INSTALL.rst b/INSTALL.rst index 2cc2a10..0ca4d58 100644 --- a/INSTALL.rst +++ b/INSTALL.rst @@ -14,6 +14,8 @@ To install also the documentation:: sudo apt install sbws-doc +You might need to check in which releases is the package available. + Continue reading to install ``sbws`` in other ways. System requirements

1 0

[sbws/maint-1.1] fix: doc: Increase RAM required
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit ffe3f317f46cd1921fd189c525fba00bb59c2c23 Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 14:35:00 2020 +0000 fix: doc: Increase RAM required ahem, because of all json it has to manage in memory. --- INSTALL.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/INSTALL.rst b/INSTALL.rst index e4e7a05..4254fab 100644 --- a/INSTALL.rst +++ b/INSTALL.rst @@ -80,7 +80,7 @@ System physical requirements ----------------------------- - Bandwidth: at least 12.5MB/s (100 Mbit/s). -- Free RAM: at least 1.5GB +- Free RAM: at least 2GB - Free disk: at least 3GB ``sbws`` and its dependencies need around 20MB of disk space.

1 0

[sbws/maint-1.1] fix: doc: Update supported Python versions
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit c676089270222f8445ba1d25ecdf7241abd85f75 Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 13:37:03 2020 +0000 fix: doc: Update supported Python versions --- INSTALL.rst | 2 +- setup.py | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/INSTALL.rst b/INSTALL.rst index 0ca4d58..6f52b13 100644 --- a/INSTALL.rst +++ b/INSTALL.rst @@ -22,7 +22,7 @@ System requirements -------------------- - Tor (last stable version is recommended) -- Python 3 (>= 3.5) +- Python 3 (>= 3.6) - virtualenv_ (while there is not ``stem`` release > 1.6.0, it is recommended to install the required python dependencies in a virtualenv) diff --git a/setup.py b/setup.py index f328d7a..10ff1fd 100755 --- a/setup.py +++ b/setup.py @@ -51,8 +51,11 @@ setup( 'Intended Audience :: Developers', 'Intended Audience :: System Administrators', 'Operating System :: OS Independent', - 'Programming Language :: Python :: 3.5', + 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.6', + 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', + 'Programming Language :: Python :: 3.9', 'Topic :: System :: Networking', ], packages=find_packages(),

1 0

[sbws/maint-1.1] fix: doc: Recommend pip only for development
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 7bbdc714c0255e067b622331b17c93c5637b5a7e Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 14:23:12 2020 +0000 fix: doc: Recommend pip only for development or testing and add links. --- INSTALL.rst | 37 ++++++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/INSTALL.rst b/INSTALL.rst index 5907d4a..e4e7a05 100644 --- a/INSTALL.rst +++ b/INSTALL.rst @@ -24,12 +24,6 @@ System requirements - Tor (last stable version is recommended) - Python 3 (>= 3.6) -- virtualenv_ (while there is not ``stem`` release > 1.6.0, it is - recommended to install the required python dependencies in a virtualenv) - -In Debian:: - - sudo apt install tor python3 virtualenv Python dependencies -------------------- @@ -37,9 +31,30 @@ Python dependencies - Stem_ >= 1.7.0 - Requests_ (with socks_ support) >= 2.10.0 -To install the Python dependencies, create a ``virtualenv`` first +It is recommend to install the dependencies from your system package manager. +If that is not possible, because the Python dependencies are not available in +your system, you can install them from their sources. +We only recommend using pip_ for development or testing. + +Installing sbws from source +--------------------------- + +Clone ``sbws``:: + + git clone https://git.torproject.org/sbws.git + +and install it:: + + cd sbws + python3 setup.py install -:: +Installing sbws for development or testing +------------------------------------------ + +If you use pip_, it is recommended to use virtualenv_, to avoid having +different versions of the same libraries in your system. + +To create a ``virtualenv``:: virtualenv venv -p /usr/bin/python3 source venv/bin/activate @@ -50,7 +65,10 @@ Clone ``sbws``:: Install the python dependencies:: - cd sbws && pip install . + cd sbws && pip install -e . + +Configuration and deployment +---------------------------- ``sbws`` needs :term:`destination` s to request files from. @@ -97,3 +115,4 @@ are 10MB and it will keep 50 rotated log files. .. _port: https://www.freshports.org/net/py-sbws/ .. _Debian: https://packages.debian.org/search?keywords=sbws&searchon=names&suite=all&s… .. _Ubuntu: https://packages.ubuntu.com/search?keywords=sbws&searchon=names&suite=all&s… +.. _pip: https://pypi.org/project/pip/

1 0

[sbws/maint-1.1] doc: fix: Update sbws availabity in OS and links
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 9adc95757c2ecd28f08a9db0526495faffe1d105 Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 13:55:27 2020 +0000 doc: fix: Update sbws availabity in OS and links --- INSTALL.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/INSTALL.rst b/INSTALL.rst index 6f52b13..5907d4a 100644 --- a/INSTALL.rst +++ b/INSTALL.rst @@ -3,10 +3,9 @@ Installing Simple Bandwidth Scanner =================================== -The recommended method is to install it from your system -distribution. +The recommended method is to install it from your system package manager. -In Debian/Ubuntu systems:: +In Debian_/Ubuntu_ systems:: sudo apt install sbws @@ -16,6 +15,8 @@ To install also the documentation:: You might need to check in which releases is the package available. +There is a port_ for FreeBSD. + Continue reading to install ``sbws`` in other ways. System requirements @@ -93,3 +94,6 @@ are 10MB and it will keep 50 rotated log files. .. _pytest: https://docs.pytest.org/ .. _tox: https://tox.readthedocs.io .. _Coverage: https://coverage.readthedocs.io/ +.. _port: https://www.freshports.org/net/py-sbws/ +.. _Debian: https://packages.debian.org/search?keywords=sbws&searchon=names&suite=all&s… +.. _Ubuntu: https://packages.ubuntu.com/search?keywords=sbws&searchon=names&suite=all&s…

1 0

[sbws/maint-1.1] fix: doc: Recomment maint-1.1 for production
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit e8a5d80b945f0988e2865147c61700cf39bc7bda Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 15:16:26 2020 +0000 fix: doc: Recomment maint-1.1 for production Closes: #33473 --- INSTALL.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/INSTALL.rst b/INSTALL.rst index 4254fab..7bf57ca 100644 --- a/INSTALL.rst +++ b/INSTALL.rst @@ -42,6 +42,10 @@ Installing sbws from source Clone ``sbws``:: git clone https://git.torproject.org/sbws.git + git checkout maint-1.1 + +The branch ``maint-1.1`` is the last stable version and the one that should be +used in production. and install it::

1 0

[sbws/maint-1.1] Merge remote-tracking branch 'gitlab/merge-requests/13' into maint-1.1
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit c5fe10e4f4725bc844fca04e233b105933d2c70b Merge: 3e86fd5 e8a5d80 Author: Georg Koppen <gk(a)torproject.org> Date: Fri Jul 10 22:59:07 2020 +0000 Merge remote-tracking branch 'gitlab/merge-requests/13' into maint-1.1 DEPLOY.rst | 16 ++++++++++++---- INSTALL.rst | 57 +++++++++++++++++++++++++++++++++++++++++++-------------- setup.py | 5 ++++- 3 files changed, 59 insertions(+), 19 deletions(-)

1 0

[sbws/maint-1.1] fix: v3bwfile: Count relay priority lists
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit a914215c7ae9dda77bb429c37141ae4a9e11db81 Author: juga0 <juga(a)riseup.net> Date: Fri Jun 5 15:34:12 2020 +0000 fix: v3bwfile: Count relay priority lists and measurement attempts from all the results. Until they get properly updated. Also change dates in tests, so that timestamps are counted correctly Closes: #34309. --- sbws/lib/v3bwfile.py | 30 ++++++++++++++++++------------ tests/unit/lib/data/results.txt | 2 +- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py index 071dc94..0c1f9b6 100644 --- a/sbws/lib/v3bwfile.py +++ b/sbws/lib/v3bwfile.py @@ -672,19 +672,25 @@ class V3BWLine(object): ]) ) - kwargs['relay_recent_priority_list_count'] = str( - max([ - len(getattr(r, 'relay_recent_priority_list', []) or []) - for r in results - ]) - ) + # Workaround for #34309. + # Because of a bug, probably in relaylist, resultdump, relayprioritizer + # or scanner, only the last timestamp is being stored in each result. + # Temporally count the number of timestamps for all results. + # If there is an unexpected failure and the result is not stored, this + # number would be lower than what would be the correct one. + # This should happen rarely or never. + ts = set([]) + for r in results: + if getattr(r, "relay_recent_priority_list", None): + ts.update(r.relay_recent_priority_list) + kwargs["relay_recent_priority_list_count"] = str(len(ts)) - kwargs['relay_recent_measurement_attempt_count'] = str( - max([ - len(getattr(r, 'relay_recent_measurement_attempt', []) or []) - for r in results - ]) - ) + # Same comment as the previous paragraph. + ts = set() + for r in results: + if getattr(r, "relay_recent_measurement_attempt", None): + ts.update(r.relay_recent_measurement_attempt) + kwargs["relay_recent_measurement_attempt_count"] = str(len(ts)) success_results = [r for r in results if isinstance(r, ResultSuccess)] diff --git a/tests/unit/lib/data/results.txt b/tests/unit/lib/data/results.txt index 36df3d9..25e30db 100644 --- a/tests/unit/lib/data/results.txt +++ b/tests/unit/lib/data/results.txt @@ -1,3 +1,3 @@ {"version": 4, "time": 1523974147, "circ": ["AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"], "type": "error-stream", "msg": "Something bad happened while measuring bandwidth", "fingerprint": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "scanner": "IDidntEditTheSBWSConfig", "dest_url": "http://y.z", "nickname": "A", "address": "111.111.111.111", "master_key_ed25519": "g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s"} -{"version": 4, "time": 1523887747, "circ": ["AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"], "type": "success", "rtts": [0.4596822261810303, 0.44872617721557617, 0.4563450813293457, 0.44872212409973145, 0.4561030864715576, 0.4765200614929199, 0.4495084285736084, 0.45711588859558105, 0.45520496368408203, 0.4635589122772217], "fingerprint": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "scanner": "IDidntEditTheSBWSConfig", "downloads": [{"amount": 590009, "duration": 6.1014368534088135}, {"amount": 590009, "duration": 8.391342878341675}, {"amount": 321663, "duration": 7.064587831497192}, {"amount": 321663, "duration": 8.266003131866455}, {"amount": 321663, "duration": 5.779450178146362}], "dest_url": "http://y.z", "nickname": "A", "address": "111.111.111.111", "master_key_ed25519": "g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s", "relay_average_bandwidth": 1000000000, "relay_burst_bandwidth": 123456, "relay_observed_bandwidth": 524288, "consensus_ba ndwidth": 600000, "consensus_bandwidth_is_unmeasured": false, "relay_in_recent_consensus": ["2018-04-07T14:09:07", "2018-04-07T14:09:07", "2018-04-07T14:09:07"], "relay_recent_measurement_attempt": ["2018-04-07T14:09:07", "2018-04-07T14:09:07"], "relay_recent_priority_list": ["2018-04-07T14:09:07", "2018-04-07T14:09:07", "2018-04-07T14:09:07"]} +{"version": 4, "time": 1523887747, "circ": ["AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"], "type": "success", "rtts": [0.4596822261810303, 0.44872617721557617, 0.4563450813293457, 0.44872212409973145, 0.4561030864715576, 0.4765200614929199, 0.4495084285736084, 0.45711588859558105, 0.45520496368408203, 0.4635589122772217], "fingerprint": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "scanner": "IDidntEditTheSBWSConfig", "downloads": [{"amount": 590009, "duration": 6.1014368534088135}, {"amount": 590009, "duration": 8.391342878341675}, {"amount": 321663, "duration": 7.064587831497192}, {"amount": 321663, "duration": 8.266003131866455}, {"amount": 321663, "duration": 5.779450178146362}], "dest_url": "http://y.z", "nickname": "A", "address": "111.111.111.111", "master_key_ed25519": "g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s", "relay_average_bandwidth": 1000000000, "relay_burst_bandwidth": 123456, "relay_observed_bandwidth": 524288, "consensus_ba ndwidth": 600000, "consensus_bandwidth_is_unmeasured": false, "relay_in_recent_consensus": ["2018-04-07T14:09:07", "2018-04-07T14:09:07", "2018-04-07T14:09:07"], "relay_recent_measurement_attempt": ["2018-04-07T14:09:07", "2018-04-08T14:09:07"], "relay_recent_priority_list": ["2018-04-07T14:09:07", "2018-04-08T14:09:07", "2018-04-09T14:09:07"]} {"version": 4, "time": 1523974147, "circ": ["AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"], "type": "error-stream", "msg": "Something bad happened while measuring bandwidth", "fingerprint": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "scanner": "IDidntEditTheSBWSConfig", "dest_url": "http://y.z", "nickname": "A", "address": "111.111.111.111", "master_key_ed25519": "g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s", "relay_in_recent_consensus": ["2018-04-07T14:09:07", "2018-04-07T14:09:07", "2018-04-07T14:09:07"], "relay_recent_measurement_attempt": ["2018-04-07T14:09:07", "2018-04-07T14:09:07"], "relay_recent_priority_list": ["2018-04-07T14:09:07", "2018-04-07T14:09:07", "2018-04-07T14:09:07"]}

1 0

[sbws/maint-1.1] fix: relaylist: Check exit to all domains/ips
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit d958915e5a1a2bd7f644ac96dc5aec6961f45d10 Author: juga <juga(a)riseup.net> Date: Thu Jul 9 10:31:35 2020 +0000 fix: relaylist: Check exit to all domains/ips When an exit policy allows to exit only to some subnet, it is not enough to check that it can exit to a port, since it can, but it might not be able to exit to the domain/ip of the sbws Web servers. To ensure that without having to check whether it can exit to a specific domain/ip, we can query the exit policy with `strict`. Closes #40006. Bugfix v1.0.3. --- sbws/lib/relaylist.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbws/lib/relaylist.py b/sbws/lib/relaylist.py index 1852199..ab7455e 100644 --- a/sbws/lib/relaylist.py +++ b/sbws/lib/relaylist.py @@ -197,7 +197,9 @@ class Relay: # Therefore, catch the exception here. try: if self.exit_policy: - return self.exit_policy.can_exit_to(port=port) + # Using `strict` to ensure it can exit to ALL domains + # and ips and that port. See #40006. + return self.exit_policy.can_exit_to(port=port, strict=True) except TypeError: return False return False

1 0

[sbws/maint-1.1] Vote on the relays with few or close measurements
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 09905b6c65712fcb58d321a0e5425e5f17e7b91d Author: juga0 <juga(a)riseup.net> Date: Tue Jun 23 12:07:29 2020 +0000 Vote on the relays with few or close measurements to vote on approximately the same numbers of relays as Torflow. Torflow does not exclude relays with few or close measurements, though it is possible that because of the way it measures, there are no few or close measurements. Closes #34393 --- sbws/globals.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/sbws/globals.py b/sbws/globals.py index 01a147b..575a19f 100644 --- a/sbws/globals.py +++ b/sbws/globals.py @@ -81,8 +81,17 @@ TORFLOW_OBS_MEAN = 1 TORFLOW_OBS_DECAYING = 3 TORFLOW_ROUND_DIG = 3 PROP276_ROUND_DIG = 2 -DAY_SECS = 86400 -NUM_MIN_RESULTS = 2 +# Number of seconds the measurements for a relay have to be distant from each +# other, otherwise the relay would be excluded from the relays to vote on. +# Ideally, this should be 86400 seconds (1 day). +# To have sbws vote on approximately the same number of relays as Torflow, +# leave it as None, to not exclude measurements. +DAY_SECS = None +# Minimum number of measurements for a relay to be included as a relay to vote +# on. +# Ideally, this should be 2. +# As the constant before, leave it as 1 to not exclude measurements. +NUM_MIN_RESULTS = 1 MIN_REPORT = 60 # Maximum difference between the total consensus bandwidth and the total in # in the bandwidth lines in percentage

1 0