July 2020 - tor-commits - lists.torproject.org

[sbws/master] doc: fix: Update sbws availabity in OS and links
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 9adc95757c2ecd28f08a9db0526495faffe1d105 Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 13:55:27 2020 +0000 doc: fix: Update sbws availabity in OS and links --- INSTALL.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/INSTALL.rst b/INSTALL.rst index 6f52b13..5907d4a 100644 --- a/INSTALL.rst +++ b/INSTALL.rst @@ -3,10 +3,9 @@ Installing Simple Bandwidth Scanner =================================== -The recommended method is to install it from your system -distribution. +The recommended method is to install it from your system package manager. -In Debian/Ubuntu systems:: +In Debian_/Ubuntu_ systems:: sudo apt install sbws @@ -16,6 +15,8 @@ To install also the documentation:: You might need to check in which releases is the package available. +There is a port_ for FreeBSD. + Continue reading to install ``sbws`` in other ways. System requirements @@ -93,3 +94,6 @@ are 10MB and it will keep 50 rotated log files. .. _pytest: https://docs.pytest.org/ .. _tox: https://tox.readthedocs.io .. _Coverage: https://coverage.readthedocs.io/ +.. _port: https://www.freshports.org/net/py-sbws/ +.. _Debian: https://packages.debian.org/search?keywords=sbws&searchon=names&suite=all&s… +.. _Ubuntu: https://packages.ubuntu.com/search?keywords=sbws&searchon=names&suite=all&s…

1 0

[sbws/master] fix: doc: Recommend pip only for development
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 7bbdc714c0255e067b622331b17c93c5637b5a7e Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 14:23:12 2020 +0000 fix: doc: Recommend pip only for development or testing and add links. --- INSTALL.rst | 37 ++++++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/INSTALL.rst b/INSTALL.rst index 5907d4a..e4e7a05 100644 --- a/INSTALL.rst +++ b/INSTALL.rst @@ -24,12 +24,6 @@ System requirements - Tor (last stable version is recommended) - Python 3 (>= 3.6) -- virtualenv_ (while there is not ``stem`` release > 1.6.0, it is - recommended to install the required python dependencies in a virtualenv) - -In Debian:: - - sudo apt install tor python3 virtualenv Python dependencies -------------------- @@ -37,9 +31,30 @@ Python dependencies - Stem_ >= 1.7.0 - Requests_ (with socks_ support) >= 2.10.0 -To install the Python dependencies, create a ``virtualenv`` first +It is recommend to install the dependencies from your system package manager. +If that is not possible, because the Python dependencies are not available in +your system, you can install them from their sources. +We only recommend using pip_ for development or testing. + +Installing sbws from source +--------------------------- + +Clone ``sbws``:: + + git clone https://git.torproject.org/sbws.git + +and install it:: + + cd sbws + python3 setup.py install -:: +Installing sbws for development or testing +------------------------------------------ + +If you use pip_, it is recommended to use virtualenv_, to avoid having +different versions of the same libraries in your system. + +To create a ``virtualenv``:: virtualenv venv -p /usr/bin/python3 source venv/bin/activate @@ -50,7 +65,10 @@ Clone ``sbws``:: Install the python dependencies:: - cd sbws && pip install . + cd sbws && pip install -e . + +Configuration and deployment +---------------------------- ``sbws`` needs :term:`destination` s to request files from. @@ -97,3 +115,4 @@ are 10MB and it will keep 50 rotated log files. .. _port: https://www.freshports.org/net/py-sbws/ .. _Debian: https://packages.debian.org/search?keywords=sbws&searchon=names&suite=all&s… .. _Ubuntu: https://packages.ubuntu.com/search?keywords=sbws&searchon=names&suite=all&s… +.. _pip: https://pypi.org/project/pip/

1 0

[sbws/master] fix: doc: Increase RAM required
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit ffe3f317f46cd1921fd189c525fba00bb59c2c23 Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 14:35:00 2020 +0000 fix: doc: Increase RAM required ahem, because of all json it has to manage in memory. --- INSTALL.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/INSTALL.rst b/INSTALL.rst index e4e7a05..4254fab 100644 --- a/INSTALL.rst +++ b/INSTALL.rst @@ -80,7 +80,7 @@ System physical requirements ----------------------------- - Bandwidth: at least 12.5MB/s (100 Mbit/s). -- Free RAM: at least 1.5GB +- Free RAM: at least 2GB - Free disk: at least 3GB ``sbws`` and its dependencies need around 20MB of disk space.

1 0

[sbws/master] fix: v3bwfile: Count relay priority lists
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit a914215c7ae9dda77bb429c37141ae4a9e11db81 Author: juga0 <juga(a)riseup.net> Date: Fri Jun 5 15:34:12 2020 +0000 fix: v3bwfile: Count relay priority lists and measurement attempts from all the results. Until they get properly updated. Also change dates in tests, so that timestamps are counted correctly Closes: #34309. --- sbws/lib/v3bwfile.py | 30 ++++++++++++++++++------------ tests/unit/lib/data/results.txt | 2 +- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py index 071dc94..0c1f9b6 100644 --- a/sbws/lib/v3bwfile.py +++ b/sbws/lib/v3bwfile.py @@ -672,19 +672,25 @@ class V3BWLine(object): ]) ) - kwargs['relay_recent_priority_list_count'] = str( - max([ - len(getattr(r, 'relay_recent_priority_list', []) or []) - for r in results - ]) - ) + # Workaround for #34309. + # Because of a bug, probably in relaylist, resultdump, relayprioritizer + # or scanner, only the last timestamp is being stored in each result. + # Temporally count the number of timestamps for all results. + # If there is an unexpected failure and the result is not stored, this + # number would be lower than what would be the correct one. + # This should happen rarely or never. + ts = set([]) + for r in results: + if getattr(r, "relay_recent_priority_list", None): + ts.update(r.relay_recent_priority_list) + kwargs["relay_recent_priority_list_count"] = str(len(ts)) - kwargs['relay_recent_measurement_attempt_count'] = str( - max([ - len(getattr(r, 'relay_recent_measurement_attempt', []) or []) - for r in results - ]) - ) + # Same comment as the previous paragraph. + ts = set() + for r in results: + if getattr(r, "relay_recent_measurement_attempt", None): + ts.update(r.relay_recent_measurement_attempt) + kwargs["relay_recent_measurement_attempt_count"] = str(len(ts)) success_results = [r for r in results if isinstance(r, ResultSuccess)] diff --git a/tests/unit/lib/data/results.txt b/tests/unit/lib/data/results.txt index 36df3d9..25e30db 100644 --- a/tests/unit/lib/data/results.txt +++ b/tests/unit/lib/data/results.txt @@ -1,3 +1,3 @@ {"version": 4, "time": 1523974147, "circ": ["AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"], "type": "error-stream", "msg": "Something bad happened while measuring bandwidth", "fingerprint": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "scanner": "IDidntEditTheSBWSConfig", "dest_url": "http://y.z", "nickname": "A", "address": "111.111.111.111", "master_key_ed25519": "g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s"} -{"version": 4, "time": 1523887747, "circ": ["AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"], "type": "success", "rtts": [0.4596822261810303, 0.44872617721557617, 0.4563450813293457, 0.44872212409973145, 0.4561030864715576, 0.4765200614929199, 0.4495084285736084, 0.45711588859558105, 0.45520496368408203, 0.4635589122772217], "fingerprint": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "scanner": "IDidntEditTheSBWSConfig", "downloads": [{"amount": 590009, "duration": 6.1014368534088135}, {"amount": 590009, "duration": 8.391342878341675}, {"amount": 321663, "duration": 7.064587831497192}, {"amount": 321663, "duration": 8.266003131866455}, {"amount": 321663, "duration": 5.779450178146362}], "dest_url": "http://y.z", "nickname": "A", "address": "111.111.111.111", "master_key_ed25519": "g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s", "relay_average_bandwidth": 1000000000, "relay_burst_bandwidth": 123456, "relay_observed_bandwidth": 524288, "consensus_ba ndwidth": 600000, "consensus_bandwidth_is_unmeasured": false, "relay_in_recent_consensus": ["2018-04-07T14:09:07", "2018-04-07T14:09:07", "2018-04-07T14:09:07"], "relay_recent_measurement_attempt": ["2018-04-07T14:09:07", "2018-04-07T14:09:07"], "relay_recent_priority_list": ["2018-04-07T14:09:07", "2018-04-07T14:09:07", "2018-04-07T14:09:07"]} +{"version": 4, "time": 1523887747, "circ": ["AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"], "type": "success", "rtts": [0.4596822261810303, 0.44872617721557617, 0.4563450813293457, 0.44872212409973145, 0.4561030864715576, 0.4765200614929199, 0.4495084285736084, 0.45711588859558105, 0.45520496368408203, 0.4635589122772217], "fingerprint": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "scanner": "IDidntEditTheSBWSConfig", "downloads": [{"amount": 590009, "duration": 6.1014368534088135}, {"amount": 590009, "duration": 8.391342878341675}, {"amount": 321663, "duration": 7.064587831497192}, {"amount": 321663, "duration": 8.266003131866455}, {"amount": 321663, "duration": 5.779450178146362}], "dest_url": "http://y.z", "nickname": "A", "address": "111.111.111.111", "master_key_ed25519": "g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s", "relay_average_bandwidth": 1000000000, "relay_burst_bandwidth": 123456, "relay_observed_bandwidth": 524288, "consensus_ba ndwidth": 600000, "consensus_bandwidth_is_unmeasured": false, "relay_in_recent_consensus": ["2018-04-07T14:09:07", "2018-04-07T14:09:07", "2018-04-07T14:09:07"], "relay_recent_measurement_attempt": ["2018-04-07T14:09:07", "2018-04-08T14:09:07"], "relay_recent_priority_list": ["2018-04-07T14:09:07", "2018-04-08T14:09:07", "2018-04-09T14:09:07"]} {"version": 4, "time": 1523974147, "circ": ["AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"], "type": "error-stream", "msg": "Something bad happened while measuring bandwidth", "fingerprint": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "scanner": "IDidntEditTheSBWSConfig", "dest_url": "http://y.z", "nickname": "A", "address": "111.111.111.111", "master_key_ed25519": "g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s", "relay_in_recent_consensus": ["2018-04-07T14:09:07", "2018-04-07T14:09:07", "2018-04-07T14:09:07"], "relay_recent_measurement_attempt": ["2018-04-07T14:09:07", "2018-04-07T14:09:07"], "relay_recent_priority_list": ["2018-04-07T14:09:07", "2018-04-07T14:09:07", "2018-04-07T14:09:07"]}

1 0

[sbws/master] fix: doc: Recomment maint-1.1 for production
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit e8a5d80b945f0988e2865147c61700cf39bc7bda Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 15:16:26 2020 +0000 fix: doc: Recomment maint-1.1 for production Closes: #33473 --- INSTALL.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/INSTALL.rst b/INSTALL.rst index 4254fab..7bf57ca 100644 --- a/INSTALL.rst +++ b/INSTALL.rst @@ -42,6 +42,10 @@ Installing sbws from source Clone ``sbws``:: git clone https://git.torproject.org/sbws.git + git checkout maint-1.1 + +The branch ``maint-1.1`` is the last stable version and the one that should be +used in production. and install it::

1 0

[sbws/master] fix: doc: Recommend using a CDN,
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 77c65cf55f5a55227bbd3506bdd7b65368faf866 Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 14:59:48 2020 +0000 fix: doc: Recommend using a CDN, add link to it and rephrase some sentences. --- DEPLOY.rst | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/DEPLOY.rst b/DEPLOY.rst index a1ab4dd..45e9d31 100644 --- a/DEPLOY.rst +++ b/DEPLOY.rst @@ -30,6 +30,9 @@ destination requirements - Bandwidth: at least 12.5MB/s (100 Mbit/s). - Network traffic: around 12-15GB/day. +If possible, use a `Content delivery network`_ (CDN) in order to make the +destination IP closer to the scanner exit. + scanner setup ---------------------- @@ -38,17 +41,20 @@ or `<INSTALL.html>`_ (local build or Read the Docs). To run the ``scanner`` it is mandatory to create a configuration file with at least one ``destination``. -It is recommended to set several ``destination``s so that the ``scanner`` can +It is recommended to set several ``destinations`` so that the ``scanner`` can continue if one fails. -If ``sbws`` is installed from the Debian package, then create a file in -``/etc/sbws/sbws.ini`` like in the following example: +If ``sbws`` is installed from the Debian package, then create the configuration +file in ``/etc/sbws/sbws.ini``. +You can see an example with all the possible options here, note that you don't +need to include all of that and that everything that starts with ``#`` and +``;`` is a comment: .. literalinclude:: /examples/sbws.example.ini :caption: Example sbws.example.ini If ``sbws`` is installed from the sources as a non-root user then create the -file in ``~/.sbws.ini``. +configuration file in ``~/.sbws.ini``. More details about the configuration file can be found in ``./docs/source/man_sbws.ini.rst`` (in the local directory or GitHub) or @@ -58,3 +64,5 @@ More details about the configuration file can be found in See also ``./docs/source/man_sbws.rst`` (in the local directory or GitHub) or `<man_sbws.html>`_ (local build or Read the Docs) or ``man sbws`` (system package). + +.. _Content delivery network: https://en.wikipedia.org/wiki/Content_delivery_network

1 0

[sbws/master] Merge branch 'maint-1.1'
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 383cf4fa9150ded0a21e64788f2e521c9ddc0173 Merge: 15fbb2b c5fe10e Author: Georg Koppen <gk(a)torproject.org> Date: Fri Jul 10 23:00:52 2020 +0000 Merge branch 'maint-1.1' DEPLOY.rst | 16 ++++++++++++---- INSTALL.rst | 57 +++++++++++++++++++++++++++++++++++++++++++-------------- setup.py | 5 ++++- 3 files changed, 59 insertions(+), 19 deletions(-)

1 0

[sbws/master] Merge remote-tracking branch 'gitlab/merge-requests/13' into maint-1.1
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit c5fe10e4f4725bc844fca04e233b105933d2c70b Merge: 3e86fd5 e8a5d80 Author: Georg Koppen <gk(a)torproject.org> Date: Fri Jul 10 22:59:07 2020 +0000 Merge remote-tracking branch 'gitlab/merge-requests/13' into maint-1.1 DEPLOY.rst | 16 ++++++++++++---- INSTALL.rst | 57 +++++++++++++++++++++++++++++++++++++++++++-------------- setup.py | 5 ++++- 3 files changed, 59 insertions(+), 19 deletions(-)

1 0

[sbws/master] fix: relaylist: Check exit to all domains/ips
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit d958915e5a1a2bd7f644ac96dc5aec6961f45d10 Author: juga <juga(a)riseup.net> Date: Thu Jul 9 10:31:35 2020 +0000 fix: relaylist: Check exit to all domains/ips When an exit policy allows to exit only to some subnet, it is not enough to check that it can exit to a port, since it can, but it might not be able to exit to the domain/ip of the sbws Web servers. To ensure that without having to check whether it can exit to a specific domain/ip, we can query the exit policy with `strict`. Closes #40006. Bugfix v1.0.3. --- sbws/lib/relaylist.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbws/lib/relaylist.py b/sbws/lib/relaylist.py index 1852199..ab7455e 100644 --- a/sbws/lib/relaylist.py +++ b/sbws/lib/relaylist.py @@ -197,7 +197,9 @@ class Relay: # Therefore, catch the exception here. try: if self.exit_policy: - return self.exit_policy.can_exit_to(port=port) + # Using `strict` to ensure it can exit to ALL domains + # and ips and that port. See #40006. + return self.exit_policy.can_exit_to(port=port, strict=True) except TypeError: return False return False

1 0

[sbws/master] Merge remote-tracking branch 'gitlab/merge-requests/14' into maint-1.1
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 748cf1c1f347b89eaab625bdebd9314dadaccc8e Merge: c5fe10e a914215 Author: Georg Koppen <gk(a)torproject.org> Date: Fri Jul 10 23:10:23 2020 +0000 Merge remote-tracking branch 'gitlab/merge-requests/14' into maint-1.1 sbws/lib/v3bwfile.py | 30 ++++++++++++++++++------------ tests/unit/lib/data/results.txt | 2 +- 2 files changed, 19 insertions(+), 13 deletions(-)

1 0