May 2021 - tor-commits - lists.torproject.org

[tor/release-0.4.5] Merge branch 'maint-0.3.5' into maint-0.4.4
by ahf＠torproject.org 25 May '21

25 May '21

commit 11c7e6573013f60d2897720ebbc8c4164e124574 Merge: 48dd87933d 42ba87d964 Author: Alexander Færøy <ahf(a)torproject.org> Date: Tue May 25 13:25:23 2021 +0000 Merge branch 'maint-0.3.5' into maint-0.4.4 changes/bug40399 | 5 +++++ src/lib/tls/tortls.h | 1 - src/lib/tls/tortls_nss.c | 7 ------- src/lib/tls/tortls_openssl.c | 13 ------------- src/test/test_tortls_openssl.c | 20 -------------------- 5 files changed, 5 insertions(+), 41 deletions(-)

1 0

[tor/release-0.4.5] Merge branch 'maint-0.4.4' into maint-0.4.5
by ahf＠torproject.org 25 May '21

25 May '21

commit 4a7379b80a3d0f61d258b26f82d894da9b8cd0f5 Merge: 97b61e21a9 11c7e65730 Author: Alexander Færøy <ahf(a)torproject.org> Date: Tue May 25 13:25:23 2021 +0000 Merge branch 'maint-0.4.4' into maint-0.4.5 changes/bug40399 | 5 +++++ src/lib/tls/tortls.h | 1 - src/lib/tls/tortls_nss.c | 7 ------- src/lib/tls/tortls_openssl.c | 13 ------------- src/test/test_tortls_openssl.c | 20 -------------------- 5 files changed, 5 insertions(+), 41 deletions(-)

1 0

[tor/release-0.4.6] Merge branch 'maint-0.3.5' into maint-0.4.4
by ahf＠torproject.org 25 May '21

25 May '21

commit 11c7e6573013f60d2897720ebbc8c4164e124574 Merge: 48dd87933d 42ba87d964 Author: Alexander Færøy <ahf(a)torproject.org> Date: Tue May 25 13:25:23 2021 +0000 Merge branch 'maint-0.3.5' into maint-0.4.4 changes/bug40399 | 5 +++++ src/lib/tls/tortls.h | 1 - src/lib/tls/tortls_nss.c | 7 ------- src/lib/tls/tortls_openssl.c | 13 ------------- src/test/test_tortls_openssl.c | 20 -------------------- 5 files changed, 5 insertions(+), 41 deletions(-)

1 0

[sbws/main] fix: CI: Build docs automatically in Gitlab
by juga＠torproject.org 25 May '21

25 May '21

commit 2944ddd3e36c904f11f54af7672c65daa1c00556 Author: juga0 <juga(a)riseup.net> Date: Tue Apr 20 11:02:21 2021 +0000 fix: CI: Build docs automatically in Gitlab also replace the links to Read the Docs to pages.torproject.net and add redirect to it. Closes: #40085. --- .gitlab-ci.yml | 10 ++++++++++ README.rst | 10 +++++----- docs/source/index.rst | 13 +++++++++++++ docs/source/man_sbws.ini.rst | 2 +- docs/source/man_sbws.rst | 2 +- 5 files changed, 30 insertions(+), 7 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 1a9ad10..3708fbf 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -148,3 +148,13 @@ release_job: ref: "$CI_COMMIT_TAG" milestones: - "sbws: 1.1.x-final" + +pages: + stage: deploy + before_script: + - pip install .[doc] + script: + - cd docs && sphinx-build source ../public + artifacts: + paths: + - public diff --git a/README.rst b/README.rst index 8849cac..4fc2ba2 100644 --- a/README.rst +++ b/README.rst @@ -23,10 +23,10 @@ authorities on the public Tor network. Please do not run this software on the public Tor network unless you are one of the Tor bandwidth authorities, to avoid creating unnecessary traffic. -**ADVICE**: It is recommended to read this documentation at `Read the -Docs <https://sbws.rtfd.io>`__. In `Tor Project -Gitlab <https://gitlab.torproject.org/tpo/network-health/sbws>`__ (tpo -Gitlab) some links won’t be properly rendered. It can also be read after +**ADVICE**: It is recommended to read this documentation at +https://tpo.pages.torproject.net/network-health/sbws. +At https://gitlab.torproject.org/tpo/network-health/sbws +some links won’t be properly rendered. It can also be read after installing the Debian package ``sbws-doc`` in ``/usr/share/doc/sbws`` or after building it locally as explained in ``./docs/source/documenting.rst``. @@ -54,7 +54,7 @@ Documentation ------------- More extensive documentation can be found in the ``./docs`` directory, -and online at `sbws.rtfd.io <https://sbws.readthedocs.io>`__. +and online at https://tpo.pages.torproject.net/network-health/sbws. License ------- diff --git a/docs/source/index.rst b/docs/source/index.rst index 85f8591..9baf73d 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,6 +3,19 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. +.. raw:: html + + <script type="text/javascript"> + if (String(window.location).indexOf("readthedocs") !== -1) { + window.alert('The documentation has moved, redirecting...'); + window.location.replace('https://tpo.pages.torproject.net/network-health/sbws'); + } + </script> + <noscript> + NOTE: documentation has moved from https://sbws.readthedocs.org to + https://tpo.pages.torproject.net/network-health/sbws + </noscript> + Welcome to Simple Bandwidth Scanner's documentation! ==================================================== diff --git a/docs/source/man_sbws.ini.rst b/docs/source/man_sbws.ini.rst index 31effc7..071d31f 100644 --- a/docs/source/man_sbws.ini.rst +++ b/docs/source/man_sbws.ini.rst @@ -230,7 +230,7 @@ Any other path to the configuration file can be specified using the SEE ALSO --------- -**sbws** (1), https://sbws.readthedocs.org. +**sbws** (1), https://tpo.pages.torproject.net/network-health/sbws. BUGS ---- diff --git a/docs/source/man_sbws.rst b/docs/source/man_sbws.rst index 3586cef..6a10a6d 100644 --- a/docs/source/man_sbws.rst +++ b/docs/source/man_sbws.rst @@ -107,7 +107,7 @@ In the following list, the first path is the default location when running SEE ALSO --------- -**sbws.ini** (5), https://sbws.readthedocs.org, +**sbws.ini** (5), https://tpo.pages.torproject.net/network-health/sbws, https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-spec.txt, **tor** (1).

1 0

[sbws/main] Merge branch 'm12'
by juga＠torproject.org 25 May '21

25 May '21

commit 77ea7105953bbf20cad284fcf247c80b1af67c8f Merge: 1089053 f3c07f2 Author: Georg Koppen <gk(a)torproject.org> Date: Tue May 4 13:06:08 2021 +0000 Merge branch 'm12' sbws/globals.py | 2 +- sbws/lib/resultdump.py | 4 ++-- tests/integration/conftest.py | 1 - tests/unit/conftest.py | 1 - 4 files changed, 3 insertions(+), 5 deletions(-)

1 0

[sbws/main] fix: Update python version for rtfd.io
by juga＠torproject.org 25 May '21

25 May '21

commit 51a90173b3c38e57b7af5679a7b7324bbe0d0001 Author: juga0 <juga(a)riseup.net> Date: Tue Apr 20 11:38:20 2021 +0000 fix: Update python version for rtfd.io --- .readthedocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.readthedocs.yml b/.readthedocs.yml index a052595..701a824 100644 --- a/.readthedocs.yml +++ b/.readthedocs.yml @@ -4,7 +4,7 @@ build: image: latest python: - version: 3.5 + version: 3.7 # To run "pip install ." in rtfd.io pip_install: true

1 0

[sbws/main] Merge branch 'm12'
by juga＠torproject.org 25 May '21

25 May '21

commit 10890538003f2b39b24d63cf53fde158c2edda6d Merge: b9b5c48 51a9017 Author: Georg Koppen <gk(a)torproject.org> Date: Tue Apr 27 12:12:20 2021 +0000 Merge branch 'm12' .gitlab-ci.yml | 10 ++++++++++ .readthedocs.yml | 2 +- README.rst | 10 +++++----- docs/source/index.rst | 13 +++++++++++++ docs/source/man_sbws.ini.rst | 2 +- docs/source/man_sbws.rst | 2 +- 6 files changed, 31 insertions(+), 8 deletions(-)

1 0

[sbws/main] fix: Reformat docstrings for black
by juga＠torproject.org 25 May '21

25 May '21

commit f3c07f29f18a25505278d09c62823b2993244cfe Author: juga0 <juga(a)riseup.net> Date: Fri Apr 30 09:18:56 2021 +0000 fix: Reformat docstrings for black To pass tox tests. This seems to have changed in black from version 20.8b1 to 21.4b2. --- sbws/globals.py | 2 +- sbws/lib/resultdump.py | 4 ++-- tests/integration/conftest.py | 1 - tests/unit/conftest.py | 1 - 4 files changed, 3 insertions(+), 5 deletions(-) diff --git a/sbws/globals.py b/sbws/globals.py index 4136970..6954fd1 100644 --- a/sbws/globals.py +++ b/sbws/globals.py @@ -191,7 +191,7 @@ MAX_RECENT_PRIORITY_RELAY_COUNT = ( def fail_hard(*a, **kw): - """ Log something ... and then exit as fast as possible """ + """Log something ... and then exit as fast as possible""" log.critical(*a, **kw) exit(1) diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py index b168cbd..33292d4 100644 --- a/sbws/lib/resultdump.py +++ b/sbws/lib/resultdump.py @@ -191,7 +191,7 @@ def load_recent_results_in_datadir( def write_result_to_datadir(result, datadir): - """ Can be called from any thread """ + """Can be called from any thread""" assert isinstance(result, Result) assert os.path.isdir(datadir) dt = datetime.utcfromtimestamp(result.time) @@ -852,7 +852,7 @@ class ResultDump: fail_hard(e) def store_result(self, result): - """ Call from ResultDump thread """ + """Call from ResultDump thread""" assert isinstance(result, Result) with self.data_lock: fp = result.fingerprint diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index c90a788..9a08fca 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -52,7 +52,6 @@ def sbwshome_dir(sbwshome_empty): @pytest.fixture(scope="session") def test_config_path(tmpdir): - """""" config = tmpdir.join(".sbws.ini") return config diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py index da6a90d..79d8f37 100644 --- a/tests/unit/conftest.py +++ b/tests/unit/conftest.py @@ -207,7 +207,6 @@ def sbwshome_only_datadir(sbwshome_empty): @pytest.fixture(scope="function") def test_config_path(tmpdir): - """""" config = tmpdir.join(".sbws.ini") return config

1 0

[sbws/main] fix: scanner: Increase time getting measurements
by juga＠torproject.org 25 May '21

25 May '21

commit 3653396c9daa9170033fd2b706cd9d290eac6f6b Author: juga0 <juga(a)riseup.net> Date: Thu May 13 12:10:35 2021 +0000 fix: scanner: Increase time getting measurements - Increase the time waiting for the last measurements queued, to avoid canceling unfinished measurements and gc maybe not releasing thread variables - Use the already declared global pool instead of passing it by args - Log more information when the last measuremetns timeout Closes: #40087 --- sbws/core/scanner.py | 30 +++++++++++++++++++++++++----- setup.cfg | 1 + 2 files changed, 26 insertions(+), 5 deletions(-) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index dbcceb9..7637028 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -15,7 +15,12 @@ from multiprocessing.dummy import Pool import sbws.util.requests as requests_utils import sbws.util.stem as stem_utils -from sbws.globals import HTTP_GET_HEADERS, TIMEOUT_MEASUREMENTS, fail_hard +from sbws.globals import ( + HTTP_GET_HEADERS, + SOCKET_TIMEOUT, + TIMEOUT_MEASUREMENTS, + fail_hard, +) from .. import settings from ..lib.circuitbuilder import GapsCircuitBuilder as CB @@ -77,7 +82,7 @@ def dumpstacks(): log.critical( "Thread: %s(%d)", thread_id2name.get(thread_id, ""), thread_id ) - log.critical(traceback.format_stack("".join(stack))) + log.critical("Traceback: %s", "".join(traceback.format_stack(stack))) # If logging level is less than DEBUG (more verbose), start pdb so that # developers can debug the issue. if log.getEffectiveLevel() < logging.DEBUG: @@ -660,7 +665,6 @@ def main_loop( result_dump, relay_prioritizer, destinations, - pool, ): """Starts and reuse the threads that measure the relays forever. @@ -696,6 +700,7 @@ def main_loop( measured. """ + global pool log.info("Started the main loop to measure the relays.") hbeat = Heartbeat(conf.getpath("paths", "state_fname")) @@ -743,6 +748,7 @@ def main_loop( # Register this measurement to the heartbeat module hbeat.register_measured_fpr(target.fingerprint) + log.debug("Measurements queued.") # After the for has finished, the pool has queued all the relays # and pending_results has the list of all the AsyncResults. # It could also be obtained with pool._cache, which contains @@ -815,6 +821,7 @@ def wait_for_results(num_relays_to_measure, pending_results): len(pending_results), num_relays_to_measure, ) + log.info("Last measured: %s", num_last_measured) time.sleep(TIMEOUT_MEASUREMENTS) old_pending_results = pending_results pending_results = [r for r in pending_results if not r.ready()] @@ -836,15 +843,28 @@ def force_get_results(pending_results): ``get`` is not call before, because it blocks and the callbacks are not call. """ + global pool log.debug("Forcing get") + # In case there are no finished AsyncResults, print the cache here + # at level info so that is visible even if debug is not enabled. + log.info("Pool cache %s", pool._cache) for r in pending_results: try: - result = r.get(timeout=0.1) + # HTTP timeout is 10 + result = r.get(timeout=SOCKET_TIMEOUT + 10) log.warning("Result %s was not stored, it took too long.", result) # TimeoutError is raised when the result is not ready, ie. has not # been processed yet except TimeoutError: log.warning("A result was not stored, it was not ready.") + # This is the only place where using psutil so far. + import psutil + + log.warning(psutil.Process(os.getpid()).memory_full_info()) + virtualMemoryInfo = psutil.virtual_memory() + availableMemory = virtualMemoryInfo.available + log.warning("Memory available %s MB.", availableMemory / 1024 ** 2) + dumpstacks() # If the result raised an exception, `get` returns it, # then log any exception so that it can be fixed. # This should not happen, since `callback_err` would have been call @@ -910,7 +930,7 @@ def run_speedtest(args, conf): max_pending_results = conf.getint("scanner", "measurement_threads") pool = Pool(max_pending_results) try: - main_loop(args, conf, controller, rl, cb, rd, rp, destinations, pool) + main_loop(args, conf, controller, rl, cb, rd, rp, destinations) except KeyboardInterrupt: log.info("Interrupted by the user.") stop_threads(signal.SIGINT, None) diff --git a/setup.cfg b/setup.cfg index 0074a1d..88d4613 100644 --- a/setup.cfg +++ b/setup.cfg @@ -32,6 +32,7 @@ include_package_data = True # See stable releases at https://www.python.org/downloads/ python_requires = >= 3.6 install_requires = + psutil >= 5.5 stem >= 1.7.0 ; # Now versioneer is also needed as dependency versioneer

1 0

[sbws/main] Merge branch 'm12' into main
by juga＠torproject.org 25 May '21

25 May '21

commit 0682250db47b689bd7758f5e9b6571b9ff1019a3 Merge: 77ea710 3653396 Author: juga0 <juga(a)riseup.net> Date: Tue May 25 13:19:20 2021 +0000 Merge branch 'm12' into main sbws/core/scanner.py | 30 +++++++++++++++++++++++++----- setup.cfg | 1 + 2 files changed, 26 insertions(+), 5 deletions(-)

1 0