July 2020 - tor-commits - lists.torproject.org

[sbws/maint-1.1] Vote on the relays with few or close measurements
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 09905b6c65712fcb58d321a0e5425e5f17e7b91d Author: juga0 <juga(a)riseup.net> Date: Tue Jun 23 12:07:29 2020 +0000 Vote on the relays with few or close measurements to vote on approximately the same numbers of relays as Torflow. Torflow does not exclude relays with few or close measurements, though it is possible that because of the way it measures, there are no few or close measurements. Closes #34393 --- sbws/globals.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/sbws/globals.py b/sbws/globals.py index 01a147b..575a19f 100644 --- a/sbws/globals.py +++ b/sbws/globals.py @@ -81,8 +81,17 @@ TORFLOW_OBS_MEAN = 1 TORFLOW_OBS_DECAYING = 3 TORFLOW_ROUND_DIG = 3 PROP276_ROUND_DIG = 2 -DAY_SECS = 86400 -NUM_MIN_RESULTS = 2 +# Number of seconds the measurements for a relay have to be distant from each +# other, otherwise the relay would be excluded from the relays to vote on. +# Ideally, this should be 86400 seconds (1 day). +# To have sbws vote on approximately the same number of relays as Torflow, +# leave it as None, to not exclude measurements. +DAY_SECS = None +# Minimum number of measurements for a relay to be included as a relay to vote +# on. +# Ideally, this should be 2. +# As the constant before, leave it as 1 to not exclude measurements. +NUM_MIN_RESULTS = 1 MIN_REPORT = 60 # Maximum difference between the total consensus bandwidth and the total in # in the bandwidth lines in percentage

1 0

[sbws/maint-1.1] fix: doc: Recommend using a CDN,
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 77c65cf55f5a55227bbd3506bdd7b65368faf866 Author: juga0 <juga(a)riseup.net> Date: Fri Jun 12 14:59:48 2020 +0000 fix: doc: Recommend using a CDN, add link to it and rephrase some sentences. --- DEPLOY.rst | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/DEPLOY.rst b/DEPLOY.rst index a1ab4dd..45e9d31 100644 --- a/DEPLOY.rst +++ b/DEPLOY.rst @@ -30,6 +30,9 @@ destination requirements - Bandwidth: at least 12.5MB/s (100 Mbit/s). - Network traffic: around 12-15GB/day. +If possible, use a `Content delivery network`_ (CDN) in order to make the +destination IP closer to the scanner exit. + scanner setup ---------------------- @@ -38,17 +41,20 @@ or `<INSTALL.html>`_ (local build or Read the Docs). To run the ``scanner`` it is mandatory to create a configuration file with at least one ``destination``. -It is recommended to set several ``destination``s so that the ``scanner`` can +It is recommended to set several ``destinations`` so that the ``scanner`` can continue if one fails. -If ``sbws`` is installed from the Debian package, then create a file in -``/etc/sbws/sbws.ini`` like in the following example: +If ``sbws`` is installed from the Debian package, then create the configuration +file in ``/etc/sbws/sbws.ini``. +You can see an example with all the possible options here, note that you don't +need to include all of that and that everything that starts with ``#`` and +``;`` is a comment: .. literalinclude:: /examples/sbws.example.ini :caption: Example sbws.example.ini If ``sbws`` is installed from the sources as a non-root user then create the -file in ``~/.sbws.ini``. +configuration file in ``~/.sbws.ini``. More details about the configuration file can be found in ``./docs/source/man_sbws.ini.rst`` (in the local directory or GitHub) or @@ -58,3 +64,5 @@ More details about the configuration file can be found in See also ``./docs/source/man_sbws.rst`` (in the local directory or GitHub) or `<man_sbws.html>`_ (local build or Read the Docs) or ``man sbws`` (system package). + +.. _Content delivery network: https://en.wikipedia.org/wiki/Content_delivery_network

1 0

[sbws/maint-1.1] fix: CI: Add .gitlab-ci.yml to run tests in Gitlab
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 8f633943f07ed71982cc95279fe1410012bafbeb Author: juga0 <juga(a)riseup.net> Date: Wed Jul 15 09:25:33 2020 +0000 fix: CI: Add .gitlab-ci.yml to run tests in Gitlab Closes: #40007 --- .gitlab-ci.yml | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tox.ini | 8 ++-- 2 files changed, 120 insertions(+), 3 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000..9b4fe40 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,115 @@ +variables: + BASE_IMAGE: python:3.7 + RELEASE: tor-nightly-master-buster + # Without version, the default available in the Debian repository will be + # installed. + # Specifying which version starts with will install the highest that start + # with that version. + TOR: tor=* + PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip" + +cache: + paths: + - .cache/pip + +image: $BASE_IMAGE + +before_script: + - "wget https://deb.torproject.org/torproject.org/\ + A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc" + - cat A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc | apt-key add - + - echo deb [signed-by=A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89] + http://deb.torproject.org/torproject.org $RELEASE + main >> /etc/apt/sources.list + - apt update -yqq + - apt install -yqq $TOR + - pip install tox + - python --version + - tor --version + +after_script: + - tox -e stats + +python36tor035: + variables: + BASE_IMAGE: python:3.6 + RELEASE: tor-nightly-0.3.5.x-buster + TOR: tor=0.3.5* + image: $BASE_IMAGE + script: + - tox -e py36 + - tox -e integration + +python36tor041: + variables: + BASE_IMAGE: python:3.6 + RELEASE: tor-nightly-0.4.1.x-buster + TOR: tor=0.4.1* + image: $BASE_IMAGE + script: + - tox -e py36 + - tox -e integration + +python36tor042: + variables: + BASE_IMAGE: python:3.6 + RELEASE: tor-nightly-0.4.2.x-buster + TOR: tor=0.4.2* + image: $BASE_IMAGE + script: + - tox -e py36 + - tox -e integration + +python36torstable: + variables: + BASE_IMAGE: python:3.6 + RELEASE: buster + TOR: tor + image: $BASE_IMAGE + script: + - tox -e py36 + - tox -e integration + +python37: + # This will overwrite the default before_script, so need to repeat the + # commands + before_script: + - "wget https://deb.torproject.org/torproject.org/\ + A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc" + - cat A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc | apt-key add - + - echo deb [signed-by=A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89] + http://deb.torproject.org/torproject.org $RELEASE + main >> /etc/apt/sources.list + - apt update -yqq + - apt install -yqq $TOR + - pip install tox + - python --version + - tor --version + # To build the docs + - apt install -yqq texlive-latex-extra + - apt install -yqq dvipng + image: $BASE_IMAGE + script: + - tox -e inst + - tox -e setup + - tox -e py37 + - tox -e integration + - tox -e lint + - tox -e doc + +python38: + variables: + BASE_IMAGE: python:3.8 + image: $BASE_IMAGE + script: + - tox -e py38 + - tox -e integration + +python39: + variables: + BASE_IMAGE: python:3.9-rc-buster + image: $BASE_IMAGE + script: + - tox -e py39 + - tox -e integration + allow_failure: true diff --git a/tox.ini b/tox.ini index f9d712f..f0aadaf 100644 --- a/tox.ini +++ b/tox.ini @@ -1,6 +1,6 @@ [tox] skip_missing_interpreters = True -envlist = py{36, 37, 38, 39, nightly}, inst, setup, unit, integration, lint, stats, doc +envlist = py{36, 37, 38, 39}, inst, setup, integration, lint, stats, doc [travis] python = @@ -10,7 +10,7 @@ python = 3.9-dev: py39, inst, setup, unit, integration, lint, doc nightly: pynightly, inst, setup, unit, integration, lint, doc -[testenv] +; [testenv] # install_command can be removed when --process-dependency-links is not # needed anymore, and this section commented # install_command = pip install {opts} {packages} @@ -32,7 +32,7 @@ skip_install = True commands = python setup.py install recreate = True -[testenv:unit] +[testenv] deps = .[test] commands = coverage run -a --rcfile={toxinidir}/.coveragerc --source=sbws -m pytest \ @@ -49,6 +49,8 @@ whitelist_externals = mkdir rm commands = + # For some reason .[test] is not copying config.* files + pip install . cp -af {toxinidir}/tests/integration/net {envtmpdir} bash {envtmpdir}/net/start.sh bash -c "time python3 {envtmpdir}/net/wait.py {envtmpdir}/net/{auth,relay,exit}*"

1 0

[sbws/maint-1.1] Merge remote-tracking branch 'gitlab/merge-requests/14' into maint-1.1
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 748cf1c1f347b89eaab625bdebd9314dadaccc8e Merge: c5fe10e a914215 Author: Georg Koppen <gk(a)torproject.org> Date: Fri Jul 10 23:10:23 2020 +0000 Merge remote-tracking branch 'gitlab/merge-requests/14' into maint-1.1 sbws/lib/v3bwfile.py | 30 ++++++++++++++++++------------ tests/unit/lib/data/results.txt | 2 +- 2 files changed, 19 insertions(+), 13 deletions(-)

1 0

[sbws/maint-1.1] Merge remote-tracking branch 'gitlab/merge-requests/18' into maint-1.1
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 40754a337887cd5c841fe07e6448e1e46f0dc1cc Merge: 16caca1 d958915 Author: Georg Koppen <gk(a)torproject.org> Date: Tue Jul 21 08:47:33 2020 +0000 Merge remote-tracking branch 'gitlab/merge-requests/18' into maint-1.1 sbws/lib/relaylist.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)

1 0

[sbws/maint-1.1] Merge remote-tracking branch 'gitlab/merge-requests/19' into maint-1.1
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 16caca1040b2120ff0381a74763fc80516795e9e Merge: 748cf1c 09905b6 Author: Georg Koppen <gk(a)torproject.org> Date: Tue Jul 21 08:46:28 2020 +0000 Merge remote-tracking branch 'gitlab/merge-requests/19' into maint-1.1 sbws/globals.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)

1 0

[sbws/maint-1.1] Merge remote-tracking branch 'gitlab/merge-requests/20' into maint-1.1
by juga＠torproject.org 22 Jul '20

22 Jul '20

commit 3fcc5892cf33b7920507edd9cca4c76ca5d1a1a1 Merge: 40754a3 8f63394 Author: Georg Koppen <gk(a)torproject.org> Date: Tue Jul 21 08:50:08 2020 +0000 Merge remote-tracking branch 'gitlab/merge-requests/20' into maint-1.1 .gitlab-ci.yml | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tox.ini | 8 ++-- 2 files changed, 120 insertions(+), 3 deletions(-)

1 0

[tor/master] relay: Remove possible ORPorts duplicate from parsed list
by nickm＠torproject.org 22 Jul '20

22 Jul '20

commit 803e769fb255f39dba07b5dbc2ca62a43b43004c Author: David Goulet <dgoulet(a)torproject.org> Date: Tue Jul 21 11:34:13 2020 -0400 relay: Remove possible ORPorts duplicate from parsed list Now that tor automatically binds to IPv4 _and_ IPv6, in order to avoid breaking configurations, we sanitize the parsed lists for duplicate ORPorts. It is possible to happen because we still allow this configuration; ORPort 9888 ORPort [4242::1]:9888 Meaning that the first ORPort value will bind to 0.0.0.0:9888 _and_ [::]:9888 which would lead to an error when attempting to bind on [4242::1]:9888. However, that configuration is accepted today and thus we must not break it. To remedy, we now sanitize the parsed list and prioritize an ORPort that has an explicit address over the global one. A warning is emitted if such configuration pattern is found. This is only for the ORPort. Related to #33246 Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- src/feature/relay/relay_config.c | 89 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 88 insertions(+), 1 deletion(-) diff --git a/src/feature/relay/relay_config.c b/src/feature/relay/relay_config.c index cddb031f8f..c8a30d63d1 100644 --- a/src/feature/relay/relay_config.c +++ b/src/feature/relay/relay_config.c @@ -133,12 +133,96 @@ port_warn_nonlocal_ext_orports(const smartlist_t *ports, const char *portname) } SMARTLIST_FOREACH_END(port); } +/** Return a static buffer containing the human readable logging string that + * describes the given port object. */ +static const char * +describe_relay_port(const port_cfg_t *port) +{ + IF_BUG_ONCE(!port) { + return "<null port>"; + } + + static char buf[256]; + const char *type, *addr; + + switch (port->type) { + case CONN_TYPE_OR_LISTENER: + type = "OR"; + break; + case CONN_TYPE_DIR_LISTENER: + type = "Dir"; + break; + case CONN_TYPE_EXT_OR_LISTENER: + type = "ExtOR"; + break; + default: + type = ""; + break; + } + + if (port->explicit_addr) { + addr = fmt_and_decorate_addr(&port->addr); + } else { + addr = ""; + } + + tor_snprintf(buf, sizeof(buf), "%sPort %s%s%d", + type, addr, (strlen(addr) > 0) ? ":" : "", port->port); + return buf; +} + +/** Attempt to find duplicate ORPort that would be superseded by another and + * remove them from the given ports list. This is possible if we have for + * instance: + * + * ORPort 9050 + * ORPort [4242::1]:9050 + * + * First one binds to both v4 and v6 address but second one is specific to an + * address superseding the global bind one. + * + * The following is O(n^2) but it is done at bootstrap or config reload and + * the list is not very long usually. */ +static void +remove_duplicate_orports(smartlist_t *ports) +{ + for (int i = 0; i < smartlist_len(ports); ++i) { + port_cfg_t *current = smartlist_get(ports, i); + + /* Skip non ORPorts. */ + if (current->type != CONN_TYPE_OR_LISTENER) { + continue; + } + + for (int j = 0; j < smartlist_len(ports); ++j) { + port_cfg_t *next = smartlist_get(ports, j); + + /* Avoid comparing the same object. */ + if (current == next) { + continue; + } + /* Same address family and same port number, we have a match. */ + if (!current->explicit_addr && next->explicit_addr && + tor_addr_family(&current->addr) == tor_addr_family(&next->addr) && + current->port == next->port) { + /* Remove current because next is explicitly set. */ + smartlist_del_keeporder(ports, i--); + char *next_str = tor_strdup(describe_relay_port(next)); + log_warn(LD_CONFIG, "Configuration port %s superseded by %s", + describe_relay_port(current), next_str); + tor_free(next_str); + port_cfg_free(current); + } + } + } +} + /** Given a list of port_cfg_t in ports, check them for internal * consistency and warn as appropriate. On Unix-based OSes, set * *n_low_ports_out to the number of sub-1024 ports we will be * binding, and warn if we may be unable to re-bind after hibernation. */ static int -check_server_ports(const smartlist_t *ports, +check_server_ports(smartlist_t *ports, const or_options_t *options, int *n_low_ports_out) { @@ -159,6 +243,9 @@ check_server_ports(const smartlist_t *ports, int n_low_port = 0; int r = 0; + /* Remove possible duplicate ORPorts before inspecting the list. */ + remove_duplicate_orports(ports); + SMARTLIST_FOREACH_BEGIN(ports, const port_cfg_t *, port) { if (port->type == CONN_TYPE_DIR_LISTENER) { if (! port->server_cfg.no_advertise)

1 0

[tor/master] relay: Change router_can_extend_over_ipv6() to look at configured port
by nickm＠torproject.org 22 Jul '20

22 Jul '20

commit 28c1b6047649dad62d269e6c2fde40f41cb9834a Author: David Goulet <dgoulet(a)torproject.org> Date: Tue Jul 21 13:11:28 2020 -0400 relay: Change router_can_extend_over_ipv6() to look at configured port In routerconf_find_ipv6_or_ap(), we check if the returned ORPort is internal but not for listening. This means that IPv6 [::] is considered internal. Thus, we can't use it, we have to look directly at the configured address and port and if they are valid, we do consider that we have a valid IPv6 ORPort and that we can thus extend in IPv6. Related #33246 Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- src/feature/relay/router.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/src/feature/relay/router.c b/src/feature/relay/router.c index 997be643dc..97a77f9578 100644 --- a/src/feature/relay/router.c +++ b/src/feature/relay/router.c @@ -1526,9 +1526,16 @@ routerconf_find_ipv6_or_ap(const or_options_t *options, bool routerconf_has_ipv6_orport(const or_options_t *options) { - tor_addr_port_t ipv6_ap; - routerconf_find_ipv6_or_ap(options, &ipv6_ap); - return tor_addr_port_is_valid_ap(&ipv6_ap, 0); + /* What we want here is to learn if we have configured an IPv6 ORPort. + * Remember, ORPort can listen on [::] and thus consider internal by + * router_get_advertised_ipv6_or_ap() since we do _not_ want to advertise + * such address. */ + const tor_addr_t *addr = + portconf_get_first_advertised_addr(CONN_TYPE_OR_LISTENER, AF_INET6); + const uint16_t port = + routerconf_find_or_port(options, AF_INET6); + + return tor_addr_port_is_valid(addr, port, 1); } /** Returns true if this router can extend over IPv6.

1 0

[tor/master] Rename check_server_ports to check_and_prune_server_ports
by nickm＠torproject.org 22 Jul '20

22 Jul '20

commit a4c5b7d7421b9f02d479e4471b527ee3b9285d8c Author: David Goulet <dgoulet(a)torproject.org> Date: Tue Jul 21 15:15:06 2020 -0400 Rename check_server_ports to check_and_prune_server_ports This is an automated commit, generated by this command: ./scripts/maint/rename_c_identifier.py \ check_server_ports check_and_prune_server_ports --- src/feature/relay/relay_config.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/feature/relay/relay_config.c b/src/feature/relay/relay_config.c index c8a30d63d1..9273a7cef0 100644 --- a/src/feature/relay/relay_config.c +++ b/src/feature/relay/relay_config.c @@ -222,7 +222,7 @@ remove_duplicate_orports(smartlist_t *ports) * *n_low_ports_out to the number of sub-1024 ports we will be * binding, and warn if we may be unable to re-bind after hibernation. */ static int -check_server_ports(smartlist_t *ports, +check_and_prune_server_ports(smartlist_t *ports, const or_options_t *options, int *n_low_ports_out) { @@ -382,7 +382,7 @@ port_parse_ports_relay(or_options_t *options, goto err; } - if (check_server_ports(ports, options, &n_low_ports) < 0) { + if (check_and_prune_server_ports(ports, options, &n_low_ports) < 0) { *msg = tor_strdup("Misconfigured server ports"); goto err; }

1 0