tor-commits December 2017

tor-commits@lists.torproject.org

14 participants
2422 discussions

[tor/release-0.3.2] Make sure bridges are definitely running before delaying directory fetches
by nickm＠torproject.org 13 Dec '17

13 Dec '17

commit 19a4abf2a99c6a3de2c9a2fdf3e6d7b7c404f8f8 Author: teor <teor2345(a)gmail.com> Date: Mon Dec 11 02:29:05 2017 +1100 Make sure bridges are definitely running before delaying directory fetches Retry directory downloads when we get our first bridge descriptor during bootstrap or while reconnecting to the network. Keep retrying every time we get a bridge descriptor, until we have a reachable bridge. Stop delaying bridge descriptor fetches when we have cached bridge descriptors. Instead, only delay bridge descriptor fetches when we have at least one reachable bridge. Fixes bug 24367; bugfix on 0.2.0.3-alpha. --- changes/bug24367 | 16 +++++++++++----- src/or/bridges.c | 10 +++++++--- src/or/directory.c | 9 +++++---- src/or/entrynodes.c | 11 +++++++++-- src/or/entrynodes.h | 2 +- src/or/networkstatus.c | 4 +++- src/test/test_dir.c | 5 +++-- 7 files changed, 39 insertions(+), 18 deletions(-) diff --git a/changes/bug24367 b/changes/bug24367 index 6081f7b5d..09ef3bb87 100644 --- a/changes/bug24367 +++ b/changes/bug24367 @@ -1,7 +1,13 @@ o Minor bugfixes (bridge clients, bootstrap): - - Stop checking for bridge descriptors when we actually want to know if - any bridges are usable. This avoids potential bootstrapping issues. + - Retry directory downloads when we get our first bridge descriptor + during bootstrap or while reconnecting to the network. Keep retrying + every time we get a bridge descriptor, until we have a reachable bridge. + Fixes bug 24367; bugfix on 0.2.0.3-alpha. + - Stop delaying bridge descriptor fetches when we have cached bridge + descriptors. Instead, only delay bridge descriptor fetches when we + have at least one reachable bridge. + Fixes bug 24367; bugfix on 0.2.0.3-alpha. + - Stop delaying directory fetches when we have cached bridge descriptors. + Instead, only delay bridge descriptor fetches when all our bridges are + definitely unreachable. Fixes bug 24367; bugfix on 0.2.0.3-alpha. - - Stop stalling when bridges are changed at runtime. Stop stalling when - old bridge descriptors are cached, but they are not in use. - Fixes bug 24367; bugfix on 23347 in 0.3.2.1-alpha. diff --git a/src/or/bridges.c b/src/or/bridges.c index ac759493c..0b1bbbd15 100644 --- a/src/or/bridges.c +++ b/src/or/bridges.c @@ -796,7 +796,11 @@ learned_bridge_descriptor(routerinfo_t *ri, int from_cache) tor_assert(ri); tor_assert(ri->purpose == ROUTER_PURPOSE_BRIDGE); if (get_options()->UseBridges) { - int first = num_bridges_usable() <= 1; + /* Retry directory downloads whenever we get a bridge descriptor: + * - when bootstrapping, and + * - when we aren't sure if any of our bridges are reachable. + * Keep on retrying until we have at least one reachable bridge. */ + int first = num_bridges_usable(0) < 1; bridge_info_t *bridge = get_configured_bridge_by_routerinfo(ri); time_t now = time(NULL); router_set_status(ri->cache_info.identity_digest, 1); @@ -826,8 +830,8 @@ learned_bridge_descriptor(routerinfo_t *ri, int from_cache) log_notice(LD_DIR, "new bridge descriptor '%s' (%s): %s", ri->nickname, from_cache ? "cached" : "fresh", router_describe(ri)); - /* set entry->made_contact so if it goes down we don't drop it from - * our entry node list */ + /* If we didn't have a reachable bridge before this one, try directory + * documents again. */ if (first) { routerlist_retry_directory_downloads(now); } diff --git a/src/or/directory.c b/src/or/directory.c index e49fd595c..f14bdde90 100644 --- a/src/or/directory.c +++ b/src/or/directory.c @@ -5337,12 +5337,13 @@ find_dl_schedule(const download_status_t *dls, const or_options_t *options) } } case DL_SCHED_BRIDGE: - /* A bridge client downloading bridge descriptors */ - if (options->UseBridges && num_bridges_usable() > 0) { - /* A bridge client with one or more running bridges */ + if (options->UseBridges && num_bridges_usable(0) > 0) { + /* A bridge client that is sure that one or more of its bridges are + * running can afford to wait longer to update bridge descriptors. */ return options->TestingBridgeDownloadSchedule; } else { - /* A bridge client with no running bridges */ + /* A bridge client which might have no running bridges, must try to + * get bridge descriptors straight away. */ return options->TestingBridgeBootstrapDownloadSchedule; } default: diff --git a/src/or/entrynodes.c b/src/or/entrynodes.c index 100286f10..90b728726 100644 --- a/src/or/entrynodes.c +++ b/src/or/entrynodes.c @@ -3135,13 +3135,16 @@ entry_list_is_constrained(const or_options_t *options) } /** Return the number of bridges that have descriptors that are marked with - * purpose 'bridge' and are running. + * purpose 'bridge' and are running. If use_maybe_reachable is + * true, include bridges that might be reachable in the count. + * Otherwise, if it is false, only include bridges that have recently been + * found running in the count. * * We use this function to decide if we're ready to start building * circuits through our bridges, or if we need to wait until the * directory "server/authority" requests finish. */ MOCK_IMPL(int, -num_bridges_usable,(void)) +num_bridges_usable,(int use_maybe_reachable)) { int n_options = 0; @@ -3154,8 +3157,12 @@ num_bridges_usable,(void)) } SMARTLIST_FOREACH_BEGIN(gs->sampled_entry_guards, entry_guard_t *, guard) { + /* Definitely not usable */ if (guard->is_reachable == GUARD_REACHABLE_NO) continue; + /* If we want to be really sure the bridges will work, skip maybes */ + if (!use_maybe_reachable && guard->is_reachable == GUARD_REACHABLE_MAYBE) + continue; if (tor_digest_is_zero(guard->identity)) continue; const node_t *node = node_get_by_id(guard->identity); diff --git a/src/or/entrynodes.h b/src/or/entrynodes.h index ad6b47936..d909115b1 100644 --- a/src/or/entrynodes.h +++ b/src/or/entrynodes.h @@ -383,7 +383,7 @@ void entry_guards_note_internet_connectivity(guard_selection_t *gs); int update_guard_selection_choice(const or_options_t *options); -MOCK_DECL(int,num_bridges_usable,(void)); +MOCK_DECL(int,num_bridges_usable,(int use_maybe_reachable)); #ifdef ENTRYNODES_PRIVATE /** diff --git a/src/or/networkstatus.c b/src/or/networkstatus.c index 2075fccc9..e0a3e4cdc 100644 --- a/src/or/networkstatus.c +++ b/src/or/networkstatus.c @@ -1209,7 +1209,9 @@ should_delay_dir_fetches(const or_options_t *options, const char **msg_out) } if (options->UseBridges) { - if (num_bridges_usable() == 0) { + /* If we know that none of our bridges can possibly work, avoid fetching + * directory documents. But if some of them might work, try again. */ + if (num_bridges_usable(1) == 0) { if (msg_out) { *msg_out = "No running bridges"; } diff --git a/src/test/test_dir.c b/src/test/test_dir.c index 0b34e8b33..997b110d6 100644 --- a/src/test/test_dir.c +++ b/src/test/test_dir.c @@ -5877,8 +5877,9 @@ mock_networkstatus_consensus_can_use_extra_fallbacks( static int mock_num_bridges_usable_value = 0; static int -mock_num_bridges_usable(void) +mock_num_bridges_usable(int use_maybe_reachable) { + (void)use_maybe_reachable; return mock_num_bridges_usable_value; } @@ -6031,7 +6032,7 @@ test_dir_find_dl_schedule(void* data) /* client */ mock_options->ClientOnly = 1; mock_options->UseBridges = 1; - if (num_bridges_usable() > 0) { + if (num_bridges_usable(0) > 0) { tt_ptr_op(find_dl_schedule(&dls, mock_options), OP_EQ, &bridge); } else { tt_ptr_op(find_dl_schedule(&dls, mock_options), OP_EQ, &bridge_bootstrap);

1 0

[tor/release-0.3.2] Run the download_status_increment test in a forked process.
by nickm＠torproject.org 13 Dec '17

13 Dec '17

commit 6f3a862966cd09b9feae357ada0ad4d51b51c0ea Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Dec 1 16:06:25 2017 -0500 Run the download_status_increment test in a forked process. It messes with global state somehow in a way that makes several of the entryconn tests fail now. --- src/test/test_dir.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/test/test_dir.c b/src/test/test_dir.c index 78bf23608..0b34e8b33 100644 --- a/src/test/test_dir.c +++ b/src/test/test_dir.c @@ -6218,7 +6218,7 @@ struct testcase_t dir_tests[] = { DIR(download_status_schedule, 0), DIR(download_status_random_backoff, 0), DIR(download_status_random_backoff_ranges, 0), - DIR(download_status_increment, 0), + DIR(download_status_increment, TT_FORK), DIR(authdir_type_to_string, 0), DIR(conn_purpose_to_string, 0), DIR(should_use_directory_guards, 0),

1 0

[tor/master] Make sure bridges are definitely running before delaying directory fetches
by nickm＠torproject.org 13 Dec '17

13 Dec '17

1 0

[tor/master] Merge branch 'maint-0.3.2'
by nickm＠torproject.org 13 Dec '17

13 Dec '17

commit 9abef516f609a619b25e8f53a413a2e3df05bde8 Merge: 13455c0f1 bf89b0893 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 12 19:27:14 2017 -0500 Merge branch 'maint-0.3.2' changes/bug24367 | 13 +++++++++++++ src/or/bridges.c | 38 +++++++------------------------------- src/or/bridges.h | 1 - src/or/circuituse.c | 15 ++++++++++++--- src/or/directory.c | 9 +++++---- src/or/entrynodes.c | 26 ++++++++++++++++++++------ src/or/entrynodes.h | 3 +-- src/or/networkstatus.c | 4 +++- src/test/test_dir.c | 22 ++++++++++++---------- 9 files changed, 73 insertions(+), 58 deletions(-) diff --cc src/or/networkstatus.c index 61cca1b9b,e0a3e4cdc..deef62bd4 --- a/src/or/networkstatus.c +++ b/src/or/networkstatus.c @@@ -1209,16 -1208,10 +1209,18 @@@ should_delay_dir_fetches(const or_optio return 1; } + if (we_are_hibernating()) { + if (msg_out) { + *msg_out = "We are hibernating or shutting down."; + } + log_info(LD_DIR, "Delaying dir fetches (Hibernating or shutting down)"); + return 1; + } + if (options->UseBridges) { - if (!any_bridge_descriptors_known()) { + /* If we know that none of our bridges can possibly work, avoid fetching + * directory documents. But if some of them might work, try again. */ + if (num_bridges_usable(1) == 0) { if (msg_out) { *msg_out = "No running bridges"; }

1 0

[stem/master] Add query example to docs
by atagar＠torproject.org 12 Dec '17

12 Dec '17

commit f5480b36cbb26a7387e6e8324a34983a6211aeda Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Dec 12 11:50:39 2017 -0800 Add query example to docs Blogged an example of using this function when we added it... http://blog.atagar.com/august2017/ Handy example to have in our docs. --- stem/manual.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/stem/manual.py b/stem/manual.py index df55e3cb..c75bfce8 100644 --- a/stem/manual.py +++ b/stem/manual.py @@ -130,6 +130,12 @@ def query(query, *param): be treated as being read-only. File permissions generally enforce this, and in the future will be enforced by this function as well. + :: + + >>> import stem.manual + >>> print(stem.manual.query('SELECT description FROM torrc WHERE key=?', 'CONTROLSOCKET').fetchone()[0]) + Like ControlPort, but listens on a Unix domain socket, rather than a TCP socket. 0 disables ControlSocket (Unix and Unix-like systems only.) + .. versionadded:: 1.6.0 :param str query: query to run on the cache

1 0

[stem/master] Fix controller sphinx errors
by atagar＠torproject.org 12 Dec '17

12 Dec '17

commit a53ec0cddde92aab452664f3824fa3e9217ea50d Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Dec 12 11:57:57 2017 -0800 Fix controller sphinx errors Couple formatting errors that caused our docs to be malformed... /home/atagar/Desktop/stem/stem/control.py:docstring of stem.control.CreateHiddenServiceOutput:12: WARNING: Definition list ends without a blank line; unexpected unindent. /home/atagar/Desktop/stem/stem/control.py:docstring of stem.control.Controller.create_ephemeral_hidden_service:45: WARNING: Definition list ends without a blank line; unexpected unindent. --- stem/control.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/stem/control.py b/stem/control.py index 1b719dd0..a4297677 100644 --- a/stem/control.py +++ b/stem/control.py @@ -453,7 +453,7 @@ class CreateHiddenServiceOutput(collections.namedtuple('CreateHiddenServiceOutpu :var str path: hidden service directory :var str hostname: content of the hostname file if available - :var dict hostname_for_client:mapping of client names to their onion address + :var dict hostname_for_client: mapping of client names to their onion address if available :var dict config: tor's new hidden service configuration """ @@ -2847,6 +2847,8 @@ class Controller(BaseController): ... while both alice and bob can access with existing credentials in the following... + :: + controller.create_ephemeral_hidden_service(80, basic_auth = { 'alice': 'l4BT016McqV2Oail+Bwe6w', 'bob': 'vGnNRpWYiMBFTWD2gbBlcA',

1 0

[stem/master] Add multiprocessing to tutorials
by atagar＠torproject.org 12 Dec '17

12 Dec '17

commit c458fcf9fafa727b9fb5b722c9f0dec2152c756a Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Dec 12 11:48:43 2017 -0800 Add multiprocessing to tutorials Ooh, this is a good candidate for our tutorials. Very handy module. Blogged these examples a few months back... http://blog.atagar.com/august2017/ --- docs/_static/example/fibonacci_multiprocessing.py | 20 +++++++++++++ docs/_static/example/fibonacci_threaded.py | 24 ++++++++++++++++ docs/tutorials/east_of_the_sun.rst | 35 +++++++++++++++++++++++ 3 files changed, 79 insertions(+) diff --git a/docs/_static/example/fibonacci_multiprocessing.py b/docs/_static/example/fibonacci_multiprocessing.py new file mode 100644 index 00000000..e909e169 --- /dev/null +++ b/docs/_static/example/fibonacci_multiprocessing.py @@ -0,0 +1,20 @@ +import stem.util.system +import time + +def fibonacci(n): + if n < 2: + return n + else: + return fibonacci(n-2) + fibonacci(n-1) + +# calculate fibonacci sequences four times in parallel + +start_time, threads = time.time(), [] + +for i in range(4): + threads.append(stem.util.system.DaemonTask(fibonacci, (35,), start = True)) + +for t in threads: + t.join() + +print('took %0.1f seconds' % (time.time() - start_time)) diff --git a/docs/_static/example/fibonacci_threaded.py b/docs/_static/example/fibonacci_threaded.py new file mode 100644 index 00000000..f40b7401 --- /dev/null +++ b/docs/_static/example/fibonacci_threaded.py @@ -0,0 +1,24 @@ +import threading +import time + +def fibonacci(n): + if n < 2: + return n + else: + return fibonacci(n-2) + fibonacci(n-1) + +# calculate fibonacci sequences four times in parallel + +start_time, threads = time.time(), [] + +for i in range(4): + t = threading.Thread(target = fibonacci, args = (35,)) + t.setDaemon(True) + t.start() + + threads.append(t) + +for t in threads: + t.join() + +print('took %0.1f seconds' % (time.time() - start_time)) diff --git a/docs/tutorials/east_of_the_sun.rst b/docs/tutorials/east_of_the_sun.rst index 845cbf4a..fe09c4d3 100644 --- a/docs/tutorials/east_of_the_sun.rst +++ b/docs/tutorials/east_of_the_sun.rst @@ -4,6 +4,7 @@ East of the Sun & West of the Moon The following is an overview of some of the utilities Stem provides. * :ref:`terminal-styling` +* :ref:`multiprocessing` * :ref:`connection-resolution` .. _terminal-styling: @@ -28,6 +29,39 @@ yourself is easy, but we also provide a module to make it `even easier .. literalinclude:: /_static/example/words_with.py :language: python +.. _multiprocessing: + +Multiprocessing +--------------- + +Python's `multiprocessing module +<https://docs.python.org/2/library/multiprocessing.html>`_ gives building +blocks to parallelize around the `Global Interpreter Lock +<https://en.wikipedia.org/wiki/Global_interpreter_lock>`_. However, honestly +it's clunky to use. + +Ever just wanted to simply turn your threads into subprocesses? `We can do +that <../api/util/system.html#stem.util.system.DaemonTask>`_. + +**Threaded** + +.. literalinclude:: /_static/example/fibonacci_threaded.py + :language: python + +:: + + % python fibonacci_threaded.py + took 21.1 seconds + +**Multi-process** + +.. literalinclude:: /_static/example/fibonacci_multiprocessing.py + :language: python + +:: + + % python fibonacci_multiprocessing.py + took 6.2 seconds .. _connection-resolution: Connection Resolution @@ -69,3 +103,4 @@ simple script that dumps Tor's present connections. 192.168.0.1:59014 => 38.229.79.2:443 192.168.0.1:58822 => 68.169.35.102:443 +

1 0

[stem/master] Unused variable in example
by atagar＠torproject.org 12 Dec '17

12 Dec '17

commit 93d92a0307810dcc7ab1c7c640d19182e86cb06b Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Dec 12 09:54:09 2017 -0800 Unused variable in example Oops, forgot to remove this while fiddling with the example. --- docs/_static/example/words_with.py | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/_static/example/words_with.py b/docs/_static/example/words_with.py index b7956cb2..ae616f9e 100644 --- a/docs/_static/example/words_with.py +++ b/docs/_static/example/words_with.py @@ -15,7 +15,6 @@ def get_words_with(target, attr): :returns: **iterable** with words containing that substring """ - matches = [] word_matcher = re.compile('(.*)(%s)(.*)' % target, re.I) with open('/etc/dictionaries-common/words') as dictionary_file:

1 0

[tor/release-0.3.2] Merge remote-tracking branch 'dgoulet/bug24595_032_02' into maint-0.3.2
by nickm＠torproject.org 12 Dec '17

12 Dec '17

commit cf5ab5934e80c6c04825d9db83329627fe6b9729 Merge: 4c54021ba b6fd78ea3 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 12 09:32:24 2017 -0500 Merge remote-tracking branch 'dgoulet/bug24595_032_02' into maint-0.3.2 src/or/hs_service.c | 50 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 33 insertions(+), 17 deletions(-)

1 0

[tor/release-0.3.2] Merge branch 'maint-0.3.2' into release-0.3.2
by nickm＠torproject.org 12 Dec '17

12 Dec '17

commit dd5ca2505599dee6a589e228b69c4e8d4f29d7e6 Merge: 706e6893d cf5ab5934 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 12 09:32:29 2017 -0500 Merge branch 'maint-0.3.2' into release-0.3.2 src/or/hs_service.c | 50 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 33 insertions(+), 17 deletions(-)

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits December 2017