May 2018 - tor-commits - lists.torproject.org

[doctor/master] Report fallback directory DirPort failures
by atagar＠torproject.org 27 May '18

27 May '18

commit 1e61af0700112e11effe7619fe40b1b416b153e4 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun May 27 12:17:54 2018 -0700 Report fallback directory DirPort failures The whole point of our fallback directory daemon is to tell us how many fallbacks are unusable, but now that one of their DirPorts is misbehaving it's resulting in uncaught exceptions. Interestingly, we're getting quite a range... Traceback (most recent call last): File "fallback_directories.py", line 82, in <module> main() File "fallback_directories.py", line 57, in main downloader.get_consensus(endpoints = [(relay.address, relay.dir_port)]).run() File "/home/atagar/Desktop/tor/doctor/stem/descriptor/remote.py", line 445, in run return list(self._run(suppress)) File "/home/atagar/Desktop/tor/doctor/stem/descriptor/remote.py", line 456, in _run raise self.error HTTPError: HTTP Error 404: Not Found ... as well as... HTTPError: HTTP Error 503: Directory busy, try again later ... or... CertificateError: hostname '85.214.62.48' doesn't match either of 'andamur.com', 'www.andamur.com' Our DescriptorDownloader raises an unpleasantly large array of exceptions so unfortunately requires a catch-all here (maybe I'll change that at some point in stem...). --- fallback_directories.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/fallback_directories.py b/fallback_directories.py index 3e4ce06..073fc85 100755 --- a/fallback_directories.py +++ b/fallback_directories.py @@ -53,10 +53,14 @@ def main(): issues.append('%s => IPv6 ORPort is unreachable (%s:%i)' % (relay.fingerprint, relay.orport_v6[0], relay.orport_v6[1])) continue - start = time.time() - downloader.get_consensus(endpoints = [(relay.address, relay.dir_port)]).run() - download_time = time.time() - start - log.info('%s download time was %0.1f seconds' % (relay.fingerprint, download_time)) + try: + start = time.time() + downloader.get_consensus(endpoints = [(relay.address, relay.dir_port)]).run() + download_time = time.time() - start + log.info('%s download time was %0.1f seconds' % (relay.fingerprint, download_time)) + except Exception as exc: + issues.append('%s => Unable to download from DirPort (%s)' % (relay.fingerprint, exc)) + continue if download_time > 15: issues.append('%s => Downloading the consensus took %0.1f seconds' % (relay.fingerprint, download_time))

1 0

[stem/master] Descriptions of new CIRC_BW fields
by atagar＠torproject.org 27 May '18

27 May '18

commit 519991aee05f15d42ba938063a674e87d5305e8c Author: Damian Johnson <atagar(a)torproject.org> Date: Sun May 27 12:01:13 2018 -0700 Descriptions of new CIRC_BW fields Thanks to clarifications from Mike on... https://trac.torproject.org/projects/tor/ticket/26110 --- stem/response/events.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/stem/response/events.py b/stem/response/events.py index a50d90f8..5623d1a7 100644 --- a/stem/response/events.py +++ b/stem/response/events.py @@ -1229,10 +1229,10 @@ class CircuitBandwidthEvent(Event): :var str id: circuit identifier :var int read: bytes received by tor that second :var int written: bytes sent by tor that second - :var int delivered_read: description pending :trac:`26110` - :var int delivered_written: description pending :trac:`26110` - :var int overhead_read: description pending :trac:`26110` - :var int overhead_written: description pending :trac:`26110` + :var int delivered_read: user payload received by tor that second + :var int delivered_written: user payload sent by tor that second + :var int overhead_read: padding so read cells will have a fixed length + :var int overhead_written: padding so written cells will have a fixed length :var datetime time: time when the measurement was recorded """

1 0

[torspec/master] Attempt to clarify new CIRC_BW fields
by nickm＠torproject.org 27 May '18

27 May '18

commit 06cbab3eae78356c8c51d61c37bfdfe7518dbfbc Author: Damian Johnson <atagar(a)torproject.org> Date: Tue May 15 11:32:34 2018 -0700 Attempt to clarify new CIRC_BW fields I didn't really follow the last description so trying one of my own... https://trac.torproject.org/projects/tor/ticket/26110 Note that this description likely is *not* accurate. I didn't dig through tor's source to puzzle out what these fields are. Rather, this is a more grokable description of what I *think* the fields are. On the above ticket the Network team and I will work together on a description that's both easily understood and accurate. --- control-spec.txt | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/control-spec.txt b/control-spec.txt index 72ceb15..d78b7ec 100644 --- a/control-spec.txt +++ b/control-spec.txt @@ -2945,15 +2945,13 @@ BytesRead and BytesWritten are the number of bytes read and written on this circuit since the last CIRC_BW event. - DeliveredBytesRead and DeliveredBytesWritten are the total length of - all valid relay cell payloads read and written on this circuit since - the last CIRC_BW event. These fields are provided only in Tor - 3.4.0-alpha and later. + Circuit data falls into two categories: delivered payloads and overhead. + DeliveredBytesRead and DeliveredBytesWritten are the total payloads + transmitted since the last CIRC_BW event, whereas OverheadBytesRead and + OverheadBytesWritten are the metadata size used for routing. - OverheadBytesRead and OverheadBytesWritten are the total length of - the unused overhead portion of valid relay cells read and written on - this circuit since the last CIRC_BW event. These fields are provided - only in Tor 0.3.4.0-alpha and later. + The sum of DeliveredBytesRead and OverheadBytesRead SHOULD match BytesRead, + and same for their written counterparts. The Time field is provided only in versions 0.3.2.1-alpha and later. It records when Tor created the bandwidth event. @@ -2964,6 +2962,9 @@ [First added in 0.2.5.2-alpha] + [DELIVERED_READ, OVERHEAD_READ, DELIVERED_WRITTEN, and OVERHEAD_WRITTEN was + added in Tor 0.3.4.0-alpha] + 4.1.23. Per-circuit cell stats The syntax is:

1 0

[torspec/master] fixup! Attempt to clarify new CIRC_BW fields
by nickm＠torproject.org 27 May '18

27 May '18

commit db81ae54b5f174972f47a3a6575db31ae6b28c0c Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Wed May 16 04:53:29 2018 +0000 fixup! Attempt to clarify new CIRC_BW fields Correct a couple statements in the clarification, and attempt to clarify further. --- control-spec.txt | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/control-spec.txt b/control-spec.txt index d78b7ec..e448918 100644 --- a/control-spec.txt +++ b/control-spec.txt @@ -2943,15 +2943,23 @@ Time = ISOTime2Frac BytesRead and BytesWritten are the number of bytes read and written - on this circuit since the last CIRC_BW event. + on this circuit since the last CIRC_BW event. These bytes have not + necessarily been validated by Tor, and can include invalid cells, + dropped cells, and ignored cells (such as padding cells). - Circuit data falls into two categories: delivered payloads and overhead. + Circuit data that has been validated and processed by Tor is further + broken down into two categories: delivered payloads and overhead. DeliveredBytesRead and DeliveredBytesWritten are the total payloads transmitted since the last CIRC_BW event, whereas OverheadBytesRead and - OverheadBytesWritten are the metadata size used for routing. + OverheadBytesWritten are the extra unused bytes at the end of each + cell in order for it to be the fixed CELL_LEN bytes long. - The sum of DeliveredBytesRead and OverheadBytesRead SHOULD match BytesRead, - and same for their written counterparts. + The sum of DeliveredBytesRead and OverheadBytesRead MUST be less than + or equal to BytesRead, and the same is true for their written + counterparts. This sum represents the total bytes on the circuit that + have been validated by Tor. Subtracting this sum from the BytesRead + (or BytesWritten) value gives the byte count that Tor has decided + to reject due to protocol errors, or has otherwise decided to ignore. The Time field is provided only in versions 0.3.2.1-alpha and later. It records when Tor created the bandwidth event. @@ -2962,8 +2970,8 @@ [First added in 0.2.5.2-alpha] - [DELIVERED_READ, OVERHEAD_READ, DELIVERED_WRITTEN, and OVERHEAD_WRITTEN was - added in Tor 0.3.4.0-alpha] + [DELIVERED_READ, OVERHEAD_READ, DELIVERED_WRITTEN, and OVERHEAD_WRITTEN + were added in Tor 0.3.4.0-alpha] 4.1.23. Per-circuit cell stats

1 0

[torspec/master] fixup! Attempt to clarify new CIRC_BW fields
by nickm＠torproject.org 27 May '18

27 May '18

commit 5a9cd80a98fa7f8379231862e0d1714a0779a960 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Wed May 16 05:58:42 2018 +0000 fixup! Attempt to clarify new CIRC_BW fields Also mention that the READ and WRITE fields do not include circuit headers, and the delivered and overhead fields do not include relay cell headers. --- control-spec.txt | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/control-spec.txt b/control-spec.txt index e448918..30160f4 100644 --- a/control-spec.txt +++ b/control-spec.txt @@ -2945,21 +2945,24 @@ BytesRead and BytesWritten are the number of bytes read and written on this circuit since the last CIRC_BW event. These bytes have not necessarily been validated by Tor, and can include invalid cells, - dropped cells, and ignored cells (such as padding cells). + dropped cells, and ignored cells (such as padding cells). These + values include the relay headers, but not circuit headers. Circuit data that has been validated and processed by Tor is further broken down into two categories: delivered payloads and overhead. - DeliveredBytesRead and DeliveredBytesWritten are the total payloads - transmitted since the last CIRC_BW event, whereas OverheadBytesRead and + DeliveredBytesRead and DeliveredBytesWritten are the total relay cell + payloads transmitted since the last CIRC_BW event, not counting relay + cell headers or circuit headers. OverheadBytesRead and OverheadBytesWritten are the extra unused bytes at the end of each cell in order for it to be the fixed CELL_LEN bytes long. The sum of DeliveredBytesRead and OverheadBytesRead MUST be less than - or equal to BytesRead, and the same is true for their written - counterparts. This sum represents the total bytes on the circuit that - have been validated by Tor. Subtracting this sum from the BytesRead - (or BytesWritten) value gives the byte count that Tor has decided - to reject due to protocol errors, or has otherwise decided to ignore. + BytesRead, and the same is true for their written counterparts. This + sum represents the total relay cell bytes on the circuit that + have been validated by Tor, not counting relay headers and cell headers. + Subtracting this sum (plus relay cell headers) from the BytesRead + (or BytesWritten) value gives the byte count that Tor has decided to + reject due to protocol errors, or has otherwise decided to ignore. The Time field is provided only in versions 0.3.2.1-alpha and later. It records when Tor created the bandwidth event.

1 0

[torspec/master] Merge remote-tracking branch 'mikeperry/ticket26110'
by nickm＠torproject.org 27 May '18

27 May '18

commit 000ce239756182c6e38543c07a45c81001def0c7 Merge: e541f2a 5a9cd80 Author: Nick Mathewson <nickm(a)torproject.org> Date: Sun May 27 12:10:22 2018 -0400 Merge remote-tracking branch 'mikeperry/ticket26110' control-spec.txt | 34 +++++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 11 deletions(-)

1 0

[tor/master] Merge remote-tracking branch 'public/bug25691_033_again_squashed' into maint-0.3.3
by nickm＠torproject.org 27 May '18

27 May '18

commit fa1890e97f19a444f8103c1207d7d18c02887b1f Merge: 0ef432d45 0941c8bfe Author: Nick Mathewson <nickm(a)torproject.org> Date: Sun May 27 10:03:11 2018 -0400 Merge remote-tracking branch 'public/bug25691_033_again_squashed' into maint-0.3.3 changes/bug25691_again | 6 +++++ src/or/circuitbuild.c | 67 ++++++++++++++++++++++++----------------------- src/or/circuituse.c | 4 +-- src/or/control.c | 9 ++++--- src/or/entrynodes.c | 7 ++--- src/or/hs_common.c | 15 ++++++++--- src/or/nodelist.c | 39 ++++++++++++++++++++++++--- src/or/nodelist.h | 4 ++- src/or/rendservice.c | 2 +- src/or/routerlist.c | 6 ++--- src/test/test_hs.c | 1 + src/test/test_hs_common.c | 3 ++- 12 files changed, 108 insertions(+), 55 deletions(-)

1 0

[tor/release-0.3.3] Use node_has_preferred_descriptor() in another case
by nickm＠torproject.org 27 May '18

27 May '18

commit 388d217c40fea4dfd9c2782ae6036126258fc292 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Apr 16 11:40:14 2018 -0400 Use node_has_preferred_descriptor() in another case In router_add_running_nodes_to_smartlist(), we had an inline implementation of the logic from node_has_descriptor(), which should be changed to node_has_preferred_descriptor(). --- src/or/routerlist.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/or/routerlist.c b/src/or/routerlist.c index bc3abb236..2ab5017b7 100644 --- a/src/or/routerlist.c +++ b/src/or/routerlist.c @@ -2335,7 +2335,7 @@ router_add_running_nodes_to_smartlist(smartlist_t *sl, int need_uptime, SMARTLIST_FOREACH_BEGIN(nodelist_get_list(), const node_t *, node) { if (!node->is_running || !node->is_valid) continue; - if (need_desc && !(node->ri || (node->rs && node->md))) + if (need_desc && !node_has_preferred_descriptor(node, direct_conn)) continue; if (node->ri && node->ri->purpose != ROUTER_PURPOSE_GENERAL) continue;

1 0

[tor/master] Merge branch 'maint-0.3.3'
by nickm＠torproject.org 27 May '18

27 May '18

commit aff49cc52dfabbd235505e12d507c6faf3262307 Merge: f42739e74 fa1890e97 Author: Nick Mathewson <nickm(a)torproject.org> Date: Sun May 27 10:04:42 2018 -0400 Merge branch 'maint-0.3.3'

1 0

[tor/release-0.3.3] Use router_crn_flags in more places, to pass direct-connect flag
by nickm＠torproject.org 27 May '18

27 May '18

commit 7915efd1b83e45c53eb06cfb384eed024df440c0 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Apr 16 11:41:45 2018 -0400 Use router_crn_flags in more places, to pass direct-connect flag In order to fix 25691 and 25692, we need to pass the "direct_conn" flag to more places -- particularly when choosing single-hop tunnels. The right way to do this involves having a couple more functions accept router_crn_flags_t, rather than a big list of boolean arguments. This commit also makes sure that choose_good_exit_server_general() honors the direct_conn flag, to fix 25691 and 25692. --- src/or/circuitbuild.c | 39 ++++++++++++++++++++++----------------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/src/or/circuitbuild.c b/src/or/circuitbuild.c index 019b9c51f..96cc8369d 100644 --- a/src/or/circuitbuild.c +++ b/src/or/circuitbuild.c @@ -1827,7 +1827,7 @@ ap_stream_wants_exit_attention(connection_t *conn) * Return NULL if we can't find any suitable routers. */ static const node_t * -choose_good_exit_server_general(int need_uptime, int need_capacity) +choose_good_exit_server_general(router_crn_flags_t flags) { int *n_supported; int n_pending_connections = 0; @@ -1837,6 +1837,9 @@ choose_good_exit_server_general(int need_uptime, int need_capacity) const or_options_t *options = get_options(); const smartlist_t *the_nodes; const node_t *selected_node=NULL; + const int need_uptime = (flags & CRN_NEED_UPTIME) != 0; + const int need_capacity = (flags & CRN_NEED_CAPACITY) != 0; + const int direct_conn = (flags & CRN_DIRECT_CONN) != 0; connections = get_connection_array(); @@ -1869,7 +1872,7 @@ choose_good_exit_server_general(int need_uptime, int need_capacity) */ continue; } - if (!node_has_preferred_descriptor(node, 0)) { + if (!node_has_preferred_descriptor(node, direct_conn)) { n_supported[i] = -1; continue; } @@ -1982,7 +1985,8 @@ choose_good_exit_server_general(int need_uptime, int need_capacity) need_capacity?", fast":"", need_uptime?", stable":""); tor_free(n_supported); - return choose_good_exit_server_general(0, 0); + flags &= ~(CRN_NEED_UPTIME|CRN_NEED_CAPACITY); + return choose_good_exit_server_general(flags); } log_notice(LD_CIRC, "All routers are down or won't exit%s -- " "choosing a doomed exit at random.", @@ -2229,17 +2233,11 @@ pick_restricted_middle_node(router_crn_flags_t flags, * toward the preferences in 'options'. */ static const node_t * -choose_good_exit_server(origin_circuit_t *circ, int need_uptime, - int need_capacity, int is_internal, int need_hs_v3) +choose_good_exit_server(origin_circuit_t *circ, + router_crn_flags_t flags, int is_internal) { const or_options_t *options = get_options(); - router_crn_flags_t flags = CRN_NEED_DESC; - if (need_uptime) - flags |= CRN_NEED_UPTIME; - if (need_capacity) - flags |= CRN_NEED_CAPACITY; - if (need_hs_v3) - flags |= CRN_RENDEZVOUS_V3; + flags |= CRN_NEED_DESC; switch (TO_CIRCUIT(circ)->purpose) { case CIRCUIT_PURPOSE_C_HSDIR_GET: @@ -2253,7 +2251,7 @@ choose_good_exit_server(origin_circuit_t *circ, int need_uptime, if (is_internal) /* pick it like a middle hop */ return router_choose_random_node(NULL, options->ExcludeNodes, flags); else - return choose_good_exit_server_general(need_uptime,need_capacity); + return choose_good_exit_server_general(flags); case CIRCUIT_PURPOSE_C_ESTABLISH_REND: { /* Pick a new RP */ @@ -2378,15 +2376,22 @@ onion_pick_cpath_exit(origin_circuit_t *circ, extend_info_t *exit_ei, extend_info_describe(exit_ei)); exit_ei = extend_info_dup(exit_ei); } else { /* we have to decide one */ + router_crn_flags_t flags = CRN_NEED_DESC; + if (state->need_uptime) + flags |= CRN_NEED_UPTIME; + if (state->need_capacity) + flags |= CRN_NEED_CAPACITY; + if (is_hs_v3_rp_circuit) + flags |= CRN_RENDEZVOUS_V3; + if (state->onehop_tunnel) + flags |= CRN_DIRECT_CONN; const node_t *node = - choose_good_exit_server(circ, state->need_uptime, - state->need_capacity, state->is_internal, - is_hs_v3_rp_circuit); + choose_good_exit_server(circ, flags, state->is_internal); if (!node) { log_warn(LD_CIRC,"Failed to choose an exit server"); return -1; } - exit_ei = extend_info_from_node(node, 0); + exit_ei = extend_info_from_node(node, state->onehop_tunnel); if (BUG(exit_ei == NULL)) return -1; }

1 0