October 2015 - tor-commits - lists.torproject.org

[torspec/master] Merge remote-tracking branch 'teor/gitignore-osx'
by nickm＠torproject.org 22 Oct '15

22 Oct '15

commit 29435e0051967a3e4b0dfe8401fd1325b5f5ddf5 Merge: 3bac19d 2f13c58 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 21 12:20:03 2015 -0400 Merge remote-tracking branch 'teor/gitignore-osx' .gitignore | 2 ++ 1 file changed, 2 insertions(+)

1 0

[doctor/master] Send fingerprint change notifications to me too
by atagar＠torproject.org 22 Oct '15

22 Oct '15

commit e846f57a20c977e9b68b412320ced55e2ef7f4e6 Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Oct 21 09:17:16 2015 -0700 Send fingerprint change notifications to me too Turns out I'm not on bad-relays@. Adding myself so I get the notices too. --- fingerprint_change_checker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fingerprint_change_checker.py b/fingerprint_change_checker.py index 6a376be..4aabfdb 100755 --- a/fingerprint_change_checker.py +++ b/fingerprint_change_checker.py @@ -90,7 +90,7 @@ def main(): body += "\n" try: - util.send(EMAIL_SUBJECT, body = body, to = ['bad-relays(a)lists.torproject.org']) + util.send(EMAIL_SUBJECT, body = body, to = ['bad-relays(a)lists.torproject.org', 'atagar(a)torproject.org']) except Exception as exc: log.warn("Unable to send email: %s" % exc)

1 0

[tor/master] Fix the return value
by nickm＠torproject.org 22 Oct '15

22 Oct '15

commit 9459ae260e3ae428283c3b45d08aaef6ef9f6fa3 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 21 12:01:05 2015 -0400 Fix the return value --- src/common/tortls.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/common/tortls.c b/src/common/tortls.c index 75ca47d..840b677 100644 --- a/src/common/tortls.c +++ b/src/common/tortls.c @@ -2678,7 +2678,7 @@ dn_indicates_v3_cert(X509_NAME *name) return 0; if (len < 4) { OPENSSL_free(s); - return 0; + return 1; } r = fast_memneq(s + len - 4, ".net", 4); OPENSSL_free(s);

1 0

[tor/master] Merge remote-tracking branch 'public/bug17404_024' into maint-0.2.7
by nickm＠torproject.org 22 Oct '15

22 Oct '15

commit 4fb49069753c37a2425427864212bc3a1f49d980 Merge: b809c26 9459ae2 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 21 12:02:42 2015 -0400 Merge remote-tracking branch 'public/bug17404_024' into maint-0.2.7 src/common/tortls.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

1 0

[tor/master] Merge remote-tracking branch 'origin/maint-0.2.7'
by nickm＠torproject.org 22 Oct '15

22 Oct '15

commit f217b24e051af2046ff03cb64fd9345967e777dd Merge: 895a98d 4fb4906 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 21 12:03:04 2015 -0400 Merge remote-tracking branch 'origin/maint-0.2.7' src/common/tortls.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --cc src/common/tortls.c index d863567,536043e..2a2a5d7 --- a/src/common/tortls.c +++ b/src/common/tortls.c @@@ -2453,12 -2519,11 +2453,12 @@@ dn_indicates_v3_cert(X509_NAME *name str = X509_NAME_ENTRY_get_data(entry); len = ASN1_STRING_to_UTF8(&s, str); - if (len < 0) + if (len < 0) { return 0; + } if (len < 4) { OPENSSL_free(s); - return 0; + return 1; } r = fast_memneq(s + len - 4, ".net", 4); OPENSSL_free(s);

1 0

[tor/maint-0.2.7] Fix the return value
by nickm＠torproject.org 22 Oct '15

22 Oct '15

commit 9459ae260e3ae428283c3b45d08aaef6ef9f6fa3 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 21 12:01:05 2015 -0400 Fix the return value --- src/common/tortls.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/common/tortls.c b/src/common/tortls.c index 75ca47d..840b677 100644 --- a/src/common/tortls.c +++ b/src/common/tortls.c @@ -2678,7 +2678,7 @@ dn_indicates_v3_cert(X509_NAME *name) return 0; if (len < 4) { OPENSSL_free(s); - return 0; + return 1; } r = fast_memneq(s + len - 4, ".net", 4); OPENSSL_free(s);

1 0

[tor/maint-0.2.7] Merge remote-tracking branch 'public/bug17404_024' into maint-0.2.7
by nickm＠torproject.org 22 Oct '15

22 Oct '15

commit 4fb49069753c37a2425427864212bc3a1f49d980 Merge: b809c26 9459ae2 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 21 12:02:42 2015 -0400 Merge remote-tracking branch 'public/bug17404_024' into maint-0.2.7 src/common/tortls.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

1 0

[torspec/master] Modify 210-faster...consensus-bootstrap for exponential backoff
by nickm＠torproject.org 21 Oct '15

21 Oct '15

commit fc4256c9073161cac5d8fe1a303b2901d18ecf25 Author: teor (Tim Wilson-Brown) <teor2345(a)gmail.com> Date: Fri Oct 2 15:46:54 2015 +0200 Modify 210-faster...consensus-bootstrap for exponential backoff To implement #4483 we need to contact multiple directory mirrors to increase bootstrap reliability. This patch implements the exponential backoff suggested in https://trac.torproject.org/projects/tor/ticket/4483#comment:22 The patch also analyses the reliability of the new scheme, and compares it to the current Tor implementation. --- .../210-faster-headless-consensus-bootstrap.txt | 124 ++++++++++++++------ 1 file changed, 86 insertions(+), 38 deletions(-) diff --git a/proposals/210-faster-headless-consensus-bootstrap.txt b/proposals/210-faster-headless-consensus-bootstrap.txt index 6b1502b..42726e5 100644 --- a/proposals/210-faster-headless-consensus-bootstrap.txt +++ b/proposals/210-faster-headless-consensus-bootstrap.txt @@ -1,9 +1,10 @@ Filename: 210-faster-headless-consensus-bootstrap.txt Title: Faster Headless Consensus Bootstrapping -Author: Mike Perry +Author: Mike Perry, Tim Wilson-Brown, Peter Palfrader Created: 01-10-2012 +Last Modified: 02-10-2015 Status: Open -Target: 0.2.4.x+ +Target: 0.2.8.x+ Overview and Motiviation @@ -19,19 +20,39 @@ Design: Bootstrap Process Changes parallel during the bootstrap process, and download the consensus from the first connection that completes. - Connection attempts will be done in batches of three. Only one - connection will be performed to one of the canonical directory - authorities. Two connections will be performed to randomly chosen hard - coded directory mirrors. - - If no connections complete within 5 seconds, another batch of three - connections will be launched. Otherwise, the first connection to - complete will be used to download the consensus document and the others - will be closed, after which bootstrapping will proceed as normal. + Connection attempts will be performed on an exponential backoff basis. + Initially, connections will be performed to randomly chosen hard + coded directory mirrors. If none of these connections complete within + 5 seconds, connections will also be performed to randomly chosen + canonical directory authorities. + + We specify that mirror connections retry after half a second, and then + double the retry time with every connection: + 0, 0.5, 1, 2, 4, 8, 16, ... + + We specify that directory authority connections start after a 5 second + delay, and retry after 5 seconds, doubling the retry time with every + connection: + 5, 10, 20, ... + + The first connection to complete will be used to download the consensus + document and the others will be closed, after which bootstrapping will + proceed as normal. + + We expect the vast majority of clients to succeed within 4 seconds, + after making up to 5 connection attempts to mirrors. Clients which can't + connect in the first 5 seconds, will then try to contact a directory + authority. We expect almost all clients to succeed within 10 seconds, + after up to 6 connection attempts to mirrors and up to 2 connection + attempts to authorities. This is a much better success rate than the + current Tor implementation, which fails k/n of clients if k of the n + directory authorities are down. (Or, if the connection fails in + certain ways, (k/n)^2.) If at any time, the total outstanding bootstrap connection attempts - exceeds 15, no new connection attempts are to be launched until existing - connection attempts experience full timeout. + exceeds 10, no new connection attempts are to be launched until an + existing connection attempt experiences full timeout. The retry time + is not doubled when a connection is skipped. Design: Fallback Dir Mirror Selection @@ -43,8 +64,8 @@ Design: Fallback Dir Mirror Selection This list of fallback dir mirrors should be updated with every major Tor release. In future releases, the number of dir mirrors - should be set at 20% of the current Guard nodes, rather than fixed at - 100. + should be set at 20% of the current Guard nodes (approximately 200 as + of October 2015), rather than fixed at 100. Performance: Additional Load with Current Parameter Choices @@ -62,19 +83,20 @@ Performance: Additional Load with Current Parameter Choices The dangerous case is in the event of a prolonged consensus failure that induces all clients to enter into the bootstrap process. In this - case, the number of initial TLS connections to the fallback dir mirrors - would be 2*C/100, or 10,000 for C=500,000 users. If no connections - complete before the five retries, this could reach as high as 50,000 - connection attempts, but this is extremely unlikely to happen in full - aggregate. + case, the number of TLS connections to the fallback dir mirrors within + the first second would be 3*C/100, or 60,000 for C=2,000,000 users. If + no connections complete before the 10 retries, 7 of which go to + mirrors, this could reach as high as 140,000 connection attempts, but + this is extremely unlikely to happen in full aggregate. However, in the no-consensus scenario today, the directory authorities - would already experience C/9 or 55,555 connection attempts. The - 5-retry scheme increases their total maximum load to about 275,000 - connection attempts, but again this is unlikely to be reached - in aggregate. Additionally, with this scheme, even if the dirauths - are taken down by this load, the dir mirrors should be able to survive - it. + would already experience 2*C/9 or 444,444 connection attempts. (Tor + currently tries 2 authorities, before delaying the next attempt.) The + 10-retry scheme, 3 of which go to authorities, increases their total + maximum load to about 666,666 connection attempts, but again this is + unlikely to be reached in aggregate. Additionally, with this scheme, + even if the dirauths are taken down by this load, the dir mirrors + should be able to survive it. Implementation Notes: Code Modifications @@ -87,12 +109,16 @@ Implementation Notes: Code Modifications a single directory server is selected and a connection is eventually made through directory_initiate_command_rend(). - There appear to be a few options for altering this code to perform - multiple connections. Without refactoring, one approach would be - to make multiple calls to directory_initiate_command_routerstatus() - from directory_get_from_dirserver() if the purpose is + There appear to be a few options for altering this code to retry multiple + simultaneous connections. Without refactoring, one approach would be to + set mirror and authority retry helper function timers in + directory_initiate_command_routerstatus() from + directory_get_from_dirserver() if the purpose is DIR_PURPOSE_FETCH_CONSENSUS and the only directory servers available - are the authorities and the fallback dir mirrors. + are the authorities and the fallback dir mirrors. (That is, there is no + valid consensus.) The retry helper function would check the list of + pending connections and, if it is 10 or greater, skip the connection + attempt, and leave the retry time constant. The code in directory_initiate_command_rend() would then need to be altered to maintain a list of the dircons created for this purpose as @@ -104,10 +130,32 @@ Implementation Notes: Code Modifications altered to examine the list of pending dircons, determine if this one is the first to complete, and if so, then call directory_send_command() to download the consensus and close the other pending dircons. - - An additional timer would need to be installed to re-call - update_consensus_networkstatus_downloads() or a related helper after 5 - seconds. connection_dir_finished_connecting() would cancel this timer. - The helper would check the list of pending connections and ensure it - never exceeds 15. - + connection_dir_finished_connecting() would also cancel both timers. + +Reliability Analysis + + We make the pessimistic assumptions that 50% of connections to directory + mirrors fail, and that 20% of connections to authorities fail. (Actual + figures depend on relay churn, age of the fallback list, and authority + uptime.) + + We expect the first 10 connection retry times to be: + Mirror: 0s 0.5s 1s 2s 4s 8s 16s + Auth: 5s 10s 20s + Success: 50% 75% 87% 94% 97% 99.4% 99.7% 99.94% 99.97% 99.99% + + 97% of clients succeed while only using directory mirrors. + 2.4% of clients succeed on their first auth connection. + 0.24% of clients succeed after one more mirror and auth connection. + 0.05% of clients succeed after two more mirror and auth connections. + 0.01% of clients remain, but in this scenario, 3 authorities are down, + so the client is most likely blocked from the Tor network. + + The current implementation makes 1 or 2 authority connections within the + first second, depending on exactly how the first connection fails. Under + the 20% authority failure assumption, these clients would have a success + rate of either 80% or 96% within a few seconds. The scheme above has a + similar success rate in the first few seconds, while spreading the load + among a larger number of directory mirrors. In addition, if all the + authorities are blocked, current clients will inevitably fail, as they + do not have a list of directory mirrors.

1 0

[torspec/master] fixup Add IPv4 and IPv6, make an auth connection early
by nickm＠torproject.org 21 Oct '15

21 Oct '15

commit e468e802980b2b4846b33fa615b19a1eab215956 Author: teor (Tim Wilson-Brown) <teor2345(a)gmail.com> Date: Fri Oct 2 17:39:46 2015 +0200 fixup Add IPv4 and IPv6, make an auth connection early Make one authority connection early so the client can check its clock. Redo the analysis for the new timing schedule. Add IPv4 and IPv6 alternation scheme for clients that have both an IPv4 and IPv6 address. Add retry timer maximum and retry timer reset events. Include min and max fallback directory weights. --- .../210-faster-headless-consensus-bootstrap.txt | 82 +++++++++++++------- 1 file changed, 54 insertions(+), 28 deletions(-) diff --git a/proposals/210-faster-headless-consensus-bootstrap.txt b/proposals/210-faster-headless-consensus-bootstrap.txt index 42726e5..79770d8 100644 --- a/proposals/210-faster-headless-consensus-bootstrap.txt +++ b/proposals/210-faster-headless-consensus-bootstrap.txt @@ -21,30 +21,53 @@ Design: Bootstrap Process Changes the first connection that completes. Connection attempts will be performed on an exponential backoff basis. - Initially, connections will be performed to randomly chosen hard - coded directory mirrors. If none of these connections complete within - 5 seconds, connections will also be performed to randomly chosen - canonical directory authorities. + Initially, connections will be performed to a randomly chosen hard + coded directory mirror and a randomly chosen canonical directory + authority. If neither of these connections complete, additional mirror + and authority connections are tried. Mirror connections are tried at + a faster rate than authority connections. We specify that mirror connections retry after half a second, and then double the retry time with every connection: - 0, 0.5, 1, 2, 4, 8, 16, ... + 0, 1, 2, 4, 8, 16, 32, ... - We specify that directory authority connections start after a 5 second - delay, and retry after 5 seconds, doubling the retry time with every - connection: - 5, 10, 20, ... + We specify that directory authority connections retry after 5 seconds, + and then double the retry time with every connection: + 0, 10, 20, ... + + If the client has both an IPv4 and IPv6 address, we try IPv4 and IPv6 + mirrors and authorities on the following schedule: + IPv4, IPv6, IPv4, IPv6, ... + + We try IPv4 first to avoid overloading IPv6-enabled authorities and + mirrors. Mirrors and auths get a separate IPv4/IPv6 schedule. This + ensures that we try an IPv6 authority within the first 10 seconds. + This helps implement #8374 and related tickets. + + The maximum retry time for both timers is 3 days + 1 hour. This places a + small load on the mirrors and authorities, while allowing a client that + regains a network connection to eventually download a consensus. + + The retry timers must reset on HUP and any network reachability events, + [ TODO: do we have network reachability events? ] + so that clients that have unreliable networks can recover from network + failures. The first connection to complete will be used to download the consensus document and the others will be closed, after which bootstrapping will proceed as normal. + A benefit of connecting to directory authorities is that clients are + warned if their clock is wrong. Therefore, when closing a directory + authority connection, we check to see if we have successfully connected + to an authority during this run of the Tor client. If not, we allow the + authority TLS connection to complete, then close the connection. + We expect the vast majority of clients to succeed within 4 seconds, - after making up to 5 connection attempts to mirrors. Clients which can't - connect in the first 5 seconds, will then try to contact a directory - authority. We expect almost all clients to succeed within 10 seconds, - after up to 6 connection attempts to mirrors and up to 2 connection - attempts to authorities. This is a much better success rate than the + after making up to 4 connection attempts to mirrors. Clients which can't + connect in the first 10 seconds, will try 1 more mirror, then try to + contact another directory authority. We expect almost all clients to + succeed within 10 seconds. This is a much better success rate than the current Tor implementation, which fails k/n of clients if k of the n directory authorities are down. (Or, if the connection fails in certain ways, (k/n)^2.) @@ -60,7 +83,11 @@ Design: Fallback Dir Mirror Selection the 100 Guard nodes with the longest uptime. The fallback weights will be set using each mirror's fraction of - consensus bandwidth out of the total of all 100 mirrors. + consensus bandwidth out of the total of all 100 mirrors, adjusted to + ensure no fallback directory sees more than 10% of clients. We will + also exclude fallback directories that are less than 1/1000 of the + consensus weight, as they are not large enough to make it worthwhile + including them. This list of fallback dir mirrors should be updated with every major Tor release. In future releases, the number of dir mirrors @@ -84,7 +111,7 @@ Performance: Additional Load with Current Parameter Choices The dangerous case is in the event of a prolonged consensus failure that induces all clients to enter into the bootstrap process. In this case, the number of TLS connections to the fallback dir mirrors within - the first second would be 3*C/100, or 60,000 for C=2,000,000 users. If + the first second would be 2*C/100, or 40,000 for C=2,000,000 users. If no connections complete before the 10 retries, 7 of which go to mirrors, this could reach as high as 140,000 connection attempts, but this is extremely unlikely to happen in full aggregate. @@ -111,7 +138,7 @@ Implementation Notes: Code Modifications There appear to be a few options for altering this code to retry multiple simultaneous connections. Without refactoring, one approach would be to - set mirror and authority retry helper function timers in + set a connection retry helper function timer in directory_initiate_command_routerstatus() from directory_get_from_dirserver() if the purpose is DIR_PURPOSE_FETCH_CONSENSUS and the only directory servers available @@ -130,7 +157,7 @@ Implementation Notes: Code Modifications altered to examine the list of pending dircons, determine if this one is the first to complete, and if so, then call directory_send_command() to download the consensus and close the other pending dircons. - connection_dir_finished_connecting() would also cancel both timers. + connection_dir_finished_connecting() would also cancel the timer. Reliability Analysis @@ -140,22 +167,21 @@ Reliability Analysis uptime.) We expect the first 10 connection retry times to be: - Mirror: 0s 0.5s 1s 2s 4s 8s 16s - Auth: 5s 10s 20s - Success: 50% 75% 87% 94% 97% 99.4% 99.7% 99.94% 99.97% 99.99% - - 97% of clients succeed while only using directory mirrors. - 2.4% of clients succeed on their first auth connection. - 0.24% of clients succeed after one more mirror and auth connection. - 0.05% of clients succeed after two more mirror and auth connections. - 0.01% of clients remain, but in this scenario, 3 authorities are down, + Mirror: 0s 1s 2s 4s 8s 16s 32s + Auth: 0s 10s 20s + Success: 90% 95% 97% 98.7% 99.4% 99.89% 99.94% 99.988% 99.994% + + 97% of clients succeed in the first 2 seconds. + 99.4% of clients succeed without trying a second authority. + 99.89% of clients succeed in the first 10 seconds. + 0.11% of clients remain, but in this scenario, 2 authorities are down, so the client is most likely blocked from the Tor network. The current implementation makes 1 or 2 authority connections within the first second, depending on exactly how the first connection fails. Under the 20% authority failure assumption, these clients would have a success rate of either 80% or 96% within a few seconds. The scheme above has a - similar success rate in the first few seconds, while spreading the load + greater success rate in the first few seconds, while spreading the load among a larger number of directory mirrors. In addition, if all the authorities are blocked, current clients will inevitably fail, as they do not have a list of directory mirrors.

1 0

[torspec/master] fixup Rewrite summary section for revised connection schedule
by nickm＠torproject.org 21 Oct '15

21 Oct '15

commit bdfce76e8a8ad5c7277300bebe8f7ed3478b304e Author: teor (Tim Wilson-Brown) <teor2345(a)gmail.com> Date: Fri Oct 2 17:53:36 2015 +0200 fixup Rewrite summary section for revised connection schedule And various other fixups --- .../210-faster-headless-consensus-bootstrap.txt | 42 +++++++++++++------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/proposals/210-faster-headless-consensus-bootstrap.txt b/proposals/210-faster-headless-consensus-bootstrap.txt index 79770d8..e5c8cb0 100644 --- a/proposals/210-faster-headless-consensus-bootstrap.txt +++ b/proposals/210-faster-headless-consensus-bootstrap.txt @@ -27,7 +27,7 @@ Design: Bootstrap Process Changes and authority connections are tried. Mirror connections are tried at a faster rate than authority connections. - We specify that mirror connections retry after half a second, and then + We specify that mirror connections retry after one second, and then double the retry time with every connection: 0, 1, 2, 4, 8, 16, 32, ... @@ -35,6 +35,12 @@ Design: Bootstrap Process Changes and then double the retry time with every connection: 0, 10, 20, ... + [ XXX: should we add random noise to these scheduled times? - teor ] + + The maximum retry time for both timers is 3 days + 1 hour. This places a + small load on the mirrors and authorities, while allowing a client that + regains a network connection to eventually download a consensus. + If the client has both an IPv4 and IPv6 address, we try IPv4 and IPv6 mirrors and authorities on the following schedule: IPv4, IPv6, IPv4, IPv6, ... @@ -44,14 +50,19 @@ Design: Bootstrap Process Changes ensures that we try an IPv6 authority within the first 10 seconds. This helps implement #8374 and related tickets. - The maximum retry time for both timers is 3 days + 1 hour. This places a - small load on the mirrors and authorities, while allowing a client that - regains a network connection to eventually download a consensus. - - The retry timers must reset on HUP and any network reachability events, + We don't want to keep on trying an IP version that always fails. + Therefore, once sufficient IPv4 and IPv6 connections have been + attempted, we select an IP version for new connections based on the ratio + of their failure rates, up to a maximum of 1:5. This may not make a + substantial difference to consensus downloads, as we only need one + successful consensus download to bootstrap. However, it is important for + future features like #17217, where clients try to automatically determine + if they can use IPv4 or IPv6 to contact the Tor network. + + The retry timers and IP version schedules must reset on HUP and any + network reachability events, so that clients that have unreliable networks + can recover from network failures. [ TODO: do we have network reachability events? ] - so that clients that have unreliable networks can recover from network - failures. The first connection to complete will be used to download the consensus document and the others will be closed, after which bootstrapping will @@ -64,13 +75,14 @@ Design: Bootstrap Process Changes authority TLS connection to complete, then close the connection. We expect the vast majority of clients to succeed within 4 seconds, - after making up to 4 connection attempts to mirrors. Clients which can't - connect in the first 10 seconds, will try 1 more mirror, then try to - contact another directory authority. We expect almost all clients to - succeed within 10 seconds. This is a much better success rate than the - current Tor implementation, which fails k/n of clients if k of the n - directory authorities are down. (Or, if the connection fails in - certain ways, (k/n)^2.) + after making up to 4 connection attempts to mirrors and 1 connection + attempt to an authority. Clients which can't connect in the first + 10 seconds, will try 1 more mirror, then try to contact another + directory authority. We expect almost all clients to succeed within + 10 seconds. This is a much better success rate than the current Tor + implementation, which fails k/n of clients if k of the n directory + authorities are down. (Or, if the connection fails in certain ways, + (k/n)^2.) If at any time, the total outstanding bootstrap connection attempts exceeds 10, no new connection attempts are to be launched until an

1 0