December 2016 - tor-commits - lists.torproject.org

[tor/master] Merge remote-tracking branch 'teor/fallbacks-20161219'
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit df87812b41abccd63eab97bd81b476ce61e16f26 Merge: 3341ad1 a0bbd72 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 20 18:38:45 2016 -0500 Merge remote-tracking branch 'teor/fallbacks-20161219' changes/fallbacks-201612 | 37 ++++ scripts/maint/fallback.blacklist | 37 +++- scripts/maint/fallback.whitelist | 208 +++++++++++------- scripts/maint/updateFallbackDirs.py | 410 +++++++++++++++++++++++++----------- 4 files changed, 494 insertions(+), 198 deletions(-)

1 0

[tor/master] Display the fingerprint when downloading consensuses from fallbacks
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 9629a25d1019cffffc39f3982161b610eab5691f Author: teor <teor2345(a)gmail.com> Date: Wed Dec 7 16:02:17 2016 +1100 Display the fingerprint when downloading consensuses from fallbacks --- changes/fallbacks-201612 | 6 ++++++ scripts/maint/updateFallbackDirs.py | 13 ++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/changes/fallbacks-201612 b/changes/fallbacks-201612 index 0306ad1..e0e412c 100644 --- a/changes/fallbacks-201612 +++ b/changes/fallbacks-201612 @@ -16,6 +16,12 @@ - Exclude relays affected by 20499 from the fallback list. Exclude known affected versions, and any relay that delivers a stale consensus. Closes ticket 20539. + - Require fallbacks to have flags for 90% of the time (weighted decaying + average), rather than 95%. This allows at least 73% of clients to + bootstrap in the first 5 seconds without contacting an authority. + Part of ticket 18828. + - Display the fingerprint when downloading consensuses from fallbacks. + Closes ticket 20908. o Minor bugfix (fallback directories): - Stop failing when OUTPUT_COMMENTS is True in updateFallbackDirs.py. Closes ticket 20877; bugfix on commit 9998343 in tor-0.2.8.3-alpha. diff --git a/scripts/maint/updateFallbackDirs.py b/scripts/maint/updateFallbackDirs.py index 40b0056..df5f049 100755 --- a/scripts/maint/updateFallbackDirs.py +++ b/scripts/maint/updateFallbackDirs.py @@ -1127,14 +1127,15 @@ class Candidate(object): # log how long it takes to download a consensus from dirip:dirport # returns True if the download failed, False if it succeeded within max_time @staticmethod - def fallback_consensus_download_speed(dirip, dirport, nickname, max_time): + def fallback_consensus_download_speed(dirip, dirport, nickname, fingerprint, + max_time): download_failed = False start = datetime.datetime.utcnow() # some directory mirrors respond to requests in ways that hang python # sockets, which is why we log this line here - logging.info('Initiating %sconsensus download from %s (%s:%d).', + logging.info('Initiating %sconsensus download from %s (%s:%d) %s.', 'microdesc ' if DOWNLOAD_MICRODESC_CONSENSUS else '', - nickname, dirip, dirport) + nickname, dirip, dirport, fingerprint) # there appears to be about 1 second of overhead when comparing stem's # internal trace time and the elapsed time calculated here TIMEOUT_SLOP = 1.0 @@ -1171,9 +1172,9 @@ class Candidate(object): else: status = 'ok' level = logging.DEBUG - logging.log(level, 'Consensus download: %0.1fs %s from %s (%s:%d), ' + + logging.log(level, 'Consensus download: %0.1fs %s from %s (%s:%d) %s, ' + 'max download time %0.1fs.', elapsed, status, nickname, - dirip, dirport, max_time) + dirip, dirport, fingerprint, max_time) return download_failed # does this fallback download the consensus fast enough? @@ -1185,12 +1186,14 @@ class Candidate(object): ipv4_failed = Candidate.fallback_consensus_download_speed(self.dirip, self.dirport, self._data['nickname'], + self._fpr, CONSENSUS_DOWNLOAD_SPEED_MAX) if self.has_ipv6() and PERFORM_IPV6_DIRPORT_CHECKS: # Clients assume the IPv6 DirPort is the same as the IPv4 DirPort ipv6_failed = Candidate.fallback_consensus_download_speed(self.ipv6addr, self.dirport, self._data['nickname'], + self._fpr, CONSENSUS_DOWNLOAD_SPEED_MAX) return ((not ipv4_failed) and (not ipv6_failed))

1 0

[tor/master] Avoid checking fallback candidates' DirPorts if they are down in OnionOO
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 2d2bbaf259f5a886ad858f2c5fc75c72e0369f5b Author: teor <teor2345(a)gmail.com> Date: Thu Dec 8 10:20:01 2016 +1100 Avoid checking fallback candidates' DirPorts if they are down in OnionOO Exclude relays that have been down for 1 or more days from the fallback candidate list. When a relay operator has multiple relays, this prioritises relays that are up over relays that are down. Fixes issue #20926. --- changes/fallbacks-201612 | 6 +++++- scripts/maint/updateFallbackDirs.py | 14 +++++++++----- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/changes/fallbacks-201612 b/changes/fallbacks-201612 index 96ae658..5bb7bdd 100644 --- a/changes/fallbacks-201612 +++ b/changes/fallbacks-201612 @@ -25,6 +25,10 @@ fallbacks.) Closes ticket 20912. - Reduce the minimum fallback bandwidth to 1 MByte/s. Part of #18828. - o Minor bugfix (fallback directories): + o Minor bugfixes (fallback directories): - Stop failing when OUTPUT_COMMENTS is True in updateFallbackDirs.py. Closes ticket 20877; bugfix on commit 9998343 in tor-0.2.8.3-alpha. + - Avoid checking fallback candidates' DirPorts if they are down in + OnionOO. When a relay operator has multiple relays, this prioritises + relays that are up over relays that are down. + Closes ticket #20926; bugfix on 0.2.8.3-alpha. diff --git a/scripts/maint/updateFallbackDirs.py b/scripts/maint/updateFallbackDirs.py index eee93c4..41d210f 100755 --- a/scripts/maint/updateFallbackDirs.py +++ b/scripts/maint/updateFallbackDirs.py @@ -81,6 +81,10 @@ PERFORM_IPV4_DIRPORT_CHECKS = False if OUTPUT_CANDIDATES else True # Don't check ~1000 candidates when OUTPUT_CANDIDATES is True PERFORM_IPV6_DIRPORT_CHECKS = False if OUTPUT_CANDIDATES else False +# Must relays be running now? +MUST_BE_RUNNING_NOW = (PERFORM_IPV4_DIRPORT_CHECKS + or PERFORM_IPV6_DIRPORT_CHECKS) + # Clients have been using microdesc consensuses by default for a while now DOWNLOAD_MICRODESC_CONSENSUS = True @@ -145,6 +149,8 @@ MAX_LIST_FILE_SIZE = 1024 * 1024 # Affected relays should upgrade to Tor 0.2.8.7 or later, which has a fix # for this issue. ADDRESS_AND_PORT_STABLE_DAYS = 7 +# We ignore relays that have been down for more than this period +MAX_DOWNTIME_DAYS = 0 if MUST_BE_RUNNING_NOW else 7 # What time-weighted-fraction of these flags must FallbackDirs # Equal or Exceed? CUTOFF_RUNNING = .90 @@ -387,8 +393,8 @@ def onionoo_fetch(what, **kwargs): params = kwargs params['type'] = 'relay' #params['limit'] = 10 - params['first_seen_days'] = '%d-'%(ADDRESS_AND_PORT_STABLE_DAYS,) - params['last_seen_days'] = '-7' + params['first_seen_days'] = '%d-'%(ADDRESS_AND_PORT_STABLE_DAYS) + params['last_seen_days'] = '-%d'%(MAX_DOWNTIME_DAYS) params['flag'] = 'V2Dir' url = ONIONOO + what + '?' + urllib.urlencode(params) @@ -843,9 +849,7 @@ class Candidate(object): self._badexit = self._avg_generic_history(badexit) / ONIONOO_SCALE_ONE def is_candidate(self): - must_be_running_now = (PERFORM_IPV4_DIRPORT_CHECKS - or PERFORM_IPV6_DIRPORT_CHECKS) - if (must_be_running_now and not self.is_running()): + if (MUST_BE_RUNNING_NOW and not self.is_running()): logging.info('%s not a candidate: not running now, unable to check ' + 'DirPort consensus download', self._fpr) return False

1 0

[tor/master] Be more tolerant of a decreased number of fallbacks
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 4c832bcfac3463f101d95b29f06e62eecaa64094 Author: teor <teor2345(a)gmail.com> Date: Wed Dec 7 16:42:42 2016 +1100 Be more tolerant of a decreased number of fallbacks --- scripts/maint/updateFallbackDirs.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/maint/updateFallbackDirs.py b/scripts/maint/updateFallbackDirs.py index a7c86d0..eee93c4 100755 --- a/scripts/maint/updateFallbackDirs.py +++ b/scripts/maint/updateFallbackDirs.py @@ -170,8 +170,8 @@ FALLBACK_PROPORTION_OF_GUARDS = None if OUTPUT_CANDIDATES else _FB_POG # Limit the number of fallbacks (eliminating lowest by advertised bandwidth) MAX_FALLBACK_COUNT = None if OUTPUT_CANDIDATES else 200 -# Emit a C #error if the number of fallbacks is more than 10% below MAX -MIN_FALLBACK_COUNT = 0 if OUTPUT_CANDIDATES else MAX_FALLBACK_COUNT*0.9 +# Emit a C #error if the number of fallbacks is less than expected +MIN_FALLBACK_COUNT = 0 if OUTPUT_CANDIDATES else MAX_FALLBACK_COUNT*0.75 # The maximum number of fallbacks on the same address, contact, or family # With 200 fallbacks, this means each operator can see 1% of client bootstraps

1 0

[tor/master] Add and update fallback directory details
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit a0bbd72caaa50cee75b820e86c7170abfea43472 Author: teor <teor2345(a)gmail.com> Date: Wed Dec 21 08:15:21 2016 +1100 Add and update fallback directory details These updates were made after the December 2016 list was created. --- scripts/maint/fallback.whitelist | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/maint/fallback.whitelist b/scripts/maint/fallback.whitelist index d333392..c993be9 100644 --- a/scripts/maint/fallback.whitelist +++ b/scripts/maint/fallback.whitelist @@ -824,4 +824,7 @@ 31.171.155.108:9030 orport=9001 id=D3E5EDDBE5159388704D6785BE51930AAFACEC6F # Email sent directly to teor, verified using relay contact info -89.163.247.43:9030 orport=9001 id=BC7ACFAC04854C77167C7D66B7E471314ED8C410 +89.163.247.43:9030 orport=9001 id=BC7ACFAC04854C77167C7D66B7E471314ED8C410 ipv6=[2001:4ba0:fff7:25::5]:9001 + +# Email sent directly to teor, verified using relay contact info +95.85.8.226:80 orport=443 id=1211AC1BBB8A1AF7CBA86BCE8689AA3146B86423

1 0

[tor/release-0.2.9] Merge branch 'maint-0.2.4' into maint-0.2.5
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 7d3d56e59a3df16615881007a77b77e30fecb831 Merge: db58d4d 8f857c2 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 20 18:19:41 2016 -0500 Merge branch 'maint-0.2.4' into maint-0.2.5 (ours merge -- there is a separate 0.2.5 patch for 20384.)

1 0

[tor/release-0.2.9] Merge branch 'maint-0.2.6' into maint-0.2.7
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 2673b4b7a87bbdc880dd235f490dfb7b8f3c64c9 Merge: a9c8a5f b6227ed Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 20 18:23:19 2016 -0500 Merge branch 'maint-0.2.6' into maint-0.2.7 changes/buf-sentinel | 11 +++++++++++ src/or/buffers.c | 40 ++++++++++++++++++++++++++++++++-------- 2 files changed, 43 insertions(+), 8 deletions(-)

1 0

[tor/release-0.2.9] Merge branch 'maint-0.2.5' into maint-0.2.6
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 746c51b6132d84fc7899e6b95b4a1666717d8cce Merge: b18bde2 39ef343 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 20 18:22:27 2016 -0500 Merge branch 'maint-0.2.5' into maint-0.2.6 ("ours" merge because there is a separate 20384 patch for 026)

1 0

[tor/release-0.2.8] Merge branch 'maint-0.2.7' into maint-0.2.8
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit cb2e2056b5637443715bf494be72d0953c75b4be Merge: 7f55aaf 2673b4b Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 20 18:23:35 2016 -0500 Merge branch 'maint-0.2.7' into maint-0.2.8

1 0

[tor/release-0.2.8] Merge branch 'maint-0.2.8' into release-0.2.8
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 17f20c7b3f385db6bec4ee8f32f050f7953fbcca Merge: 5d63ceb cb2e205 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 20 18:23:49 2016 -0500 Merge branch 'maint-0.2.8' into release-0.2.8

1 0