December 2016 - tor-commits - lists.torproject.org

[tor/master] Avoid an error in the fallback script when a fallback doesn't have any uptime
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 53ec0874504133f089ec422f6a97b2b3f58ae8db Author: teor <teor2345(a)gmail.com> Date: Sun Dec 11 22:47:00 2016 +1100 Avoid an error in the fallback script when a fallback doesn't have any uptime Sometimes, the fallback generation script doesn't add attributes to the fallbacks in the list. If this happens, log an error, and avoid selecting that fallback. This is a rare issue: it should not change selection behaviour. Fixes issue #20945. --- changes/fallbacks-201612 | 2 ++ scripts/maint/updateFallbackDirs.py | 67 ++++++++++++++++++++----------------- 2 files changed, 38 insertions(+), 31 deletions(-) diff --git a/changes/fallbacks-201612 b/changes/fallbacks-201612 index 03d9843..d9a67fd 100644 --- a/changes/fallbacks-201612 +++ b/changes/fallbacks-201612 @@ -33,3 +33,5 @@ OnionOO. When a relay operator has multiple relays, this prioritises relays that are up over relays that are down. Closes ticket #20926; bugfix on 0.2.8.3-alpha. + - Stop failing when a relay has no uptime data in updateFallbackDirs.py. + Closes ticket 20945; bugfix on tor-0.2.8.1-alpha. diff --git a/scripts/maint/updateFallbackDirs.py b/scripts/maint/updateFallbackDirs.py index 4f60fd4..e0bc939 100755 --- a/scripts/maint/updateFallbackDirs.py +++ b/scripts/maint/updateFallbackDirs.py @@ -862,37 +862,42 @@ class Candidate(object): self._badexit = self._avg_generic_history(badexit) / ONIONOO_SCALE_ONE def is_candidate(self): - if (MUST_BE_RUNNING_NOW and not self.is_running()): - logging.info('%s not a candidate: not running now, unable to check ' + - 'DirPort consensus download', self._fpr) - return False - if (self._data['last_changed_address_or_port'] > - self.CUTOFF_ADDRESS_AND_PORT_STABLE): - logging.info('%s not a candidate: changed address/port recently (%s)', - self._fpr, self._data['last_changed_address_or_port']) - return False - if self._running < CUTOFF_RUNNING: - logging.info('%s not a candidate: running avg too low (%lf)', - self._fpr, self._running) - return False - if self._v2dir < CUTOFF_V2DIR: - logging.info('%s not a candidate: v2dir avg too low (%lf)', - self._fpr, self._v2dir) - return False - if self._badexit is not None and self._badexit > PERMITTED_BADEXIT: - logging.info('%s not a candidate: badexit avg too high (%lf)', - self._fpr, self._badexit) - return False - # this function logs a message depending on which check fails - if not self.is_valid_version(): - return False - if self._guard < CUTOFF_GUARD: - logging.info('%s not a candidate: guard avg too low (%lf)', - self._fpr, self._guard) - return False - if (not self._data.has_key('consensus_weight') - or self._data['consensus_weight'] < 1): - logging.info('%s not a candidate: consensus weight invalid', self._fpr) + try: + if (MUST_BE_RUNNING_NOW and not self.is_running()): + logging.info('%s not a candidate: not running now, unable to check ' + + 'DirPort consensus download', self._fpr) + return False + if (self._data['last_changed_address_or_port'] > + self.CUTOFF_ADDRESS_AND_PORT_STABLE): + logging.info('%s not a candidate: changed address/port recently (%s)', + self._fpr, self._data['last_changed_address_or_port']) + return False + if self._running < CUTOFF_RUNNING: + logging.info('%s not a candidate: running avg too low (%lf)', + self._fpr, self._running) + return False + if self._v2dir < CUTOFF_V2DIR: + logging.info('%s not a candidate: v2dir avg too low (%lf)', + self._fpr, self._v2dir) + return False + if self._badexit is not None and self._badexit > PERMITTED_BADEXIT: + logging.info('%s not a candidate: badexit avg too high (%lf)', + self._fpr, self._badexit) + return False + # this function logs a message depending on which check fails + if not self.is_valid_version(): + return False + if self._guard < CUTOFF_GUARD: + logging.info('%s not a candidate: guard avg too low (%lf)', + self._fpr, self._guard) + return False + if (not self._data.has_key('consensus_weight') + or self._data['consensus_weight'] < 1): + logging.info('%s not a candidate: consensus weight invalid', self._fpr) + return False + except BaseException as e: + logging.warning("Exception %s when checking if fallback is a candidate", + str(e)) return False return True

1 0

[tor/master] Merge remote-tracking branch 'teor/fallbacks-20161219'
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit df87812b41abccd63eab97bd81b476ce61e16f26 Merge: 3341ad1 a0bbd72 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 20 18:38:45 2016 -0500 Merge remote-tracking branch 'teor/fallbacks-20161219' changes/fallbacks-201612 | 37 ++++ scripts/maint/fallback.blacklist | 37 +++- scripts/maint/fallback.whitelist | 208 +++++++++++------- scripts/maint/updateFallbackDirs.py | 410 +++++++++++++++++++++++++----------- 4 files changed, 494 insertions(+), 198 deletions(-)

1 0

[tor/master] Display the fingerprint when downloading consensuses from fallbacks
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 9629a25d1019cffffc39f3982161b610eab5691f Author: teor <teor2345(a)gmail.com> Date: Wed Dec 7 16:02:17 2016 +1100 Display the fingerprint when downloading consensuses from fallbacks --- changes/fallbacks-201612 | 6 ++++++ scripts/maint/updateFallbackDirs.py | 13 ++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/changes/fallbacks-201612 b/changes/fallbacks-201612 index 0306ad1..e0e412c 100644 --- a/changes/fallbacks-201612 +++ b/changes/fallbacks-201612 @@ -16,6 +16,12 @@ - Exclude relays affected by 20499 from the fallback list. Exclude known affected versions, and any relay that delivers a stale consensus. Closes ticket 20539. + - Require fallbacks to have flags for 90% of the time (weighted decaying + average), rather than 95%. This allows at least 73% of clients to + bootstrap in the first 5 seconds without contacting an authority. + Part of ticket 18828. + - Display the fingerprint when downloading consensuses from fallbacks. + Closes ticket 20908. o Minor bugfix (fallback directories): - Stop failing when OUTPUT_COMMENTS is True in updateFallbackDirs.py. Closes ticket 20877; bugfix on commit 9998343 in tor-0.2.8.3-alpha. diff --git a/scripts/maint/updateFallbackDirs.py b/scripts/maint/updateFallbackDirs.py index 40b0056..df5f049 100755 --- a/scripts/maint/updateFallbackDirs.py +++ b/scripts/maint/updateFallbackDirs.py @@ -1127,14 +1127,15 @@ class Candidate(object): # log how long it takes to download a consensus from dirip:dirport # returns True if the download failed, False if it succeeded within max_time @staticmethod - def fallback_consensus_download_speed(dirip, dirport, nickname, max_time): + def fallback_consensus_download_speed(dirip, dirport, nickname, fingerprint, + max_time): download_failed = False start = datetime.datetime.utcnow() # some directory mirrors respond to requests in ways that hang python # sockets, which is why we log this line here - logging.info('Initiating %sconsensus download from %s (%s:%d).', + logging.info('Initiating %sconsensus download from %s (%s:%d) %s.', 'microdesc ' if DOWNLOAD_MICRODESC_CONSENSUS else '', - nickname, dirip, dirport) + nickname, dirip, dirport, fingerprint) # there appears to be about 1 second of overhead when comparing stem's # internal trace time and the elapsed time calculated here TIMEOUT_SLOP = 1.0 @@ -1171,9 +1172,9 @@ class Candidate(object): else: status = 'ok' level = logging.DEBUG - logging.log(level, 'Consensus download: %0.1fs %s from %s (%s:%d), ' + + logging.log(level, 'Consensus download: %0.1fs %s from %s (%s:%d) %s, ' + 'max download time %0.1fs.', elapsed, status, nickname, - dirip, dirport, max_time) + dirip, dirport, fingerprint, max_time) return download_failed # does this fallback download the consensus fast enough? @@ -1185,12 +1186,14 @@ class Candidate(object): ipv4_failed = Candidate.fallback_consensus_download_speed(self.dirip, self.dirport, self._data['nickname'], + self._fpr, CONSENSUS_DOWNLOAD_SPEED_MAX) if self.has_ipv6() and PERFORM_IPV6_DIRPORT_CHECKS: # Clients assume the IPv6 DirPort is the same as the IPv4 DirPort ipv6_failed = Candidate.fallback_consensus_download_speed(self.ipv6addr, self.dirport, self._data['nickname'], + self._fpr, CONSENSUS_DOWNLOAD_SPEED_MAX) return ((not ipv4_failed) and (not ipv6_failed))

1 0

[tor/master] Avoid checking fallback candidates' DirPorts if they are down in OnionOO
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 2d2bbaf259f5a886ad858f2c5fc75c72e0369f5b Author: teor <teor2345(a)gmail.com> Date: Thu Dec 8 10:20:01 2016 +1100 Avoid checking fallback candidates' DirPorts if they are down in OnionOO Exclude relays that have been down for 1 or more days from the fallback candidate list. When a relay operator has multiple relays, this prioritises relays that are up over relays that are down. Fixes issue #20926. --- changes/fallbacks-201612 | 6 +++++- scripts/maint/updateFallbackDirs.py | 14 +++++++++----- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/changes/fallbacks-201612 b/changes/fallbacks-201612 index 96ae658..5bb7bdd 100644 --- a/changes/fallbacks-201612 +++ b/changes/fallbacks-201612 @@ -25,6 +25,10 @@ fallbacks.) Closes ticket 20912. - Reduce the minimum fallback bandwidth to 1 MByte/s. Part of #18828. - o Minor bugfix (fallback directories): + o Minor bugfixes (fallback directories): - Stop failing when OUTPUT_COMMENTS is True in updateFallbackDirs.py. Closes ticket 20877; bugfix on commit 9998343 in tor-0.2.8.3-alpha. + - Avoid checking fallback candidates' DirPorts if they are down in + OnionOO. When a relay operator has multiple relays, this prioritises + relays that are up over relays that are down. + Closes ticket #20926; bugfix on 0.2.8.3-alpha. diff --git a/scripts/maint/updateFallbackDirs.py b/scripts/maint/updateFallbackDirs.py index eee93c4..41d210f 100755 --- a/scripts/maint/updateFallbackDirs.py +++ b/scripts/maint/updateFallbackDirs.py @@ -81,6 +81,10 @@ PERFORM_IPV4_DIRPORT_CHECKS = False if OUTPUT_CANDIDATES else True # Don't check ~1000 candidates when OUTPUT_CANDIDATES is True PERFORM_IPV6_DIRPORT_CHECKS = False if OUTPUT_CANDIDATES else False +# Must relays be running now? +MUST_BE_RUNNING_NOW = (PERFORM_IPV4_DIRPORT_CHECKS + or PERFORM_IPV6_DIRPORT_CHECKS) + # Clients have been using microdesc consensuses by default for a while now DOWNLOAD_MICRODESC_CONSENSUS = True @@ -145,6 +149,8 @@ MAX_LIST_FILE_SIZE = 1024 * 1024 # Affected relays should upgrade to Tor 0.2.8.7 or later, which has a fix # for this issue. ADDRESS_AND_PORT_STABLE_DAYS = 7 +# We ignore relays that have been down for more than this period +MAX_DOWNTIME_DAYS = 0 if MUST_BE_RUNNING_NOW else 7 # What time-weighted-fraction of these flags must FallbackDirs # Equal or Exceed? CUTOFF_RUNNING = .90 @@ -387,8 +393,8 @@ def onionoo_fetch(what, **kwargs): params = kwargs params['type'] = 'relay' #params['limit'] = 10 - params['first_seen_days'] = '%d-'%(ADDRESS_AND_PORT_STABLE_DAYS,) - params['last_seen_days'] = '-7' + params['first_seen_days'] = '%d-'%(ADDRESS_AND_PORT_STABLE_DAYS) + params['last_seen_days'] = '-%d'%(MAX_DOWNTIME_DAYS) params['flag'] = 'V2Dir' url = ONIONOO + what + '?' + urllib.urlencode(params) @@ -843,9 +849,7 @@ class Candidate(object): self._badexit = self._avg_generic_history(badexit) / ONIONOO_SCALE_ONE def is_candidate(self): - must_be_running_now = (PERFORM_IPV4_DIRPORT_CHECKS - or PERFORM_IPV6_DIRPORT_CHECKS) - if (must_be_running_now and not self.is_running()): + if (MUST_BE_RUNNING_NOW and not self.is_running()): logging.info('%s not a candidate: not running now, unable to check ' + 'DirPort consensus download', self._fpr) return False

1 0

[tor/master] Be more tolerant of a decreased number of fallbacks
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 4c832bcfac3463f101d95b29f06e62eecaa64094 Author: teor <teor2345(a)gmail.com> Date: Wed Dec 7 16:42:42 2016 +1100 Be more tolerant of a decreased number of fallbacks --- scripts/maint/updateFallbackDirs.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/maint/updateFallbackDirs.py b/scripts/maint/updateFallbackDirs.py index a7c86d0..eee93c4 100755 --- a/scripts/maint/updateFallbackDirs.py +++ b/scripts/maint/updateFallbackDirs.py @@ -170,8 +170,8 @@ FALLBACK_PROPORTION_OF_GUARDS = None if OUTPUT_CANDIDATES else _FB_POG # Limit the number of fallbacks (eliminating lowest by advertised bandwidth) MAX_FALLBACK_COUNT = None if OUTPUT_CANDIDATES else 200 -# Emit a C #error if the number of fallbacks is more than 10% below MAX -MIN_FALLBACK_COUNT = 0 if OUTPUT_CANDIDATES else MAX_FALLBACK_COUNT*0.9 +# Emit a C #error if the number of fallbacks is less than expected +MIN_FALLBACK_COUNT = 0 if OUTPUT_CANDIDATES else MAX_FALLBACK_COUNT*0.75 # The maximum number of fallbacks on the same address, contact, or family # With 200 fallbacks, this means each operator can see 1% of client bootstraps

1 0

[tor/master] Add and update fallback directory details
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit a0bbd72caaa50cee75b820e86c7170abfea43472 Author: teor <teor2345(a)gmail.com> Date: Wed Dec 21 08:15:21 2016 +1100 Add and update fallback directory details These updates were made after the December 2016 list was created. --- scripts/maint/fallback.whitelist | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/maint/fallback.whitelist b/scripts/maint/fallback.whitelist index d333392..c993be9 100644 --- a/scripts/maint/fallback.whitelist +++ b/scripts/maint/fallback.whitelist @@ -824,4 +824,7 @@ 31.171.155.108:9030 orport=9001 id=D3E5EDDBE5159388704D6785BE51930AAFACEC6F # Email sent directly to teor, verified using relay contact info -89.163.247.43:9030 orport=9001 id=BC7ACFAC04854C77167C7D66B7E471314ED8C410 +89.163.247.43:9030 orport=9001 id=BC7ACFAC04854C77167C7D66B7E471314ED8C410 ipv6=[2001:4ba0:fff7:25::5]:9001 + +# Email sent directly to teor, verified using relay contact info +95.85.8.226:80 orport=443 id=1211AC1BBB8A1AF7CBA86BCE8689AA3146B86423

1 0

[tor/release-0.2.9] Merge branch 'maint-0.2.4' into maint-0.2.5
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 7d3d56e59a3df16615881007a77b77e30fecb831 Merge: db58d4d 8f857c2 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 20 18:19:41 2016 -0500 Merge branch 'maint-0.2.4' into maint-0.2.5 (ours merge -- there is a separate 0.2.5 patch for 20384.)

1 0

[tor/release-0.2.9] Merge branch 'maint-0.2.6' into maint-0.2.7
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 2673b4b7a87bbdc880dd235f490dfb7b8f3c64c9 Merge: a9c8a5f b6227ed Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 20 18:23:19 2016 -0500 Merge branch 'maint-0.2.6' into maint-0.2.7 changes/buf-sentinel | 11 +++++++++++ src/or/buffers.c | 40 ++++++++++++++++++++++++++++++++-------- 2 files changed, 43 insertions(+), 8 deletions(-)

1 0

[tor/release-0.2.9] Merge branch 'maint-0.2.5' into maint-0.2.6
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit 746c51b6132d84fc7899e6b95b4a1666717d8cce Merge: b18bde2 39ef343 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 20 18:22:27 2016 -0500 Merge branch 'maint-0.2.5' into maint-0.2.6 ("ours" merge because there is a separate 20384 patch for 026)

1 0

[tor/release-0.2.8] Merge branch 'maint-0.2.7' into maint-0.2.8
by nickm＠torproject.org 20 Dec '16

20 Dec '16

commit cb2e2056b5637443715bf494be72d0953c75b4be Merge: 7f55aaf 2673b4b Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Dec 20 18:23:35 2016 -0500 Merge branch 'maint-0.2.7' into maint-0.2.8

1 0