May 2015 - tor-commits - lists.torproject.org

[bridgedb/master] Fix bug in proxy.loadProxiesFromFile().
by isis＠torproject.org 01 May '15

01 May '15

commit d9f44fabf0922f1d76a6d313b42ef7ed61ce4977 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat Mar 28 02:38:42 2015 +0000 Fix bug in proxy.loadProxiesFromFile(). Checking `if proxySet` when the ProxySet was empty (as at server startup, particularly when loading the PROXY_LIST_FILES), would evaluate to False, cause the proxies from the files *not* to be loaded into the empty ProxySet. This did not effect loading the list of Tor Exit relays, since we use scripts/get-tor-exits and load the list into memory while calling ProxySet.add() for each one (rather than loading Tor Exit relays from a downloaded file, like we used to do). --- lib/bridgedb/proxy.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/bridgedb/proxy.py b/lib/bridgedb/proxy.py index fe575b3..3940ed6 100644 --- a/lib/bridgedb/proxy.py +++ b/lib/bridgedb/proxy.py @@ -79,14 +79,17 @@ def loadProxiesFromFile(filename, proxySet=None, removeStale=False): logging.info("Reloading proxy lists...") addresses = [] - if proxySet: + + # We have to check the instance because, if the ProxySet was newly + # created, it will likely be empty, causing it to evaluate to False: + if isinstance(proxySet, ProxySet): oldProxySet = proxySet.copy() try: with open(filename, 'r') as proxyFile: for line in proxyFile.readlines(): line = line.strip() - if proxySet: + if isinstance(proxySet, ProxySet): # ProxySet.add() will validate the IP address if proxySet.add(line, tag=filename): logging.info("Added %s to the proxy list." % line) @@ -98,7 +101,7 @@ def loadProxiesFromFile(filename, proxySet=None, removeStale=False): except Exception as error: logging.warn("Error while reading a proxy list file: %s" % str(error)) - if proxySet: + if isinstance(proxySet, ProxySet): stale = list(oldProxySet.difference(addresses)) if removeStale:

1 0

[bridgedb/master] Remove use of logSafely() for area in getBridgesForIP().
by isis＠torproject.org 01 May '15

01 May '15

commit 5261334d1be22c56d653db85715904b77ce60f49 Author: Isis Lovecruft <isis(a)torproject.org> Date: Mon Mar 30 01:28:53 2015 +0000 Remove use of logSafely() for area in getBridgesForIP(). IP address log sanitisation happens automatically now, so we don't need to use logSafely(). --- lib/bridgedb/Dist.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/bridgedb/Dist.py b/lib/bridgedb/Dist.py index c2a8620..9f730c0 100644 --- a/lib/bridgedb/Dist.py +++ b/lib/bridgedb/Dist.py @@ -273,8 +273,7 @@ class IPBasedDistributor(Distributor): % ' '.join([x.func_name for x in bridgeFilterRules])) area = self.areaMapper(ip) - logging.debug("IP mapped to area:\t%s" - % logSafely("{0}.0/24".format(area))) + logging.debug("IP mapped to area:\t%s.0 /24" % area) key1 = '' pos = 0

1 0

[bridgedb/develop] Add IGNORE_NETWORKSTATUS config option.
by isis＠torproject.org 01 May '15

01 May '15

commit bb3a3db7d8b2126bc5543286252e5fbafe750c16 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Apr 30 06:02:43 2015 +0000 Add IGNORE_NETWORKSTATUS config option. --- bridgedb.conf | 20 +++++++++++++++++++- lib/bridgedb/configure.py | 4 ++++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/bridgedb.conf b/bridgedb.conf index cbdead0..51d9047 100644 --- a/bridgedb.conf +++ b/bridgedb.conf @@ -20,10 +20,14 @@ # # CHANGELOG: # ~~~~~~~~~~ -# Changed in version 0.3.2 - 2015-03-30 +# Changed in version 0.3.2 - 2015-04-30 # * CHANGE to using BridgeDB release versions for bridgedb.conf file versions. # * ADD support for specifying bridge rotation periods via the # EMAIL_ROTATION_PERIOD and HTTPS_ROTATION_PERIOD settings. +# * ADD a new option, IGNORE_NETWORKSTATUS, which controls if BridgeDB cares +# whether or not a bridge was present in the BridgeAuthority's +# networkstatus-bridges file, as well as what information is gleaned from +# the bridge-networkstatus documents within that file. # # Changed in version 0.0.15 - 2015-03-26 # * ADD new SUPPORTED_TRANSPORTS and DEFAULT_TRANSPORT settings. @@ -123,6 +127,20 @@ EXTRA_INFO_FILES = ["cached-extrainfo", "cached-extrainfo.new"] # learning which current bridges are Running, as well as their IPv6 addresses. STATUS_FILE = "networkstatus-bridges" +# If ``True``, then the only information which we obtain from the +# networkstatus file (the ``STATUS_FILE`` option) created by the +# BridgeAuthority are the bridge flags (i.e. "Running", "Stable", etc.), and +# we ignore all errors which have anything to do with networkstatus being +# wrong, out-of-date, or missing. +# +# Otherwise, if ``True``, use the networkstatus to determine which bridges +# should be distributed. +# +# See https://bugs.torproject.org/9380#comment:39, +# https://bugs.torproject.org/11216, and https://bugs.torproject.org/15866. +# +IGNORE_NETWORKSTATUS = True + # Certificate file and private key for the HTTPS Distributor. To create a # self-signed cert, run ``scripts/make-ssl-cert`` it will create these files # in your current directory. diff --git a/lib/bridgedb/configure.py b/lib/bridgedb/configure.py index 2992e1c..0056107 100644 --- a/lib/bridgedb/configure.py +++ b/lib/bridgedb/configure.py @@ -120,6 +120,10 @@ def loadConfig(configFile=None, configCls=None): setting = getattr(config, attr, None) # Default to None setattr(config, attr, setting) + for attr in ["IGNORE_NETWORKSTATUS"]: + setting = getattr(config, attr, True) # Default to True + setattr(config, attr, setting) + for attr in ["FORCE_PORTS", "FORCE_FLAGS", "NO_DISTRIBUTION_COUNTRIES"]: setting = getattr(config, attr, []) # Default to empty lists setattr(config, attr, setting)

1 0

[bridgedb/develop] Ignore networkstatus documents (mostly), if configured to do so.
by isis＠torproject.org 01 May '15

01 May '15

commit ccff2c10464b61d7f4654dd70529de1cc97946dc Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Apr 30 06:16:44 2015 +0000 Ignore networkstatus documents (mostly), if configured to do so. * FIXES #15866: https://bugs.torproject.org/15866 --- lib/bridgedb/Main.py | 17 +++++++++++++---- lib/bridgedb/bridges.py | 32 +++++++++++++++++++++++--------- 2 files changed, 36 insertions(+), 13 deletions(-) diff --git a/lib/bridgedb/Main.py b/lib/bridgedb/Main.py index 44413d2..88b386b 100644 --- a/lib/bridgedb/Main.py +++ b/lib/bridgedb/Main.py @@ -110,6 +110,10 @@ def load(state, splitter, clear=False): logging.info("Loading bridges...") + ignoreNetworkstatus = state.IGNORE_NETWORKSTATUS + if ignoreNetworkstatus: + logging.info("Ignoring BridgeAuthority networkstatus documents.") + bridges = {} timestamps = {} @@ -120,8 +124,7 @@ def load(state, splitter, clear=False): logging.info("Processing networkstatus descriptors...") for router in networkstatuses: bridge = Bridge() - bridge.updateFromNetworkStatus(router) - + bridge.updateFromNetworkStatus(router, ignoreNetworkstatus) try: bridge.assertOK() except MalformedBridgeInfo as error: @@ -136,12 +139,18 @@ def load(state, splitter, clear=False): for router in serverdescriptors: try: - bridges[router.fingerprint].updateFromServerDescriptor(router) + bridge = bridges[router.fingerprint] except KeyError: logging.warn( ("Received server descriptor for bridge '%s' which wasn't " "in the networkstatus!") % router.fingerprint) - continue + if ignoreNetworkstatus: + bridge = Bridge() + else: + continue + + try: + bridge.updateFromServerDescriptor(router, ignoreNetworkstatus) except (ServerDescriptorWithoutNetworkstatus, MissingServerDescriptorDigest, ServerDescriptorDigestMismatch) as error: diff --git a/lib/bridgedb/bridges.py b/lib/bridgedb/bridges.py index 93b9fe9..65a462c 100644 --- a/lib/bridgedb/bridges.py +++ b/lib/bridgedb/bridges.py @@ -1379,33 +1379,38 @@ class Bridge(BridgeBackwardsCompatibility): """ return list(set([pt.methodname for pt in self.transports])) - def updateFromNetworkStatus(self, descriptor): + def updateFromNetworkStatus(self, descriptor, ignoreNetworkstatus=False): """Update this bridge's attributes from a parsed networkstatus document. :type descriptor: :api:`stem.descriptors.router_status_entry.RouterStatusEntry` :param descriptor: The networkstatus document for this bridge. + :param bool ignoreNetworkstatus: If ``True``, then ignore most of the + information in the networkstatus document. """ self.descriptors['networkstatus'] = descriptor # These fields are *only* found in the networkstatus document: - self.descriptorDigest = descriptor.digest self.flags.update(descriptor.flags) - self.bandwidth = descriptor.bandwidth + self.descriptorDigest = descriptor.digest + + if not ignoreNetworkstatus: + self.bandwidth = descriptor.bandwidth # These fields are also found in the server-descriptor. We will prefer # to use the information taken later from the server-descriptor # because it is signed by the bridge. However, for now, we harvest all # the info we can: self.fingerprint = descriptor.fingerprint - self.nickname = descriptor.nickname - self.address = descriptor.address - self.orPort = descriptor.or_port - self._updateORAddresses(descriptor.or_addresses) + if not ignoreNetworkstatus: + self.nickname = descriptor.nickname + self.address = descriptor.address + self.orPort = descriptor.or_port + self._updateORAddresses(descriptor.or_addresses) - def updateFromServerDescriptor(self, descriptor): + def updateFromServerDescriptor(self, descriptor, ignoreNetworkstatus=False): """Update this bridge's info from an ``@type bridge-server-descriptor``. .. info:: @@ -1425,7 +1430,16 @@ class Bridge(BridgeBackwardsCompatibility): networkstatus entry, or its **descriptor** digest didn't match the expected digest (from the networkstatus entry). """ - self._checkServerDescriptor(descriptor) + if ignoreNetworkstatus: + try: + self._checkServerDescriptor(descriptor) + except (ServerDescriptorWithoutNetworkstatus, + MissingServerDescriptorDigest, + ServerDescriptorDigestMismatch) as ignored: + logging.warn(ignored) + else: + self._checkServerDescriptor(descriptor) + self.descriptors['server'] = descriptor # Replace the values which we harvested from the networkstatus

1 0

[bridgedb/develop] Add unittests for Bridge.updateFrom*(ignoreNetworkstatus=True).
by isis＠torproject.org 01 May '15

01 May '15

commit f128f46de25361ec51341f850b1fc43de06cbf2b Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Apr 22 01:25:42 2015 +0000 Add unittests for Bridge.updateFrom*(ignoreNetworkstatus=True). --- lib/bridgedb/test/test_bridges.py | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/lib/bridgedb/test/test_bridges.py b/lib/bridgedb/test/test_bridges.py index 5a79d5e..7778b02 100644 --- a/lib/bridgedb/test/test_bridges.py +++ b/lib/bridgedb/test/test_bridges.py @@ -1029,6 +1029,25 @@ class BridgeTests(unittest.TestCase): self.assertIn((ipaddr.IPAddress('123.34.56.78'), 36488, 4), self.bridge.allVanillaAddresses) + def test_Bridge_updateFromNetworkStatus_ignoreNetworkstatus(self): + """Calling updateFromNetworkStatus([…], ignoreNetworkstatus=True) + should update the Bridge's flags, its fingerprint, and its + descriptorDigest, and nothing else. + """ + self.bridge.updateFromNetworkStatus(self.networkstatus, + ignoreNetworkstatus=True) + + self.assertTrue(self.bridge.flags.running) + self.assertTrue(self.bridge.flags.stable) + self.assertIsNotNone(self.bridge.descriptorDigest) + self.assertEqual(self.bridge.fingerprint, + '2C3225C4805331025E211F4B6E5BF45C333FDD2C') + + self.assertIsNone(self.bridge.nickname) + self.assertIsNone(self.bridge.address) + self.assertIsNone(self.bridge.orPort) + self.assertIsNone(self.bridge.orPort) + def test_Bridge_updateFromServerDescriptor(self): """ """ self.bridge.updateFromNetworkStatus(self.networkstatus) @@ -1046,6 +1065,16 @@ class BridgeTests(unittest.TestCase): self.bridge.updateFromServerDescriptor, self.serverdescriptor) + def test_Bridge_updateFromServerDescriptor_ignoreNetworkstatus_no_networkstatus(self): + """Parsing a server descriptor for a bridge which wasn't included in + the networkstatus document from the BridgeAuthority, when + ignoreNetworkstatus=True, should not raise any warnings. + """ + self.bridge.updateFromServerDescriptor(self.serverdescriptor, + ignoreNetworkstatus=True) + self.assertIsNone(self.bridge.descriptors['networkstatus']) + self.assertIsNotNone(self.bridge.descriptors['server']) + def test_Bridge_verifyExtraInfoSignature_good_signature(self): """Calling _verifyExtraInfoSignature() with a descriptor which has a good signature should return None.

1 0

[bridgedb/develop] Merge branch 'fix/15866-broken-bridgeauth' into develop
by isis＠torproject.org 01 May '15

01 May '15

commit ee9a76d18c4328c3f3ce44ec62f166cf870d4003 Merge: 9b2a724 f128f46 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Apr 22 01:25:42 2015 +0000 Merge branch 'fix/15866-broken-bridgeauth' into develop bridgedb.conf | 20 +++++++++++++++++++- lib/bridgedb/Main.py | 17 +++++++++++++---- lib/bridgedb/bridges.py | 32 +++++++++++++++++++++++--------- lib/bridgedb/configure.py | 4 ++++ lib/bridgedb/test/test_bridges.py | 29 +++++++++++++++++++++++++++++ 5 files changed, 88 insertions(+), 14 deletions(-)

1 0

[bridgedb/develop] Merge branch 'release-0.3.2'
by isis＠torproject.org 01 May '15

01 May '15

1 0

[bridgedb/develop] Update CHANGELOG for BridgeDB-0.3.2.
by isis＠torproject.org 01 May '15

01 May '15

commit 251945414b7a7f166b9528b4620d4b51ba7a645a Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Apr 22 01:25:42 2015 +0000 Update CHANGELOG for BridgeDB-0.3.2. --- CHANGELOG | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/CHANGELOG b/CHANGELOG index 8c1bb0e..240794e 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,10 +1,19 @@ -Changes in version 0.3.2 - 2015-03-?? +Changes in version 0.3.2 - 2015-05-01 * FIXES a problem with the calculation of Levenshtein distances between blacklisted email addresses and those on incoming email. This fixes a problem with the fuzzy matching implemented in #9385: https://bugs.torproject.org/9385. + * FIXES #1839 https://bugs.torproject.org/1839 + BridgeDB's distributors now rotate their hashrings at + configurable scheduled intervals. + + * FIXES #4771 https://bugs.torproject.org/4771 + BridgeDB now records which of the HTTPS Distributor's + sub-hashrings are used for clients coming from Tor Exit nodes and + other known proxies. + * FIXES #12504 https://bugs.torproject.org/12504 Which Pluggable Transports BridgeDB distributes is now easily configurable via the bridgedb.conf configuration file. @@ -24,11 +33,31 @@ Changes in version 0.3.2 - 2015-03-?? work. This fixes the issue by excluding broken transports from being distributed to clients. + * FIXES #15522 https://bugs.torproject.org/15522 + For all clients who are coming from IPv6 addresses and are not + using Tor, who go to https://bridges.torproject.org, BridgeDB now + groups these clients together by /32. This "grouping" causes all + IPv6 clients within the same IPv6 /32 to get the same bridges. + Previously, BridgeDB grouped IPv6 clients by /64 (which is + ridiculously small, considering standard IPv6 allocation sizes). + + For all clients who are coming from IPv4 addresses and are not + using Tor, BridgeDB now groups these clients together by /16. + Previously, BridgeDB grouped IPv4 clients by /24. (This latter + change was technically made as part of #4771.) + * FIXES #15464 https://bugs.torproject.org/15464 The setup procedure for creating a BridgeDB Continuous Integration build machine is now simplified and generalised to include build environments like Jenkins, not just TravisCI. + * FIXES #15866 https://bugs.torproject.org/15866 + BridgeDB now ignores nearly all the information in the + networkstatus-bridges file created by the BridgeAuthority. + + * ADDS benchmark tests to BridgeDB's test suite, and some of + BridgeDB's algorithms have been revised to improve their speed. + Changes in version 0.3.1 - 2015-03-24

1 0

[bridgedb/develop] Merge branch 'master' into develop
by isis＠torproject.org 01 May '15

01 May '15

commit 2d3685220f4b473fec47480bc859bc1953cb5199 Merge: 2519454 94dd106 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Apr 22 01:25:42 2015 +0000 Merge branch 'master' into develop

1 0

[bridgedb/develop] Change default Makefile directive.
by isis＠torproject.org 01 May '15

01 May '15

commit 9b2a72469175d92cca13d6559918ba1b0a8c0749 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Apr 22 01:25:42 2015 +0000 Change default Makefile directive. Conflicts: Makefile --- Makefile | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/Makefile b/Makefile index c6c0442..2c09bbe 100644 --- a/Makefile +++ b/Makefile @@ -5,8 +5,7 @@ TRIAL:=$(shell which trial) VERSION:=$(shell git describe) -all: - python setup.py build +all: uninstall clean install coverage-test test: python setup.py test

1 0