July 2015 - tor-commits - lists.torproject.org

[bridgedb/master] Update copyright dates and version number for example bridgedb.conf.
by isis＠torproject.org 25 Jul '15

25 Jul '15

commit 8037df3a2d8475185175a22da66ba245f76fc934 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sun May 10 22:05:44 2015 +0000 Update copyright dates and version number for example bridgedb.conf. --- bridgedb.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/bridgedb.conf b/bridgedb.conf index 51d9047..2c83582 100644 --- a/bridgedb.conf +++ b/bridgedb.conf @@ -13,9 +13,9 @@ # :authors: The Tor Project, Inc. # :license: This file is freely distributed as part of BridgeDB, see LICENSE # for details. -# :copyright: (c) 2007-2014 The Tor Project, Inc. -# (c) 2007-2014, all sentient entities within the AUTHORS file -# :version: 0.0.14 +# :copyright: (c) 2007-2015 The Tor Project, Inc. +# (c) 2007-2015, all sentient entities within the AUTHORS file +# :version: 0.3.2 #=============================================================================== # # CHANGELOG:

1 0

[bridgedb/master] Remove extra whitespace from bridgedb.bridges.Bridge docstring.
by isis＠torproject.org 25 Jul '15

25 Jul '15

commit 7c142b4d3a6646f531574bba77c283ffc0a479d0 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sun May 10 22:23:00 2015 +0000 Remove extra whitespace from bridgedb.bridges.Bridge docstring. Conflicts: lib/bridgedb/bridges.py --- lib/bridgedb/bridges.py | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/lib/bridgedb/bridges.py b/lib/bridgedb/bridges.py index 65acbc7..37e17cd 100644 --- a/lib/bridgedb/bridges.py +++ b/lib/bridgedb/bridges.py @@ -775,16 +775,12 @@ class Bridge(BridgeBackwardsCompatibility): :type fingerprint: str or ``None`` :ivar fingerprint: This ``Bridge``'s fingerprint, in lowercased hexadecimal format. - :type nickname: str or ``None`` :ivar nickname: This ``Bridge``'s router nickname. - :type socksPort: int :ivar socksPort: This ``Bridge``'s SOCKSPort. Should always be ``0``. - :type dirPort: int :ivar dirPort: This ``Bridge``'s DirPort. Should always be ``0``. - :type orAddresses: list :ivar orAddresses: A list of 3-tuples in the form:: (ADDRESS, PORT, IP_VERSION) @@ -792,39 +788,30 @@ class Bridge(BridgeBackwardsCompatibility): * ADDRESS is an :class:`ipaddr.IPAddress`, * PORT is an ``int``, * IP_VERSION is either ``4`` or ``6``. - :type transports: list :ivar transports: A list of :class:`PluggableTransport`s, one for each transport that this :class:`Bridge` currently supports. - :type flags: :class:`~bridgedb.bridges.Flags` :ivar flags: All flags assigned by the BridgeAuthority to this :class:`Bridge`. - :type hibernating: bool :ivar hibernating: ``True`` if this :class:`Bridge` is hibernating and not currently serving clients (e.g. if the Bridge hit its configured ``RelayBandwidthLimit``); ``False`` otherwise. - :type _blockedIn: dict :ivar _blockedIn: A dictionary of ``ADDRESS:PORT`` pairs to lists of lowercased, two-letter country codes (e.g. ``"us"``, ``"gb"``, ``"cn"``, etc.) which that ``ADDRESS:PORT`` pair is blocked in. - :type contact: str or ``None`` :ivar contact: The contact information for the this Bridge's operator. - :type family: set or ``None`` :ivar family: The fingerprints of other Bridges related to this one. - :type platform: str or ``None`` :ivar platform: The ``platform`` line taken from the ``@type bridge-server-descriptor``, e.g. ``'Tor 0.2.5.4-alpha on Linux'``. - :type software: :api:`stem.version.Version` or ``None`` :ivar software: The OR version portion of the ``platform`` line. - :type os: str or None :ivar os: The OS portion of the ``platform`` line. """

1 0

[bridgedb/master] Decrease logging in bridgedb.bridges.Flags.update().
by isis＠torproject.org 25 Jul '15

25 Jul '15

commit ce6bb3db25117178e38e4b418df48a9efdb35623 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sun May 10 22:20:42 2015 +0000 Decrease logging in bridgedb.bridges.Flags.update(). --- lib/bridgedb/bridges.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/lib/bridgedb/bridges.py b/lib/bridgedb/bridges.py index 65a462c..65acbc7 100644 --- a/lib/bridgedb/bridges.py +++ b/lib/bridgedb/bridges.py @@ -120,11 +120,6 @@ class Flags(object): self.stable = 'Stable' in flags self.valid = 'Valid' in flags - if not self.running: - logging.debug("Bridge doesn't have the Running flag.") - if not self.stable: - logging.debug("Bridge doesn't have the Stable flag.") - class BridgeAddressBase(object): """A base class for describing one of a :class:`Bridge`'s or a

1 0

[bridgedb/master] Fix a unittest for PluggableTransport.
by isis＠torproject.org 25 Jul '15

25 Jul '15

commit 31a6b5a18ba9b5319d1c32a1fda147f502b52827 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sun May 10 22:57:28 2015 +0000 Fix a unittest for PluggableTransport. The ptargs was supposed to be a list, and previously it was erroneously a string. --- lib/bridgedb/test/test_bridges.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/bridgedb/test/test_bridges.py b/lib/bridgedb/test/test_bridges.py index 99fc458..88b858e 100644 --- a/lib/bridgedb/test/test_bridges.py +++ b/lib/bridgedb/test/test_bridges.py @@ -1715,7 +1715,8 @@ class BridgeTests(unittest.TestCase): self.bridge.updateFromServerDescriptor(self.serverdescriptor) obfs4 = self.extrainfo.transport['obfs4'] - obfs4 = (u'1.1.1.1', 1111, obfs4[-1][-1].replace('iat-mode=0,', '')) + ptargs = [str(obfs4[-1][-1].replace('iat-mode=0', '')),] + obfs4 = (u'1.1.1.1', 1111, ptargs) self.extrainfo.transport['obfs4'] = obfs4 self.bridge.updateFromExtraInfoDescriptor(self.extrainfo)

1 0

[bridgedb/master] Cleanup bridgedb.Stability module docstring and add file header.
by isis＠torproject.org 25 Jul '15

25 Jul '15

commit 62701b3c3673ad474831363827bcb2a22580bef6 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sun May 10 23:13:42 2015 +0000 Cleanup bridgedb.Stability module docstring and add file header. --- lib/bridgedb/Stability.py | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/lib/bridgedb/Stability.py b/lib/bridgedb/Stability.py index 981e005..ad6fd0c 100644 --- a/lib/bridgedb/Stability.py +++ b/lib/bridgedb/Stability.py @@ -1,13 +1,30 @@ +# -*- coding: utf-8 ; test-case-name: bridgedb.test.test_Stability -*- +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: please see the AUTHORS file for attributions +# :copyright: (c) 2013-2015, Isis Lovecruft +# (c) 2013-2015, Matthew Finkel +# (c) 2012-2015, Aaron Gibson +# (c) 2007-2015, Nick Mathewson +# (c) 2007-2015, The Tor Project, Inc. +# :license: see LICENSE for licensing information + +"""This module provides functionality for tracking bridge stability metrics. + +Bridge stability metrics are calculated using the model introduced in +`"An Analysis of Tor Bridge Stability"`_ and +`implemented in the Tor Metrics library`_. + +.. An Analysis of Tor Bridge Stability: + https://metrics.torproject.org/papers/bridge-stability-2011-10-31.pdf + Karsten Loesing, An Analysis of Tor Bridge Stability. Technical Report. + The Tor Project, October 2011. + +.. implemented in the Tor Metrics library: + https://gitweb.torproject.org/metrics-tasks/task-4255/SimulateBridgeStabili… """ -This module provides functionality for tracking bridge stability metrics, using -the model introduced in [1] and implemented in [2]. -[1] Karsten Loesing, An Analysis of Tor Bridge Stability. Technical Report. -The Tor Project, October 2011. -https://metrics.torproject.org/papers/bridge-stability-2011-10-31.pdf - -[2] https://gitweb.torproject.org/metrics-tasks/task-4255/SimulateBridgeStabili… -""" import logging import bridgedb.Storage @@ -15,6 +32,7 @@ import bridgedb.Storage weighting_factor = float(19)/float(20) discountIntervalMillis = long(60*60*12*1000) + class BridgeHistory(object): """ Record Class that tracks a single Bridge The fields stored are:

1 0

[bridgedb/master] Removed unused import of twisted.internet.defer from test_safelog.
by isis＠torproject.org 25 Jul '15

25 Jul '15

commit 57aa6d31d2c606e2cd3a0b8c6d97b9662236f675 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sun May 10 22:16:03 2015 +0000 Removed unused import of twisted.internet.defer from test_safelog. --- lib/bridgedb/test/test_safelog.py | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/bridgedb/test/test_safelog.py b/lib/bridgedb/test/test_safelog.py index bc92305..247db06 100644 --- a/lib/bridgedb/test/test_safelog.py +++ b/lib/bridgedb/test/test_safelog.py @@ -4,7 +4,6 @@ import re -from twisted.internet import defer from twisted.test.proto_helpers import StringTransport from twisted.trial import unittest

1 0

[bridgedb/master] Decrease logging for extrainfo descriptor deduplication.
by isis＠torproject.org 25 Jul '15

25 Jul '15

commit ccb709c56eb42dfb2a3fb10540c7d2d303c78c68 Author: Isis Lovecruft <isis(a)torproject.org> Date: Mon May 11 20:19:32 2015 +0000 Decrease logging for extrainfo descriptor deduplication. It was *really* noisy. --- lib/bridgedb/parse/descriptors.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/lib/bridgedb/parse/descriptors.py b/lib/bridgedb/parse/descriptors.py index 3f8d48e..b90e2e7 100644 --- a/lib/bridgedb/parse/descriptors.py +++ b/lib/bridgedb/parse/descriptors.py @@ -189,9 +189,6 @@ def deduplicate(descriptors, statistics=False): for descriptor in descriptors: fingerprint = descriptor.fingerprint - logging.debug("Deduplicating %s descriptor for router %s" - % (descriptor.__class__.__name__.rsplit('.', 1)[-1], - safelog.logSafely(fingerprint))) if fingerprint in duplicates: duplicates[fingerprint].append(descriptor) else:

1 0

[bridgedb/master] Move Main.updateBridgeHistory() → Stability.updateBridgeHistory().
by isis＠torproject.org 25 Jul '15

25 Jul '15

commit 6417ffe9681138ac1fb9dbccd897df61ef358ae1 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sun May 10 23:59:42 2015 +0000 Move Main.updateBridgeHistory() → Stability.updateBridgeHistory(). Conflicts: lib/bridgedb/Main.py --- lib/bridgedb/Main.py | 42 ++-------------------------------------- lib/bridgedb/Stability.py | 40 ++++++++++++++++++++++++++++++++++++++ lib/bridgedb/test/test_Main.py | 28 ++++++++++++++------------- 3 files changed, 57 insertions(+), 53 deletions(-) diff --git a/lib/bridgedb/Main.py b/lib/bridgedb/Main.py index 88b386b..4a00799 100644 --- a/lib/bridgedb/Main.py +++ b/lib/bridgedb/Main.py @@ -40,51 +40,13 @@ from bridgedb.configure import Conf from bridgedb.parse import descriptors from bridgedb.parse import options from bridgedb.parse.addr import isIPAddress -from bridgedb.schedule import toUnixSeconds import bridgedb.Storage from bridgedb import Bridges from bridgedb import Dist -from bridgedb.Stability import addOrUpdateBridgeHistory - - -def updateBridgeHistory(bridges, timestamps): - """Process all the timestamps and update the bridge stability statistics in - the database. - - .. warning: This function is extremely expensive, and will keep getting - more and more expensive, on a linearithmic scale, every time it is - called. Blame the :mod:`bridgedb.Stability` module. - - :param dict bridges: All bridges from the descriptors, parsed into - :class:`bridgedb.bridges.Bridge`s. - :param dict timestamps: A dictionary whose keys are bridge fingerprints, - and whose values are lists of integers, each integer being a timestamp - (in seconds since Unix Epoch) for when a descriptor for that bridge - was published. - :rtype: dict - :returns: The original **timestamps**, but which each list of integers - (re)sorted. - """ - logging.debug("Beginning bridge stability calculations") - sortedTimestamps = {} - - for fingerprint, stamps in timestamps.items()[:]: - stamps.sort() - bridge = bridges[fingerprint] - for timestamp in stamps: - logging.debug( - ("Adding/updating timestamps in BridgeHistory for %s in " - "database: %s") % (fingerprint, timestamp)) - timestamp = toUnixSeconds(timestamp.timetuple()) - addOrUpdateBridgeHistory(bridge, timestamp) - # Replace the timestamps so the next sort is (hopefully) less - # expensive: - sortedTimestamps[fingerprint] = stamps - - logging.debug("Stability calculations complete") - return sortedTimestamps +from bridgedb.Stability import updateBridgeHistory + def load(state, splitter, clear=False): """Read and parse all descriptors, and load into a bridge splitter. diff --git a/lib/bridgedb/Stability.py b/lib/bridgedb/Stability.py index ad6fd0c..fbc8e88 100644 --- a/lib/bridgedb/Stability.py +++ b/lib/bridgedb/Stability.py @@ -28,6 +28,9 @@ Bridge stability metrics are calculated using the model introduced in import logging import bridgedb.Storage +from bridgedb.schedule import toUnixSeconds + + # tunables weighting_factor = float(19)/float(20) discountIntervalMillis = long(60*60*12*1000) @@ -253,3 +256,40 @@ def updateWeightedTime(statusPublicationMillis): for bh in bhToUpdate: db.updateIntoBridgeHistory(bh) + +def updateBridgeHistory(bridges, timestamps): + """Process all the timestamps and update the bridge stability statistics in + the database. + + .. warning: This function is extremely expensive, and will keep getting + more and more expensive, on a linearithmic scale, every time it is + called. Blame the :mod:`bridgedb.Stability` module. + + :param dict bridges: All bridges from the descriptors, parsed into + :class:`bridgedb.bridges.Bridge`s. + :param dict timestamps: A dictionary whose keys are bridge fingerprints, + and whose values are lists of integers, each integer being a timestamp + (in seconds since Unix Epoch) for when a descriptor for that bridge + was published. + :rtype: dict + :returns: The original **timestamps**, but which each list of integers + (re)sorted. + """ + logging.debug("Beginning bridge stability calculations") + sortedTimestamps = {} + + for fingerprint, stamps in timestamps.items()[:]: + stamps.sort() + bridge = bridges[fingerprint] + for timestamp in stamps: + logging.debug( + ("Adding/updating timestamps in BridgeHistory for %s in " + "database: %s") % (fingerprint, timestamp)) + timestamp = toUnixSeconds(timestamp.timetuple()) + addOrUpdateBridgeHistory(bridge, timestamp) + # Replace the timestamps so the next sort is (hopefully) less + # expensive: + sortedTimestamps[fingerprint] = stamps + + logging.debug("Stability calculations complete") + return sortedTimestamps diff --git a/lib/bridgedb/test/test_Main.py b/lib/bridgedb/test/test_Main.py index 0b14319..5ae4025 100644 --- a/lib/bridgedb/test/test_Main.py +++ b/lib/bridgedb/test/test_Main.py @@ -48,13 +48,15 @@ w Bandwidth=2094050 p reject 1-65535 ''' -def mockAddOrUpdateBridgeHistory(bridge, timestamp): - """A mocked version of :func:`bridgedb.Stability.addOrUpdateBridgeHistory` +def mockUpdateBridgeHistory(bridges, timestamps): + """A mocked version of :func:`bridgedb.Stability.updateBridgeHistory` which doesn't access the database (so that we can test functions which - call it, like :func:`bridgedb.Main.updateBridgeHistory`). + call it, like :func:`bridgedb.Main.load`). """ - print("Pretending to update Bridge %s with timestamp %s..." % - (bridge.fingerprint, timestamp)) + for fingerprint, stamps in timestamps.items()[:]: + for timestamp in stamps: + print("Pretending to update Bridge %s with timestamp %s..." % + (fingerprint, timestamp)) class MockBridgeHolder(BridgeHolder): @@ -170,23 +172,23 @@ class MainTests(unittest.TestCase): # Functions which some tests mock, which we'll need to re-replace # later in tearDown(): - self._orig_addOrUpdateBridgeHistory = Main.addOrUpdateBridgeHistory + self._orig_updateBridgeHistory = Main.updateBridgeHistory self._orig_sys_argv = sys.argv def tearDown(self): - """Replace the mocked mockAddOrUpdateBridgeHistory() function with the - real function, Stability.addOrUpdateBridgeHistory(). + """Replace the mocked mockUpdateBridgeHistory() function with the + real function, Stability.updateBridgeHistory(). """ - Main.addOrUpdateBridgeHistory = self._orig_addOrUpdateBridgeHistory + Main.updateBridgeHistory = self._orig_updateBridgeHistory sys.argv = self._orig_sys_argv def test_Main_updateBridgeHistory(self): """Main.updateBridgeHistory should update some timestamps for some bridges. """ - # Mock the addOrUpdateBridgeHistory() function so that we don't try to + # Mock the updateBridgeHistory() function so that we don't try to # access the database: - Main.addOrUpdateBridgeHistory = mockAddOrUpdateBridgeHistory + Main.updateBridgeHistory = mockUpdateBridgeHistory # Get the bridges into the mocked splitter d = deferToThread(Main.load, self.state, self.splitter) @@ -222,12 +224,12 @@ class MainTests(unittest.TestCase): """ # Mock the addOrUpdateBridgeHistory() function so that we don't try to # access the database: - Main.addOrUpdateBridgeHistory = mockAddOrUpdateBridgeHistory + Main.updateBridgeHistory = mockUpdateBridgeHistory state = self.state state.COLLECT_TIMESTAMPS = True # The reactor is deferring this to a thread, so the test execution - # here isn't actually covering the Main.updateBridgeHistory() + # here isn't actually covering the Storage.updateBridgeHistory() # function: Main.load(state, self.splitter)

1 0

[bridgedb/master] Spend even less time running benchmark tests.
by isis＠torproject.org 25 Jul '15

25 Jul '15

commit 80939644b8833243e64a7cb51783f83dc7603e73 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sun May 10 22:09:01 2015 +0000 Spend even less time running benchmark tests. --- lib/bridgedb/test/test_parse_descriptors.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/bridgedb/test/test_parse_descriptors.py b/lib/bridgedb/test/test_parse_descriptors.py index a7b22cf..46d86f1 100644 --- a/lib/bridgedb/test/test_parse_descriptors.py +++ b/lib/bridgedb/test/test_parse_descriptors.py @@ -499,6 +499,9 @@ class ParseDescriptorsTests(unittest.TestCase): def test_parse_descriptors_parseExtraInfoFiles_benchmark_1000_bridges(self): """Benchmark test for ``b.p.descriptors.parseExtraInfoFiles``.""" + raise SkipTest(("This test can take several minutes to complete. " + "Run it on your own free time.")) + print() for i in range(1, 6): descFiles = self.createDuplicatesForBenchmark(b=1000, n=i) @@ -509,7 +512,7 @@ class ParseDescriptorsTests(unittest.TestCase): """Benchmark test for ``b.p.descriptors.parseExtraInfoFiles``. The algorithm should grow linearly in the number of duplicates. """ - raise SkipTest(("This test takes ~7 minutes to complete. " + raise SkipTest(("This test can take several minutes to complete. " "Run it on your own free time.")) print()

1 0

[bridgedb/master] Fix idempotency and reentrancy issues with Bridge.allVanillaAddresses.
by isis＠torproject.org 25 Jul '15

25 Jul '15

commit 6e60cc33bac91442ed7b45994ef312ebd86916bd Author: Isis Lovecruft <isis(a)torproject.org> Date: Sun May 10 22:38:38 2015 +0000 Fix idempotency and reentrancy issues with Bridge.allVanillaAddresses. Conflicts: lib/bridgedb/bridges.py --- lib/bridgedb/bridges.py | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/lib/bridgedb/bridges.py b/lib/bridgedb/bridges.py index 37e17cd..e1c8738 100644 --- a/lib/bridgedb/bridges.py +++ b/lib/bridgedb/bridges.py @@ -1105,10 +1105,23 @@ class Bridge(BridgeBackwardsCompatibility): :returns: All of this bridge's ORAddresses, as well as its ORPort IP address and port. """ - addresses = self.orAddresses - # Add the default ORPort address (it will always be IPv4, otherwise - # Stem should have raised a ValueError during parsing): - addresses.append((self.address, self.orPort, 4)) + # Force deep-copying of the orAddresses. Otherwise, the later use of + # ``addresses.append()`` is both non-reentrant and non-idempotent, as + # it would change the value of ``Bridge.orAddresses``, as well as + # append a (possibly updated, if ``Bridge.address`` or + # ``Bridge.orPort`` changed!) new copy of the bridge's primary + # ORAddress each time this property is called. + addresses = self.orAddresses[:] + + # Add the default ORPort address. It will always be IPv4, otherwise + # Stem should have raised a ValueError during parsing. But for + # testability, check which type it is: + version = 4 + if isIPv6(self.address): + version = 6 + + addresses.append((self.address, self.orPort, version)) + return addresses def assertOK(self):

1 0