tor-commits December 2017

tor-commits@lists.torproject.org

14 participants
2422 discussions

[bridgedb/master] Add a human-readable __str__() method to BridgeRequest classes.
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit f37c22c6284f13fb9f26f78926f0d115b468fb71 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Dec 20 21:38:12 2017 +0000 Add a human-readable __str__() method to BridgeRequest classes. --- bridgedb/bridgerequest.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/bridgedb/bridgerequest.py b/bridgedb/bridgerequest.py index e555e0a..d5af05f 100644 --- a/bridgedb/bridgerequest.py +++ b/bridgedb/bridgerequest.py @@ -117,6 +117,15 @@ class BridgeRequestBase(object): self.client = 'default' self.valid = False + def __str__(self): + """Return a human-readable string describing this bridge request.""" + return "%s(ipVersion=%d, transports=%s, notBlockedIn=%s, valid=%s)" % \ + (self.__class__.__name__, + self.ipVersion, + self.transports, + self.notBlockedIn, + self.valid) + @property def ipVersion(self): """The IP version of bridge addresses to distribute to the client.

1 0

[bridgedb/master] Merge branch 'fix/24637' into develop
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 6ace9a350f5f700d9f1f6ff88c6a275c1fb18414 Merge: 91305fa 4c79973 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Dec 20 22:59:15 2017 +0000 Merge branch 'fix/24637' into develop README.rst | 9 ++-- bridgedb/bridgerequest.py | 9 ++++ bridgedb/distributors/moat/server.py | 73 +++++++++++++++++++++----- bridgedb/test/test_distributors_moat_server.py | 55 +++++++++++++++++-- 4 files changed, 126 insertions(+), 20 deletions(-)

1 0

[bridgedb/master] Revert "Change "client-transport" moat resource type to "moat-transports"."
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 6aaa2ed270e432efddd7cdc9ea81b83c00fe9c75 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Dec 13 20:10:32 2017 +0000 Revert "Change "client-transport" moat resource type to "moat-transports"." This reverts commit 50a9321fadcfd0e9a4f30bc1919c1bf46d27d8e1. --- bridgedb/distributors/moat/server.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/bridgedb/distributors/moat/server.py b/bridgedb/distributors/moat/server.py index 096b8c0..eb356fe 100644 --- a/bridgedb/distributors/moat/server.py +++ b/bridgedb/distributors/moat/server.py @@ -386,12 +386,12 @@ class CaptchaFetchResource(CaptchaResource): :param str request: A JSON blob containing the following fields: * "version": The moat protocol version. - * "type": "moat-transports". - * "supported": ["TRANSPORT", … ] + * "type": "client-transports". + * "supported": ['TRANSPORT', … ] where: - * ``TRANSPORT`` is a string identifying a transport, e.g. - "obfs3" or "obfs4". Currently supported transport identifiers - are: "vanilla", "fte", "obfs3", "obfs4", "scramblesuit". + * TRANSPORT is a string identifying a transport, e.g. "obfs3" or + "obfs4". Currently supported transport identifiers are: + "vanilla", "fte", "obfs3", "obfs4", "scramblesuit". :rtype: list :returns: The list of transports the client supports. """ @@ -401,10 +401,10 @@ class CaptchaFetchResource(CaptchaResource): encoded_data = request.content.read() data = json.loads(encoded_data)["data"][0] - if data["type"] != "moat-transports": + if data["type"] != "client-transports": raise ValueError( "Bad JSON API object type: expected %s got %s" % - ('moat-transports', data["type"])) + ('client-transports', data["type"])) elif data["version"] != MOAT_API_VERSION: raise ValueError( "Client requested protocol version %s, but we're using %s" %

1 0

[bridgedb/master] Merge branch 'fix/24636' into develop
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 91305fa523c8b9e6a631d8c29f583e3da722832e Merge: 9a926f2 da84c25 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Dec 20 22:59:11 2017 +0000 Merge branch 'fix/24636' into develop README.rst | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-)

1 0

[bridgedb/develop] Add CHANGELOG entries for 0.6.1.
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit b828d5777f02dc6f544a0ea821952dc1f06e2d7d Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Dec 13 18:55:50 2017 +0000 Add CHANGELOG entries for 0.6.1. --- CHANGELOG | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index 7dbe222..effc405 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,3 +1,56 @@ +Changes in version 0.6.1 - 2017-12-13 + + * ADDS a shell script, scripts/test-moat, for testing either a + locally-running moat server, or a remote one through a meek + tunnel. + Thanks to David Fifield for his work on meek, assistance setting + it up, and providing the first version of this script. + + * FIXES #24433 https://bugs.torproject.org/24433 + The test-moat script wasn't sending an X-Forwarded-For header, + which triggered a bug in the moat server, since the CAPTCHA + solution includes an HMAC based on the client's IP (forwarded + through all the several layers of tunnels/proxies). + + * FIXES #24443 https://bugs.torproject.org/24443 + Due to a difference between how booleans are parsed by Python's + json library and normal Python booleans, the moat server was + generating and returning QRCodes… regardless of whether the remote + client application asked for one. This is now fixed. + + * FIXES #24460 https://bugs.torproject.org/24460 + There was an unhandled error when sending certain (what appears to + be possibly malicious? but in a very strange way) requests to + BridgeDB's HTTPS distributor. The robots making the requests were + attempting to request bridges, but were presenting a CAPTCHA + solution without the correct HTML form field parameters present, + which isn't possible through normal usage of the web interface. + Whoever or whatever is doing this is now going to be endlessly + redirected so that they may forever spiral in their own private + internet hell. I reserve the right come up with a worse fate for + them later, should I get bored. + + * FIXES #3015 https://bugs.torproject.org/3015 + BridgeDB has had a partially-implemented concept of "buckets" + since the age of the dinosaurs: write some of the unallocated + bridges to a file which should (somehow) be manually distributed. + In addition to be unused and untested, there were several issues + with the buckets, the most significant of which were the inability + to request pluggable transports in a bucket and the fact that + buckets were not persistent in any way (e.g. if i request a bucket + of 50 bridges for Gomez and another with 50 for Morticia, they + might end up with some of the same bridges, further, tomorrow + they'll end up with 50 possibly different bridges than those they + received today). All of this code is now removed. + +And includes the following general changes: + + * FIXES issues with JSON quote syntax and a mistaken JSON API + "type" parameter in the specification of the moat server (in the + README). + Thanks to Mark Smith and Kathy Brade for pointing out the issues. + + Changes in version 0.6.0 - 2017-11-15 * ADDS a new JSON API distributor called "moat", which is intended

1 0

[bridgedb/develop] Merge branch 'fix/3015-remove-buckets' into develop
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit e3a2ea593dcc6614b865572d9fc1d3b333b5770d Merge: 16e7a2d f4d5591 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 28 21:22:24 2017 +0000 Merge branch 'fix/3015-remove-buckets' into develop README.rst | 23 +-- bridgedb.conf | 6 - bridgedb/Bridges.py | 18 +-- bridgedb/Bucket.py | 288 ---------------------------------- bridgedb/main.py | 51 +++--- bridgedb/parse/options.py | 29 ++-- bridgedb/runner.py | 14 -- bridgedb/test/test_Bucket.py | 63 -------- bridgedb/test/test_bridgedb_script.py | 19 --- bridgedb/test/test_main.py | 26 +-- doc/sphinx/source/bridgedb.Bucket.rst | 8 - doc/sphinx/source/bridgedb.rst | 1 - doc/sphinx/source/conf.py | 1 - 13 files changed, 41 insertions(+), 506 deletions(-)

1 0

[bridgedb/develop] Fix quoting in moat specification.
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 8d8db4a06d3dff5e9f82adcb4e0e921fe3577098 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Dec 13 18:18:55 2017 +0000 Fix quoting in moat specification. --- README.rst | 74 +++++++++++++++++++++++++++++++------------------------------- 1 file changed, 37 insertions(+), 37 deletions(-) diff --git a/README.rst b/README.rst index c2f32c1..8481382 100644 --- a/README.rst +++ b/README.rst @@ -429,10 +429,10 @@ Requesting Bridges The client MUST send a ``POST /meek/moat/fetch`` containing the following JSON:: { - 'data': { - 'version': '0.1.0', - 'type': 'client-transports', - 'supported': [ 'TRANSPORT', 'TRANSPORT', ... ], + "data": { + "version": "0.1.0", + "type": "client-transports", + "supported": [ "TRANSPORT", "TRANSPORT", ... ], } } @@ -456,10 +456,10 @@ If there is no overlap with the transports which BridgeDB supports, the moat server will respond with the list of transports which is *does* support:: { - 'data': { - 'version': '0.1.0', - 'type': 'moat-transports', - 'supported': [ 'TRANSPORT', 'TRANSPORT', ... ], + "data": { + "version": "0.1.0", + "type": "moat-transports", + "supported": [ "TRANSPORT", "TRANSPORT", ... ], } } @@ -468,13 +468,13 @@ the "best" transport from the list of supported transports, and respond with the following JSON containing a CAPTCHA challenge:: { - 'data': { - 'id': 1, - 'type': 'moat-challenge', - 'version': '0.1.0', - 'transport': TRANSPORT, - 'image': CAPTCHA, - 'challenge': CHALLENGE, + "data": { + "id": "1", + "type": "moat-challenge", + "version": "0.1.0", + "transport": "TRANSPORT", + "image": "CAPTCHA", + "challenge": "CHALLENGE", } } @@ -498,14 +498,14 @@ To propose a solution to a CAPTCHA, the client MUST send a request for ``POST /meek/moat/check``, where the body of the request contains the following JSON:: { - 'data': { - 'id': 2, - 'type': 'moat-solution', - 'version': '0.1.0', - 'transport': TRANSPORT, - 'challenge': CHALLENGE, - 'solution': SOLUTION, - 'qrcode': BOOLEAN, + "data": { + "id": "2", + "type": "moat-solution", + "version": "0.1.0", + "transport": "TRANSPORT", + "challenge": "CHALLENGE", + "solution": "SOLUTION", + "qrcode": "BOOLEAN", } } @@ -518,8 +518,8 @@ where: * ``SOLUTION`` is a valid unicode string, up to 20 bytes in length, containing the client's answer (i.e. what characters the CAPTCHA image displayed). The solution is *not* case-sensitive. -* ``BOOLEAN`` is ``'true'`` if the client wants a qrcode containing the bridge - lines to be generated and returned; ``'false'`` otherwise. +* ``BOOLEAN`` is ``"true"`` if the client wants a qrcode containing the bridge + lines to be generated and returned; ``"false"`` otherwise. Receiving Bridges @@ -532,12 +532,12 @@ If the ``SOLUTION`` was successful for the supplied ``CHALLENGE``, the server responds ``200 OK`` with the following JSON:: { - 'data': { - 'id': 3, - 'type': 'moat-bridges', - 'version': '0.1.0', - 'bridges': [ 'BRIDGE_LINE', ... ], - 'qrcode': QRCODE, + "data": { + "id": "3", + "type": "moat-bridges", + "version": "0.1.0", + "bridges": [ "BRIDGE_LINE", ... ], + "qrcode": "QRCODE", } } @@ -556,12 +556,12 @@ server is unable to distribute the requested Bridges, the server responds ``200 OK`` with the following JSON:: { - 'error': { - 'id': 1, - 'code': '404', - 'status': 'Not Found', - 'title': 'Could not fetch the type of bridges you requested', - 'detail': DETAILS, + "error": { + "id": "1", + "code": "404", + "status": "Not Found", + "title": "Could not fetch the type of bridges you requested", + "detail": "DETAILS", } }

1 0

[bridgedb/develop] Change "client-transport" moat resource type to "moat-transports".
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 50a9321fadcfd0e9a4f30bc1919c1bf46d27d8e1 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Dec 13 18:24:04 2017 +0000 Change "client-transport" moat resource type to "moat-transports". --- bridgedb/distributors/moat/server.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/bridgedb/distributors/moat/server.py b/bridgedb/distributors/moat/server.py index eb356fe..096b8c0 100644 --- a/bridgedb/distributors/moat/server.py +++ b/bridgedb/distributors/moat/server.py @@ -386,12 +386,12 @@ class CaptchaFetchResource(CaptchaResource): :param str request: A JSON blob containing the following fields: * "version": The moat protocol version. - * "type": "client-transports". - * "supported": ['TRANSPORT', … ] + * "type": "moat-transports". + * "supported": ["TRANSPORT", … ] where: - * TRANSPORT is a string identifying a transport, e.g. "obfs3" or - "obfs4". Currently supported transport identifiers are: - "vanilla", "fte", "obfs3", "obfs4", "scramblesuit". + * ``TRANSPORT`` is a string identifying a transport, e.g. + "obfs3" or "obfs4". Currently supported transport identifiers + are: "vanilla", "fte", "obfs3", "obfs4", "scramblesuit". :rtype: list :returns: The list of transports the client supports. """ @@ -401,10 +401,10 @@ class CaptchaFetchResource(CaptchaResource): encoded_data = request.content.read() data = json.loads(encoded_data)["data"][0] - if data["type"] != "client-transports": + if data["type"] != "moat-transports": raise ValueError( "Bad JSON API object type: expected %s got %s" % - ('client-transports', data["type"])) + ('moat-transports', data["type"])) elif data["version"] != MOAT_API_VERSION: raise ValueError( "Client requested protocol version %s, but we're using %s" %

1 0

[bridgedb/develop] Remove unused bucket functionality.
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit f4d559197e06bc4b2e783634d6e79232a440025f Author: Isis Lovecruft <isis(a)torproject.org> Date: Mon Nov 20 23:13:47 2017 +0000 Remove unused bucket functionality. * CLOSES #3015: https://bugs.torproject.org/3015 --- README.rst | 23 +-- bridgedb.conf | 6 - bridgedb/Bridges.py | 18 +-- bridgedb/Bucket.py | 288 ---------------------------------- bridgedb/main.py | 51 +++--- bridgedb/parse/options.py | 29 ++-- bridgedb/runner.py | 14 -- bridgedb/test/test_Bucket.py | 63 -------- bridgedb/test/test_bridgedb_script.py | 19 --- bridgedb/test/test_main.py | 26 +-- doc/sphinx/source/bridgedb.Bucket.rst | 8 - doc/sphinx/source/bridgedb.rst | 1 - doc/sphinx/source/conf.py | 1 - 13 files changed, 41 insertions(+), 506 deletions(-) diff --git a/README.rst b/README.rst index 3dbe647..c2f32c1 100644 --- a/README.rst +++ b/README.rst @@ -361,25 +361,12 @@ Or just give it a SIGHUP:: kill -s SIGHUP `cat .../run/bridgedb.pid` ---------------------------------------------------- -To extract bucket files of all unallocated bridges: ---------------------------------------------------- - -Edit the configuration file value ``FILE_BUCKETS`` according to your -needs. For example, the following is a possible configuration:: - - FILE_BUCKETS = { "name1": 10, "name2": 15, "foobar": 3 } - -This configuration for buckets would result in 3 files being created for -bridge distribution: name1-2010-07-17.brdgs, name2-2010-07-17.brdgs and -foobar-2010-07-17.brdgs. The first file would contain 10 bridges from -BridgeDB's 'unallocated' pool. The second file would contain 15 bridges -from the same pool and the third one similarly 3 bridges. These files -can then be handed out to trusted parties via mail or fed to other -distribution mechanisms such as Twitter. +---------------------------------- +To extract all bridge assignments: +---------------------------------- -To dump all buckets to their files, send BridgeDB a ``SIGUSR1`` signal -by doing:: +To dump all bridge assignments to files, send BridgeDB a ``SIGUSR1`` +signal by doing:: kill -s SIGUSR1 `cat .../run/bridgedb.pid` diff --git a/bridgedb.conf b/bridgedb.conf index d0366cf..724c802 100644 --- a/bridgedb.conf +++ b/bridgedb.conf @@ -683,9 +683,3 @@ EMAIL_SHARE = 2 # An integer specifying the proportion of bridges which should remain # unallocated, for backup usage and manual distribution. RESERVED_SHARE = 2 - -# A dictionary of {FILENAME: NUMBER} where FILENAME is a string specifying the -# filename to store a certain NUMBER (an integer) of bridges in. The number of -# bridges here is *not* a share/proportion, as above; instead it's literally -# the number of bridges. See the ``README`` for more details. -FILE_BUCKETS = {} diff --git a/bridgedb/Bridges.py b/bridgedb/Bridges.py index 50d70ae..13bffd2 100644 --- a/bridgedb/Bridges.py +++ b/bridgedb/Bridges.py @@ -20,7 +20,6 @@ import ipaddr import random import bridgedb.Storage -import bridgedb.Bucket from bridgedb.bridges import Bridge from bridgedb.crypto import getHMACFunc @@ -388,10 +387,7 @@ class UnallocatedHolder(object): continue dist = bridge.distributor desc = [ description ] - if dist.startswith(bridgedb.Bucket.PSEUDO_DISTRI_PREFIX): - dist = dist.replace(bridgedb.Bucket.PSEUDO_DISTRI_PREFIX, "") - desc.append("bucket=%s" % dist) - elif dist != "unallocated": + if dist != "unallocated": continue f.write("%s %s\n" % (bridge.hex_key, " ".join(desc).strip())) @@ -407,7 +403,6 @@ class BridgeSplitter(object): self.totalP = 0 self.pValues = [] self.rings = [] - self.pseudoRings = [] self.statsHolders = [] def __len__(self): @@ -430,11 +425,6 @@ class BridgeSplitter(object): self.rings.append(ringname) self.totalP += p - def addPseudoRing(self, ringname): - """Add a pseudo ring to the list of pseudo rings. - """ - self.pseudoRings.append(bridgedb.Bucket.PSEUDO_DISTRI_PREFIX + ringname) - def addTracker(self, t): """Adds a statistics tracker that gets told about every bridge we see. """ @@ -454,7 +444,7 @@ class BridgeSplitter(object): if not bridge.flags.running: return - validRings = self.rings + self.pseudoRings + validRings = self.rings distribution_method = None # If the bridge already has a distributor, use that. @@ -498,10 +488,6 @@ class BridgeSplitter(object): time.time(), validRings) db.commit() - # Pseudo distributors are always held in the "unallocated" ring - if ringname in self.pseudoRings: - ringname = "unallocated" - ring = self.ringsByName.get(ringname) ring.insert(bridge) diff --git a/bridgedb/Bucket.py b/bridgedb/Bucket.py deleted file mode 100644 index 1382188..0000000 --- a/bridgedb/Bucket.py +++ /dev/null @@ -1,288 +0,0 @@ -# -*- coding: utf-8 -*- - -""" -This module is responsible for everything concerning file bucket bridge -distribution. File bucket bridge distribution means that unallocated bridges -are allocated to a certain pseudo-distributor and later written to a file. - -For example, the following is a dict of pseudo-distributors (also called -'bucket identifiers') with numbers of bridges assigned to them: - - FILE_BUCKETS = { "name1": 10, "name2": 15, "foobar": 3 } - -This configuration for buckets would result in 3 files being created for bridge -distribution: name1-2010-07-17.brdgs, name2-2010-07-17.brdgs and -foobar-2010-07-17.brdgs. The first file would contain 10 bridges from BridgeDB's -'unallocated' pool. The second file would contain 15 bridges from the same pool -and the third one similarly 3 bridges. These files can then be handed out to -trusted parties via mail or fed to other distribution mechanisms such as -twitter. - -Note that in BridgeDB slang, the _distributor_ would still be 'unallocated', -even though in the database, there would now by 'name1', 'name2' or 'foobar' -instead of 'unallocated'. This is why they are called pseudo-distributors. -""" - -import logging -import time -import bridgedb.Storage -import bridgedb.Bridges -import binascii -import sqlite3 -from gettext import gettext as _ -toHex = binascii.b2a_hex - - -# What should pseudo distributors be prefixed with in the database so we can -# distinguish them from real distributors? -PSEUDO_DISTRI_PREFIX = "pseudo_" - -# Set to rediculously high number -BUCKET_MAX_BRIDGES = 1000000 - - -class BucketData(object): - """Configures a bridge bucket with the number of bridges which should be - allocated, the name of the bucket, and other similar data. - - :param str name: The name of this bucket (from the config file). This will - be prefixed by the :data:`PSEUDO_DISTRIBUTOR_PREFIX`. - :type needed: str or int - :param needed: The number of bridges needed for this bucket (also from the - config file). - :param int allocated: Number of bridges already allocated for this bucket. - """ - def __init__(self, name, needed): - self.name = name - if needed == "*": - needed = BUCKET_MAX_BRIDGES - self.needed = int(needed) - self.allocated = 0 - -class BucketManager(object): - """BucketManager reads a number of file bucket identifiers from the config. - - They're expected to be in the following format:: - - FILE_BUCKETS = { "name1": 10, "name2": 15, "foobar": 3 } - - This syntax means that certain buckets ("name1", "name2" and so on) are - given a number of bridges (10, 15 and so on). Names can be anything. The - name will later be the prefix of the file that is written with the - assigned number of bridges in it. Instead of a number, a wildcard item - ("*") is allowed, too. This means that the corresponsing bucket file will - get the maximum number of possible bridges (as many as are left in the - unallocated bucket). - - The files will be written in ip:port format, one bridge per line. - - The way this works internally is as follows: - - First of all, the assignBridgesToBuckets() routine runs through the - database of bridges and looks up the 'distributor' field of each - bridge. Unallocated bridges are sent to a pool for later assignement. - Already allocated bridges for file bucket distribution are sorted and - checked. They're checked for whether their bucket identifier still exists - in the current config and also whether the number of assigned bridges is - still valid. If either the bucket identifier is not existing anymore or - too many bridges are currently assigned to it, bridges will go to the - unassigned pool. - - In the second step, after bridges are sorted and the unassigned pool is - ready, the assignBridgesToBuckets() routine assigns one bridge from the - unassigned pool to a known bucket identifier at a time until it either - runs out of bridges in the unallocated pool or the number of needed - bridges for that bucket is reached. - - When all bridges are assigned in this way, they can then be dumped into - files by calling the dumpBridges() routine. - - :type cfg: :class:`bridgedb.persistent.Conf` - :ivar cfg: The central configuration instance. - :ivar list bucketList: A list of BucketData instances, holding all - configured (and thus requested) buckets with their respective numbers. - :ivar list unallocatedList: Holds all bridges from the 'unallocated' pool. - :ivar bool unallocated_available: Is at least one unallocated bridge - available? - :ivar str distributor_prefix: The 'distributor' field in the database will - hold the name of our pseudo-distributor, prefixed by this string. By - default, this uses :data:`PSEUDO_DISTRIBUTOR_PREFIX`. - :ivar db: The bridge database instance. - """ - - def __init__(self, cfg): - """Create a ``BucketManager``. - - :type cfg: :class:`bridgedb.persistent.Conf` - :param cfg: The central configuration instance. - """ - self.cfg = cfg - self.bucketList = [] - self.unallocatedList = [] - self.unallocated_available = False - self.distributor_prefix = PSEUDO_DISTRI_PREFIX - - def addToUnallocatedList(self, hex_key): - """Add a bridge by **hex_key** into the unallocated pool.""" - with bridgedb.Storage.getDB() as db: - try: - db.updateDistributorForHexKey("unallocated", hex_key) - except: - db.rollback() - raise - else: - db.commit() - self.unallocatedList.append(hex_key) - self.unallocated_available = True - - def getBucketByIdent(self, bucketIdent): - """If we know this bucket identifier, then return the corresponding - :class:`BucketData` object. - """ - for d in self.bucketList: - if d.name == bucketIdent: - return d - return None - - def assignUnallocatedBridge(self, bucket): - """Assign an unallocated bridge to a certain **bucket**.""" - hex_key = self.unallocatedList.pop() - # Mark pseudo-allocators in the database as such - allocator_name = bucket.name - #print "KEY: %d NAME: %s" % (hex_key, allocator_name) - logging.debug("Moving %s to %s" % (hex_key, allocator_name)) - with bridgedb.Storage.getDB() as db: - try: - db.updateDistributorForHexKey(allocator_name, hex_key) - except: - db.rollback() - logging.warn("Failed to move %s to new distributor (%s)" - % (hex_key, allocator_name)) - - # Ok, this seems useless, but for consistancy's sake, we'll - # re-assign the bridge from this missed db update attempt to the - # unallocated list. Remember? We pop()'d it before. - self.addToUnallocatedList(hex_key) - raise - else: - db.commit() - bucket.allocated += 1 - if len(self.unallocatedList) < 1: - self.unallocated_available = False - return True - - def assignBridgesToBuckets(self): - """Read file bucket identifiers from the configuration, sort them, and - write necessary changes to the database. - """ - logging.debug("Assigning bridges to buckets for pseudo-distributors") - # Build distributor list - for k, v in self.cfg.FILE_BUCKETS.items(): - prefixed_key = self.distributor_prefix + k - d = BucketData(prefixed_key, v) - self.bucketList.append(d) - - # Loop through all bridges and sort out distributors - with bridgedb.Storage.getDB() as db: - allBridges = db.getAllBridges() - for bridge in allBridges: - if bridge.distributor == "unallocated": - self.addToUnallocatedList(bridge.hex_key) - continue - - # Filter non-pseudo distributors (like 'https' and 'email') early, - # too - if not bridge.distributor.startswith(self.distributor_prefix): - continue - - # Return the bucket in case we know it already - d = self.getBucketByIdent(bridge.distributor) - if d is not None: - # Does this distributor need another bridge? If not, re-inject - # it into the 'unallocated' pool for for later assignment - if d.allocated < d.needed: - d.allocated += 1 - else: - # Bucket has enough members already, free this one - self.addToUnallocatedList(bridge.hex_key) - # We don't know it. Maybe an old entry. Free it. - else: - self.addToUnallocatedList(bridge.hex_key) - - # Loop through bucketList while we have and need unallocated - # bridges, assign one bridge at a time - while self.unallocated_available and len(self.bucketList) > 0: - logging.debug("We have %d unallocated bridges and %d buckets to " \ - "fill. Let's do it." - % (len(self.unallocatedList), len(self.bucketList))) - for d in self.bucketList: - if d.allocated < d.needed: - try: - if not self.assignUnallocatedBridge(d): - break - except sqlite3.DatabaseError as e: - dist = d.name.replace(self.distributor_prefix, "") - logging.warn("Couldn't assign unallocated bridge to " \ - "%s: %s" % (dist, e)) - else: - # When we have enough bridges, remove bucket identifier - # from list - self.bucketList.remove(d) - - def dumpBridgesToFile(self, filename, bridges): - """Dump a list of given **bridges** into **filename**.""" - logging.debug("Dumping bridge assignments to file: %r" % filename) - # get the bridge histories and sort by Time On Same Address - bridgeHistories = [] - with bridgedb.Storage.getDB() as db: - for b in bridges: - if self.cfg.COLLECT_TIMESTAMPS: - bh = db.getBridgeHistory(b.hex_key) - if bh: bridgeHistories.append(bh) - bridgeHistories.sort(lambda x,y: cmp(x.weightedFractionalUptime, - y.weightedFractionalUptime)) - - try: - f = open(filename, 'w') - if self.cfg.COLLECT_TIMESTAMPS: - for bh in bridgeHistories: - days = bh.tosa / long(60*60*24) - line = "%s:%s\t(%d days at this address)" % \ - (bh.ip, bh.port, days) - if str(bh.fingerprint) in blocklist.keys(): - line = line + "\t(Might be blocked): (%s)" % \ - ",".join(blocklist[bh.fingerprint]) - f.write(line + '\n') - else: - for bridge in bridges: - line = "%s:%d %s" \ - % (bridge.address, bridge.or_port, bridge.hex_key) - f.write(line + '\n') - f.close() - except IOError: - print "I/O error: %s" % filename - - def dumpBridges(self): - """Dump all known file distributors to files, sorted by distributor.""" - logging.info("Dumping all distributors to file.") - with bridgedb.Storage.getDB() as db: - allBridges = db.getAllBridges() - bridgeDict = {} - # Sort returned bridges by distributor - for bridge in allBridges: - dist = str(bridge.distributor) - if dist in bridgeDict.keys(): - bridgeDict[dist].append(bridge) - else: - bridgeDict[dist] = [bridge] - - # Now dump to file(s) - for k in bridgeDict.keys(): - dist = k - if (dist.startswith(self.distributor_prefix)): - # Subtract the pseudo distributor prefix - dist = dist.replace(self.distributor_prefix, "") - # Be safe. Replace all '/' in distributor names - dist = dist.replace("/", "_") - filename = dist + "-" + time.strftime("%Y-%m-%d") + ".brdgs" - self.dumpBridgesToFile(filename, bridgeDict[k]) diff --git a/bridgedb/main.py b/bridgedb/main.py index 905af2d..586ed83 100644 --- a/bridgedb/main.py +++ b/bridgedb/main.py @@ -54,6 +54,24 @@ def expandBridgeAuthDir(authdir, filename): return path +def writeAssignments(hashring, filename): + """Dump bridge distributor assignments to disk. + + :type hashring: A :class:`~bridgedb.Bridges.BridgeSplitter` + :ivar hashring: A class which takes an HMAC key and splits bridges + into their hashring assignments. + :param str filename: The filename to write the assignments to. + """ + logging.debug("Dumping pool assignments to file: '%s'" % filename) + + try: + with open(filename, 'a') as fh: + fh.write("bridge-pool-assignment %s\n" % + time.strftime("%Y-%m-%d %H:%M:%S")) + hashring.dumpAssignments(fh) + except IOError: + logging.info("I/O error while writing assignments to: '%s'" % filename) + def load(state, hashring, clear=False): """Read and parse all descriptors, and load into a bridge hashring. @@ -192,19 +210,6 @@ def _handleSIGHUP(*args): """Called when we receive a SIGHUP; invokes _reloadFn.""" reactor.callInThread(_reloadFn) -def _handleSIGUSR1(*args): - """Handler for SIGUSR1. Calls :func:`~bridgedb.runner.doDumpBridges`.""" - logging.debug("Caught SIGUSR1 signal") - - from bridgedb import runner - - logging.info("Loading saved state...") - state = persistent.load() - cfg = loadConfig(state.CONFIG_FILE, state.config) - - logging.info("Dumping bridge assignments to files...") - reactor.callInThread(runner.doDumpBridges, cfg) - def replaceBridgeRings(current, replacement): """Replace the current thing with the new one""" current.hashring = replacement.hashring @@ -274,10 +279,6 @@ def createBridgeRings(cfg, proxyList, key): "unallocated", cfg.RESERVED_SHARE) - # Add pseudo distributors to hashring - for pseudoRing in cfg.FILE_BUCKETS.keys(): - hashring.addPseudoRing(pseudoRing) - return hashring, emailDistributor, ipDistributor, moatDistributor def run(options, reactor=reactor): @@ -317,7 +318,7 @@ def run(options, reactor=reactor): # :func:`logging.basicConfig` will be ignored. util.configureLogging(config) - if options['dump-bridges'] or (options.subCommand is not None): + if options.subCommand is not None: runSubcommand(options, config) # Write the pidfile only after any options.subCommands are run (because @@ -428,18 +429,7 @@ def run(options, reactor=reactor): logging.warn("No Moat distributor created!") # Dump bridge pool assignments to disk. - try: - logging.debug("Dumping pool assignments to file: '%s'" - % state.ASSIGNMENTS_FILE) - fh = open(state.ASSIGNMENTS_FILE, 'a') - fh.write("bridge-pool-assignment %s\n" % - time.strftime("%Y-%m-%d %H:%M:%S")) - hashring.dumpAssignments(fh) - fh.flush() - fh.close() - except IOError: - logging.info("I/O error while writing assignments to: '%s'" - % state.ASSIGNMENTS_FILE) + writeAssignments(hashring, state.ASSIGNMENTS_FILE) state.save() if inThread: @@ -462,7 +452,6 @@ def run(options, reactor=reactor): global _reloadFn _reloadFn = reload signal.signal(signal.SIGHUP, _handleSIGHUP) - signal.signal(signal.SIGUSR1, _handleSIGUSR1) if reactor: # pragma: no cover # And actually load it to start parsing. Get back our distributors. diff --git a/bridgedb/parse/options.py b/bridgedb/parse/options.py index d866ac8..e5c4d1e 100644 --- a/bridgedb/parse/options.py +++ b/bridgedb/parse/options.py @@ -268,34 +268,31 @@ class MockOptions(BaseOptions): class SIGHUPOptions(BaseOptions): """Options menu to explain usage and handling of SIGHUP signals.""" - longdesc = """If you send a SIGHUP to a running BridgeDB process, the - servers will parse and reload all bridge descriptor files into the - databases. - - Note that this command WILL NOT handle sending the signal for you; see - signal(7) and kill(1) for additional help.""" + longdesc = ( + "If you send a SIGHUP to a running BridgeDB process, the servers will " + "parse and reload all bridge descriptor files into the databases." + "\n\n" + "Note that this command WILL NOT handle sending the signal for you; " + "see signal(7) and kill(1) for additional help.") class SIGUSR1Options(BaseOptions): """Options menu to explain usage and handling of SIGUSR1 signals.""" - longdesc = """If you send a SIGUSR1 to a running BridgeDB process, the - servers will dump all bridge assignments by distributor from the - databases to files. - - Note that this command WILL NOT handle sending the signal for you; see - signal(7) and kill(1) for additional help.""" + longdesc = ( + "If you send a SIGUSR1 to a running BridgeDB process, the servers will " + "dump all bridge assignments by distributor from the databases to files." + "\n\n" + "Note that this command WILL NOT handle sending the signal for you; see " + "signal(7) and kill(1) for additional help.") class MainOptions(BaseOptions): """Main commandline options parser for BridgeDB.""" optFlags = [ - ['dump-bridges', 'd', 'Dump bridges by hashring assignment into files'], ['reload', 'R', 'Reload bridge descriptors into running servers']] subCommands = [ ['mock', None, MockOptions, "Generate a testing environment"], ['SIGHUP', None, SIGHUPOptions, - "Reload bridge descriptors into running servers"], - ['SIGUSR1', None, SIGUSR1Options, - "Dump bridges by hashring assignment into files"]] + "Reload bridge descriptors into running servers"]] diff --git a/bridgedb/runner.py b/bridgedb/runner.py index 57297cb..35865e5 100644 --- a/bridgedb/runner.py +++ b/bridgedb/runner.py @@ -122,17 +122,3 @@ def generateDescriptors(count=None, rundir=None): print("Sucessfully generated %s descriptors." % str(count)) del subprocess return statuscode - -def doDumpBridges(config): - """Dump bridges by assignment to a file. - - This function handles the commandline '--dump-bridges' option. - - :type config: :class:`bridgedb.main.Conf` - :param config: The current configuration. - """ - import bridgedb.Bucket as bucket - - bucketManager = bucket.BucketManager(config) - bucketManager.assignBridgesToBuckets() - bucketManager.dumpBridges() diff --git a/bridgedb/test/test_Bucket.py b/bridgedb/test/test_Bucket.py deleted file mode 100644 index 01f3f26..0000000 --- a/bridgedb/test/test_Bucket.py +++ /dev/null @@ -1,63 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of BridgeDB, a Tor bridge distribution system. -# -# :copyright: (c) 2007-2017, The Tor Project, Inc. -# (c) 2007-2017, all entities within the AUTHORS file -# :license: 3-Clause BSD, see LICENSE for licensing information - -"""Unittests for the :mod:`bridgedb.Bucket` module. - -These tests are meant to ensure that the :mod:`bridgedb.Bucket` module is -functioning as expected. -""" - -from __future__ import print_function - -from io import StringIO - -import sure -from sure import this -from sure import the -from sure import expect - -from bridgedb import Bucket - -from twisted.trial import unittest - - -class BucketDataTest(unittest.TestCase): - """Tests for :class:`bridgedb.Bucket.BucketData`.""" - - def test_alloc_some_of_the_bridges(self): - """Set the needed number of bridges""" - needed = 10 - distname = "test-distributor" - bucket = Bucket.BucketData(distname, needed) - this(bucket.name).should.be.equal(distname) - this(bucket.needed).should.be.equal(needed) - - def test_alloc_all_the_bridges(self): - """Set the needed number of bridges to the default""" - needed = '*' - distname = "test-distributor" - bucket = Bucket.BucketData(distname, needed) - this(bucket.name).should.be.equal(distname) - this(bucket.needed).should.be.equal(Bucket.BUCKET_MAX_BRIDGES) - - -class BucketManagerTest(unittest.TestCase): - """Tests for :class:`bridgedb.Bucket.BucketManager`.""" - TEST_CONFIG_FILE = StringIO(unicode("""\ - FILE_BUCKETS = { 'test1': 7, 'test2': 11 } - COLLECT_TIMESTAMPS = False - COUNTRY_BLOCK_FILE = []""")) - - def setUp(self): - configuration = {} - TEST_CONFIG_FILE.seek(0) - compiled = compile(TEST_CONFIG_FILE.read(), '<string>', 'exec') - exec compiled in configuration - self.config = persistent.Conf(**configuration) - self.state = persistent.State(**config.__dict__) - self.bucket = Bucket.BucketManager(self.config) diff --git a/bridgedb/test/test_bridgedb_script.py b/bridgedb/test/test_bridgedb_script.py index 70c71f0..fb8bb3e 100644 --- a/bridgedb/test/test_bridgedb_script.py +++ b/bridgedb/test/test_bridgedb_script.py @@ -82,22 +82,3 @@ class BridgeDBCliTest(unittest.TestCase): os.kill(self.pid, signal.SIGHUP) self.doSleep() self.assertTrue(os.path.isfile(self.assignmentsFile)) - - def test_bridgedb_SIGUSR1_buckets(self): - """Test that BridgeDB dumps buckets appropriately after a SIGUSR1.""" - if os.environ.get("CI"): - if not self.pid or not processExists(self.pid): - raise FailTest("Could not start BridgeDB process on CI server!") - if not self.pid or not processExists(self.pid): - raise SkipTest("Can't run test: no BridgeDB process running.") - - os.kill(self.pid, signal.SIGUSR1) - self.doSleep() - buckets = [['email', False], ['https', False], ['unallocated', False]] - for rundirfile in os.listdir(self.rundir): - for bucket in buckets: - if rundirfile.startswith(bucket[0]): - bucket[1] = True - break - for bucket in buckets: - self.assertTrue(bucket[1], "%s bucket was not dumped!" % bucket[0]) diff --git a/bridgedb/test/test_main.py b/bridgedb/test/test_main.py index 2a124b1..b6eae2b 100644 --- a/bridgedb/test/test_main.py +++ b/bridgedb/test/test_main.py @@ -372,29 +372,6 @@ class BridgedbTests(unittest.TestCase): self.assertEqual(len(hashring.ringsByName.keys()), 3) self.assertNotIn('unallocated', hashring.rings) - def test_main_createBridgeRings_two_file_buckets(self): - """When FILE_BUCKETS has two filenames in it, main.createBridgeRings() - should add three hashrings to the hashring, then add two - "pseudo-rings". - """ - proxyList = main.proxy.ProxySet() - config = self.config - config.FILE_BUCKETS = { - 'bridges-for-support-desk': 10, - 'bridges-for-ooni-tests': 10, - } - (hashring, emailDist, httpsDist, moatDist) = main.createBridgeRings( - config, proxyList, self.key) - - # Should have an HTTPSDistributor ring, an EmailDistributor, a - # MoatDistributor and an UnallocatedHolder ring: - self.assertEqual(len(hashring.ringsByName.keys()), 4) - - # Should have two pseudoRings: - self.assertEqual(len(hashring.pseudoRings), 2) - self.assertIn('pseudo_bridges-for-support-desk', hashring.pseudoRings) - self.assertIn('pseudo_bridges-for-ooni-tests', hashring.pseudoRings) - def test_main_run(self): """main.run() should run and then finally raise SystemExit.""" config = """ @@ -473,8 +450,7 @@ EMAiL_GPG_PASSPHRASE = None EMAIL_GPG_PASSPHRASE_FILE = None HTTPS_SHARE = 10 EMAIL_SHARE = 5 -RESERVED_SHARE = 2 -FILE_BUCKETS = {}""" +RESERVED_SHARE = 2""" configFile = self._writeConfig(config) # Fake some options: diff --git a/doc/sphinx/source/bridgedb.Bucket.rst b/doc/sphinx/source/bridgedb.Bucket.rst deleted file mode 100644 index 753bafd..0000000 --- a/doc/sphinx/source/bridgedb.Bucket.rst +++ /dev/null @@ -1,8 +0,0 @@ -bridgedb.Bucket ----------------- - -.. automodule:: bridgedb.Bucket - :members: - :undoc-members: - :private-members: - :show-inheritance: diff --git a/doc/sphinx/source/bridgedb.rst b/doc/sphinx/source/bridgedb.rst index 1f272b3..9a8f092 100644 --- a/doc/sphinx/source/bridgedb.rst +++ b/doc/sphinx/source/bridgedb.rst @@ -9,7 +9,6 @@ Packages & Modules bridgedb.bridgerequest bridgedb.bridges bridgedb.Bridges - bridgedb.Bucket bridgedb.captcha bridgedb.configure bridgedb.distribute diff --git a/doc/sphinx/source/conf.py b/doc/sphinx/source/conf.py index e98d2b2..31ae0b4 100644 --- a/doc/sphinx/source/conf.py +++ b/doc/sphinx/source/conf.py @@ -32,7 +32,6 @@ import bridgedb.bridgerequest import bridgedb.bridges import bridgedb.captcha import bridgedb.Bridges -import bridgedb.Bucket import bridgedb.crypto import bridgedb.distribute import bridgedb.distributors

1 0

[bridgedb/develop] Merge branch 'release/0.6.1'
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 9bb5de6b07d33bef6ca7db22a70398018523b6e4 Merge: 659b4ed 6aaa2ed Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Dec 13 20:12:08 2017 +0000 Merge branch 'release/0.6.1' CHANGELOG | 53 +++++ README.rst | 97 ++++----- bridgedb.conf | 6 - bridgedb/Bridges.py | 18 +- bridgedb/Bucket.py | 288 ------------------------- bridgedb/bridgerequest.py | 2 +- bridgedb/distributors/https/server.py | 6 +- bridgedb/distributors/moat/server.py | 2 +- bridgedb/main.py | 51 ++--- bridgedb/parse/options.py | 29 ++- bridgedb/runner.py | 14 -- bridgedb/test/test_Bucket.py | 63 ------ bridgedb/test/test_bridgedb_script.py | 19 -- bridgedb/test/test_distributors_moat_server.py | 2 +- bridgedb/test/test_https_server.py | 14 ++ bridgedb/test/test_main.py | 26 +-- doc/sphinx/source/bridgedb.Bucket.rst | 8 - doc/sphinx/source/bridgedb.rst | 1 - doc/sphinx/source/conf.py | 1 - scripts/test-moat | 56 +++++ 20 files changed, 208 insertions(+), 548 deletions(-)

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits December 2017