December 2017 - tor-commits - lists.torproject.org

[bridgedb/master] Add script for using curl to test moat's JSON API.
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 0c24fa3719aa29e967ca5804e8eaaab80af17e32 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Sep 12 21:33:22 2017 +0000 Add script for using curl to test moat's JSON API. --- scripts/test-moat | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/scripts/test-moat b/scripts/test-moat new file mode 100755 index 0000000..6bfa500 --- /dev/null +++ b/scripts/test-moat @@ -0,0 +1,36 @@ +#!/usr/bin/env bash + +function usage() { + printf "Usage: %s [fetch] [check]\n" "$(basename $0)" +} + +if test "$#" -lt 1 ; then + usage + exit 1 +fi + +function do_fetch() { + curl -H 'Content-Type: application/vnd.api+json' \ + -H 'Accept: application/vnd.api+json' \ + --data '{"data": [{"supported": ["obfs4"], "version": "0.1.0", "type": "client-transports"}]}' \ + http://127.0.0.1:6790/meek/moat/fetch + echo +} + +function do_check() { + curl -H 'Content-Type: application/vnd.api+json' \ + -H 'Accept: application/vnd.api+json' \ + --data '{"data": [{"challenge": "foo", "solution": "bar", "version": "0.1.0", "qrcode": "false", "type": "moat-solution", "id": 2, "transport": "obfs4"}]}' \ + http://127.0.0.1:6790/meek/moat/check + echo +} + +while test -n "$1" ; do + case "$1" in + fetch) do_fetch ;; + check) do_check ;; + *) usage ;; + esac + + shift +done

1 0

[bridgedb/develop] Merge branch 'fix/24636' into develop
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 91305fa523c8b9e6a631d8c29f583e3da722832e Merge: 9a926f2 da84c25 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Dec 20 22:59:11 2017 +0000 Merge branch 'fix/24636' into develop README.rst | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-)

1 0

[bridgedb/develop] Merge branch 'release/0.6.2'
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 2ae288d6f7e3051b3aa3c85e9a8341c3e21baf4b Merge: 9bb5de6 6ace9a3 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Dec 20 23:07:12 2017 +0000 Merge branch 'release/0.6.2' README.rst | 34 ++++++------ bridgedb/bridgerequest.py | 9 ++++ bridgedb/distributors/moat/server.py | 73 +++++++++++++++++++++----- bridgedb/test/test_distributors_moat_server.py | 55 +++++++++++++++++-- 4 files changed, 140 insertions(+), 31 deletions(-)

1 0

[bridgedb/master] Merge branch 'master' into develop
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit bbd40684b566faafdc87255d5358deca8a1aeb38 Merge: b4d32c7 659b4ed Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Sep 12 21:33:22 2017 +0000 Merge branch 'master' into develop

1 0

[bridgedb/master] Merge branch 'fix/24433' into develop
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 76d3c5bca779c78236a8dffb389304412e97af5e Merge: 8245b6a a1597da Author: Isis Lovecruft <isis(a)torproject.org> Date: Mon Nov 27 19:38:27 2017 +0000 Merge branch 'fix/24433' into develop bridgedb/bridgerequest.py | 2 +- scripts/test-moat | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)

1 0

[bridgedb/master] Fix the moat server to only generate QRcodes when asked to do so.
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 039884c4f19bb3f52b076e1bc5901800b5258fe1 Author: Isis Lovecruft <isis(a)torproject.org> Date: Mon Nov 27 23:22:33 2017 +0000 Fix the moat server to only generate QRcodes when asked to do so. * FIXES #24443: https://bugs.torproject.org/24443 --- bridgedb/distributors/moat/server.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bridgedb/distributors/moat/server.py b/bridgedb/distributors/moat/server.py index daf62fa..eb356fe 100644 --- a/bridgedb/distributors/moat/server.py +++ b/bridgedb/distributors/moat/server.py @@ -560,7 +560,7 @@ class CaptchaCheckResource(CaptchaResource): "Transport '%s' is not currently supported" % data["transport"]) else: - qrcode = data["qrcode"] + qrcode = True if data["qrcode"] == "true" else False transport = type('')(data["transport"]) challenge = type('')(data["challenge"]) solution = type('')(data["solution"])

1 0

[bridgedb/master] Make the test-moat script easier to input challenges and solutions.
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 8245b6a2c4f4697a33b1c169529aed66b6e1a55a Author: Isis Lovecruft <isis(a)torproject.org> Date: Mon Nov 27 19:13:54 2017 +0000 Make the test-moat script easier to input challenges and solutions. --- scripts/test-moat | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/scripts/test-moat b/scripts/test-moat index 6bfa500..490edc6 100755 --- a/scripts/test-moat +++ b/scripts/test-moat @@ -1,7 +1,10 @@ #!/usr/bin/env bash +CHALLENGE= +SOLUTION= + function usage() { - printf "Usage: %s [fetch] [check]\n" "$(basename $0)" + printf "Usage: %s [fetch] [check [challenge solution]]\n" "$(basename $0)" } if test "$#" -lt 1 ; then @@ -20,17 +23,32 @@ function do_fetch() { function do_check() { curl -H 'Content-Type: application/vnd.api+json' \ -H 'Accept: application/vnd.api+json' \ - --data '{"data": [{"challenge": "foo", "solution": "bar", "version": "0.1.0", "qrcode": "false", "type": "moat-solution", "id": 2, "transport": "obfs4"}]}' \ + --data '{"data": [{"challenge": "'$CHALLENGE'", "solution": "'$SOLUTION'", "version": "0.1.0", "qrcode": "false", "type": "moat-solution", "id": 2, "transport": "obfs4"}]}' \ http://127.0.0.1:6790/meek/moat/check echo } while test -n "$1" ; do + OPTSHIFT=1 + case "$1" in fetch) do_fetch ;; - check) do_check ;; + check) if [[ "$2" != "fetch" ]] ; then + CHALLENGE="$2" + if [[ "$3" != "fetch" ]] ; then + SOLUTION="$3" + fi + fi + for var in "$CHALLENGE" "$SOLUTION" ; do + if test -n "$var"; then + OPTSHIFT=$(( OPTSHIFT + 1 )) + fi + done + do_check ;; + *) usage ;; esac - shift + shift $OPTSHIFT + OPTSHIFT=1 done

1 0

[bridgedb/master] Merge branch 'fix/24443' into develop
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit cb3f4b4aa27b527bfb000cc254d08622b90a526e Merge: 76d3c5b 039884c Author: Isis Lovecruft <isis(a)torproject.org> Date: Mon Nov 27 23:26:17 2017 +0000 Merge branch 'fix/24443' into develop bridgedb/distributors/moat/server.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

1 0

[bridgedb/master] Remove unused bucket functionality.
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit f4d559197e06bc4b2e783634d6e79232a440025f Author: Isis Lovecruft <isis(a)torproject.org> Date: Mon Nov 20 23:13:47 2017 +0000 Remove unused bucket functionality. * CLOSES #3015: https://bugs.torproject.org/3015 --- README.rst | 23 +-- bridgedb.conf | 6 - bridgedb/Bridges.py | 18 +-- bridgedb/Bucket.py | 288 ---------------------------------- bridgedb/main.py | 51 +++--- bridgedb/parse/options.py | 29 ++-- bridgedb/runner.py | 14 -- bridgedb/test/test_Bucket.py | 63 -------- bridgedb/test/test_bridgedb_script.py | 19 --- bridgedb/test/test_main.py | 26 +-- doc/sphinx/source/bridgedb.Bucket.rst | 8 - doc/sphinx/source/bridgedb.rst | 1 - doc/sphinx/source/conf.py | 1 - 13 files changed, 41 insertions(+), 506 deletions(-) diff --git a/README.rst b/README.rst index 3dbe647..c2f32c1 100644 --- a/README.rst +++ b/README.rst @@ -361,25 +361,12 @@ Or just give it a SIGHUP:: kill -s SIGHUP `cat .../run/bridgedb.pid` ---------------------------------------------------- -To extract bucket files of all unallocated bridges: ---------------------------------------------------- - -Edit the configuration file value ``FILE_BUCKETS`` according to your -needs. For example, the following is a possible configuration:: - - FILE_BUCKETS = { "name1": 10, "name2": 15, "foobar": 3 } - -This configuration for buckets would result in 3 files being created for -bridge distribution: name1-2010-07-17.brdgs, name2-2010-07-17.brdgs and -foobar-2010-07-17.brdgs. The first file would contain 10 bridges from -BridgeDB's 'unallocated' pool. The second file would contain 15 bridges -from the same pool and the third one similarly 3 bridges. These files -can then be handed out to trusted parties via mail or fed to other -distribution mechanisms such as Twitter. +---------------------------------- +To extract all bridge assignments: +---------------------------------- -To dump all buckets to their files, send BridgeDB a ``SIGUSR1`` signal -by doing:: +To dump all bridge assignments to files, send BridgeDB a ``SIGUSR1`` +signal by doing:: kill -s SIGUSR1 `cat .../run/bridgedb.pid` diff --git a/bridgedb.conf b/bridgedb.conf index d0366cf..724c802 100644 --- a/bridgedb.conf +++ b/bridgedb.conf @@ -683,9 +683,3 @@ EMAIL_SHARE = 2 # An integer specifying the proportion of bridges which should remain # unallocated, for backup usage and manual distribution. RESERVED_SHARE = 2 - -# A dictionary of {FILENAME: NUMBER} where FILENAME is a string specifying the -# filename to store a certain NUMBER (an integer) of bridges in. The number of -# bridges here is *not* a share/proportion, as above; instead it's literally -# the number of bridges. See the ``README`` for more details. -FILE_BUCKETS = {} diff --git a/bridgedb/Bridges.py b/bridgedb/Bridges.py index 50d70ae..13bffd2 100644 --- a/bridgedb/Bridges.py +++ b/bridgedb/Bridges.py @@ -20,7 +20,6 @@ import ipaddr import random import bridgedb.Storage -import bridgedb.Bucket from bridgedb.bridges import Bridge from bridgedb.crypto import getHMACFunc @@ -388,10 +387,7 @@ class UnallocatedHolder(object): continue dist = bridge.distributor desc = [ description ] - if dist.startswith(bridgedb.Bucket.PSEUDO_DISTRI_PREFIX): - dist = dist.replace(bridgedb.Bucket.PSEUDO_DISTRI_PREFIX, "") - desc.append("bucket=%s" % dist) - elif dist != "unallocated": + if dist != "unallocated": continue f.write("%s %s\n" % (bridge.hex_key, " ".join(desc).strip())) @@ -407,7 +403,6 @@ class BridgeSplitter(object): self.totalP = 0 self.pValues = [] self.rings = [] - self.pseudoRings = [] self.statsHolders = [] def __len__(self): @@ -430,11 +425,6 @@ class BridgeSplitter(object): self.rings.append(ringname) self.totalP += p - def addPseudoRing(self, ringname): - """Add a pseudo ring to the list of pseudo rings. - """ - self.pseudoRings.append(bridgedb.Bucket.PSEUDO_DISTRI_PREFIX + ringname) - def addTracker(self, t): """Adds a statistics tracker that gets told about every bridge we see. """ @@ -454,7 +444,7 @@ class BridgeSplitter(object): if not bridge.flags.running: return - validRings = self.rings + self.pseudoRings + validRings = self.rings distribution_method = None # If the bridge already has a distributor, use that. @@ -498,10 +488,6 @@ class BridgeSplitter(object): time.time(), validRings) db.commit() - # Pseudo distributors are always held in the "unallocated" ring - if ringname in self.pseudoRings: - ringname = "unallocated" - ring = self.ringsByName.get(ringname) ring.insert(bridge) diff --git a/bridgedb/Bucket.py b/bridgedb/Bucket.py deleted file mode 100644 index 1382188..0000000 --- a/bridgedb/Bucket.py +++ /dev/null @@ -1,288 +0,0 @@ -# -*- coding: utf-8 -*- - -""" -This module is responsible for everything concerning file bucket bridge -distribution. File bucket bridge distribution means that unallocated bridges -are allocated to a certain pseudo-distributor and later written to a file. - -For example, the following is a dict of pseudo-distributors (also called -'bucket identifiers') with numbers of bridges assigned to them: - - FILE_BUCKETS = { "name1": 10, "name2": 15, "foobar": 3 } - -This configuration for buckets would result in 3 files being created for bridge -distribution: name1-2010-07-17.brdgs, name2-2010-07-17.brdgs and -foobar-2010-07-17.brdgs. The first file would contain 10 bridges from BridgeDB's -'unallocated' pool. The second file would contain 15 bridges from the same pool -and the third one similarly 3 bridges. These files can then be handed out to -trusted parties via mail or fed to other distribution mechanisms such as -twitter. - -Note that in BridgeDB slang, the _distributor_ would still be 'unallocated', -even though in the database, there would now by 'name1', 'name2' or 'foobar' -instead of 'unallocated'. This is why they are called pseudo-distributors. -""" - -import logging -import time -import bridgedb.Storage -import bridgedb.Bridges -import binascii -import sqlite3 -from gettext import gettext as _ -toHex = binascii.b2a_hex - - -# What should pseudo distributors be prefixed with in the database so we can -# distinguish them from real distributors? -PSEUDO_DISTRI_PREFIX = "pseudo_" - -# Set to rediculously high number -BUCKET_MAX_BRIDGES = 1000000 - - -class BucketData(object): - """Configures a bridge bucket with the number of bridges which should be - allocated, the name of the bucket, and other similar data. - - :param str name: The name of this bucket (from the config file). This will - be prefixed by the :data:`PSEUDO_DISTRIBUTOR_PREFIX`. - :type needed: str or int - :param needed: The number of bridges needed for this bucket (also from the - config file). - :param int allocated: Number of bridges already allocated for this bucket. - """ - def __init__(self, name, needed): - self.name = name - if needed == "*": - needed = BUCKET_MAX_BRIDGES - self.needed = int(needed) - self.allocated = 0 - -class BucketManager(object): - """BucketManager reads a number of file bucket identifiers from the config. - - They're expected to be in the following format:: - - FILE_BUCKETS = { "name1": 10, "name2": 15, "foobar": 3 } - - This syntax means that certain buckets ("name1", "name2" and so on) are - given a number of bridges (10, 15 and so on). Names can be anything. The - name will later be the prefix of the file that is written with the - assigned number of bridges in it. Instead of a number, a wildcard item - ("*") is allowed, too. This means that the corresponsing bucket file will - get the maximum number of possible bridges (as many as are left in the - unallocated bucket). - - The files will be written in ip:port format, one bridge per line. - - The way this works internally is as follows: - - First of all, the assignBridgesToBuckets() routine runs through the - database of bridges and looks up the 'distributor' field of each - bridge. Unallocated bridges are sent to a pool for later assignement. - Already allocated bridges for file bucket distribution are sorted and - checked. They're checked for whether their bucket identifier still exists - in the current config and also whether the number of assigned bridges is - still valid. If either the bucket identifier is not existing anymore or - too many bridges are currently assigned to it, bridges will go to the - unassigned pool. - - In the second step, after bridges are sorted and the unassigned pool is - ready, the assignBridgesToBuckets() routine assigns one bridge from the - unassigned pool to a known bucket identifier at a time until it either - runs out of bridges in the unallocated pool or the number of needed - bridges for that bucket is reached. - - When all bridges are assigned in this way, they can then be dumped into - files by calling the dumpBridges() routine. - - :type cfg: :class:`bridgedb.persistent.Conf` - :ivar cfg: The central configuration instance. - :ivar list bucketList: A list of BucketData instances, holding all - configured (and thus requested) buckets with their respective numbers. - :ivar list unallocatedList: Holds all bridges from the 'unallocated' pool. - :ivar bool unallocated_available: Is at least one unallocated bridge - available? - :ivar str distributor_prefix: The 'distributor' field in the database will - hold the name of our pseudo-distributor, prefixed by this string. By - default, this uses :data:`PSEUDO_DISTRIBUTOR_PREFIX`. - :ivar db: The bridge database instance. - """ - - def __init__(self, cfg): - """Create a ``BucketManager``. - - :type cfg: :class:`bridgedb.persistent.Conf` - :param cfg: The central configuration instance. - """ - self.cfg = cfg - self.bucketList = [] - self.unallocatedList = [] - self.unallocated_available = False - self.distributor_prefix = PSEUDO_DISTRI_PREFIX - - def addToUnallocatedList(self, hex_key): - """Add a bridge by **hex_key** into the unallocated pool.""" - with bridgedb.Storage.getDB() as db: - try: - db.updateDistributorForHexKey("unallocated", hex_key) - except: - db.rollback() - raise - else: - db.commit() - self.unallocatedList.append(hex_key) - self.unallocated_available = True - - def getBucketByIdent(self, bucketIdent): - """If we know this bucket identifier, then return the corresponding - :class:`BucketData` object. - """ - for d in self.bucketList: - if d.name == bucketIdent: - return d - return None - - def assignUnallocatedBridge(self, bucket): - """Assign an unallocated bridge to a certain **bucket**.""" - hex_key = self.unallocatedList.pop() - # Mark pseudo-allocators in the database as such - allocator_name = bucket.name - #print "KEY: %d NAME: %s" % (hex_key, allocator_name) - logging.debug("Moving %s to %s" % (hex_key, allocator_name)) - with bridgedb.Storage.getDB() as db: - try: - db.updateDistributorForHexKey(allocator_name, hex_key) - except: - db.rollback() - logging.warn("Failed to move %s to new distributor (%s)" - % (hex_key, allocator_name)) - - # Ok, this seems useless, but for consistancy's sake, we'll - # re-assign the bridge from this missed db update attempt to the - # unallocated list. Remember? We pop()'d it before. - self.addToUnallocatedList(hex_key) - raise - else: - db.commit() - bucket.allocated += 1 - if len(self.unallocatedList) < 1: - self.unallocated_available = False - return True - - def assignBridgesToBuckets(self): - """Read file bucket identifiers from the configuration, sort them, and - write necessary changes to the database. - """ - logging.debug("Assigning bridges to buckets for pseudo-distributors") - # Build distributor list - for k, v in self.cfg.FILE_BUCKETS.items(): - prefixed_key = self.distributor_prefix + k - d = BucketData(prefixed_key, v) - self.bucketList.append(d) - - # Loop through all bridges and sort out distributors - with bridgedb.Storage.getDB() as db: - allBridges = db.getAllBridges() - for bridge in allBridges: - if bridge.distributor == "unallocated": - self.addToUnallocatedList(bridge.hex_key) - continue - - # Filter non-pseudo distributors (like 'https' and 'email') early, - # too - if not bridge.distributor.startswith(self.distributor_prefix): - continue - - # Return the bucket in case we know it already - d = self.getBucketByIdent(bridge.distributor) - if d is not None: - # Does this distributor need another bridge? If not, re-inject - # it into the 'unallocated' pool for for later assignment - if d.allocated < d.needed: - d.allocated += 1 - else: - # Bucket has enough members already, free this one - self.addToUnallocatedList(bridge.hex_key) - # We don't know it. Maybe an old entry. Free it. - else: - self.addToUnallocatedList(bridge.hex_key) - - # Loop through bucketList while we have and need unallocated - # bridges, assign one bridge at a time - while self.unallocated_available and len(self.bucketList) > 0: - logging.debug("We have %d unallocated bridges and %d buckets to " \ - "fill. Let's do it." - % (len(self.unallocatedList), len(self.bucketList))) - for d in self.bucketList: - if d.allocated < d.needed: - try: - if not self.assignUnallocatedBridge(d): - break - except sqlite3.DatabaseError as e: - dist = d.name.replace(self.distributor_prefix, "") - logging.warn("Couldn't assign unallocated bridge to " \ - "%s: %s" % (dist, e)) - else: - # When we have enough bridges, remove bucket identifier - # from list - self.bucketList.remove(d) - - def dumpBridgesToFile(self, filename, bridges): - """Dump a list of given **bridges** into **filename**.""" - logging.debug("Dumping bridge assignments to file: %r" % filename) - # get the bridge histories and sort by Time On Same Address - bridgeHistories = [] - with bridgedb.Storage.getDB() as db: - for b in bridges: - if self.cfg.COLLECT_TIMESTAMPS: - bh = db.getBridgeHistory(b.hex_key) - if bh: bridgeHistories.append(bh) - bridgeHistories.sort(lambda x,y: cmp(x.weightedFractionalUptime, - y.weightedFractionalUptime)) - - try: - f = open(filename, 'w') - if self.cfg.COLLECT_TIMESTAMPS: - for bh in bridgeHistories: - days = bh.tosa / long(60*60*24) - line = "%s:%s\t(%d days at this address)" % \ - (bh.ip, bh.port, days) - if str(bh.fingerprint) in blocklist.keys(): - line = line + "\t(Might be blocked): (%s)" % \ - ",".join(blocklist[bh.fingerprint]) - f.write(line + '\n') - else: - for bridge in bridges: - line = "%s:%d %s" \ - % (bridge.address, bridge.or_port, bridge.hex_key) - f.write(line + '\n') - f.close() - except IOError: - print "I/O error: %s" % filename - - def dumpBridges(self): - """Dump all known file distributors to files, sorted by distributor.""" - logging.info("Dumping all distributors to file.") - with bridgedb.Storage.getDB() as db: - allBridges = db.getAllBridges() - bridgeDict = {} - # Sort returned bridges by distributor - for bridge in allBridges: - dist = str(bridge.distributor) - if dist in bridgeDict.keys(): - bridgeDict[dist].append(bridge) - else: - bridgeDict[dist] = [bridge] - - # Now dump to file(s) - for k in bridgeDict.keys(): - dist = k - if (dist.startswith(self.distributor_prefix)): - # Subtract the pseudo distributor prefix - dist = dist.replace(self.distributor_prefix, "") - # Be safe. Replace all '/' in distributor names - dist = dist.replace("/", "_") - filename = dist + "-" + time.strftime("%Y-%m-%d") + ".brdgs" - self.dumpBridgesToFile(filename, bridgeDict[k]) diff --git a/bridgedb/main.py b/bridgedb/main.py index 905af2d..586ed83 100644 --- a/bridgedb/main.py +++ b/bridgedb/main.py @@ -54,6 +54,24 @@ def expandBridgeAuthDir(authdir, filename): return path +def writeAssignments(hashring, filename): + """Dump bridge distributor assignments to disk. + + :type hashring: A :class:`~bridgedb.Bridges.BridgeSplitter` + :ivar hashring: A class which takes an HMAC key and splits bridges + into their hashring assignments. + :param str filename: The filename to write the assignments to. + """ + logging.debug("Dumping pool assignments to file: '%s'" % filename) + + try: + with open(filename, 'a') as fh: + fh.write("bridge-pool-assignment %s\n" % + time.strftime("%Y-%m-%d %H:%M:%S")) + hashring.dumpAssignments(fh) + except IOError: + logging.info("I/O error while writing assignments to: '%s'" % filename) + def load(state, hashring, clear=False): """Read and parse all descriptors, and load into a bridge hashring. @@ -192,19 +210,6 @@ def _handleSIGHUP(*args): """Called when we receive a SIGHUP; invokes _reloadFn.""" reactor.callInThread(_reloadFn) -def _handleSIGUSR1(*args): - """Handler for SIGUSR1. Calls :func:`~bridgedb.runner.doDumpBridges`.""" - logging.debug("Caught SIGUSR1 signal") - - from bridgedb import runner - - logging.info("Loading saved state...") - state = persistent.load() - cfg = loadConfig(state.CONFIG_FILE, state.config) - - logging.info("Dumping bridge assignments to files...") - reactor.callInThread(runner.doDumpBridges, cfg) - def replaceBridgeRings(current, replacement): """Replace the current thing with the new one""" current.hashring = replacement.hashring @@ -274,10 +279,6 @@ def createBridgeRings(cfg, proxyList, key): "unallocated", cfg.RESERVED_SHARE) - # Add pseudo distributors to hashring - for pseudoRing in cfg.FILE_BUCKETS.keys(): - hashring.addPseudoRing(pseudoRing) - return hashring, emailDistributor, ipDistributor, moatDistributor def run(options, reactor=reactor): @@ -317,7 +318,7 @@ def run(options, reactor=reactor): # :func:`logging.basicConfig` will be ignored. util.configureLogging(config) - if options['dump-bridges'] or (options.subCommand is not None): + if options.subCommand is not None: runSubcommand(options, config) # Write the pidfile only after any options.subCommands are run (because @@ -428,18 +429,7 @@ def run(options, reactor=reactor): logging.warn("No Moat distributor created!") # Dump bridge pool assignments to disk. - try: - logging.debug("Dumping pool assignments to file: '%s'" - % state.ASSIGNMENTS_FILE) - fh = open(state.ASSIGNMENTS_FILE, 'a') - fh.write("bridge-pool-assignment %s\n" % - time.strftime("%Y-%m-%d %H:%M:%S")) - hashring.dumpAssignments(fh) - fh.flush() - fh.close() - except IOError: - logging.info("I/O error while writing assignments to: '%s'" - % state.ASSIGNMENTS_FILE) + writeAssignments(hashring, state.ASSIGNMENTS_FILE) state.save() if inThread: @@ -462,7 +452,6 @@ def run(options, reactor=reactor): global _reloadFn _reloadFn = reload signal.signal(signal.SIGHUP, _handleSIGHUP) - signal.signal(signal.SIGUSR1, _handleSIGUSR1) if reactor: # pragma: no cover # And actually load it to start parsing. Get back our distributors. diff --git a/bridgedb/parse/options.py b/bridgedb/parse/options.py index d866ac8..e5c4d1e 100644 --- a/bridgedb/parse/options.py +++ b/bridgedb/parse/options.py @@ -268,34 +268,31 @@ class MockOptions(BaseOptions): class SIGHUPOptions(BaseOptions): """Options menu to explain usage and handling of SIGHUP signals.""" - longdesc = """If you send a SIGHUP to a running BridgeDB process, the - servers will parse and reload all bridge descriptor files into the - databases. - - Note that this command WILL NOT handle sending the signal for you; see - signal(7) and kill(1) for additional help.""" + longdesc = ( + "If you send a SIGHUP to a running BridgeDB process, the servers will " + "parse and reload all bridge descriptor files into the databases." + "\n\n" + "Note that this command WILL NOT handle sending the signal for you; " + "see signal(7) and kill(1) for additional help.") class SIGUSR1Options(BaseOptions): """Options menu to explain usage and handling of SIGUSR1 signals.""" - longdesc = """If you send a SIGUSR1 to a running BridgeDB process, the - servers will dump all bridge assignments by distributor from the - databases to files. - - Note that this command WILL NOT handle sending the signal for you; see - signal(7) and kill(1) for additional help.""" + longdesc = ( + "If you send a SIGUSR1 to a running BridgeDB process, the servers will " + "dump all bridge assignments by distributor from the databases to files." + "\n\n" + "Note that this command WILL NOT handle sending the signal for you; see " + "signal(7) and kill(1) for additional help.") class MainOptions(BaseOptions): """Main commandline options parser for BridgeDB.""" optFlags = [ - ['dump-bridges', 'd', 'Dump bridges by hashring assignment into files'], ['reload', 'R', 'Reload bridge descriptors into running servers']] subCommands = [ ['mock', None, MockOptions, "Generate a testing environment"], ['SIGHUP', None, SIGHUPOptions, - "Reload bridge descriptors into running servers"], - ['SIGUSR1', None, SIGUSR1Options, - "Dump bridges by hashring assignment into files"]] + "Reload bridge descriptors into running servers"]] diff --git a/bridgedb/runner.py b/bridgedb/runner.py index 57297cb..35865e5 100644 --- a/bridgedb/runner.py +++ b/bridgedb/runner.py @@ -122,17 +122,3 @@ def generateDescriptors(count=None, rundir=None): print("Sucessfully generated %s descriptors." % str(count)) del subprocess return statuscode - -def doDumpBridges(config): - """Dump bridges by assignment to a file. - - This function handles the commandline '--dump-bridges' option. - - :type config: :class:`bridgedb.main.Conf` - :param config: The current configuration. - """ - import bridgedb.Bucket as bucket - - bucketManager = bucket.BucketManager(config) - bucketManager.assignBridgesToBuckets() - bucketManager.dumpBridges() diff --git a/bridgedb/test/test_Bucket.py b/bridgedb/test/test_Bucket.py deleted file mode 100644 index 01f3f26..0000000 --- a/bridgedb/test/test_Bucket.py +++ /dev/null @@ -1,63 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of BridgeDB, a Tor bridge distribution system. -# -# :copyright: (c) 2007-2017, The Tor Project, Inc. -# (c) 2007-2017, all entities within the AUTHORS file -# :license: 3-Clause BSD, see LICENSE for licensing information - -"""Unittests for the :mod:`bridgedb.Bucket` module. - -These tests are meant to ensure that the :mod:`bridgedb.Bucket` module is -functioning as expected. -""" - -from __future__ import print_function - -from io import StringIO - -import sure -from sure import this -from sure import the -from sure import expect - -from bridgedb import Bucket - -from twisted.trial import unittest - - -class BucketDataTest(unittest.TestCase): - """Tests for :class:`bridgedb.Bucket.BucketData`.""" - - def test_alloc_some_of_the_bridges(self): - """Set the needed number of bridges""" - needed = 10 - distname = "test-distributor" - bucket = Bucket.BucketData(distname, needed) - this(bucket.name).should.be.equal(distname) - this(bucket.needed).should.be.equal(needed) - - def test_alloc_all_the_bridges(self): - """Set the needed number of bridges to the default""" - needed = '*' - distname = "test-distributor" - bucket = Bucket.BucketData(distname, needed) - this(bucket.name).should.be.equal(distname) - this(bucket.needed).should.be.equal(Bucket.BUCKET_MAX_BRIDGES) - - -class BucketManagerTest(unittest.TestCase): - """Tests for :class:`bridgedb.Bucket.BucketManager`.""" - TEST_CONFIG_FILE = StringIO(unicode("""\ - FILE_BUCKETS = { 'test1': 7, 'test2': 11 } - COLLECT_TIMESTAMPS = False - COUNTRY_BLOCK_FILE = []""")) - - def setUp(self): - configuration = {} - TEST_CONFIG_FILE.seek(0) - compiled = compile(TEST_CONFIG_FILE.read(), '<string>', 'exec') - exec compiled in configuration - self.config = persistent.Conf(**configuration) - self.state = persistent.State(**config.__dict__) - self.bucket = Bucket.BucketManager(self.config) diff --git a/bridgedb/test/test_bridgedb_script.py b/bridgedb/test/test_bridgedb_script.py index 70c71f0..fb8bb3e 100644 --- a/bridgedb/test/test_bridgedb_script.py +++ b/bridgedb/test/test_bridgedb_script.py @@ -82,22 +82,3 @@ class BridgeDBCliTest(unittest.TestCase): os.kill(self.pid, signal.SIGHUP) self.doSleep() self.assertTrue(os.path.isfile(self.assignmentsFile)) - - def test_bridgedb_SIGUSR1_buckets(self): - """Test that BridgeDB dumps buckets appropriately after a SIGUSR1.""" - if os.environ.get("CI"): - if not self.pid or not processExists(self.pid): - raise FailTest("Could not start BridgeDB process on CI server!") - if not self.pid or not processExists(self.pid): - raise SkipTest("Can't run test: no BridgeDB process running.") - - os.kill(self.pid, signal.SIGUSR1) - self.doSleep() - buckets = [['email', False], ['https', False], ['unallocated', False]] - for rundirfile in os.listdir(self.rundir): - for bucket in buckets: - if rundirfile.startswith(bucket[0]): - bucket[1] = True - break - for bucket in buckets: - self.assertTrue(bucket[1], "%s bucket was not dumped!" % bucket[0]) diff --git a/bridgedb/test/test_main.py b/bridgedb/test/test_main.py index 2a124b1..b6eae2b 100644 --- a/bridgedb/test/test_main.py +++ b/bridgedb/test/test_main.py @@ -372,29 +372,6 @@ class BridgedbTests(unittest.TestCase): self.assertEqual(len(hashring.ringsByName.keys()), 3) self.assertNotIn('unallocated', hashring.rings) - def test_main_createBridgeRings_two_file_buckets(self): - """When FILE_BUCKETS has two filenames in it, main.createBridgeRings() - should add three hashrings to the hashring, then add two - "pseudo-rings". - """ - proxyList = main.proxy.ProxySet() - config = self.config - config.FILE_BUCKETS = { - 'bridges-for-support-desk': 10, - 'bridges-for-ooni-tests': 10, - } - (hashring, emailDist, httpsDist, moatDist) = main.createBridgeRings( - config, proxyList, self.key) - - # Should have an HTTPSDistributor ring, an EmailDistributor, a - # MoatDistributor and an UnallocatedHolder ring: - self.assertEqual(len(hashring.ringsByName.keys()), 4) - - # Should have two pseudoRings: - self.assertEqual(len(hashring.pseudoRings), 2) - self.assertIn('pseudo_bridges-for-support-desk', hashring.pseudoRings) - self.assertIn('pseudo_bridges-for-ooni-tests', hashring.pseudoRings) - def test_main_run(self): """main.run() should run and then finally raise SystemExit.""" config = """ @@ -473,8 +450,7 @@ EMAiL_GPG_PASSPHRASE = None EMAIL_GPG_PASSPHRASE_FILE = None HTTPS_SHARE = 10 EMAIL_SHARE = 5 -RESERVED_SHARE = 2 -FILE_BUCKETS = {}""" +RESERVED_SHARE = 2""" configFile = self._writeConfig(config) # Fake some options: diff --git a/doc/sphinx/source/bridgedb.Bucket.rst b/doc/sphinx/source/bridgedb.Bucket.rst deleted file mode 100644 index 753bafd..0000000 --- a/doc/sphinx/source/bridgedb.Bucket.rst +++ /dev/null @@ -1,8 +0,0 @@ -bridgedb.Bucket ----------------- - -.. automodule:: bridgedb.Bucket - :members: - :undoc-members: - :private-members: - :show-inheritance: diff --git a/doc/sphinx/source/bridgedb.rst b/doc/sphinx/source/bridgedb.rst index 1f272b3..9a8f092 100644 --- a/doc/sphinx/source/bridgedb.rst +++ b/doc/sphinx/source/bridgedb.rst @@ -9,7 +9,6 @@ Packages & Modules bridgedb.bridgerequest bridgedb.bridges bridgedb.Bridges - bridgedb.Bucket bridgedb.captcha bridgedb.configure bridgedb.distribute diff --git a/doc/sphinx/source/conf.py b/doc/sphinx/source/conf.py index e98d2b2..31ae0b4 100644 --- a/doc/sphinx/source/conf.py +++ b/doc/sphinx/source/conf.py @@ -32,7 +32,6 @@ import bridgedb.bridgerequest import bridgedb.bridges import bridgedb.captcha import bridgedb.Bridges -import bridgedb.Bucket import bridgedb.crypto import bridgedb.distribute import bridgedb.distributors

1 0

[bridgedb/master] Merge branch 'fix/24460' into develop
by isis＠torproject.org 20 Dec '17

20 Dec '17

commit 16e7a2d5fad1658244c204bdfb54e5e385e4632f Merge: a316083 77776de Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 28 19:44:42 2017 +0000 Merge branch 'fix/24460' into develop bridgedb/distributors/https/server.py | 6 ++++-- bridgedb/test/test_https_server.py | 14 ++++++++++++++ 2 files changed, 18 insertions(+), 2 deletions(-)

1 0