February 2014 - tor-commits - lists.torproject.org

[bridgedb/master] Update Bridge.__init__() docstring for ip
by isis＠torproject.org 21 Feb '14

21 Feb '14

commit b5c8cfb94c06eea8571f035df1ba957e72e40bbb Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Sat Feb 8 17:07:51 2014 +0000 Update Bridge.__init__() docstring for ip --- lib/bridgedb/Bridges.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/bridgedb/Bridges.py b/lib/bridgedb/Bridges.py index e8201b5..f44be55 100644 --- a/lib/bridgedb/Bridges.py +++ b/lib/bridgedb/Bridges.py @@ -112,7 +112,8 @@ class Bridge: """Holds information for a single bridge :attr str nickname: The bridge's nickname. Not currently used. - :attr str ip: The bridge's IPv4 address, as a dotted quad. + :type ip: :class:`ipaddr.IPAddress` + :attr ip: The bridge's IPv4 address, specified on 'r'-line in NS :attr int orport: The bridge's OR port. :attr dict or_addresses: The bridges alternate IP addresses, keys the address, values are the port(s) on

1 0

[bridgedb/master] Add draft of volunteer page text for GSoC 2014 applications.
by isis＠torproject.org 21 Feb '14

21 Feb '14

commit 90d759d291907e87f2364a59ca1ee57f604f579c Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Feb 21 21:00:02 2014 +0000 Add draft of volunteer page text for GSoC 2014 applications. --- doc/gsoc2014-volunteer.txt | 92 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/doc/gsoc2014-volunteer.txt b/doc/gsoc2014-volunteer.txt new file mode 100644 index 0000000..eb93611 --- /dev/null +++ b/doc/gsoc2014-volunteer.txt @@ -0,0 +1,92 @@ + +Volunteer Posting +----------------- + +Number of Positions Available: 2 + +Project: BridgeDB https://gitweb.torproject.org/bridgedb.git + +Difficulty Level: Moderate - Difficult, dependent on proposal and previous experience + +Myself (isis) and sysrqb are both looking for students who wish to help Tor by +contributing code to BridgeDB (the program which distributes Tor bridges and +pluggable transport bridges to clients connecting to Tor) through +participation in Google Summer of Code 2014. We are each taking a student, so +if we each find someone someone we want to mentor, then we'll take two +students total. If we only find one student we like, then I suppose sysrqb and +I will just have to fight over that person. + +Required Skills and Prerequisite Tasks for Application: +------------------------------------------------------- + * IRC skills. Lurk in irc.oftc.net#tor-dev, and possibly irc.freenode.net#gsoc as well. + + * Fluent in Python (version 2.7). + + * Proven experience and comfort with asynchronous network programming using + Twisted. + + * Be comfortable doing git things. Not saying you have to have mad + git-fu, but we shouldn't have to explain the difference between + merge/rebase or fetch/pull. + + * Familiarity with documentating code in Sphinx ReStructured Text + format. (See recent commits in the BridgeDB repository for syntax + examples, and https://para.noid.cat/bridgedb/ for documentation builds.) + + * Familiarity with writing unittests. (And willingness to do so!) + Preferably, using `twisted.trial.unittest.TestCase`s. See + https://coveralls.io/r/isislovecruft/bridgedb for current test coverage. + + * Submit a brief sample of Python code that you're proud of (written by you, + and for which you've got whatever necessary legal whatever is needed for + you to share it). It helps if it's Twisted Python code, but it doesn't + have to be. + + * Fix a BridgeDB ticket. Contact isis or sysrqb on ircs://irc.oftc.net:6697 + in #tor-dev to ask if a particular ticket is a good idea, or to ask for a + suggestion or help. Eventually, sysrqb and I will tag some tickets with + "bridgedb-gsoc-application" that we think are good ones to try. + + * Submit a proposal for a distributor (an object which distributes Tor + bridges to clients in possibly censored/surveilled regions). Current + distributors include the `bridgedb.Dist.EmailBasedDistributor` at + bridges(a)bridges.torproject.org, and the `bridgedb.Dist.IPBasedDistributor` + which serves pages at https://bridges.torproject.org . Other example ideas + for bridge distributors are: + - A Twitter bot which interacts with Chinese and Farsi speaking + Twitter users through PMs. + - A distributor which uses XMPP+OTR to give bridges to users. + - Basically, all the nifty protocol obfuscation tricks which could + apply to a pluggable transport would likely also apply here. + - Keep in mind that various adversaries (automated and/or human and/or + extraterrestrial and/or unknown) will be trying to trick your + distributor into giving them bridges. Be prepared to come up with + some sort of safety machanism(s) to protect against this. + - You should plan to have your code finished, reviewed (and + revised…), tested, approved, and deployed by the end of GSoC. (I + think this means "at the end of six weeks".) + + * Please send your code sample, a link to your completed BridgeDB ticket + (which should have your patch(es) attached, or have a link to your git + repository), and proposal to both isis(a)torproject.org and + sysrqb(a)torproject.org. Additionally CC'ing tor-dev(a)lists.torproject.org is + probably a good idea. + + * If accepted, you'll need a google-melange account: + https://www.google-melange.com/gsoc/homepage/google/gsoc2014?ModPagespeed=n… + +Additional Ways to Earn Bonus Points: +------------------------------------- + * OTR skills (over IRC/XMPP). + + * Knowledge about the current state and technical mechanisms of real-world, + deployed censorship/surveillance systems is helpful, though not necessary. + We will gladly provide continual feedback on design proposals and + implementation to ensure that the threats posed by various adversaries + have been taken into account. + + * Major bonus points if you're into test-driven development. + + * Propose building more than one distributor. Each of them still needs to be + a good idea, and we'll need to make certain it's something you're capable + of finishing in the allotted timespan.

1 0

[bridgedb/master] Merge branch 'release-0.1.3'
by isis＠torproject.org 21 Feb '14

21 Feb '14

commit 564f4b9fdf0c63128c1fd6d1e7f4e36a42848fb4 Merge: 9871642 26ba67b Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Feb 21 21:54:41 2014 +0000 Merge branch 'release-0.1.3' .travis.yml | 3 +- CHANGELOG | 14 ++++ doc/DESCRIPTORS.md | 4 +- doc/gsoc2014-volunteer.txt | 92 +++++++++++++++++++++ lib/bridgedb/Bridges.py | 151 ++++++++++++++++++++++++++++++----- lib/bridgedb/HTTPServer.py | 55 +++++++++++-- lib/bridgedb/Main.py | 72 ++++++++--------- lib/bridgedb/parse/networkstatus.py | 2 +- lib/bridgedb/persistent.py | 11 ++- lib/bridgedb/templates/base.html | 5 ++ lib/bridgedb/templates/index.html | 11 +-- lib/bridgedb/templates/options.html | 39 +++++++++ lib/bridgedb/test/test_Bridges.py | 111 +++++++++++++++++++++++++ 13 files changed, 491 insertions(+), 79 deletions(-)

1 0

[bridgedb/master] Update CHANGELOG with entries for version 0.1.3.
by isis＠torproject.org 21 Feb '14

21 Feb '14

commit 26ba67b41cb7340d70164b3238882b730b2df0aa Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Feb 21 21:50:51 2014 +0000 Update CHANGELOG with entries for version 0.1.3. --- CHANGELOG | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index 4ac962b..bd2a28e 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,3 +1,17 @@ +Changes in version 0.1.3 - 2014-02-21 +BridgeDB 0.1.3 includes fixes for the following bugs: + * FIXES #9264 Problem with transport lines in BridgeDB's bridge + pool assignment files + * FIXES a bug caused by attempts to convert descriptor digests + which could not be parsed into hexadecimal. + * FIXES a unicode decoding error within the bridgedb.persistent + module. + * REMOVES continuous integration testing for Python 2.6. + +Including the following general changes: + * ADD volunteer page text for GSoC 2014 applications. + + Changes in version 0.1.2 - 2014-02-05 BridgeDB 0.1.2 includes fixes for the following bugs: * FIXES #10811 BridgeDB's assignments.log files are not being updated

1 0

[bridgedb/master] Fix error when desc digest parsed from networkstatus is None.
by isis＠torproject.org 21 Feb '14

21 Feb '14

commit a2192c4d0fb0101526d96c3ad9267fcc653659ca Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Feb 21 20:50:01 2014 +0000 Fix error when desc digest parsed from networkstatus is None. * FIXES a bug where, if the nickname was not to spec (longer than 19 characters), then the nickname and descriptor digest are both set to ``None`` and ``bridgedb.Bridges.parseStatusFile()`` attempts to format the null descriptor digest into hexadecimal. --- lib/bridgedb/Bridges.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/bridgedb/Bridges.py b/lib/bridgedb/Bridges.py index 4aaf86f..e5d5921 100644 --- a/lib/bridgedb/Bridges.py +++ b/lib/bridgedb/Bridges.py @@ -641,12 +641,12 @@ def parseStatusFile(networkstatusFile): logging.debug(" Nickname: %s" % nickname) logging.debug(" Identity: %s" % hexID) if descDigest: - logging.debug(" Descriptor: {0}".format(toHex(descDigest))) + descDigest = toHex(descDigest) + logging.debug(" Descriptor: {0}".format(descDigest)) logging.debug(" Timestamp: {0}".format(timestamp)) logging.debug(" ORAddress: {0}".format(ORaddr)) logging.debug(" ORport: {0}".format(ORport)) logging.debug(" dirport: {0}".format(dirport)) - descDigest = toHex(descDigest) elif ID and line.startswith("a "): try:

1 0

[bridgedb/develop] We need to compute the hash digest of the documents during (re)loading
by isis＠torproject.org 21 Feb '14

21 Feb '14

commit e2d15df828ac6f4678ca5b8c132ab6c466465bc4 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Fri Jan 31 18:13:31 2014 +0000 We need to compute the hash digest of the documents during (re)loading --- lib/bridgedb/Bridges.py | 71 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/lib/bridgedb/Bridges.py b/lib/bridgedb/Bridges.py index c8ecc9f..2c1b34e 100644 --- a/lib/bridgedb/Bridges.py +++ b/lib/bridgedb/Bridges.py @@ -15,6 +15,7 @@ import socket import time import ipaddr import random +import hashlib import bridgedb.Storage import bridgedb.Bucket @@ -23,6 +24,11 @@ import bridgedb.Util as Util from bridgedb.parse import addr from bridgedb.parse import networkstatus +try: + from cStringIO import StringIO +except ImportError: + from io import StringIO + HEX_FP_LEN = 40 ID_LEN = 20 @@ -316,6 +322,71 @@ class Bridge: db = bridgedb.Storage.getDB() return db.getBridgeHistory(self.fingerprint).weightedUptime +def getDescriptorDigests(desc): + """Return the SHA-1 hash hexdigests of all descriptor descs + + :param File desc: A string containing the contents of one + or more bridge descriptors concatenated + together. + :returns: A dict indexed by the SHA-1 hexdigest of the bridge + descriptor, equivalent to that which was published + on the 'r' line of the networkstatus for this bridge. + The value is the bridge's extra-info document digest, + or None, if not provided. + """ + if not desc: return None + + descriptors = {} + sha1hash = hashlib.sha1() + ei_digest = None + + for line in desc: + if line != '-----BEGIN SIGNATURE-----\n': + sha1hash.update(line) + if line.startswith('extra-info-digest'): + parts = line.split() + if len(parts) == 2: + ei_digest = parts[1].lower() + else: + digest = sha1hash.hexdigest().lower() + descriptors[digest] = ei_digest + while line != '-----END SIGNATURE-----\n': + line = next(desc) + sha1hash = hashlib.sha1() + ei_digest = None + return descriptors + +def getExtraInfoDigests(doc): + """Return the SHA-1 hash hexdigests of all extra-info documents + + :param File doc: A string containing the contents of one + or more bridge extra-info documents concatenated + together. + :returns: A dict indexed by the SHA-1 hexdigest of the bridge + extra-info doc, equivalent to that which was published + on the 'extra-info-digest' line of the bridge's + descriptor. The value is the bridge's extra-info document + digest, or None, if not provided. + """ + if not doc: return None + + documents = {} + sha1hash = hashlib.sha1() + document_content = '' + + for line in doc: + if line != '-----BEGIN SIGNATURE-----\n': + sha1hash.update(line) + document_content += line + else: + digest = sha1hash.hexdigest().lower() + documents[digest] = StringIO(document_content) + while line != '-----END SIGNATURE-----\n': + line = next(doc) + sha1hash = hashlib.sha1() + document_content = '' + return documents + def parseDescFile(f, bridge_purpose='bridge'): """Generator. Parses a cached-descriptors file 'f' and yeilds a Bridge object for every entry whose purpose matches bridge_purpose.

1 0

[bridgedb/develop] Prefer to use information from NS rather than descriptor
by isis＠torproject.org 21 Feb '14

21 Feb '14

commit bafeaaa06f5a0f70b922b93f3d14baa2e6ff33fe Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Fri Jan 31 18:12:48 2014 +0000 Prefer to use information from NS rather than descriptor --- lib/bridgedb/Bridges.py | 14 ++++++++++---- lib/bridgedb/Main.py | 5 +++-- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/lib/bridgedb/Bridges.py b/lib/bridgedb/Bridges.py index 6bf4b2e..c8ecc9f 100644 --- a/lib/bridgedb/Bridges.py +++ b/lib/bridgedb/Bridges.py @@ -540,6 +540,7 @@ def parseStatusFile(networkstatusFile): logging.debug(" ORAddress: {0}".format(ORaddr)) logging.debug(" ORport: {0}".format(ORport)) logging.debug(" dirport: {0}".format(dirport)) + descDigest = toHex(descDigest) elif ID and line.startswith("a "): try: @@ -556,10 +557,15 @@ def parseStatusFile(networkstatusFile): elif ID and timestamp and line.startswith("s "): running, stable = networkstatus.parseSLine(line) - logging.debug("Bridges.parseStatusFile(): "\ - "yielding %s running=%s stable=%s oraddrs=%s ts=%s" - % (toHex(ID), running, stable, or_addresses, timestamp)) - yield ID, running, stable, or_addresses, timestamp + logging.debug("Bridges.parseStatusFile(): " + "yielding %s nickname=%s descDigest=%s " + "running=%s stable=%s oraddr=%s orport=%s " + "oraddrs=%s ts=%s" + % (toHex(ID), nickname, descDigest, running, + stable, ORaddr, ORport, or_addresses, + timestamp)) + yield (ID, nickname, descDigest, running, stable, ORaddr, ORport, + or_addresses, timestamp) (nickname, ID, descDigest, timestamp, ORaddr, ORport, dirport, addr, portlist) = (None for x in xrange(9)) diff --git a/lib/bridgedb/Main.py b/lib/bridgedb/Main.py index 4ab4d02..c3a90ea 100644 --- a/lib/bridgedb/Main.py +++ b/lib/bridgedb/Main.py @@ -98,8 +98,9 @@ def load(state, splitter, clear=False): logging.info("Opening network status file: %s" % state.STATUS_FILE) f = open(state.STATUS_FILE, 'r') - for (ID, running, stable, - or_addresses, timestamp) in Bridges.parseStatusFile(f): + for (ID, nickname, descDigest, running, stable, + ORaddr, ORport, or_addresses, + timestamp) in Bridges.parseStatusFile(f): status[ID] = running, stable addresses[ID] = or_addresses

1 0

[bridgedb/develop] We need the ORPort to be an int, convert it from the get-go
by isis＠torproject.org 21 Feb '14

21 Feb '14

commit 36af115dd5aef145c1fdd4d4db75eca86ad7f635 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Sat Feb 1 22:46:19 2014 +0000 We need the ORPort to be an int, convert it from the get-go --- lib/bridgedb/parse/networkstatus.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bridgedb/parse/networkstatus.py b/lib/bridgedb/parse/networkstatus.py index 0b73569..9304009 100644 --- a/lib/bridgedb/parse/networkstatus.py +++ b/lib/bridgedb/parse/networkstatus.py @@ -132,7 +132,7 @@ def parseRLine(line): timestamp = time.mktime(time.strptime(" ".join(fields[3:5]), "%Y-%m-%d %H:%M:%S")) ORaddr = fields[5] - ORport = fields[6] + ORport = int(fields[6]) dirport = fields[7] except InvalidBase64 as error: logging.error(error)

1 0

[bridgedb/develop] We need to remember the digests of the documents the bridge published
by isis＠torproject.org 21 Feb '14

21 Feb '14

commit 1bf55b5aa58e89b8654d6bb7d9458bbdcd37b261 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Sun Feb 2 02:55:04 2014 +0000 We need to remember the digests of the documents the bridge published --- lib/bridgedb/Bridges.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/lib/bridgedb/Bridges.py b/lib/bridgedb/Bridges.py index 2c1b34e..9e07ad8 100644 --- a/lib/bridgedb/Bridges.py +++ b/lib/bridgedb/Bridges.py @@ -131,6 +131,9 @@ class Bridge: self.transports = transports self.running = self.stable = None self.blockingCountries = {} + self.desc_digest = None + self.ei_digest = None + self.verified = False if id_digest is not None: assert fingerprint is None @@ -145,6 +148,22 @@ class Bridge: else: raise TypeError("Bridge with no ID") + def setDescriptorDigest(self, digest): + """Set the descriptor digest, specified in the NS.""" + self.desc_digest = digest + + def setExtraInfoDigest(self, digest): + """Set the extra-info digest, specified in the descriptor.""" + self.ei_digest = digest + + def setVerified(self): + """Call when the bridge's descriptor is parsed""" + self.verified = True + + def isVerified(self): + """Returns the truthiness of ``verified``""" + return self.verified + def getID(self): """Return the bridge's identity digest.""" return fromHex(self.fingerprint)

1 0

[bridgedb/develop] Fix UnicodeDecodeErrors from unittests for bridgedb.persistent.
by isis＠torproject.org 21 Feb '14

21 Feb '14

commit 1a8fb997ad2a61df32cd633e727401a7fea85d03 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sun Feb 9 00:35:05 2014 +0000 Fix UnicodeDecodeErrors from unittests for bridgedb.persistent. --- lib/bridgedb/persistent.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/lib/bridgedb/persistent.py b/lib/bridgedb/persistent.py index 311240a..84260f6 100644 --- a/lib/bridgedb/persistent.py +++ b/lib/bridgedb/persistent.py @@ -252,12 +252,16 @@ class State(jelly.Jellyable): if value != self.config.__dict__[key]: setattr(self, key, value) updated.append(key) - logging.debug("Updated %s setting: %r → %r" - % (key, self.config.__dict__[key], value)) + logging.debug("Updated %s setting: %r → %r" % + (key.decode('ascii', 'replace'), + self.config.__dict__[key], + value)) except (KeyError, AttributeError): setattr(self, key, value) new.append(key) - logging.debug("New setting: %s = %r" % (key, value)) + logging.debug("New setting: %s = %r" % + (key.decode('ascii', 'replace'), + value)) logging.info("Updated setting(s): %s" % ' '.join([x for x in updated])) logging.info("New setting(s): %s" % ' '.join([x for x in new]))

1 0