April 2020 - tor-commits - lists.torproject.org

[bridgedb/master] Merge branch 'defect/31967' into develop
by phw＠torproject.org 01 Apr '20

01 Apr '20

commit 5afd164ba046173e23a6c7be57a3183b41243cf1 Merge: 580222d a1f0e51 Author: Philipp Winter <phw(a)nymity.ch> Date: Tue Mar 24 09:40:57 2020 -0700 Merge branch 'defect/31967' into develop CHANGELOG | 3 +++ bridgedb/captcha.py | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)

1 0

[bridgedb/master] Add CHANGELOG entry for #31967.
by phw＠torproject.org 01 Apr '20

01 Apr '20

commit a1f0e51fd7a082676f880e0783cc9ec9c94bae2f Author: Philipp Winter <phw(a)nymity.ch> Date: Tue Mar 24 09:28:50 2020 -0700 Add CHANGELOG entry for #31967. --- CHANGELOG | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index b0e8275..5db8fb2 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,3 +1,6 @@ + * FIXES https://bugs.torproject.org/31967 + Use a CSPRNG for selecting cached CAPTCHAs. + * FIXES https://bugs.torproject.org/33008 Add an info page, available at bridges.torproject.org/info. Relay Search links to this info page to explain to bridge operators what their

1 0

[bridgedb/master] Update instructions on how to add bridges.
by phw＠torproject.org 01 Apr '20

01 Apr '20

commit 9c1175441de0c7a9a9c5ca6ff327cfb9556b9842 Author: Philipp Winter <phw(a)nymity.ch> Date: Tue Mar 24 10:23:25 2020 -0700 Update instructions on how to add bridges. In addition to updating the existing instructions, this patch adds instructions for adding bridges on Android. This fixes <https://bugs.torproject.org/30317>. --- CHANGELOG | 5 +++ bridgedb/distributors/email/templates.py | 9 +---- bridgedb/distributors/https/templates/bridges.html | 20 ++++----- bridgedb/distributors/https/templates/howto.html | 20 ++++----- bridgedb/i18n/templates/bridgedb.pot | 47 +++++----------------- bridgedb/strings.py | 27 ++++--------- bridgedb/test/test_https_server.py | 8 ++-- 7 files changed, 44 insertions(+), 92 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index 5db8fb2..f57c168 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,3 +1,8 @@ + * FIXES https://bugs.torproject.org/30317 + Update our "howto" box, which explains how one adds bridges to Tor + Browser. In addition to updating the instructions, this patch also + links to instructions for Android. + * FIXES https://bugs.torproject.org/31967 Use a CSPRNG for selecting cached CAPTCHAs. diff --git a/bridgedb/distributors/email/templates.py b/bridgedb/distributors/email/templates.py index 188f052..570341a 100644 --- a/bridgedb/distributors/email/templates.py +++ b/bridgedb/distributors/email/templates.py @@ -96,14 +96,9 @@ def addHowto(template): """ howToTBB = template.gettext(strings.HOWTO_TBB[1]) % strings.EMAIL_SPRINTF["HOWTO_TBB1"] howToTBB += u'\n\n' - howToTBB += template.gettext(strings.HOWTO_TBB[2]) - howToTBB += u'\n\n' - howToTBB += u'\n'.join(["> {0}".format(ln) for ln in - template.gettext(strings.HOWTO_TBB[3]).split('\n')]) - howToTBB += u'\n\n' - howToTBB += template.gettext(strings.HOWTO_TBB[4]) - howToTBB += u'\n\n' howToTBB += strings.EMAIL_REFERENCE_LINKS.get("HOWTO_TBB1") + howToTBB += strings.EMAIL_REFERENCE_LINKS.get("HOWTO_TBB2") + howToTBB += strings.EMAIL_REFERENCE_LINKS.get("HOWTO_TBB3") howToTBB += u'\n\n' return howToTBB diff --git a/bridgedb/distributors/https/templates/bridges.html b/bridgedb/distributors/https/templates/bridges.html index a7503f5..55a9e79 100644 --- a/bridgedb/distributors/https/templates/bridges.html +++ b/bridgedb/distributors/https/templates/bridges.html @@ -91,21 +91,15 @@ ${_("""This QRCode contains your bridge lines. Scan it with a QRCode """ \ <div class="container-fluid" id="howto"> <p> ${_(strings.HOWTO_TBB[1]) % \ - ("""<a href="https://www.torproject.org/projects/torbrowser.html" + ("""<a href="https://www.torproject.org/download/" + target="_blank">""", + """</a>""", + """<a href="https://tb-manual.torproject.org/bridges/#entering-bridge-addresses" + target="_blank">""", + """</a>""", + """<a href="https://tb-manual.torproject.org/mobile-tor/#circumvention" target="_blank">""", """</a>""")} - ${_(strings.HOWTO_TBB[2])} - </p> - <br /> - <div class="bs-component"> - <blockquote> - <p> - ${_(strings.HOWTO_TBB[3])} - </p> - </blockquote> - </div> - <p> - ${_(strings.HOWTO_TBB[4])} </p> </div> </div> diff --git a/bridgedb/distributors/https/templates/howto.html b/bridgedb/distributors/https/templates/howto.html index 70fca6a..1d31491 100644 --- a/bridgedb/distributors/https/templates/howto.html +++ b/bridgedb/distributors/https/templates/howto.html @@ -15,21 +15,15 @@ <div class="container-fluid" id="howto"> <p> ${_(strings.HOWTO_TBB[1]) % \ - ("""<a href="https://www.torproject.org/projects/torbrowser.html" + ("""<a href="https://www.torproject.org/download/" + target="_blank">""", + """</a>""", + """<a href="https://tb-manual.torproject.org/bridges/#entering-bridge-addresses" + target="_blank">""", + """</a>""", + """<a href="https://tb-manual.torproject.org/mobile-tor/#circumvention" target="_blank">""", """</a>""")} - ${_(strings.HOWTO_TBB[2])} - </p> - <br /> - <div class="bs-component"> - <blockquote> - <p> - ${_(strings.HOWTO_TBB[3])} - </p> - </blockquote> - </div> - <p> - ${_(strings.HOWTO_TBB[4])} </p> </div> </div> diff --git a/bridgedb/i18n/templates/bridgedb.pot b/bridgedb/i18n/templates/bridgedb.pot index 8d442df..8fa69c6 100644 --- a/bridgedb/i18n/templates/bridgedb.pot +++ b/bridgedb/i18n/templates/bridgedb.pot @@ -5,11 +5,11 @@ # msgid "" msgstr "" -"Project-Id-Version: bridgedb 0.9.4+6.g963ce80.dirty\n" +"Project-Id-Version: bridgedb 0.9.4+8.g5afd164.dirty\n" "Report-Msgid-Bugs-To: " "'https://trac.torproject.org/projects/tor/newticket?component=BridgeDB&keywo…" "=bridgedb-reported,msgid&cc=isis,sysrqb&owner=isis'\n" -"POT-Creation-Date: 2020-03-18 09:23-0700\n" +"POT-Creation-Date: 2020-03-24 10:22-0700\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n" "Language-Team: LANGUAGE <LL(a)li.org>\n" @@ -428,60 +428,35 @@ msgstr "" #: bridgedb/strings.py:184 #, python-format msgid "" -"To enter bridges into Tor Browser, first go to the %s Tor Browser download\n" -"page %s and then follow the instructions there for downloading and starting\n" -"Tor Browser." +" First, you need to %sdownload Tor Browser%s. Our Tor Browser User\n" +" Manual explains how you can add your bridges to Tor Browser. If you are\n" +" using Windows, Linux, or OS X, %sclick here%s to learn more. If you\n" +" are using Android, %sclick here%s." msgstr "" -#. TRANSLATORS: Please DO NOT translate "Tor". -#: bridgedb/strings.py:189 -msgid "" -"When the 'Tor Network Settings' dialogue pops up, click 'Configure' and " -"follow\n" -"the wizard until it asks:" -msgstr "" - -#. TRANSLATORS: Please DO NOT translate "Tor". -#: bridgedb/strings.py:193 -msgid "" -"Does your Internet Service Provider (ISP) block or otherwise censor " -"connections\n" -"to the Tor network?" -msgstr "" - -#. TRANSLATORS: Please DO NOT translate "Tor". -#: bridgedb/strings.py:197 -msgid "" -"Select 'Yes' and then click 'Next'. To configure your new bridges, copy and\n" -"paste the bridge lines into the text input box. Finally, click 'Connect', and" -"\n" -"you should be good to go! If you experience trouble, try clicking the 'Help'\n" -"button in the 'Tor Network Settings' wizard for further assistance." -msgstr "" - -#: bridgedb/strings.py:205 +#: bridgedb/strings.py:192 msgid "Displays this message." msgstr "" #. TRANSLATORS: Please try to make it clear that "vanilla" here refers to the #. same non-Pluggable Transport bridges described above as being #. "plain-ol'-vanilla" bridges. -#: bridgedb/strings.py:209 +#: bridgedb/strings.py:196 msgid "Request vanilla bridges." msgstr "" -#: bridgedb/strings.py:210 +#: bridgedb/strings.py:197 msgid "Request IPv6 bridges." msgstr "" #. TRANSLATORS: Please DO NOT translate the word the word "TYPE". -#: bridgedb/strings.py:212 +#: bridgedb/strings.py:199 msgid "Request a Pluggable Transport by TYPE." msgstr "" #. TRANSLATORS: Please DO NOT translate "BridgeDB". #. TRANSLATORS: Please DO NOT translate "GnuPG". -#: bridgedb/strings.py:215 +#: bridgedb/strings.py:202 msgid "Get a copy of BridgeDB's public GnuPG key." msgstr "" diff --git a/bridgedb/strings.py b/bridgedb/strings.py index ca2feaa..b6a7f1f 100644 --- a/bridgedb/strings.py +++ b/bridgedb/strings.py @@ -182,23 +182,10 @@ HOWTO_TBB = { 0: _("""How to start using your bridges"""), # TRANSLATORS: Please DO NOT translate "Tor Browser". 1: _("""\ -To enter bridges into Tor Browser, first go to the %s Tor Browser download -page %s and then follow the instructions there for downloading and starting -Tor Browser."""), - # TRANSLATORS: Please DO NOT translate "Tor". - 2: _("""\ -When the 'Tor Network Settings' dialogue pops up, click 'Configure' and follow -the wizard until it asks:"""), - # TRANSLATORS: Please DO NOT translate "Tor". - 3: _("""\ -Does your Internet Service Provider (ISP) block or otherwise censor connections -to the Tor network?"""), - # TRANSLATORS: Please DO NOT translate "Tor". - 4: _("""\ -Select 'Yes' and then click 'Next'. To configure your new bridges, copy and -paste the bridge lines into the text input box. Finally, click 'Connect', and -you should be good to go! If you experience trouble, try clicking the 'Help' -button in the 'Tor Network Settings' wizard for further assistance."""), + First, you need to %sdownload Tor Browser%s. Our Tor Browser User + Manual explains how you can add your bridges to Tor Browser. If you are + using Windows, Linux, or OS X, %sclick here%s to learn more. If you + are using Android, %sclick here%s."""), } EMAIL_COMMANDS = { @@ -307,7 +294,7 @@ EMAIL_SPRINTF = { # Goes into the "%s without Pluggable Transport %s" part of ``WELCOME[2]`` "WELCOME2": ("-", "-"), # For the "%s Tor Browser download page %s" part of ``HOWTO_TBB[1]`` - "HOWTO_TBB1": ("", "[0]"), + "HOWTO_TBB1": ("", "[0]", "", "[1]", "", "[2]"), # For the "you should email %s" in ``HELP[0]`` "HELP0": ("frontdesk(a)torproject.org"), } @@ -344,7 +331,9 @@ BridgeDB can provide bridges with several types of Pluggable Transports[0], EMAIL_REFERENCE_LINKS = { "WELCOME0": "[0]: https://www.torproject.org/docs/pluggable-transports.html", - "HOWTO_TBB1": "[0]: https://www.torproject.org/projects/torbrowser.html", + "HOWTO_TBB1": "[0]: https://www.torproject.org/download/", + "HOWTO_TBB2": "[1]: https://tb-manual.torproject.org/bridges/#entering-bridge-addresses", + "HOWTO_TBB3": "[2]: https://tb-manual.torproject.org/mobile-tor/#circumvention", } BRIDGEDB_OPENPGP_KEY = """\ diff --git a/bridgedb/test/test_https_server.py b/bridgedb/test/test_https_server.py index ff7a1e3..54bda2e 100644 --- a/bridgedb/test/test_https_server.py +++ b/bridgedb/test/test_https_server.py @@ -243,7 +243,7 @@ class HowtoResourceTests(unittest.TestCase): request = DummyRequest([self.pagename]) request.method = b'GET' page = self.howtoResource.render_GET(request) - self.assertSubstring(b"the wizard", page) + self.assertSubstring(b"How to start using your bridges", page) def test_HowtoResource_render_GET_lang_ru(self): """renderGet() with ?lang=ru should return the howto page in Russian.""" @@ -255,7 +255,7 @@ class HowtoResourceTests(unittest.TestCase): request.method = b'GET' request.addArg(b'lang', 'ru') page = self.howtoResource.render_GET(request) - self.assertSubstring("следовать инструкциям установщика".encode("utf-8"), page) + self.assertSubstring("Как использовать мосты".encode("utf-8"), page) class CaptchaProtectedResourceTests(unittest.TestCase): @@ -765,7 +765,7 @@ class BridgesResourceTests(unittest.TestCase): page = self.bridgesResource.render(request) # The response should explain how to use the bridge lines: - self.assertTrue("To enter bridges into Tor Browser" in str(page)) + self.assertTrue("How to start using your bridges" in str(page)) for b in self.parseBridgesFromHTMLPage(page): # Check that each bridge line had the expected number of fields: @@ -798,7 +798,7 @@ class BridgesResourceTests(unittest.TestCase): self.bridgesResource.useForwardedHeader = False # Reset it # The response should explain how to use the bridge lines: - self.assertTrue("To enter bridges into Tor Browser" in str(page)) + self.assertTrue("How to start using your bridges" in str(page)) def test_render_GET_RTLlang(self): """Test rendering a request for plain bridges in Arabic."""

1 0

[bridgedb/master] Merge branch 'defect/30317' into develop
by phw＠torproject.org 01 Apr '20

01 Apr '20

commit 484422015af0fff03b515b7c08b64c78567e5d31 Merge: 5afd164 9c11754 Author: Philipp Winter <phw(a)nymity.ch> Date: Thu Mar 26 09:33:02 2020 -0700 Merge branch 'defect/30317' into develop CHANGELOG | 5 +++ bridgedb/distributors/email/templates.py | 9 +---- bridgedb/distributors/https/templates/bridges.html | 20 ++++----- bridgedb/distributors/https/templates/howto.html | 20 ++++----- bridgedb/i18n/templates/bridgedb.pot | 47 +++++----------------- bridgedb/strings.py | 27 ++++--------- bridgedb/test/test_https_server.py | 8 ++-- 7 files changed, 44 insertions(+), 92 deletions(-)

1 0

[bridgedb/master] Remove unused class FixedBridgeSplitter.
by phw＠torproject.org 01 Apr '20

01 Apr '20

commit d866aa9ecc4b0cdf1c6a1665a359d0e11586fea9 Author: Philipp Winter <phw(a)nymity.ch> Date: Thu Mar 26 16:49:19 2020 -0700 Remove unused class FixedBridgeSplitter. --- bridgedb/Bridges.py | 40 ---------------------------------------- bridgedb/test/test_Bridges.py | 42 ------------------------------------------ 2 files changed, 82 deletions(-) diff --git a/bridgedb/Bridges.py b/bridgedb/Bridges.py index 84a6064..2126a7d 100644 --- a/bridgedb/Bridges.py +++ b/bridgedb/Bridges.py @@ -372,46 +372,6 @@ class BridgeRing(object): f.write("%s %s\n" % (b.fingerprint, " ".join(desc).strip())) -class FixedBridgeSplitter(object): - """Splits bridges up based on an HMAC and assigns them to one of several - subhashrings with equal probability. - """ - def __init__(self, key, rings): - self.hmac = getHMACFunc(key, hex=True) - self.rings = rings[:] - - def insert(self, bridge): - # Grab the first 4 bytes - digest = self.hmac(bridge.identity) - pos = int( digest[:8], 16 ) - which = pos % len(self.rings) - self.rings[which].insert(bridge) - - def clear(self): - """Clear all bridges from every ring in ``rings``.""" - for r in self.rings: - r.clear() - - def __len__(self): - """Returns the total number of bridges in all ``rings``.""" - total = 0 - for ring in self.rings: - total += len(ring) - return total - - def dumpAssignments(self, filename, description=""): - """Write all bridges assigned to this hashring to ``filename``. - - :param string description: If given, include a description next to the - index number of the ring from :attr:`FilteredBridgeSplitter.rings` - the following bridges were assigned to. For example, if the - description is ``"IPv6 obfs2 bridges"`` the line would read: - ``"IPv6 obfs2 bridges ring=3"``. - """ - for index, ring in zip(range(len(self.rings)), self.rings): - ring.dumpAssignments(filename, "%s ring=%s" % (description, index)) - - class UnallocatedHolder(object): """A pseudo-bridgeholder that ignores its bridges and leaves them unassigned. diff --git a/bridgedb/test/test_Bridges.py b/bridgedb/test/test_Bridges.py index 2deeec7..96e6a72 100644 --- a/bridgedb/test/test_Bridges.py +++ b/bridgedb/test/test_Bridges.py @@ -120,48 +120,6 @@ class BridgeRingTests(unittest.TestCase): self.assertIn(first, data) -class FixedBridgeSplitterTests(unittest.TestCase): - """Unittests for :class:`bridgedb.Bridges.FixedBridgeSplitter`.""" - - def setUp(self): - self.rings = [Bridges.BridgeRing('fake-hmac-key-1'), - Bridges.BridgeRing('fake-hmac-key-2')] - self.splitter = Bridges.FixedBridgeSplitter('fake-hmac-key', self.rings) - - def addRandomBridges(self): - bridges = copy.deepcopy(util.generateFakeBridges()) - - [self.splitter.insert(bridge) for bridge in bridges] - - def test_insert(self): - self.addRandomBridges() - self.assertGreater(len(self.splitter), 0) - - def test_clear(self): - """Clear should get rid of all the inserted bridges.""" - self.addRandomBridges() - self.assertGreater(len(self.splitter), 0) - self.splitter.clear() - self.assertEqual(len(self.splitter), 0) - - def test_dumpAssignments(self): - """This should dump the bridges to the file.""" - self.addRandomBridges() - - f = io.StringIO() - - self.splitter.dumpAssignments(f) - - f.flush() - f.seek(0) - - data = f.read() - first = list(self.splitter.rings[0].bridges.values())[0].fingerprint - - # The first bridge's fingerprint should be within the data somewhere - self.assertIn(first, data) - - class BridgeSplitterTests(unittest.TestCase): """Unittests for :class:`bridgedb.Bridges.BridgeSplitter`."""

1 0

[bridgedb/master] Allow change of distribution mechanisms.
by phw＠torproject.org 01 Apr '20

01 Apr '20

commit 35ca1365111923dd661793175054685548b6e5a4 Author: Philipp Winter <phw(a)nymity.ch> Date: Tue Mar 17 13:53:12 2020 -0700 Allow change of distribution mechanisms. This was more work than expected because in addition to inserting a bridge into a new hashring, we had to implement new methods that allow us to remove bridges from their previous hashrings. This fixes <https://bugs.torproject.org/33631>. --- CHANGELOG | 7 ++ bridgedb/Bridges.py | 122 +++++++++++++++++++++++++---------- bridgedb/Storage.py | 29 ++++----- bridgedb/test/test_Bridges.py | 144 ++++++++++++++++++++++++++++++++++++++++++ bridgedb/test/test_Storage.py | 2 +- 5 files changed, 251 insertions(+), 53 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index f57c168..4a05fc6 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -3,6 +3,13 @@ Browser. In addition to updating the instructions, this patch also links to instructions for Android. + * FIXES https://bugs.torproject.org/33631 + So far, BridgeDB remembered only the first distribution mechanism it + ever learned for a given bridge. That means that if a bridge would + change its mind and re-configure its distribution mechanism using + BridgeDistribution, BridgeDB would ignore it. This patch changes this + behavior, so bridges can actually change their distribution mechanism. + * FIXES https://bugs.torproject.org/31967 Use a CSPRNG for selecting cached CAPTCHAs. diff --git a/bridgedb/Bridges.py b/bridgedb/Bridges.py index c29a2bf..bbc9b5f 100644 --- a/bridgedb/Bridges.py +++ b/bridgedb/Bridges.py @@ -180,6 +180,14 @@ class BridgeRing(object): for tp, val, count, subring in self.subrings: subring.clear() + def remove(self, bridge): + """Remove a **bridge** from this hashring.""" + for tp, val, _, subring in self.subrings: + subring.remove(bridge) + pos = self.hmac(bridge.identity) + if pos in self.bridges: + del self.bridges[pos] + def insert(self, bridge): """Add a **bridge** to this hashring. @@ -411,8 +419,17 @@ class UnallocatedHolder(object): def __init__(self): self.fingerprints = [] + def remove(self, bridge): + logging.debug("Removing %s from unallocated" % bridge.fingerprint) + i = -1 + try: + i = self.fingerprints.index(bridge.fingerprint) + except ValueError: + return + del self.fingerprints[i] + def insert(self, bridge): - logging.debug("Leaving %s unallocated", bridge.fingerprint) + logging.debug("Leaving %s unallocated" % bridge.fingerprint) if not bridge.fingerprint in self.fingerprints: self.fingerprints.append(bridge.fingerprint) @@ -488,43 +505,52 @@ class BridgeSplitter(object): return validRings = self.rings - distribution_method = None + distribution_method = orig_method = None - # If the bridge already has a distributor, use that. with bridgedb.Storage.getDB() as db: - distribution_method = db.getBridgeDistributor(bridge, validRings) + orig_method = db.getBridgeDistributor(bridge, validRings) + if orig_method is not None: + distribution_method = orig_method + logging.info("So far, bridge %s was in hashring %s" % + (bridge, orig_method)) + + # Check if the bridge requested a distribution method and if so, try to + # use it. + if bridge.distribution_request: + distribution_method = bridge.distribution_request + logging.info("Bridge %s requested placement in hashring %s" % + (bridge, distribution_method)) + + # Is the bridge requesting a distribution method that's different + # from the one we have on record? If so, we have to delete it from + # its previous ring. + if orig_method is not None and \ + orig_method != distribution_method and \ + distribution_method in (validRings + ["none"]): + logging.info("Bridge %s is in %s but wants to be in %s." % + (bridge, orig_method, distribution_method)) + prevRing = self.ringsByName.get(orig_method) + prevRing.remove(bridge) + + # If they requested not to be distributed, honor the request: + if distribution_method == "none": + logging.info("Bridge %s requested to not be distributed." % bridge) + return - if distribution_method: - logging.info("%s bridge %s was already in hashring %s" % - (self.__class__.__name__, bridge, distribution_method)) - else: - # Check if the bridge requested a specific distribution method. - if bridge.distribution_request: - distribution_method = bridge.distribution_request - logging.info("%s bridge %s requested placement in hashring %s" - % (self.__class__.__name__, bridge, - distribution_method)) - - # If they requested not to be distributed, honor the request: - if distribution_method == "none": - logging.info("%s bridge %s requested to not be distributed." - % (self.__class__.__name__, bridge)) - return - - # If we didn't know what they are talking about, or they requested - # "any" distribution method, and we've never seen this bridge - # before, then determine where to place it. - if ((distribution_method not in validRings) or - (distribution_method == "any")): - - pos = self.hmac(bridge.identity) - n = int(pos[:8], 16) % self.totalP - pos = bisect.bisect_right(self.pValues, n) - 1 - assert 0 <= pos < len(self.rings) - distribution_method = self.rings[pos] - logging.info(("%s placing bridge %s into hashring %s (via n=%s," - " pos=%s).") % (self.__class__.__name__, bridge, - distribution_method, n, pos)) + # If we didn't know what they are talking about, or they requested + # "any" distribution method, and we've never seen this bridge + # before, then deterministically determine where to place it. + if ((distribution_method not in validRings) or + (distribution_method == "any")): + + pos = self.hmac(bridge.identity) + n = int(pos[:8], 16) % self.totalP + pos = bisect.bisect_right(self.pValues, n) - 1 + assert 0 <= pos < len(self.rings) + distribution_method = self.rings[pos] + logging.info(("%s placing bridge %s into hashring %s (via n=%s," + " pos=%s).") % (self.__class__.__name__, bridge, + distribution_method, n, pos)) with bridgedb.Storage.getDB() as db: ringname = db.insertBridgeAndGetRing(bridge, distribution_method, @@ -588,6 +614,32 @@ class FilteredBridgeSplitter(object): self.bridges = [] self.filterRings = {} + def remove(self, bridge): + """Remove a bridge from all appropriate sub-hashrings. + + :type bridge: :class:`~bridgedb.bridges.Bridge` + :param bridge: The bridge to add. + """ + logging.debug("Removing %s from hashring..." % bridge) + + all_fingerprints = [b.fingerprint for b in self.bridges] + index = -1 + try: + index = all_fingerprints.index(bridge.fingerprint) + except ValueError: + # The given bridge doesn't exist, so there's nothing to remove. + logging.warn("Was asked to remove non-existant bridge %s " + "from ring." % bridge) + return + assert index > -1 + del self.bridges[index] + + for ringname, (filterFn, subring) in self.filterRings.items(): + if filterFn(bridge): + subring.remove(bridge) + logging.debug("Removed bridge %s from %s subhashring." % + (bridge, ringname)) + def insert(self, bridge): """Insert a bridge into all appropriate sub-hashrings. diff --git a/bridgedb/Storage.py b/bridgedb/Storage.py index cfd60bb..d725fa3 100644 --- a/bridgedb/Storage.py +++ b/bridgedb/Storage.py @@ -166,9 +166,8 @@ class Database(object): def insertBridgeAndGetRing(self, bridge, setRing, seenAt, validRings, defaultPool="unallocated"): - '''Updates info about bridge, setting ring to setRing if none was set. - Also sets distributor to `defaultPool' if the bridge was found in - the database, but its distributor isn't valid anymore. + '''Updates info about bridge, setting ring to setRing. Also sets + distributor to `defaultPool' if setRing isn't a valid ring. Returns the name of the distributor the bridge is assigned to. ''' @@ -178,26 +177,22 @@ class Database(object): h = bridge.fingerprint assert len(h) == HEX_ID_LEN - cur.execute("SELECT id, distributor " - "FROM Bridges WHERE hex_key = ?", (h,)) + # Check if this is currently a valid ring name. If not, move into + # default pool. + if setRing not in validRings: + setRing = defaultPool + + cur.execute("SELECT id FROM Bridges WHERE hex_key = ?", (h,)) v = cur.fetchone() if v is not None: - i, ring = v - # Check if this is currently a valid ring name. If not, move back - # into default pool. - if ring not in validRings: - ring = defaultPool + bridgeId = v[0] # Update last_seen, address, port and (possibly) distributor. cur.execute("UPDATE Bridges SET address = ?, or_port = ?, " "distributor = ?, last_seen = ? WHERE id = ?", - (str(bridge.address), bridge.orPort, ring, - timeToStr(seenAt), i)) - return ring + (str(bridge.address), bridge.orPort, setRing, + timeToStr(seenAt), bridgeId)) + return setRing else: - # Check if this is currently a valid ring name. If not, move back - # into default pool. - if setRing not in validRings: - setRing = defaultPool # Insert it. cur.execute("INSERT INTO Bridges (hex_key, address, or_port, " "distributor, first_seen, last_seen) " diff --git a/bridgedb/test/test_Bridges.py b/bridgedb/test/test_Bridges.py index 77eff1c..2deeec7 100644 --- a/bridgedb/test/test_Bridges.py +++ b/bridgedb/test/test_Bridges.py @@ -17,11 +17,18 @@ import copy import io import ipaddr import logging +import tempfile +import os from twisted.trial import unittest +import bridgedb.Storage + from bridgedb import Bridges +from bridgedb import crypto from bridgedb.test import util +from bridgedb.distributors.https.distributor import HTTPSDistributor +from bridgedb.distributors.moat.distributor import MoatDistributor # For additional logger output for debugging, comment out the following: logging.disable(50) @@ -153,3 +160,140 @@ class FixedBridgeSplitterTests(unittest.TestCase): # The first bridge's fingerprint should be within the data somewhere self.assertIn(first, data) + + +class BridgeSplitterTests(unittest.TestCase): + """Unittests for :class:`bridgedb.Bridges.BridgeSplitter`.""" + + def setUp(self): + + self.bridges = copy.deepcopy(util.generateFakeBridges()) + + self.fd, self.fname = tempfile.mkstemp(suffix=".sqlite", dir=os.getcwd()) + bridgedb.Storage.initializeDBLock() + self.db = bridgedb.Storage.openDatabase(self.fname) + bridgedb.Storage.setDBFilename(self.fname) + + key = 'fake-hmac-key' + self.splitter = Bridges.BridgeSplitter(key) + ringParams = Bridges.BridgeRingParameters(needPorts=[(443, 1)], + needFlags=[("Stable", 1)]) + self.https_distributor = HTTPSDistributor( + 4, + crypto.getHMAC(key, "HTTPS-IP-Dist-Key"), + None, + answerParameters=ringParams) + self.moat_distributor = MoatDistributor( + 4, + crypto.getHMAC(key, "Moat-Dist-Key"), + None, + answerParameters=ringParams) + self.unallocated_distributor = Bridges.UnallocatedHolder() + + self.splitter.addRing(self.https_distributor.hashring, "https", p=10) + self.splitter.addRing(self.moat_distributor.hashring, "moat", p=10) + self.splitter.addRing(self.unallocated_distributor, "unallocated", p=10) + self.https_ring = self.splitter.ringsByName.get("https") + self.moat_ring = self.splitter.ringsByName.get("moat") + self.unallocated_ring = self.splitter.ringsByName.get("unallocated") + + def tearDown(self): + self.db.close() + os.close(self.fd) + os.unlink(self.fname) + + def _len_all_subrings(self, ring): + """Return the sum of the length of all subrings.""" + all_subrings = [subring for _, subring in ring.filterRings.values()] + return sum([len(subring) for subring in all_subrings]) + + def test_no_distribution(self): + """Make sure that bridges can un-distribute themselves.""" + bridge = self.bridges[0] + bridge.distribution_request = "https" + + # Assume a bridge wants to be distributed over HTTPS. + self.splitter.insert(bridge) + self.assertEqual(len(self.https_ring), 1) + + # ...and now the bridge no longer wants to be distributed. + bridge.distribution_request = "none" + self.splitter.insert(bridge) + self.assertEqual(len(self.https_ring), 0) + self.assertEqual(len(self.moat_ring), 0) + self.assertEqual(len(self.unallocated_ring), 0) + + def test_change_distribution(self): + """Make sure that bridges can change their distribution mechanism.""" + bridge = self.bridges[0] + # We hard-code our identity to make this test deterministic. + bridge.identity = b"\xfd{\xe7\x90a'\n\xf483@H\xd6-\x9c\xf3\x8f\x12~$" + bridge.distribution_request = "https" + + # Assume a bridge wants to be distributed over HTTPS. + self.splitter.insert(bridge) + self.assertEqual(len(self.https_ring), 1) + self.assertEqual(len(self.moat_ring), 0) + self.assertEqual(self._len_all_subrings(self.moat_ring), 0) + + # ...and now the bridge changes its mind and wants Moat. + bridge.distribution_request = "moat" + self.splitter.insert(bridge) + self.assertEqual(len(self.https_ring), 0) + self.assertEqual(self._len_all_subrings(self.https_ring), 0) + self.assertEqual(len(self.moat_ring), 1) + + # ...if the bridge uses "any", it should stay where it is. + bridge.distribution_request = "any" + self.splitter.insert(bridge) + self.assertEqual(len(self.https_ring), 0) + self.assertEqual(self._len_all_subrings(self.https_ring), 0) + self.assertEqual(len(self.moat_ring), 1) + + # ...if it uses "none", it shouldn't be distributed at all. + bridge.distribution_request = "none" + self.splitter.insert(bridge) + self.assertEqual(len(self.https_ring), 0) + self.assertEqual(self._len_all_subrings(self.https_ring), 0) + self.assertEqual(len(self.moat_ring), 0) + self.assertEqual(self._len_all_subrings(self.moat_ring), 0) + + # ...if the distribution method is unrecognised, it should be treated + # as "any". + bridge.distribution_request = "foobar" + self.splitter.insert(bridge) + self.assertEqual(len(self.https_ring), 0) + self.assertEqual(self._len_all_subrings(self.https_ring), 0) + self.assertEqual(len(self.moat_ring), 1) + + # ...and finally, it wants HTTPS again. + bridge.distribution_request = "https" + self.splitter.insert(bridge) + self.assertEqual(len(self.https_ring), 1) + self.assertEqual(len(self.moat_ring), 0) + self.assertEqual(self._len_all_subrings(self.moat_ring), 0) + + def test_https_remove(self): + """Make sure that we can remove bridges from our BridgeRing.""" + bridge = self.bridges[0] + + self.assertEqual(len(self.https_ring), 0) + self.https_ring.insert(bridge) + self.https_distributor.prepopulateRings() + self.assertEqual(len(self.https_ring), 1) + + self.https_ring.remove(bridge) + self.assertEqual(len(self.https_ring), 0) + self.assertEqual(self._len_all_subrings(self.https_ring), 0) + + def test_unallocated_remove(self): + """Make sure that we can remove bridges from our UnallocatedHolder.""" + bridge = self.bridges[0] + bridge.distribution_request = "unallocated" + + self.assertEqual(len(self.unallocated_distributor), 0) + self.splitter.insert(bridge) + self.assertEqual(len(self.unallocated_distributor), 1) + + self.unallocated_distributor.remove(bridge) + self.assertEqual(len(self.unallocated_distributor), 0) diff --git a/bridgedb/test/test_Storage.py b/bridgedb/test/test_Storage.py index ca10bb9..720afdd 100644 --- a/bridgedb/test/test_Storage.py +++ b/bridgedb/test/test_Storage.py @@ -82,7 +82,7 @@ class DatabaseTest(unittest.TestCase): time.time(), self.validRings) self.assertIn(ringname, self.validRings) - self.assertEqual(ringname, 'moat') + self.assertEqual(ringname, 'https') def test_getBridgeDistributor_recognised(self): bridge = self.fakeBridges[0]

1 0

[bridgedb/master] Merge branch 'defect/33631' into develop
by phw＠torproject.org 01 Apr '20

01 Apr '20

commit f015596ed72131293207b29088ba0391477a6c00 Merge: 4844220 43e931f Author: Philipp Winter <phw(a)nymity.ch> Date: Wed Apr 1 14:34:07 2020 -0700 Merge branch 'defect/33631' into develop .travis.requirements.txt | 2 +- CHANGELOG | 7 ++ bridgedb/Bridges.py | 178 +++++++++++++++++++++--------------------- bridgedb/Storage.py | 29 +++---- bridgedb/test/test_Bridges.py | 176 ++++++++++++++++++++++++++++++++--------- bridgedb/test/test_Storage.py | 2 +- requirements.txt | 2 +- 7 files changed, 251 insertions(+), 145 deletions(-)

1 0

[bridgedb/master] Turn self.bridges from a list into dictionary.
by phw＠torproject.org 01 Apr '20

01 Apr '20

commit d0db599891c281ec2fa37591c0ec67ff07bf2a1c Author: Philipp Winter <phw(a)nymity.ch> Date: Thu Mar 26 10:50:00 2020 -0700 Turn self.bridges from a list into dictionary. This simplifies the code significantly. --- bridgedb/Bridges.py | 32 +++++++++++--------------------- 1 file changed, 11 insertions(+), 21 deletions(-) diff --git a/bridgedb/Bridges.py b/bridgedb/Bridges.py index bbc9b5f..84a6064 100644 --- a/bridgedb/Bridges.py +++ b/bridgedb/Bridges.py @@ -593,7 +593,8 @@ class FilteredBridgeSplitter(object): I-guess-it-passes-for-some-sort-of-hashring classes in this module. :ivar hmac: DOCDOC - :ivar bridges: DOCDOC + :ivar bridges: A dictionary mapping a bridge's fingerprint to its + :class:`~bridgedb.bridges.Bridge` object. :type distributorName: str :ivar distributorName: The name of this splitter's distributor. See :meth:`~bridgedb.distributors.https.distributor.HTTPSDistributor.setDistributorName`. @@ -601,7 +602,7 @@ class FilteredBridgeSplitter(object): self.key = key self.filterRings = {} self.hmac = getHMACFunc(key, hex=True) - self.bridges = [] + self.bridges = {} self.distributorName = '' #XXX: unused @@ -611,28 +612,23 @@ class FilteredBridgeSplitter(object): return len(self.bridges) def clear(self): - self.bridges = [] + self.bridges = {} self.filterRings = {} def remove(self, bridge): """Remove a bridge from all appropriate sub-hashrings. :type bridge: :class:`~bridgedb.bridges.Bridge` - :param bridge: The bridge to add. + :param bridge: The bridge to remove. """ logging.debug("Removing %s from hashring..." % bridge) - all_fingerprints = [b.fingerprint for b in self.bridges] - index = -1 try: - index = all_fingerprints.index(bridge.fingerprint) - except ValueError: - # The given bridge doesn't exist, so there's nothing to remove. + del self.bridges[bridge.fingerprint] + except KeyError: logging.warn("Was asked to remove non-existant bridge %s " "from ring." % bridge) return - assert index > -1 - del self.bridges[index] for ringname, (filterFn, subring) in self.filterRings.items(): if filterFn(bridge): @@ -655,15 +651,9 @@ class FilteredBridgeSplitter(object): "bridge: %s") % bridge) return - index = 0 logging.debug("Inserting %s into hashring..." % bridge) - for old_bridge in self.bridges[:]: - if bridge.fingerprint == old_bridge.fingerprint: - self.bridges[index] = bridge - break - index += 1 - else: - self.bridges.append(bridge) + self.bridges[bridge.fingerprint] = bridge + for ringname, (filterFn, subring) in self.filterRings.items(): if filterFn(bridge): subring.insert(bridge) @@ -747,7 +737,7 @@ class FilteredBridgeSplitter(object): if populate_from: inserted = 0 - for bridge in populate_from: + for bridge in populate_from.values(): if isinstance(bridge, Bridge) and filterFn(bridge): subring.insert(bridge) inserted += 1 @@ -761,7 +751,7 @@ class FilteredBridgeSplitter(object): # bridges may be present in multiple filter sets # only one line should be dumped per bridge - for b in self.bridges: + for b in self.bridges.values(): # gather all the filter descriptions desc = [] for n,(g,r) in self.filterRings.items():

1 0

[bridgedb/master] Upgrade Twisted to fix CVE-2020-10108.
by phw＠torproject.org 01 Apr '20

01 Apr '20

commit 43e931fa49c548d1f29c8c235a21fbd1f571ae19 Author: Philipp Winter <phw(a)nymity.ch> Date: Tue Mar 31 13:28:45 2020 -0700 Upgrade Twisted to fix CVE-2020-10108. --- .travis.requirements.txt | 2 +- requirements.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.travis.requirements.txt b/.travis.requirements.txt index 1a1d7ff..8308ce2 100644 --- a/.travis.requirements.txt +++ b/.travis.requirements.txt @@ -18,7 +18,7 @@ Babel==2.8.0 beautifulsoup4==4.8.2 Mako==1.1.1 pycryptodome==3.9.6 -Twisted==19.10.0 +Twisted==20.3.0 coverage==5.0.3 coveralls==1.10.0 gnupg==2.3.1 diff --git a/requirements.txt b/requirements.txt index eed4889..ef72ed5 100644 --- a/requirements.txt +++ b/requirements.txt @@ -3,7 +3,7 @@ beautifulsoup4==4.8.2 html5lib==1.0b8 Mako==1.1.1 pycryptodome==3.9.6 -Twisted==19.10.0 +Twisted==20.3.0 gnupg==2.3.1 ipaddr==2.2.0 Pillow==6.2.2

1 0

[bridgedb/master] Bump version number to 0.10.0.
by phw＠torproject.org 01 Apr '20

01 Apr '20

commit f9a9a9cdb6fec89daf7592f6c44db814875e0bca Author: Philipp Winter <phw(a)nymity.ch> Date: Wed Apr 1 14:48:22 2020 -0700 Bump version number to 0.10.0. --- CHANGELOG | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index 4a05fc6..a92824c 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,3 +1,5 @@ +Changes in version 0.10.0 - 2020-04-01 + * FIXES https://bugs.torproject.org/30317 Update our "howto" box, which explains how one adds bridges to Tor Browser. In addition to updating the instructions, this patch also

1 0