January 2018 - tor-commits - lists.torproject.org

[bridgedb/develop] Create CHANGELOG entry for 0.6.3.
by isis＠torproject.org 23 Jan '18

23 Jan '18

commit bc61afb1ec44f402a86c670d99e749164b5f3f3e Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Jan 23 21:40:33 2018 +0000 Create CHANGELOG entry for 0.6.3. --- CHANGELOG | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index 82cf769..9ffeac1 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,3 +1,27 @@ +Changes in version 0.6.3 - 2018-01-23 + + * FIXES #24432 https://bugs.torproject.org/24432 + The production moat server had issues related to redirecting to + resources properly, which are now fixed. + + * FIXES #24701 https://bugs.torproject.org/24701 + Adds a special surprise for the special someone who has been + automatedly requesting bridges not through driving a browser, but + through a script which is so thoroughly stupid that it doesn't + even send the URL parameters for the CAPTCHA challenge and solution. + Their script will now be delayed for quite some time and then + rickrolled. Mess with the best, die like the rest. + + * FIXES #24704 https://bugs.torproject.org/24704 + Bridges returned to a single request are now filtered such that + there will never be two bridges from the same IPv4 /16 or IPv6 /64. + +And includes the following general changes: + + * ADDS unittests for the legacy code in bridgedb/Bridges.py, + bringing the total test coverage above 90% for the first time. + + Changes in version 0.6.2 - 2017-12-20 * FIXES #24636 https://bugs.torproject.org/24636

1 0

[bridgedb/develop] Merge branch 'release/0.6.3'
by isis＠torproject.org 23 Jan '18

23 Jan '18

commit d49e90455c1fd6d66d057b0ba6e6ee4f23129b9f Merge: 2ae288d bc61afb Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Jan 23 21:41:55 2018 +0000 Merge branch 'release/0.6.3' CHANGELOG | 44 ++++++++ bridgedb/Bridges.py | 52 ++++++++-- bridgedb/distributors/email/distributor.py | 2 +- bridgedb/distributors/https/distributor.py | 2 +- bridgedb/distributors/https/server.py | 43 ++++++-- bridgedb/distributors/moat/server.py | 1 + bridgedb/test/test_Bridges.py | 155 +++++++++++++++++++++++++++++ bridgedb/test/test_https_server.py | 11 +- scripts/externalize-pt-client | 12 +++ scripts/test-moat | 49 +++++++-- 10 files changed, 337 insertions(+), 34 deletions(-)

1 0

[bridgedb/master] Merge branch 'fix/24704' into develop
by isis＠torproject.org 23 Jan '18

23 Jan '18

commit 94327eb3eca56785f865483d1ccdf4aa6e03752e Merge: c6c9f47 8aedf0b Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Jan 23 02:02:09 2018 +0000 Merge branch 'fix/24704' into develop bridgedb/test/test_Bridges.py | 95 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 93 insertions(+), 2 deletions(-)

1 0

[bridgedb/master] Merge branch 'fix/24432' into develop
by isis＠torproject.org 23 Jan '18

23 Jan '18

commit 3a5dce31dc43f1ea7292e67372b57385ab5a4788 Merge: 94327eb 19c111c Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Jan 23 21:05:21 2018 +0000 Merge branch 'fix/24432' into develop bridgedb/distributors/moat/server.py | 1 + scripts/externalize-pt-client | 12 +++++++++ scripts/test-moat | 49 ++++++++++++++++++++++++++++-------- 3 files changed, 52 insertions(+), 10 deletions(-)

1 0

[bridgedb/master] Add David's script for running an external meek client.
by isis＠torproject.org 23 Jan '18

23 Jan '18

commit 2783933bf9935b6a7ab4ce97386887865a45589a Author: David Fifield <david(a)bamsoftware.com> Date: Tue Nov 28 02:05:58 2017 +0000 Add David's script for running an external meek client. --- scripts/externalize-pt-client | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/scripts/externalize-pt-client b/scripts/externalize-pt-client new file mode 100755 index 0000000..bbe01c6 --- /dev/null +++ b/scripts/externalize-pt-client @@ -0,0 +1,12 @@ +#!/bin/bash +set -e +CMDLINE="${@:?need a meek-client command line}" +TRANSPORTS=meek +# This could be controlled by a command-line arg. +PORT=10000 +TOR_PT_MANAGED_TRANSPORT_VER=1 \ + TOR_PT_CLIENT_TRANSPORTS="$TRANSPORTS" $CMDLINE \ + | sed -n -u -e '/^CMETHOD /{s/^.*127\.0\.0\.1://;p}' | while read pt_port; do + echo "forwarding port $PORT -> $pt_port" + socat -v -v TCP-LISTEN:$PORT,fork,reuseaddr TCP-CONNECT:127.0.0.1:$pt_port & +done

1 0

[bridgedb/master] Create CHANGELOG entry for 0.6.3.
by isis＠torproject.org 23 Jan '18

23 Jan '18

commit bc61afb1ec44f402a86c670d99e749164b5f3f3e Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Jan 23 21:40:33 2018 +0000 Create CHANGELOG entry for 0.6.3. --- CHANGELOG | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index 82cf769..9ffeac1 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,3 +1,27 @@ +Changes in version 0.6.3 - 2018-01-23 + + * FIXES #24432 https://bugs.torproject.org/24432 + The production moat server had issues related to redirecting to + resources properly, which are now fixed. + + * FIXES #24701 https://bugs.torproject.org/24701 + Adds a special surprise for the special someone who has been + automatedly requesting bridges not through driving a browser, but + through a script which is so thoroughly stupid that it doesn't + even send the URL parameters for the CAPTCHA challenge and solution. + Their script will now be delayed for quite some time and then + rickrolled. Mess with the best, die like the rest. + + * FIXES #24704 https://bugs.torproject.org/24704 + Bridges returned to a single request are now filtered such that + there will never be two bridges from the same IPv4 /16 or IPv6 /64. + +And includes the following general changes: + + * ADDS unittests for the legacy code in bridgedb/Bridges.py, + bringing the total test coverage above 90% for the first time. + + Changes in version 0.6.2 - 2017-12-20 * FIXES #24636 https://bugs.torproject.org/24636

1 0

[bridgedb/master] Merge branch 'fix/24704' into develop
by isis＠torproject.org 23 Jan '18

23 Jan '18

commit 637e1fc2a4e09ec8ad5a2442db2298d3666897eb Merge: 1096bf4 ad26965 Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Dec 22 02:41:46 2017 +0000 Merge branch 'fix/24704' into develop bridgedb/Bridges.py | 52 ++++++++++++++++++++---- bridgedb/distributors/email/distributor.py | 2 +- bridgedb/distributors/https/distributor.py | 2 +- bridgedb/test/test_Bridges.py | 64 ++++++++++++++++++++++++++++++ 4 files changed, 111 insertions(+), 9 deletions(-)

1 0

[bridgedb/master] Add CHANGELOG entries for 0.6.2.
by isis＠torproject.org 23 Jan '18

23 Jan '18

commit 1096bf4d9ed1cbded7ce8ad2a183338f969b1166 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Dec 20 23:22:31 2017 +0000 Add CHANGELOG entries for 0.6.2. --- CHANGELOG | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index effc405..82cf769 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,3 +1,23 @@ +Changes in version 0.6.2 - 2017-12-20 + + * FIXES #24636 https://bugs.torproject.org/24636 + The moat API specification included an extra response type which + could be sent if there was no overlap between transports the + client supported and those which the server supported. This has + been removed from the specification, which now describes the + behaviour moat has always exhibited: if there is no overlap, the + server responds with a CAPTCHA image response which includes the + list of transports it does support. + + * FIXES #24637 https://bugs.torproject.org/24637 + The moat server did not respond correctly with the specified JSON + API error type when there were no bridges available. It now + responds correctly with a 404 error whose details describe why the + request could not be fulfilled. The moat server also now logs + messages if there were not the configured + MOAT_BRIDGES_PER_RESPONSE number of bridges available. + + Changes in version 0.6.1 - 2017-12-13 * ADDS a shell script, scripts/test-moat, for testing either a

1 0

[bridgedb/master] Delay and then redirect malicious requests.
by isis＠torproject.org 23 Jan '18

23 Jan '18

commit 0f03fb65518353e1409e08def7201ea17fe1318f Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Dec 22 01:45:52 2017 +0000 Delay and then redirect malicious requests. * FIXES #24701: https://bugs.torproject.org/24701 --- bridgedb/distributors/https/server.py | 43 +++++++++++++++++++++++++++-------- bridgedb/test/test_https_server.py | 11 ++++----- 2 files changed, 39 insertions(+), 15 deletions(-) diff --git a/bridgedb/distributors/https/server.py b/bridgedb/distributors/https/server.py index d4771a6..8c50bc1 100644 --- a/bridgedb/distributors/https/server.py +++ b/bridgedb/distributors/https/server.py @@ -37,7 +37,9 @@ import mako.exceptions from mako.template import Template from mako.lookup import TemplateLookup +from twisted.internet import defer from twisted.internet import reactor +from twisted.internet import task from twisted.internet.error import CannotListenError from twisted.web import resource from twisted.web import static @@ -138,6 +140,20 @@ def replaceErrorPage(request, error, template_name=None, html=True): return rendered +def redirectMaliciousRequest(request): + '''Redirect the client to a "daring work of art" which "in true + post-modern form, […] tends to raise more questions than answers." + ''' + logging.debug("Redirecting %s to a daring work of art..." % getClientIP(request)) + request.write(redirectTo(base64.b64decode("aHR0cDovLzJnaXJsczFjdXAuY2Ev"), request)) + request.finish() + return request + + +class MaliciousRequest(Exception): + """Raised when we received a possibly malicious request.""" + + class CSPResource(resource.Resource): """A resource which adds a ``'Content-Security-Policy:'`` header. @@ -411,9 +427,9 @@ class CaptchaProtectedResource(CustomErrorHandlingResource, CSPResource): challenge = request.args['captcha_challenge_field'][0] response = request.args['captcha_response_field'][0] except Exception as error: - logging.debug(("Client CAPTCHA solution to HTTPS distributor server" - "didn't include correct HTTP arguments: %s" % error)) - return redirectTo(type(b'')(request.URLPath()), request) + raise MaliciousRequest( + ("Client CAPTCHA solution to HTTPS distributor server " + "didn't include correct HTTP arguments: %s" % error)) return (challenge, response) def checkSolution(self, request): @@ -477,12 +493,21 @@ class CaptchaProtectedResource(CustomErrorHandlingResource, CSPResource): self.setCSPHeader(request) request.setHeader("Content-Type", "text/html; charset=utf-8") - if self.checkSolution(request) is True: - try: - rendered = self.resource.render(request) - except Exception as err: - rendered = replaceErrorPage(request, err) - return rendered + try: + if self.checkSolution(request) is True: + return self.resource.render(request) + except ValueError as err: + logging.debug(err.message) + except MaliciousRequest as err: + logging.debug(err.message) + # Make them wait a bit, then redirect them to a "daring + # work of art" as pennance for their sins. + d = task.deferLater(reactor, 1, lambda: request) + d.addCallback(redirectMaliciousRequest) + return NOT_DONE_YET + except Exception as err: + logging.debug(err.message) + return replaceErrorPage(request, err) logging.debug("Client failed a CAPTCHA; returning redirect to %s" % request.uri) diff --git a/bridgedb/test/test_https_server.py b/bridgedb/test/test_https_server.py index 13ec20e..ba555e8 100644 --- a/bridgedb/test/test_https_server.py +++ b/bridgedb/test/test_https_server.py @@ -373,18 +373,17 @@ class GimpCaptchaProtectedResourceTests(unittest.TestCase): self.assertEqual(response, expectedResponse) def test_extractClientSolution_missing_arguments(self): - """A solution with missing arguments (the solution field) should - return a very agressive redirect to the originally requested, - CAPTCHA-protected page. + """A solution with missing arguments (the solution/challenge fields) + should raise a MaliciousRequest exception. """ expectedChallenge = '23232323232323232323' self.request.method = b'POST' self.request.addArg('captcha_challenge_field', expectedChallenge) - response = self.captchaResource.extractClientSolution(self.request) - - self.assertIn("click here", response) + self.assertRaises(server.MaliciousRequest, + self.captchaResource.extractClientSolution, + self.request) def test_checkSolution(self): """checkSolution() should return False is the solution is invalid."""

1 0

[bridgedb/master] Filter returned bridges to avoid multiples from the same /16 or /32.
by isis＠torproject.org 23 Jan '18

23 Jan '18

commit ad269657fe16f2603f207bdb2a979266c3b918e8 Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Dec 22 02:34:32 2017 +0000 Filter returned bridges to avoid multiples from the same /16 or /32. * FIXES #24704 https://bugs.torproject.org/24704 --- bridgedb/Bridges.py | 52 ++++++++++++++++++++---- bridgedb/distributors/email/distributor.py | 2 +- bridgedb/distributors/https/distributor.py | 2 +- bridgedb/test/test_Bridges.py | 64 ++++++++++++++++++++++++++++++ 4 files changed, 111 insertions(+), 9 deletions(-) diff --git a/bridgedb/Bridges.py b/bridgedb/Bridges.py index 13bffd2..dfc6ac5 100644 --- a/bridgedb/Bridges.py +++ b/bridgedb/Bridges.py @@ -268,7 +268,39 @@ class BridgeRing(object): assert len(r) == N return r - def getBridges(self, pos, N=1): + def filterDistinctSubnets(self, fingerprints): + """Given a chosen set of ``fingerprints`` of bridges to distribute, + filter the bridges such that they are in distinct subnets. + """ + logging.debug("Got %d possible bridges to filter" % len(fingerprints)) + + bridges = [] + subnets = [] + + for fingerprint in fingerprints: + bridge = self.bridges[fingerprint] + jump = False + for subnet in subnets: + if bridge.address in subnet: + jump = True + logging.debug( + ("Skipping distribution of bridge %s in a subnet which " + "contains another bridge we're already distributing") + % bridge) + break + if jump: + continue + + bridges.append(bridge) + if bridge.address.version == 4: + cidr = str(bridge.address) + "/16" + else: + cidr = str(bridge.address) + "/32" + subnets.append(ipaddr.IPNetwork(cidr)) + + return bridges + + def getBridges(self, pos, N=1, filterBySubnet=False): """Return **N** bridges appearing in this hashring after a position. :param bytes pos: The position to jump to. Any bridges returned will @@ -285,19 +317,25 @@ class BridgeRing(object): count = len(subring) forced.extend(subring._getBridgeKeysAt(pos, count)) - keys = [ ] - for k in forced + self._getBridgeKeysAt(pos, N): + keys = [] + + # Oversample double the number we need, in case we need to + # filter them and some are within the same subnet. + for k in forced + self._getBridgeKeysAt(pos, N + N): if k not in keys: keys.append(k) else: logging.debug( "Got duplicate bridge %r in main hashring for position %r." % (logSafely(k.encode('hex')), pos.encode('hex'))) - keys = keys[:N] keys.sort() - #Do not return bridges from the same /16 - bridges = [ self.bridges[k] for k in keys ] + if filterBySubnet: + bridges = self.filterDistinctSubnets(keys) + else: + bridges = [self.bridges[k] for k in keys] + + bridges = bridges[:N] return bridges @@ -551,7 +589,7 @@ class FilteredBridgeSplitter(object): For all sub-hashrings, the ``bridge`` will only be added iff it passes the filter functions for that sub-hashring. - :type bridge: :class:`~bridgedb.Bridges.Bridge` + :type bridge: :class:`~bridgedb.bridges.Bridge` :param bridge: The bridge to add. """ # The bridge must be running to insert it: diff --git a/bridgedb/distributors/email/distributor.py b/bridgedb/distributors/email/distributor.py index b76c26c..fbf1a50 100644 --- a/bridgedb/distributors/email/distributor.py +++ b/bridgedb/distributors/email/distributor.py @@ -191,7 +191,7 @@ class EmailDistributor(Distributor): populate_from=self.hashring.bridges) returnNum = self.bridgesPerResponse(ring) - result = ring.getBridges(pos, returnNum) + result = ring.getBridges(pos, returnNum, filterBySubnet=False) db.setEmailTime(bridgeRequest.client, now) db.commit() diff --git a/bridgedb/distributors/https/distributor.py b/bridgedb/distributors/https/distributor.py index 5ff9d83..7bce8d9 100644 --- a/bridgedb/distributors/https/distributor.py +++ b/bridgedb/distributors/https/distributor.py @@ -342,6 +342,6 @@ class HTTPSDistributor(Distributor): # Determine the appropriate number of bridges to give to the client: returnNum = self.bridgesPerResponse(ring) - answer = ring.getBridges(position, returnNum) + answer = ring.getBridges(position, returnNum, filterBySubnet=True) return answer diff --git a/bridgedb/test/test_Bridges.py b/bridgedb/test/test_Bridges.py new file mode 100644 index 0000000..092b033 --- /dev/null +++ b/bridgedb/test/test_Bridges.py @@ -0,0 +1,64 @@ +# -*- coding: utf-8 -*- +#_____________________________________________________________________________ +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: Isis Lovecruft 0xA3ADB67A2CDB8B35 <isis(a)torproject.org> +# :copyright: (c) 2017, Isis Lovecruft +# (c) 2017, The Tor Project, Inc. +# :license: see LICENSE for licensing information +#_____________________________________________________________________________ + +"""Unittests for :mod:`bridgedb.Bridges`.""" + +from __future__ import print_function + +import copy +import ipaddr +import logging + +from twisted.trial import unittest + +from bridgedb import Bridges +from bridgedb.test import util + +# For additional logger output for debugging, comment out the following: +#logging.disable(50) +# and then uncomment the following line: +Bridges.logging.getLogger().setLevel(10) + + +class BridgeRingTests(unittest.TestCase): + """Unittests for :class:`bridgedb.Bridges.BridgeRing`.""" + + def setUp(self): + self.ring = Bridges.BridgeRing('fake-hmac-key') + + def addBridgesFromSameSubnet(self): + bridges = copy.deepcopy(util.generateFakeBridges()) + subnet = "5.5.%d.%d" + i = 1 + j = 1 + + for bridge in bridges: + bridge.address = ipaddr.IPAddress(subnet % (i, j)) + self.ring.insert(bridge) + + if j == 255: + j = 1 + i += 1 + else: + j += 1 + + def test_filterDistinctSubnets(self): + """If there are bridges in the same subnet then they should be + filtered out of the results. + """ + self.addBridgesFromSameSubnet() + + chosen = self.ring.bridges.keys()[:10] + bridges = self.ring.filterDistinctSubnets(chosen) + + # Since they're all in the same /16, we should only get one + # bridge back: + self.assertEqual(len(bridges), 1)

1 0