March 2015 - tor-commits - lists.torproject.org

[bridgedb/develop] Update CHANGELOG entries for bridgedb-0.3.0.
by isis＠torproject.org 21 Mar '15

21 Mar '15

commit cdc0ffab82864ca4c2ad75492da029b70927395e Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat Mar 21 01:53:50 2015 +0000 Update CHANGELOG entries for bridgedb-0.3.0. --- CHANGELOG | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index 081c79d..d0f99e1 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -42,6 +42,14 @@ Changes in version 0.3.0 - XXX the Mechanize-based integrations tests in lib/bridgedb/test/test_https.py. hotfix/0.2.4-mechanize-tags + * FIXES part of #12507 https://bugs.torproject.org/12507 + BridgeDB now has semi-automated developer documentation builds at + https://pythonhosted.org/bridgedb/. + + * FIXES #12805 https://bugs.torproject.org/12805 + BridgeDB is now packaged on PyPI, in the hopes that someday other + organisations will be able to run their own BridgeDBs. + * FIXES #12843 https://bugs.torproject.org/12843 BridgeDB will no longer distribute bridges which it believes are located in Iran or Syria.

1 0

[bridgedb/develop] Make a note of the developer documentation in README.rst.
by isis＠torproject.org 21 Mar '15

21 Mar '15

commit 464009411e7ef4ce5c005f88851afbb90df1bf1a Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat Mar 21 01:23:34 2015 +0000 Make a note of the developer documentation in README.rst. --- README.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.rst b/README.rst index 01b4e2c..b4472c7 100644 --- a/README.rst +++ b/README.rst @@ -66,6 +66,10 @@ instructions for getting extra Bridges. Maintainer Setup ================ +If you'd like to hack on BridgeDB, you might wish to read BridgeDB's +`developer documentation <https://pythonhosted.org/bridgedb/>`__. The rest of +this document mainly concerns mainenance and installation instructions. + ----------------------------- Dependencies and installation -----------------------------

1 0

[bridgedb/master] Update to-be-translated strings in …/i18n/templates/bridgedb.pot.
by isis＠torproject.org 21 Mar '15

21 Mar '15

commit 303c3f9183a0132a0832bc8eb47ad59415642c95 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Feb 3 03:27:32 2015 +0000 Update to-be-translated strings in …/i18n/templates/bridgedb.pot. --- lib/bridgedb/i18n/templates/bridgedb.pot | 54 +++++++++++++++++++++--------- 1 file changed, 38 insertions(+), 16 deletions(-) diff --git a/lib/bridgedb/i18n/templates/bridgedb.pot b/lib/bridgedb/i18n/templates/bridgedb.pot index c1db8e2..276841e 100644 --- a/lib/bridgedb/i18n/templates/bridgedb.pot +++ b/lib/bridgedb/i18n/templates/bridgedb.pot @@ -1,17 +1,17 @@ # Translations template for BridgeDB. -# Copyright (C) 2014 'The Tor Project, Inc.' +# Copyright (C) 2015 'The Tor Project, Inc.' # This file is distributed under the same license as the BridgeDB project. -# Isis Lovecruft <isis(a)torproject.org>, 2014. +# Isis Lovecruft <isis(a)torproject.org>, 2015. # #, fuzzy msgid "" msgstr "" -"Project-Id-Version: bridgedb 0.2.2-84-ge798a44\n" +"Project-Id-Version: bridgedb 0.2.4-3-ge333d91\n" "Report-Msgid-Bugs-To: " "'https://trac.torproject.org/projects/tor/newticket?component=BridgeDB&keywo…" "=bridgedb-reported,msgid&cc=isis,sysrqb&owner=isis'\n" -"POT-Creation-Date: 2014-07-26 02:11+0000\n" -"PO-Revision-Date: 2014-07-26 02:11+0000\n" +"POT-Creation-Date: 2015-02-03 03:24+0000\n" +"PO-Revision-Date: 2015-02-03 03:24+0000\n" "Last-Translator: Isis Lovecruft <isis(a)torproject.org>\n" "Language-Team: LANGUAGE <LL(a)li.org>\n" "MIME-Version: 1.0\n" @@ -30,7 +30,7 @@ msgstr "" #. "fteproxy" #. "Tor" #. "Tor Browser" -#: lib/bridgedb/HTTPServer.py:121 +#: lib/bridgedb/HTTPServer.py:122 msgid "Sorry! Something went wrong with your request." msgstr "" @@ -73,7 +73,7 @@ msgstr "" msgid "Hello, friend!" msgstr "" -#: lib/bridgedb/strings.py:33 lib/bridgedb/templates/base.html:101 +#: lib/bridgedb/strings.py:33 lib/bridgedb/templates/base.html:100 msgid "Public Keys" msgstr "" @@ -261,35 +261,57 @@ msgstr "" msgid "Get a copy of BridgeDB's public GnuPG key." msgstr "" -#: lib/bridgedb/templates/base.html:92 +#: lib/bridgedb/templates/base.html:89 msgid "Report a Bug" msgstr "" -#: lib/bridgedb/templates/base.html:94 +#: lib/bridgedb/templates/base.html:92 msgid "Source Code" msgstr "" -#: lib/bridgedb/templates/base.html:97 +#: lib/bridgedb/templates/base.html:95 msgid "Changelog" msgstr "" -#: lib/bridgedb/templates/base.html:99 +#: lib/bridgedb/templates/base.html:98 msgid "Contact" msgstr "" +#: lib/bridgedb/templates/bridges.html:81 +msgid "Select All" +msgstr "" + +#: lib/bridgedb/templates/bridges.html:87 +msgid "Show QRCode" +msgstr "" + +#: lib/bridgedb/templates/bridges.html:100 +msgid "QRCode for your bridge lines" +msgstr "" + #. TRANSLATORS: Please translate this into some silly way to say #. "There was a problem!" in your language. For example, #. for Italian, you might translate this into "Mama mia!", #. or for French: "Sacrebleu!". :) -#: lib/bridgedb/templates/bridges.html:66 +#: lib/bridgedb/templates/bridges.html:115 lib/bridgedb/templates/bridges.html:175 msgid "Uh oh, spaghettios!" msgstr "" -#: lib/bridgedb/templates/bridges.html:72 +#: lib/bridgedb/templates/bridges.html:116 +msgid "It seems there was an error getting your QRCode." +msgstr "" + +#: lib/bridgedb/templates/bridges.html:121 +msgid "" +"This QRCode contains your bridge lines. Scan it with a QRCode reader to copy " +"your bridge lines onto mobile and other devices." +msgstr "" + +#: lib/bridgedb/templates/bridges.html:181 msgid "There currently aren't any bridges available..." msgstr "" -#: lib/bridgedb/templates/bridges.html:73 +#: lib/bridgedb/templates/bridges.html:182 #, python-format msgid " Perhaps you should try %s going back %s and choosing a different bridge type!" msgstr "" @@ -348,7 +370,7 @@ msgstr "" #. TRANSLATORS: Please make sure the '%s' surrounding single letters at the #. beginning of words are present in your final translation. Thanks! #. TRANSLATORS: Translate "Yes!" as in "Yes! I do need IPv6 addresses." -#: lib/bridgedb/templates/options.html:130 +#: lib/bridgedb/templates/options.html:127 #, python-format msgid "%sY%ses!" msgstr "" @@ -356,7 +378,7 @@ msgstr "" #. TRANSLATORS: Please make sure the '%s' surrounding single letters at the #. beginning of words are present in your final translation. Thanks! #. TRANSLATORS: Please do NOT translate the word "bridge"! -#: lib/bridgedb/templates/options.html:154 +#: lib/bridgedb/templates/options.html:151 #, python-format msgid "%sG%set Bridges" msgstr ""

1 0

[bridgedb/master] Add exit-list download as a repeating, timed task on the reactor.
by isis＠torproject.org 21 Mar '15

21 Mar '15

commit 14f8fd11adbd3849c67cc030a3c0d5dc42b2560f Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 5 05:04:45 2013 +0000 Add exit-list download as a repeating, timed task on the reactor. --- bridgedb.conf | 23 +++++++++++++++++++++++ lib/bridgedb/Main.py | 49 +++++++++++++++++++++++++++++++++---------------- 2 files changed, 56 insertions(+), 16 deletions(-) diff --git a/bridgedb.conf b/bridgedb.conf index c8c0de1..1fe61ab 100644 --- a/bridgedb.conf +++ b/bridgedb.conf @@ -20,6 +20,10 @@ # # CHANGELOG: # ~~~~~~~~~~ +# Changes in version 0.0.12 - 2015-02-06 +# * ADD a dictionary of TASKS to be run and the intervals at which they +# should be repeated. +# # Changes in version 0.0.11 - 2014-07-06 # * ADD EMAIL_BLACKLIST and EMAIL_FUZZY_MATCH settings. # @@ -206,6 +210,16 @@ FORCE_FLAGS = [("Stable", 1)] # Only consider routers whose purpose matches this string. BRIDGE_PURPOSE = "bridge" +# TASKS is a dictionary mapping the names of tasks to the frequency with which +# they should be run (in seconds). If a task's value is set to 0, it will not +# be scheduled to run. +TASKS = { + # Download a list of Tor exit relays once every three hours (by running + # scripts/get-exit-list) and add those exit relays to the list of proxies + # loaded from the PROXY_LIST_FILES: + 'GET_TOR_EXIT_LIST': 3 * 60 * 60, +} + #------------------------------- # HTTP(S) Distribution Options \ #------------------------------------------------------------------------------ @@ -216,6 +230,15 @@ BRIDGE_PURPOSE = "bridge" # your SSL certificate and key! #------------------------------------------------------------------------------ +# (string) The Fully-Qualified Domain Name (FQDN) of the server that the HTTP +# and/or HTTPS distributor(s) is/are publicly reachable at. +SERVER_PUBLIC_FQDN = 'bridges.torproject.org' + +# (string) The IP address that the HTTP and/or HTTPS distributor(s) is/are +# publicly reachable at. This is used for fetching the list of Tor exit +# relays which allow exits to this IP. +SERVER_PUBLIC_EXTERNAL_IP = '38.229.72.19' + # (boolean) True to enable distribution via HTTP or HTTPS; False otherwise. HTTPS_DIST = True diff --git a/lib/bridgedb/Main.py b/lib/bridgedb/Main.py index c720d05..ff4aca8 100644 --- a/lib/bridgedb/Main.py +++ b/lib/bridgedb/Main.py @@ -19,15 +19,18 @@ import logging import gettext from twisted.internet import reactor +from twisted.internet import task from bridgedb import crypto from bridgedb import persistent +from bridgedb import proxy from bridgedb import safelog from bridgedb import schedule from bridgedb import util from bridgedb.configure import loadConfig from bridgedb.configure import Conf from bridgedb.parse import options +from bridgedb.parse.addr import isIPAddress import bridgedb.Bridges as Bridges import bridgedb.Dist as Dist @@ -189,7 +192,7 @@ def loadProxyList(cfg): line = line.strip() if line.startswith("#"): continue - elif Bridges.is_valid_ip(line): + elif isIPAddress(line): ipset[line] = True elif line: logging.info("Skipping line %r in %s: not an IP.", @@ -218,15 +221,6 @@ def _handleSIGUSR1(*args): logging.info("Dumping bridge assignments to files...") reactor.callInThread(runner.doDumpBridges, cfg) - -class ProxyCategory: - def __init__(self): - self.ipset = {} - def contains(self, ip): - return self.ipset.has_key(ip) - def replaceProxyList(self, ipset): - self.ipset = ipset - def replaceBridgeRings(current, replacement): """Replace the current thing with the new one""" current.splitter = replacement.splitter @@ -238,7 +232,7 @@ def createBridgeRings(cfg, proxyList, key): :param cfg: The current configuration, including any in-memory settings (i.e. settings whose values were not obtained from the config file, but were set via a function somewhere) - :type proxyList: :class:`ProxyCategory` + :type proxyList: :class:`~bridgedb.proxy.ProxySet` :param proxyList: The container for the IP addresses of any currently known open proxies. :param bytes key: Splitter master key @@ -261,7 +255,7 @@ def createBridgeRings(cfg, proxyList, key): if cfg.HTTPS_DIST and cfg.HTTPS_SHARE: logging.debug("Setting up HTTPS Distributor...") categories = [] - if proxyList.ipset: + if proxyList: logging.debug("Adding proxyList to HTTPS Distributor categories.") categories.append(proxyList) logging.debug("HTTPS Distributor categories: '%s'" % categories) @@ -349,8 +343,9 @@ def startup(options): key = crypto.getKey(config.MASTER_KEY_FILE) # Get a proxy list. - proxyList = ProxyCategory() - proxyList.replaceProxyList(loadProxyList(config)) + proxyList = proxy.ProxySet() + for proxyfile in config.PROXY_LIST_FILES: + proxy.loadProxiesFromFile(proxyfile, proxyList) emailDistributor = ipDistributor = None @@ -377,7 +372,7 @@ def startup(options): :type splitter: A :class:`bridgedb.Bridges.BridgeHolder` :ivar splitter: A class which takes an HMAC key and splits bridges into their hashring assignments. - :type proxyList: :class:`ProxyCategory` + :type proxyList: :class:`~bridgedb.proxy.ProxySet` :ivar proxyList: The container for the IP addresses of any currently known open proxies. :ivar ipDistributor: A :class:`Dist.IPBasedDistributor`. @@ -418,8 +413,10 @@ def startup(options): state = persistent.load() logging.info("Bridges loaded: %d" % len(splitter)) + logging.debug("Replacing the list of open proxies...") - state.proxyList.replaceProxyList(loadProxyList(cfg)) + for proxyfile in cfg.PROXY_LIST_FILES: + proxy.loadProxiesFromFile(proxyfile, state.proxyList, removeStale=True) if emailDistributorTmp is not None: emailDistributorTmp.prepopulateRings() # create default rings @@ -497,6 +494,26 @@ def startup(options): emailSchedule = schedule.Unscheduled() addSMTPServer(config, emailDistributor, emailSchedule) + tasks = {} + + # Setup all our repeating tasks: + if config.TASKS['GET_TOR_EXIT_LIST']: + tasks['GET_TOR_EXIT_LIST'] = task.LoopingCall( + proxy.downloadTorExits, + proxyList, + config.SERVER_PUBLIC_EXTERNAL_IP) + + # Schedule all configured repeating tasks: + for name, seconds in config.TASKS.items(): + if seconds: + try: + tasks[name].start(abs(seconds)) + except KeyError: + logging.info("Task %s is disabled and will not run." % name) + else: + logging.info("Scheduled task %s to run every %s seconds." + % (name, seconds)) + # Actually run the servers. try: logging.info("Starting reactors.")

1 0

[bridgedb/master] Add t.i.process.protocols for handling Tor exit-list download.
by isis＠torproject.org 21 Mar '15

21 Mar '15

commit 59c0772f275debce5df690a3bac919484b4ae1e2 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 5 01:32:58 2013 +0000 Add t.i.process.protocols for handling Tor exit-list download. * ADD bridgedb.proxy module for handling open proxy lists. * ADD bridgedb.proxy.ExitListProtocol class for asyncronously processing the results read from an HTTPS connection to a remote TorBulkExitList.py. * ADD bridgedb.proxy.ProxySet for storing a MutableSet of proxy addresses, which functions better than bridgedb.Main.ProxyCatagory because it will not allow duplicates to be stored. It also enables us to add its methods as callbacks on deferreds, meansing that the ExitListProtocol can callback its protocol's deferred with a ProxySet.addExitRelays(), which automatically parses and adds the IP:port pairs as they are received from the wire, and adds them to the ProxySet's MutableSet if they do not already exist. The ProxySet class can also store whether an IP:port pair is a known Tor exit relay (with the ``ProxySet.addExitRelay`` method mentioned above), this is *most* of the fix for #4771. * FIXES #4771: bridgedb's list of tor exit relays is down since bulk exit list is down. * ADD several IP address parsers/validators to bridgedb/parse/__init__.py, because I wasn't sure yet if they should be their own module. These checks are improved versions of the old IP address checks in bridgedb.Bridges, because they do not use regexes, and also check that IPv6 addresses are not site-local. --- lib/bridgedb/proxy.py | 466 ++++++++++++++++++++++++++++++++ lib/bridgedb/test/test_proxy.py | 560 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 1026 insertions(+) diff --git a/lib/bridgedb/proxy.py b/lib/bridgedb/proxy.py new file mode 100644 index 0000000..d954213 --- /dev/null +++ b/lib/bridgedb/proxy.py @@ -0,0 +1,466 @@ +# -*- coding: utf-8 -*- +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: Isis Lovecruft 0xA3ADB67A2CDB8B35 <isis(a)torproject.org> +# please also see AUTHORS file +# :copyright: (c) 2013-2015 Isis Lovecruft +# (c) 2007-2015, The Tor Project, Inc. +# (c) 2007-2015, all entities within the AUTHORS file +# :license: 3-clause BSD, see included LICENSE for information + +"""Classes for finding and managing lists of open proxies.""" + +from __future__ import print_function +from collections import MutableSet +from functools import update_wrapper +from functools import wraps + +import ipaddr +import logging +import os +import time + +from twisted.internet import defer +from twisted.internet import protocol +from twisted.internet import reactor +from twisted.internet import utils as txutils +from bridgedb.runner import find +from bridgedb.parse.addr import isIPAddress + + +def downloadTorExits(proxyList, ipaddress, port=443, protocol=None): + """Run a script which downloads a list of Tor exit relays which allow their + clients to exit to the given **ipaddress** and **port**. + + :param proxyList: The :class:`ProxySet` instance from :mod:`bridgedb.Main`. + :param str ipaddress: The IP address that each Tor exit relay should be + capable of connecting to for clients, as specified by its ExitPolicy. + :param int port: The port corresponding to the above **ipaddress** that + each Tor exit relay should allow clients to exit to. (See + https://check.torproject.org/cgi-bin/TorBulkExitList.py.) + :type protocol: :api:`twisted.internet.protocol.Protocol` + :param protocol: A :class:`~bridgedb.proxy.ExitListProtocol`, or any other + :api:`~twisted.internet.protocol.Protocol` implementation for + processing the results of a process which downloads a list of Tor exit + relays. This parameter is mainly meant for use in testing, and should + not be changed. + :rtype: :class:`~twisted.internet.defer.Deferred` + :returns: A deferred which will callback with a list, each item in the + list is a string containing an IP of a Tor exit relay. + """ + proto = ExitListProtocol() if protocol is None else protocol() + args = [proto.script, '--stdout', '-a', ipaddress, '-p', str(port)] + proto.deferred.addCallback(proxyList.addExitRelays) + proto.deferred.addErrback(logging.exception) + transport = reactor.spawnProcess(proto, proto.script, args=args, env={}) + return proto.deferred + +def loadProxiesFromFile(filename, proxySet=None, removeStale=False): + """Load proxy IP addresses from a list of files. + + :param str filename: A filename whose path can be either absolute or + relative to the current working directory. The file should contain the + IP addresses of various open proxies, one per line, for example:: + + 11.11.11.11 + 22.22.22.22 + 123.45.67.89 + + :type proxySet: None or :class:`~bridgedb.proxy.ProxySet`. + :param proxySet: If given, load the addresses read from the files into + this ``ProxySet``. + :param bool removeStale: If ``True``, remove proxies from the **proxySet** + which were not listed in any of the **files**. + (default: ``False``) + :returns: A list of all the proxies listed in the **files* (regardless of + whether they were added or removed). + """ + logging.info("Reloading proxy lists...") + + addresses = [] + if proxySet: + oldProxySet = proxySet.copy() + + with open(filename, 'r') as proxyFile: + for line in proxyFile.readlines(): + line = line.strip() + if proxySet: + # ProxySet.add() will validate the IP address + if proxySet.add(line, tag=filename): + logging.info("Added %s to the proxy list." % line) + addresses.append(line) + else: + ip = isIPAddress(line) + if ip: + addresses.append(ip) + + if proxySet: + stale = list(oldProxySet.difference(addresses)) + + if removeStale: + for ip in stale: + if proxySet.getTag(ip) == filename: + logging.info("Removing stale IP %s from proxy list." % ip) + proxySet.remove(ip) + else: + logging.info("Tag %s didn't match %s" + % (proxySet.getTag(ip), filename)) + + return addresses + + +class ProxySet(MutableSet): + """A :class:`collections.MutableSet` for storing validated IP addresses.""" + + #: A tag to apply to IP addresses within this ``ProxySet`` which are known + #: Tor exit relays. + _exitTag = 'exit_relay' + + def __init__(self, proxies=dict()): + """Initialise a ``ProxySet``. + + :type proxies: A tuple, list, dict, or set. + :param proxies: Optionally, initialise with an iterable, ``proxies``. + For each ``item`` in that iterable, ``item`` must either: + 1. be a string or int representing an IP address, or, + 2. be another iterable, whose first item satisfies #1. + """ + super(ProxySet, self).__init__() + self._proxydict = dict() + self._proxies = set() + self.addProxies(proxies) + + @property + def proxies(self): + """All proxies in this set, regardless of tags.""" + return list(self._proxies) + + @property + def exitRelays(self): + """Get all proxies in this ``ProxySet`` tagged as Tor exit relays. + + :rtype: set + :returns: A set of all known Tor exit relays which are contained + within this :class:`~bridgedb.proxy.ProxySet`. + """ + return self.getAllWithTag(self._exitTag) + + def __add__(self, ip=None, tag=None): + """Add an **ip** to this set, with an optional **tag**. + + This has no effect if the **ip** is already present. The **ip** is + only added if it passes the checks in + :func:`~bridgedb.parse.addr.isIPAddress`. + + :type ip: basestring or int + :param ip: The IP address to add. + :param tag: An optional value to link to **ip**. If not given, it will + be a timestamp (seconds since epoch, as a float) for when **ip** + was first added to this set. + :rtype: bool + :returns: ``True`` if **ip** is in this set; ``False`` otherwise. + """ + ip = isIPAddress(ip) + if ip: + if self._proxies.isdisjoint(set(ip)): + self._proxies.add(ip) + self._proxydict[ip] = tag if tag else time.time() + return True + return False + + def __radd__(self, *args, **kwargs): self.__add__(*args, **kwargs) + + def __contains__(self, ip): + """x.__contains__(y) <==> y in x. + + :type ip: basestring or int + :param ip: The IP address to check. + :rtype: boolean + :returns: True if ``ip`` is in this set; False otherwise. + """ + ipset = [isIPAddress(ip),] + if ipset and len(self._proxies.intersection(ipset)) == len(ipset): + return True + return False + + def __sub__(self, ip): + """Entirely remove **ip** from this set. + + :type ip: basestring or int + :param ip: The IP address to remove. + """ + try: + self._proxydict.pop(ip) + self._proxies.discard(ip) + except KeyError: + pass + + def __rsub__(self, *args, **kwargs): raise NotImplemented + + def _getErrorMessage(self, x=None, y=None): + """Make an error message describing how this class works.""" + message = """\nParameter 'proxies' must be one of: + - a {1} of {0} + - a {2} of {0} + - a {3}, whose keys are {0} (the values can be anything) + - a {4} of {1}s, whose first object in each {1} must be a {0} + - a {4} of {0} + """.format(type(''), type(()), type([]), type({}), type(set(()))) + end = "You gave: a {0}".format(type(y)) + end += " of {0}".format(type(x)) + return os.linesep.join((message, end)) + + def addProxies(self, proxies, tag=None): + """Add proxies to this set. + + This calls :func:`add` for each item in the iterable **proxies**. + Each proxy, if added, will be tagged with a current timestamp. + + :type proxies: A tuple, list, dict, or set. + :param proxies: An iterable. For each ``item`` in that iterable, + ``item`` must either: + 1. be a string or int representing an IP address, or, + 2. be another iterable, whose first item satisfies #1. + :keyword tag: An optional value to link to all untagged + **proxies**. If ``None``, it will be a timestamp (seconds since + epoch, as a float) for when the proxy was first added to this set. + """ + if isinstance(proxies, dict): + [self.add(ip, value) for (ip, value) in proxies.items()] + else: + try: + for x in proxies: + if isinstance(x, (tuple, list, set)): + if len(x) == 2: self.add(x[0], x[1]) + elif len(x) == 1: self.add(x, tag) + else: raise ValueError(self._getErrorMessage(x, proxies)) + elif isinstance(x, (basestring, int)): + self.add(x, tag) + else: + raise ValueError(self._getErrorMessage(x, proxies)) + except TypeError: + raise ValueError(self._getErrorMessage(proxies, None)) + + @wraps(addProxies) + def addExitRelays(self, relays): + logging.info("Loading exit relays into proxy list...") + [self.add(x, self._exitTag) for x in relays] + + def getTag(self, ip): + """Get the tag for an **ip** in this ``ProxySet``, if available. + + :type ip: basestring or int + :param ip: The IP address to obtain the tag for. + :rtype: ``None`` or basestring or int + :returns: The tag for that **ip**, iff **ip** exists in this + ``ProxySet`` and it has a tag. + """ + return self._proxydict.get(ip) + + def getAllWithTag(self, tag): + """Get all proxies in this ``ProxySet`` with a given tag. + + :param basestring tag: A tag to search for. + :rtype: set + :returns: A set of all proxies which are contained within this + :class:`~bridgedb.proxy.ProxySet` which are also tagged with + **tag**. + """ + return set([key for key, value in filter(lambda x: x[1] == tag, + self._proxydict.items())]) + + def firstSeen(self, ip): + """Get the timestamp when **ip** was first seen, if available. + + :type ip: basestring or int + :param ip: The IP address to obtain a timestamp for. + :rtype: float or None + :returns: The timestamp (in seconds since epoch) if available. + Otherwise, returns None. + """ + when = self.getTag(ip) + if isinstance(when, float): + return when + + def isExitRelay(self, ip): + """Check if ``ip`` is a known Tor exit relay. + + :type ip: basestring or int + :param ip: The IP address to check. + :rtype: boolean + :returns: True if ``ip`` is a known Tor exit relay; False otherwise. + """ + if self.getTag(ip) == self._exitTag: + return True + return False + + def replaceProxyList(self, proxies, tag=None): + """Clear everything and add all ``proxies``. + + :type proxies: A tuple, list, dict, or set. + :param proxies: An iterable. For each ``item`` in that iterable, + ``item`` must either: + 1. be a string or int representing an IP address, or, + 2. be another iterable, whose first item satisfies #1. + """ + try: + self.clear() + self.addProxies(proxies, tag=tag) + except Exception as error: + logging.error(str(error)) + + _assigned = ('__name__', '__doc__') + + @wraps(MutableSet._hash) + def __hash__(self): return self._hash() + def __iter__(self): return self._proxies.__iter__() + def __len__(self): return len(self._proxydict.items()) + def __repr__(self): return type('')(self.proxies) + def __str__(self): return os.linesep.join(self.proxies) + update_wrapper(__iter__, set.__iter__, _assigned) + update_wrapper(__len__, len, _assigned) + update_wrapper(__repr__, repr, _assigned) + update_wrapper(__str__, str, _assigned) + + def add(self, ip, tag=None): return self.__add__(ip, tag) + def copy(self): return self.__class__(self._proxydict.copy()) + def contains(self, ip): return self.__contains__(ip) + def discard(self, ip): return self.__sub__(ip) + def remove(self, other): return self.__sub__(other) + update_wrapper(add, __add__) + update_wrapper(copy, __init__) + update_wrapper(contains, __contains__) + update_wrapper(discard, __sub__) + update_wrapper(remove, __sub__) + + def difference(self, other): return self._proxies.difference(other) + def issubset(self, other): return self._proxies.issubset(other) + def issuperset(self, other): return self._proxies.issuperset(other) + def intersection(self, other): return self._proxies.intersection(other) + def symmetric_difference(self, other): return self._proxies.symmetric_difference(other) + def union(self, other): return self._proxies.union(other) + update_wrapper(difference, set.difference, _assigned) + update_wrapper(issubset, set.issubset, _assigned) + update_wrapper(issuperset, set.issuperset, _assigned) + update_wrapper(intersection, set.intersection, _assigned) + update_wrapper(symmetric_difference, set.symmetric_difference, _assigned) + update_wrapper(union, set.union, _assigned) + + +class ExitListProtocol(protocol.ProcessProtocol): + """A :class:`~twisted.internet.protocol.Protocol` for ``get-exit-list``. + + :attr boolean connected: True if our ``transport`` is connected. + + :type transport: An implementer of + :interface:`twisted.internet.interface.IProcessTransport`. + :attr transport: If :func:`twisted.internet.reactor.spawnProcess` is + called with an instance of this class as it's ``protocol``, then + :func:`~twisted.internet.reactor.spawnProcess` will return this + ``transport``. + """ + + def __init__(self): + """Create a protocol for downloading a list of current Tor exit relays. + + :type exitlist: :class:`ProxySet` + :ivar exitlist: A :class:`~collections.MutableSet` containing the IP + addresses of known Tor exit relays which can reach our public IP + address. + :ivar list data: A list containing a ``bytes`` object for each chuck + of data received from the ``transport``. + :ivar deferred: A deferred which will callback with the ``exitlist`` + when the process has ended. + + :param string script: The full pathname of the script to run. + """ + self.data = [] + self.script = find('get-tor-exits') + self.exitlist = ProxySet() + self.deferred = defer.Deferred() + + def childConnectionLost(self, childFD): + """See :func:`t.i.protocol.ProcessProtocol.childConnectionLost`.""" + protocol.ProcessProtocol.childConnectionLost(self, childFD) + + def connectionMade(self): + """Called when a connection is made. + + This may be considered the initializer of the protocol, because it is + called when the connection is completed. For clients, this is called + once the connection to the server has been established; for servers, + this is called after an accept() call stops blocking and a socket has + been received. If you need to send any greeting or initial message, + do it here. + """ + logging.debug("ExitListProtocol: Connection made with remote server") + self.transport.closeStdin() + + def errReceived(self, data): + """Some data was received from stderr.""" + # The get-exit-list script uses twisted.python.log to log to stderr: + logging.debug(data) # pragma: no cover + + def outReceived(self, data): + """Some data was received from stdout.""" + self.data.append(data) + + def outConnectionLost(self): + """This will be called when stdout is closed.""" + logging.debug("Finished downloading list of Tor exit relays.") + self.transport.loseConnection() + self.parseData() + + def parseData(self): + """Parse all data received so far into our + :class:`<bridgedb.proxy.ProxySet> exitlist`. + """ + unparseable = [] + + data = ''.join(self.data).split('\n') + + for line in data: + line.strip() + if not line: continue + # If it reached an errorpage, then we grabbed raw HTML that starts + # with an HTML tag: + if line.startswith('<'): break + if line.startswith('#'): continue + ip = isIPAddress(line) + if ip: + logging.info("Discovered Tor exit relay: %s" % ip) + self.exitlist.add(ip) + else: + logging.debug("Got exitlist line that wasn't an IP: %s" % line) + unparseable.append(line) + + if unparseable: + logging.warn(("There were unparseable lines in the downloaded " + "list of Tor exit relays: %r") % unparseable) + + def processEnded(self, reason): + """Called when the child process exits and all file descriptors + associated with it have been closed. + + :type reason: :class:`twisted.python.failure.Failure` + """ + self.transport.loseConnection() + if reason.value.exitCode != 0: # pragma: no cover + logging.debug(reason.getTraceback()) + logging.error("There was an error downloading Tor exit list: %s" + % reason.value) + else: + logging.info("Finished processing list of Tor exit relays.") + logging.debug("Transferring exit list to storage...") + # Avoid triggering the deferred twice, e.g. on processExited(): + if not self.deferred.called: + self.deferred.callback(list(self.exitlist.proxies)) + + def processExited(self, reason): + """This will be called when the subprocess exits. + + :type reason: :class:`twisted.python.failure.Failure` + """ + logging.debug("%s exited with status code %d" + % (self.script, reason.value.exitCode)) diff --git a/lib/bridgedb/test/test_proxy.py b/lib/bridgedb/test/test_proxy.py new file mode 100644 index 0000000..78e24ff --- /dev/null +++ b/lib/bridgedb/test/test_proxy.py @@ -0,0 +1,560 @@ +# -*- coding: utf-8 -*- +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: Isis Lovecruft 0xA3ADB67A2CDB8B35 <isis(a)torproject.org> +# please also see AUTHORS file +# :copyright: (c) 2013-2015 Isis Lovecruft +# (c) 2007-2015, The Tor Project, Inc. +# (c) 2007-2015, all entities within the AUTHORS file +# :license: 3-clause BSD, see included LICENSE for information + +"""Tests for :mod:`bridgedb.proxy`.""" + +import sure + +from twisted.internet import defer +from twisted.trial import unittest + +from bridgedb import proxy + + +EXIT_LIST_0 = """\ +11.11.11.11 +22.22.22.22 +123.45.67.89""" + +EXIT_LIST_1 = """\ +33.33.33.33 +44.44.44.44 +55.55.55.55 +66.66.66.66 +77.77.77.77""" + +EXIT_LIST_BAD = """\ +foo +bar +baz""" + + +class MockExitListProtocol(proxy.ExitListProtocol): + """A mocked version of :class:`~bridgedb.proxy.ExitListProtocol`.""" + + def __init__(self): + proxy.ExitListProtocol.__init__(self) + self._data = EXIT_LIST_0 + self.script = '/bin/echo' + print() + + def _log(self, msg): + print("%s: %s" % (self.__class__.__name__, msg)) + + def childConnectionLost(self, childFD): + self._log("childConnectionLost() called with childFD=%s" % childFD) + proxy.ExitListProtocol.childConnectionLost(self, childFD) + + def connectionMade(self): + self._log("connectionMade() called") + proxy.ExitListProtocol.connectionMade(self) + + def errReceived(self, data): + self._log("errReceived() called with %s" % data) + proxy.ExitListProtocol.errReceived(self, data) + + def outReceivedData(self, data): + self._log("outReceivedData() called with %s" % data) + proxy.ExitListProtocol.outReceivedData(self, data) + + def outConnectionLost(self): + self._log("outConnectionLost() called") + proxy.ExitListProtocol.outConnectionLost(self) + + def parseData(self): + data = self._data.split('\n') + for line in data: + line = line.strip() + if not line: continue + if line.startswith('<'): break + if line.startswith('#'): continue + ip = proxy.isIPAddress(line) + if ip: + self._log("adding IP %s to exitlist..." % ip) + self.exitlist.add(ip) + + def processEnded(self, reason): + self._log("processEnded() called with %s" % reason) + proxy.ExitListProtocol.processEnded(self, reason) + + def processExited(self, reason): + self._log("processExited() called with %s" % reason) + proxy.ExitListProtocol.processExited(self, reason) + + +class ProxySetImplementationTest(unittest.TestCase): + + def setUp(self): + # We have to put something in it, otherwise self.ps.should.be.ok won't + # think it's truthy: + self.ps = proxy.ProxySet(['1.1.1.1']) + + def test_instantiation(self): + self.ps.should.be.ok + self.ps.should.have.property('__contains__').being.callable + self.ps.should.have.property('__hash__').being.callable + self.ps.should.have.property('__iter__').being.callable + self.ps.should.have.property('__len__').being.callable + self.ps.should.have.property('add').being.callable + self.ps.should.have.property('copy').being.callable + self.ps.should.have.property('contains').being.callable + self.ps.should.have.property('discard').being.callable + self.ps.should.have.property('remove').being.callable + + self.ps.should.have.property('difference').being.callable + self.ps.should.have.property('issubset').being.callable + self.ps.should.have.property('issuperset').being.callable + self.ps.should.have.property('intersection').being.callable + self.ps.should.have.property('symmetric_difference').being.callable + self.ps.should.have.property('union').being.callable + + def test_attributes(self): + self.ps.should.have.property('proxies').being.a(list) + self.ps.should.have.property('exitRelays').being.a(set) + self.ps.should.have.property('_proxies').being.a(set) + self.ps.should.have.property('_proxydict').being.a(dict) + + +class LoadProxiesFromFileIntegrationTests(unittest.TestCase): + """Unittests for :class:`~bridgedb.proxy.loadProxiesFromFile()`.""" + + def setUp(self): + self.fn0 = '%s-0' % self.__class__.__name__ + self.fn1 = '%s-1' % self.__class__.__name__ + self.badfile = '%s-badfile' % self.__class__.__name__ + self.writeFiles() + + def writeFiles(self): + with open(self.fn0, 'w') as fh: + fh.write(EXIT_LIST_0) + fh.flush() + with open(self.fn1, 'w') as fh: + fh.write(EXIT_LIST_1) + fh.flush() + with open(self.badfile, 'w') as fh: + fh.write(EXIT_LIST_BAD) + fh.flush() + + def emptyFile(self, filename): + """We have to do this is a separate method, otherwise Twisted doesn't + actually do it. + """ + fh = open(filename, 'w') + fh.truncate() + fh.close() + + def test_proxy_loadProxiesFromFile_1_file(self): + """Test loading proxies from one file.""" + proxies = proxy.loadProxiesFromFile(self.fn0) + self.assertEqual(len(proxies), 3) + + def test_proxy_loadProxiesFromFile_1_file_and_proxyset(self): + """Test loading proxies from one file.""" + proxyList = proxy.ProxySet(['1.1.1.1']) + proxies = proxy.loadProxiesFromFile(self.fn0, proxySet=proxyList) + self.assertEqual(len(proxies), 3) + self.assertEqual(len(proxyList), 4) + + def test_proxy_loadProxiesFromFile_2_files_and_proxyset(self): + """Test loading proxies from two files.""" + proxyList = proxy.ProxySet(['1.1.1.1']) + proxy.loadProxiesFromFile(self.fn0, proxySet=proxyList) + proxies = proxy.loadProxiesFromFile(self.fn1, proxySet=proxyList) + self.assertEqual(len(proxies), 5) + self.assertEqual(len(proxyList), 9) + + def test_proxy_loadProxiesFromFile_removeStale(self): + """Test loading proxies from two files and removing the stale ones.""" + proxyList = proxy.ProxySet(['1.1.1.1']) + self.assertEqual(len(proxyList), 1) + proxies = proxy.loadProxiesFromFile(self.fn0, proxySet=proxyList) + self.assertEqual(len(proxies), 3) + self.assertEqual(len(proxyList), 4) + proxies = proxy.loadProxiesFromFile(self.fn1, proxySet=proxyList) + self.assertEqual(len(proxies), 5) + self.assertEqual(len(proxyList), 9) + + self.emptyFile(self.fn0) + proxies = proxy.loadProxiesFromFile(self.fn0, proxySet=proxyList, + removeStale=True) + self.assertEqual(len(proxies), 0) + self.assertEqual(len(proxyList), 6) + + def test_proxy_loadProxiesFromFile_duplicates(self): + """Loading proxies from the same file twice shouldn't store + duplicates. + """ + proxyList = proxy.ProxySet(['1.1.1.1']) + proxy.loadProxiesFromFile(self.fn1, proxySet=proxyList) + self.assertEqual(len(proxyList), 6) + proxy.loadProxiesFromFile(self.fn1, proxySet=proxyList) + self.assertEqual(len(proxyList), 6) + + def test_proxy_loadProxiesFromFile_bad_file(self): + """Loading proxies from a file with invalid IPs in it should do + nothing. + """ + proxyList = proxy.ProxySet() + proxy.loadProxiesFromFile(self.badfile, proxySet=proxyList) + self.assertEqual(len(proxyList), 0) + + +class DownloadTorExitsTests(unittest.TestCase): + """Tests for `~bridgedb.proxy.downloadTorExits()`.""" + + def test_proxy_downloadTorExits(self): + proxyList = proxy.ProxySet() + d = proxy.downloadTorExits(proxyList, + 'OurIPWouldGoHere', + protocol=MockExitListProtocol) + self.assertIsInstance(d, defer.Deferred) + + +class ProxySetUnittests(unittest.TestCase): + """Unittests for :class:`~bridgedb.proxy.ProxySet`.""" + + def setUp(self): + self.proxies = EXIT_LIST_1.split('\n') + self.moarProxies = EXIT_LIST_0.split('\n') + + self.proxyList = proxy.ProxySet() + for p in self.proxies: + self.proxyList.add(p) + + def test_ProxySet_init(self): + """When initialised (after setUp() has run), the ProxySet should + contain a number of proxies equal to the number we added in the setUp() + method. + """ + self.assertEquals(len(self.proxyList), len(self.proxies)) + + def test_ProxySet_proxies_getter(self): + """ProxySet.proxies should list all proxies.""" + self.assertItemsEqual(self.proxyList.proxies, set(self.proxies)) + + def test_ProxySet_proxies_setter(self): + """``ProxySet.proxies = ['foo']`` should raise an ``AttributeError``.""" + self.assertRaises(AttributeError, self.proxyList.__setattr__, 'proxies', ['foo']) + + def test_ProxySet_proxies_deleter(self): + """``del(ProxySet.proxies)`` should raise an AttributeError.""" + self.assertRaises(AttributeError, self.proxyList.__delattr__, 'proxies') + + def test_ProxySet_exitRelays_issubset_proxies(self): + """ProxySet.exitRelays should always be a subset of ProxySet.proxies.""" + self.assertTrue(self.proxyList.exitRelays.issubset(self.proxyList.proxies)) + self.proxyList.addExitRelays(self.moarProxies) + self.assertTrue(self.proxyList.exitRelays.issubset(self.proxyList.proxies)) + + def test_ProxySet_exitRelays_getter(self): + """ProxySet.exitRelays should list all exit relays.""" + self.proxyList.addExitRelays(self.moarProxies) + self.assertItemsEqual(self.proxyList.exitRelays, set(self.moarProxies)) + + def test_ProxySet_exitRelays_setter(self): + """``ProxySet.exitRelays = ['foo']`` should raise an ``AttributeError``.""" + self.assertRaises(AttributeError, self.proxyList.__setattr__, 'exitRelays', ['foo']) + + def test_ProxySet_exitRelays_deleter(self): + """``del(ProxySet.exitRelays)`` should raise an AttributeError.""" + self.assertRaises(AttributeError, self.proxyList.__delattr__, 'exitRelays') + + def test_ProxySet_add_new(self): + """ProxySet.add() should add a new proxy.""" + self.proxyList.add('110.110.110.110') + self.assertEquals(len(self.proxyList), len(self.proxies) + 1) + self.assertIn('110.110.110.110', self.proxyList) + + def test_ProxySet_add_None(self): + """ProxySet.add() called with None should return False.""" + self.assertFalse(self.proxyList.add(None)) + self.assertEquals(len(self.proxyList), len(self.proxies)) + + def test_ProxySet_add_duplicate(self): + """ProxySet.add() shouldn't add the same proxy twice.""" + self.proxyList.add(self.proxies[0]) + self.assertEquals(len(self.proxyList), len(self.proxies)) + self.assertIn(self.proxies[0], self.proxyList) + + def test_ProxySet_addExitRelays(self): + """ProxySet.addExitRelays() should add the new proxies.""" + self.proxyList.addExitRelays(self.moarProxies) + self.assertIn(self.moarProxies[0], self.proxyList) + + def test_ProxySet_radd_new(self): + """ProxySet.radd() should add a new proxy.""" + self.proxyList.__radd__('110.110.110.110') + self.assertEquals(len(self.proxyList), len(self.proxies) + 1) + self.assertIn('110.110.110.110', self.proxyList) + + def test_ProxySet_addExitRelays_tagged(self): + """ProxySet.addExitRelays() should add the new proxies, and they should + be tagged as being Tor exit relays. + """ + self.proxyList.addExitRelays(self.moarProxies) + self.assertTrue(self.proxyList.isExitRelay(self.moarProxies[0])) + self.assertEquals(self.proxyList.getTag(self.moarProxies[0]), + self.proxyList._exitTag) + + def test_ProxySet_addExitRelays_length(self): + """ProxySet.addExitRelays() should add the new proxies and then the + total number should be equal to the previous number of proxies plus the + new exit relays added. + """ + self.proxyList.addExitRelays(self.moarProxies) + self.assertEquals(len(self.proxyList), len(self.proxies) + len(self.moarProxies)) + + def test_ProxySet_addExitRelays_previous_proxies_kept(self): + """ProxySet.addExitRelays() should add the new proxies and keep ones that + were already in the ProxySet. + """ + self.proxyList.addExitRelays(self.moarProxies) + self.assertIn(self.proxies[0], self.proxyList) + + def test_ProxySet_addExitRelays_previous_proxies_not_tagged(self): + """ProxySet.addExitRelays() should add the new proxies and tag them, + but any previous non-exit relays in the ProxySet shouldn't be tagged as + being Tor exit relays. + """ + self.proxyList.addExitRelays(self.moarProxies) + self.assertFalse(self.proxyList.isExitRelay(self.proxies[0])) + self.assertNotEquals(self.proxyList.getTag(self.proxies[0]), + self.proxyList._exitTag) + + def test_ProxySet_addProxies_tuple_individual_tags(self): + """ProxySet.addProxies() should add the new proxies and tag them with + whatever tags we want. + """ + tags = ['foo', 'bar', 'baz'] + extraProxies = zip(self.moarProxies, tags) + self.proxyList.addProxies(extraProxies) + self.assertEquals(len(self.proxyList), len(self.proxies) + len(extraProxies)) + self.assertIn(extraProxies[0][0], self.proxyList) + self.assertEquals(self.proxyList._proxydict[extraProxies[0][0]], extraProxies[0][1]) + self.assertEquals(self.proxyList._proxydict[extraProxies[1][0]], extraProxies[1][1]) + self.assertEquals(self.proxyList._proxydict[extraProxies[2][0]], extraProxies[2][1]) + + def test_ProxySet_addProxies_tuple_too_many_items(self): + """``ProxySet.addProxies()`` where the tuples have >2 items should + raise a ValueError. + """ + extraProxies = zip(self.moarProxies, + ['sometag' for _ in range(len(self.moarProxies))], + ['othertag' for _ in range(len(self.moarProxies))]) + self.assertRaises(ValueError, self.proxyList.addProxies, extraProxies) + + def test_ProxySet_addProxies_list(self): + """``ProxySet.addProxies(..., tag='sometag')`` should add the new + proxies and tag them all with the same tag. + """ + self.proxyList.addProxies(self.moarProxies, tag='sometag') + self.assertEquals(len(self.proxyList), len(self.proxies) + len(self.moarProxies)) + self.assertIn(self.moarProxies[0], self.proxyList) + for p in self.moarProxies: + self.assertEquals(self.proxyList.getTag(p), 'sometag') + for p in self.proxies: + self.assertNotEqual(self.proxyList.getTag(p), 'sometag') + + def test_ProxySet_addProxies_set(self): + """``ProxySet.addProxies(..., tag=None)`` should add the new + proxies and tag them all with timestamps. + """ + self.proxyList.addProxies(set(self.moarProxies)) + self.assertEquals(len(self.proxyList), len(self.proxies) + len(self.moarProxies)) + self.assertIn(self.moarProxies[0], self.proxyList) + for p in self.moarProxies: + self.assertIsInstance(self.proxyList.getTag(p), float) + for p in self.proxies: + self.assertNotEqual(self.proxyList.getTag(p), 'sometag') + + def test_ProxySet_addProxies_bad_type(self): + """``ProxySet.addProxies()`` called with something which is neither an + iterable, a basestring, or an int should raise a ValueError. + """ + self.assertRaises(ValueError, self.proxyList.addProxies, object) + + def test_ProxySet_addProxies_list_of_bad_types(self): + """``ProxySet.addProxies()`` called with something which is neither an + iterable, a basestring, or an int should raise a ValueError. + """ + self.assertRaises(ValueError, self.proxyList.addProxies, [object, object, object]) + + def test_ProxySet_getTag(self): + """ProxySet.getTag() should get the tag for a proxy in the set.""" + self.proxyList.add('1.1.1.1', 'bestproxyevar') + self.assertEquals(self.proxyList.getTag('1.1.1.1'), 'bestproxyevar') + + def test_ProxySet_getTag_nonexistent(self): + """ProxySet.getTag() should get None for a proxy not in the set.""" + self.assertIsNone(self.proxyList.getTag('1.1.1.1')) + + def test_ProxySet_clear(self): + """ProxySet.clear() should clear the set of proxies.""" + self.proxyList.clear() + self.assertEquals(len(self.proxyList), 0) + self.assertEquals(len(self.proxyList.proxies), 0) + self.assertEquals(len(self.proxyList._proxies), 0) + self.assertEquals(len(self.proxyList._proxydict.items()), 0) + + def test_ProxySet_contains_list(self): + """Calling ``list() is in ProxySet()`` should return False.""" + self.assertFalse(self.proxyList.contains(list(self.proxies[0],))) + + def test_ProxySet_contains_nonexistent(self): + """``ProxySet().contains()`` with a proxy not in the set should + return False. + """ + self.assertFalse(self.proxyList.contains(self.moarProxies[0])) + + def test_ProxySet_contains_nonexistent(self): + """``ProxySet().contains()`` with a proxy in the set should + return True. + """ + self.assertTrue(self.proxyList.contains(self.proxies[0])) + + def test_ProxySet_copy(self): + """ProxySet.copy() should create an exact copy.""" + newProxyList = self.proxyList.copy() + self.assertEquals(newProxyList, self.proxyList) + + def test_ProxySet_difference(self): + """ProxySet.difference() should list the items in ProxySetA which + aren't in ProxySetB. + """ + proxySetA = self.proxyList + proxySetB = proxy.ProxySet(self.moarProxies) + self.assertItemsEqual(proxySetA.difference(proxySetB), + set(self.proxies)) + self.assertItemsEqual(proxySetB.difference(proxySetA), + set(self.moarProxies)) + + def test_ProxySet_firstSeen_returns_timestamp(self): + """ProxySet.firstSeen() should return a timestamp for a proxy with a + timestamp tag. + """ + self.proxyList.add(self.moarProxies[0]) + self.assertIsNotNone(self.proxyList.firstSeen(self.moarProxies[0])) + + def test_ProxySet_firstSeen_returns_float(self): + """ProxySet.firstSeen() should return a timestamp for a proxy with a + timestamp tag. + """ + self.proxyList.add(self.moarProxies[1]) + self.assertIsInstance(self.proxyList.firstSeen(self.moarProxies[1]), float) + + def test_ProxySet_firstSeen_other_tags(self): + """ProxySet.firstSeen() should return None when a proxy doesn't have a + timestamp. + """ + self.proxyList.add(self.moarProxies[2], 'sometag') + self.assertIsNone(self.proxyList.firstSeen(self.moarProxies[2])) + + def test_ProxySet_issubset(self): + """ProxySet.issubset() on a superset should return True.""" + self.assertTrue(self.proxyList.issubset(set(self.proxies + self.moarProxies[:0]))) + + def test_ProxySet_issuperset(self): + """ProxySet.issubset() on a subset should return True.""" + self.assertTrue(self.proxyList.issuperset(set(self.proxies[:1]))) + + def test_ProxySet_remove(self): + """ProxySet.remove() should subtract proxies which were already added + to the set. + """ + self.proxyList.remove(self.proxies[0]) + self.assertEquals(len(self.proxyList), len(self.proxies) - 1) + self.assertNotIn(self.proxies[0], self.proxyList) + + def test_ProxySet_remove_nonexistent(self): + """ProxySet.remove() shouldn't subtract proxies which aren't already in + the set. + """ + self.proxyList.remove('110.110.110.110') + self.assertEquals(len(self.proxyList), len(self.proxies)) + self.assertNotIn('110.110.110.110', self.proxyList) + + def test_ProxySet_replaceProxyList(self): + """ProxySet.replaceProxyList should remove all the current proxies and + add all the new ones. + """ + self.proxyList.replaceProxyList(self.moarProxies, 'seven proxies') + for p in self.moarProxies: + self.assertIn(p, self.proxyList) + self.assertEqual(self.proxyList.getTag(p), 'seven proxies') + for p in self.proxies: + self.assertNotIn(p, self.proxyList) + + def test_ProxySet_replaceProxyList_bad_type(self): + """ProxySet.replaceProxyList should remove all the current proxies and + then since we're giving it a bad type it should do nothing else. + """ + self.proxyList.replaceProxyList([object, object, object]) + self.assertEqual(len(self.proxyList), 0) + + def test_ProxySet_hash(self): + """Two equal ProxySets should return the same hash.""" + proxyListA = proxy.ProxySet(self.proxies) + proxyListB = proxy.ProxySet(self.proxies) + self.assertEqual(proxyListA, proxyListB) + self.assertItemsEqual(proxyListA, proxyListB) + self.assertEqual(hash(proxyListA), hash(proxyListB)) + + +class ExitListProtocolTests(unittest.TestCase): + """Unittests for :class:`~bridgedb.proxy.ExitListProtocol`.""" + + def setUp(self): + self.proto = proxy.ExitListProtocol() + + def test_ExitListProtocol_parseData_error_page(self): + """ """ + self.proto.data = """\ +<!doctype html> +<html lang="en"> +<body> + <div class="content"> + <img src="/torcheck/img/tor-on.png" class="onion" /> + <h4>Welcome to the Tor Bulk Exit List exporting tool.</h4> + </div> +</body> +</html>""" + self.proto.parseData() + self.assertEqual(len(self.proto.exitlist), 0) + + def test_ExitListProtocol_parseData_page_with_3_ips_with_comments(self): + """ """ + self.proto.data = """\ +# This is a list of all Tor exit nodes from the past 16 hours that can contact 1.1.1.1 on port 443 # +# You can update this list by visiting https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1&port=443 # +# This file was generated on Fri Feb 6 02:04:27 UTC 2015 # +101.99.64.150 +103.10.197.50 +103.240.91.7""" + self.proto.parseData() + self.assertEqual(len(self.proto.exitlist), 3) + + def test_ExitListProtocol_parseData_page_with_3_ips(self): + """ """ + self.proto.data = """ +101.99.64.150 +103.10.197.50 +103.240.91.7""" + self.proto.parseData() + self.assertEqual(len(self.proto.exitlist), 3) + + def test_ExitListProtocol_parseData_page_with_bad_ip(self): + """ """ + self.proto.data = """ +192.168.0.1 +127.0.0.1 +103.240.91.7""" + self.proto.parseData() + self.assertEqual(len(self.proto.exitlist), 1)

1 0

[bridgedb/master] Add mechanism/script for querying a remote TorBulkExitList.py.
by isis＠torproject.org 21 Mar '15

21 Mar '15

commit 5bf4a24c6715ab6c35ff4fae017c852e4b013749 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat Nov 2 07:15:33 2013 +0000 Add mechanism/script for querying a remote TorBulkExitList.py. * This uses a twisted.internet.protocol.ProcessProtocol to handle, process, and store within a ProxyCategory class, the data asynchronously as it is received from the connection, and uses SSL at the transport layer. --- lib/bridgedb/Bridges.py | 3 +- scripts/get-tor-exits | 199 +++++++++++++++++++++++++++++++++++++++++++++++ setup.py | 3 +- 3 files changed, 203 insertions(+), 2 deletions(-) diff --git a/lib/bridgedb/Bridges.py b/lib/bridgedb/Bridges.py index a3f50b4..fa718c0 100644 --- a/lib/bridgedb/Bridges.py +++ b/lib/bridgedb/Bridges.py @@ -64,7 +64,8 @@ def is_valid_ip(ip): :param str ip: A string representing an IPv4 or IPv6 address. """ - + logging.warn(PendingDeprecationWarning( + "Bridges.is_valid_ip() is replaced with parse.isIPAddress()")) # ipaddr does not treat "1.2" as a synonym for "0.0.1.2" try: ipaddr.IPAddress(ip) diff --git a/scripts/get-tor-exits b/scripts/get-tor-exits new file mode 100755 index 0000000..32b457d --- /dev/null +++ b/scripts/get-tor-exits @@ -0,0 +1,199 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: Isis Lovecruft 0xA3ADB67A2CDB8B35 <isis(a)torproject.org> +# please also see AUTHORS file +# :copyright: (c) 2013 Isis Lovecruft +# (c) 2007-2013, The Tor Project, Inc. +# (c) 2007-2013, all entities within the AUTHORS file +# :license: 3-clause BSD, see included LICENSE for information + +"""get-tor-exits -- Download the current list of Tor exit relays.""" + +from __future__ import print_function + +import os.path +import socket +import sys + +from ipaddr import IPAddress + +from OpenSSL import SSL + +from twisted.python import log +from twisted.python import usage +from twisted.names import client as dnsclient +from twisted.names import error as dnserror +from twisted.web import client +from twisted.internet import defer +from twisted.internet import protocol +from twisted.internet import reactor +from twisted.internet import ssl +from twisted.internet.error import TimeoutError + + +log.startLogging(sys.stderr) + + +def backupFile(filename): + """Move our old exit list file so that we don't append to it.""" + if os.path.isfile(filename): + backup = filename + '.bak' + log.msg("get-tor-exits: Moving old exit list file to %s" + % backup) + os.renames(filename, backup) + +def getSelfIPAddress(): + """Get external IP address, to ask check which relays can exit to here. + + TODO: This is blocking. Make it use Twisted. + """ + ip = s = None + try: + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + s.connect(('bridges.torproject.org', 443)) + name = s.getsockname()[0] + ip = IPAddress(name) + if ip.is_link_local or ip.is_private or ip.is_reserved: + name = s.getpeername()[0] + ip = IPAddress(name) + except ValueError as error: + log.err("get-tor-exits: A socket gave us something that wasn't an IP: %s" + % error) + except Exception as error: + log.err("get-tor-exits: Unhandled Exception: %s\n%s\n" + % (error.message, error)) + finally: + if s is not None: + s.close() + return ip.compressed + +def handleTimeout(failure): + """Handle a **failure** due to a timedout connection attempt.""" + if failure.type == TimeoutError: + log.msg("get-tor-exits: Could not download exitlist; connection timed out.") + failure.trap(TimeoutError) + +def writeToFile(response, filename): + log.msg("get-tor-exits: Downloading list of Tor exit relays.") + finished = defer.Deferred() + response.deliverBody(FileWriter(finished, filename)) + return finished + + +class GetTorExitsOptions(usage.Options): + """Options for this script""" + optFlags = [['stdout', 's', "Write results to stdout instead of file"]] + optParameters = [['file', 'f', 'exit-list', "File to write results to"], + ['address', 'a', '1.1.1.1', "Only exits which can reach this address"], + ['port', 'p', 443, "Only exits which can reach this port"]] + + +class FileWriter(protocol.Protocol): + """Read a downloaded file incrementally and write to file.""" + def __init__(self, finished, file): + """Create a FileWriter. + + .. warning:: We currently only handle the first 2MB of a file. Files + over 2MB will be truncated prematurely. + + :param finished: A :class:`~twisted.internet.defer.Deferred` which + will fire when another portion of the download is complete. + """ + self.finished = finished + self.remaining = 1024 * 1024 * 2 + self.fh = file + + def dataReceived(self, bytes): + """Write a portion of the download with ``bytes`` size to disk.""" + if self.remaining: + display = bytes[:self.remaining] + self.fh.write(display) + self.fh.flush() + self.remaining -= len(display) + + def connectionLost(self, reason): + """Called when the download is complete.""" + log.msg('get-tor-exits: Finished receiving exit list: %s' + % reason.getErrorMessage()) + self.finished.callback(None) + + +class WebClientContextFactory(ssl.ClientContextFactory): + """An HTTPS client.""" + + def getContext(self, hostname, port): + """Get this connection's OpenSSL context. + + By default, :api:`twisted.internet.ssl.ClientContextFactory` uses + ``OpenSSL.SSL.SSLv23_METHOD``, which allows SSLv2, SSLv3, and TLSv1, + then they disable SSLv2 in the + :api:`twisted.internet.ssl.ClientContextFactory.getContext` method. + + We disable SSLv3 also. + + :rtype: ``OpenSSL.SSL.Context`` + :returns: An OpenSSL context with options set. + """ + ctx = self._contextFactory(self.method) + ctx.set_options(SSL.OP_NO_SSLv2 ^ SSL.OP_NO_SSLv3) + return ctx + + +def main(filename=None, address=None, port=None): + + fh = filename + if filename: + if (not isinstance(filename, file)) and (filename is not sys.stdout): + fh = open(filename, 'w') + + if not address: + address = getSelfIPAddress() + + check = "https://check.torproject.org/cgi-bin/TorBulkExitList.py" + params = [] + + params.append('ip=%s' % address) + if port is not None: + params.append("port=%s" % port) + + check += '?' + '&'.join(params) + + log.msg("get-tor-exits: Requesting %s..." % check) + + contextFactory = WebClientContextFactory() + agent = client.Agent(reactor, contextFactory) + d = agent.request("GET", check) + d.addCallback(writeToFile, fh) + d.addErrback(handleTimeout) + d.addCallbacks(log.msg, log.err) + + if not reactor.running: + d.addCallback(lambda ignored: reactor.stop()) + reactor.run() + + if filename: + if (not isinstance(filename, file)) and (filename is not sys.stdout): + fh.flush() + fh.close() + + +if __name__ == "__main__": + try: + options = GetTorExitsOptions() + options.parseOptions() + except usage.UsageError as error: + log.err(error) + raise SystemExit(options.getUsage()) + + if options['stdout']: + filename = sys.stdout + elif options['file']: + filename = options['file'] + log.msg("get-tor-exits: Saving Tor exit relay list to file: '%s'" + % filename) + backupFile(filename) + + main(filename, options['address'], options['port']) diff --git a/setup.py b/setup.py index d6f9a20..1e293bd 100644 --- a/setup.py +++ b/setup.py @@ -282,7 +282,8 @@ setuptools.setup( 'bridgedb.email', 'bridgedb.parse', 'bridgedb.test'], - scripts=['scripts/bridgedb'], + scripts=['scripts/bridgedb', + 'scripts/get-tor-exits'], extras_require={'test': ["sure==1.2.2", "coverage==3.7.1", "leekspin==1.1.3"]},

1 0

[bridgedb/master] Ignore any downloaded Tor exit-list files.
by isis＠torproject.org 21 Mar '15

21 Mar '15

commit 6e5fa28f2a8012e43e1e9e61292fd365b1790163 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 5 00:57:07 2013 +0000 Ignore any downloaded Tor exit-list files. --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index c214b82..61d4b0a 100644 --- a/.gitignore +++ b/.gitignore @@ -101,3 +101,6 @@ _trial_temp.lock # default captcha cache directory: captchas/* + +# ignore any downloaded exit-list files: +*exit-list*

1 0

[bridgedb/master] Add utility bridgedb.runner.find().
by isis＠torproject.org 21 Mar '15

21 Mar '15

commit cd5961494295553efe64d265fae486860202fa89 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 5 01:01:07 2013 +0000 Add utility bridgedb.runner.find(). * ADD utility for finding executables by name, uses Twisted's procutils, which functions like doing ``which nameofprogram``. This is used to find additional scripts, like BridgeDB's ``get-tor-exits`` script. --- lib/bridgedb/runner.py | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/lib/bridgedb/runner.py b/lib/bridgedb/runner.py index d0eeaa9..9bc5d5b 100644 --- a/lib/bridgedb/runner.py +++ b/lib/bridgedb/runner.py @@ -16,8 +16,37 @@ from __future__ import print_function +import logging import sys +import os +from twisted.python import procutils + + +def find(filename): + """Find the executable ``filename``. + + :param string filename: The executable to search for. Must be in the + effective user ID's $PATH. + :rtype: string + :returns: The location of the executable, if found. Otherwise, returns + None. + """ + executable = None + + logging.debug("Searching for installed '%s'..." % filename) + which = procutils.which(filename, os.X_OK) + + if len(which) > 0: + for that in which: + if os.stat(that).st_uid == os.geteuid(): + executable = that + break + if not executable: + return None + + logging.debug("Found installed script at '%s'" % executable) + return executable def generateDescriptors(count=None, rundir=None): """Run a script which creates fake bridge descriptors for testing purposes.

1 0

[bridgedb/master] Merge branch 'fix/4405-tor-exit-check_2_r1' into develop
by isis＠torproject.org 21 Mar '15

21 Mar '15

commit a9732719b14b308f9e2e9af15bda65e2e3dd067a Merge: 303c3f9 14f8fd1 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Feb 10 02:35:59 2015 +0000 Merge branch 'fix/4405-tor-exit-check_2_r1' into develop .gitignore | 3 + bridgedb.conf | 23 ++ lib/bridgedb/Bridges.py | 3 +- lib/bridgedb/Main.py | 49 ++-- lib/bridgedb/proxy.py | 466 ++++++++++++++++++++++++++++++++ lib/bridgedb/runner.py | 29 ++ lib/bridgedb/test/test_proxy.py | 560 +++++++++++++++++++++++++++++++++++++++ scripts/get-tor-exits | 199 ++++++++++++++ setup.py | 3 +- 9 files changed, 1317 insertions(+), 18 deletions(-)

1 0

[bridgedb/master] Add unittests for bridgedb.qrcodes module.
by isis＠torproject.org 21 Mar '15

21 Mar '15

commit 129dc997439dfb4c218c482b1838078c51e41f00 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Feb 10 04:27:38 2015 +0000 Add unittests for bridgedb.qrcodes module. --- lib/bridgedb/qrcodes.py | 4 +- lib/bridgedb/test/test_qrcodes.py | 73 +++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+), 2 deletions(-) diff --git a/lib/bridgedb/qrcodes.py b/lib/bridgedb/qrcodes.py index 0b576e3..e09a705 100644 --- a/lib/bridgedb/qrcodes.py +++ b/lib/bridgedb/qrcodes.py @@ -18,7 +18,7 @@ import logging try: import qrcode -except ImportError: +except ImportError: # pragma: no cover qrcode = False logging.warn("Could not import Python qrcode module.") logging.debug(("You'll need the qrcode Python module for this to " @@ -72,6 +72,6 @@ def generateQR(bridgelines, imageFormat=u'JPEG', bridgeSchema=False): logging.error(str(error)) logging.debug(("It seems python-imaging doesn't understand how to " "save in the %s format.") % imageFormat) - except Exception as error: + except Exception as error: # pragma: no cover logging.error(("There was an error while attempting to generate the " "QRCode: %s") % str(error)) diff --git a/lib/bridgedb/test/test_qrcodes.py b/lib/bridgedb/test/test_qrcodes.py new file mode 100644 index 0000000..4abe8cc --- /dev/null +++ b/lib/bridgedb/test/test_qrcodes.py @@ -0,0 +1,73 @@ +# -*- coding: utf-8 -*- +#_____________________________________________________________________________ +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: Isis Lovecruft 0xA3ADB67A2CDB8B35 <isis(a)torproject.org> +# please also see AUTHORS file +# :copyright: (c) 2007-2015, The Tor Project, Inc. +# (c) 2014-2015, Isis Lovecruft +# :license: see LICENSE for licensing information +#_____________________________________________________________________________ + +"""Tests for :mod:`bridgedb.qrcodes`.""" + + +from twisted.trial import unittest + +from bridgedb import qrcodes + + +class GenerateQRTests(unittest.TestCase): + """Unittests for :func:`bridgedb.qrcodes.generateQR`.""" + + def setUp(self): + self.qrcodeModule = qrcodes.qrcode + bridgelines = [ + "obfs4 63.125.48.205:26573 441a151632806a3cc42adfecc2e6e823299db7b2 iat-mode=1 public-key=8d27ba37c5d810106b55f3fd6cdb35842007e88754184bfc0e6035f9bcede633 node-id=42d2a6ad49f93ab4b987b1a9e738425aacb8d2af", + "obfs4 103.111.131.45:43288 cb1362f8eaf5d3c2b6fad5da300f66c4197f23e5 iat-mode=0 public-key=c75cb66ae28d8ebc6eded002c28a8ba0d06d3a78c6b5cbf9b2ade051f0775ac4 node-id=4cd66dfabbd964f8c6c4414b07cdb45dae692e19", + "obfs4 17.194.28.21:5530 86ef8ab343e76bcd3c57ee32febe4482c98141c7 iat-mode=0 public-key=36ebe205bcdfc499a25e6923f4450fa8d48196ceb4fa0ce077d9d8ec4a36926d node-id=6b6277afcb65d33525545904e95c2fa240632660", + ] + self.bridgelines = '\n'.join(bridgelines) + + def tearDown(self): + """Replace the qrcode module to its original form.""" + qrcodes.qrcode = self.qrcodeModule + + def test_generateQR(self): + """Calling generateQR() should generate an image.""" + self.assertTrue(qrcodes.generateQR(self.bridgelines)) + + def test_generateQR_bad_bridgelines(self): + """Calling generateQR() with a bad type for the bridgelines should + return None. + """ + self.assertIsNone(qrcodes.generateQR(list())) + + def test_generateQR_no_bridgelines(self): + """Calling generateQR() without bridgelines should return None.""" + self.assertIsNone(qrcodes.generateQR("")) + + def test_generateQR_no_qrcode_module(self): + """Calling generateQR() without the qrcode module installed should + return None. + """ + qrcodes.qrcode = None + self.assertIsNone(qrcodes.generateQR(self.bridgelines)) + + def test_generateQR_bridgeSchema(self): + """Calling generateQR() with bridgeSchema=True should prepend + ``'bridge://`` to each of the QR encoded bridge lines. + """ + # If we were to install the python-qrtools Debian package, we'd be + # able to decode the resulting QRCode to check that it contains the + # 'bridge://' prefix for each bridge line… but that would add another + # Debian dependency just to unittest 5 lines of code. + # + # Instead: + self.assertTrue(qrcodes.generateQR(self.bridgelines, bridgeSchema=True)) + + def test_generateQR_save_nonexistent_format(self): + """Calling generateQR() with imageFormat=u'FOOBAR' should return None. + """ + self.assertIsNone(qrcodes.generateQR(self.bridgelines, imageFormat=u'FOOBAR'))

1 0