January 2014 - tor-commits - lists.torproject.org

[bridgedb/master] Merge branch 'fix/10446-geoip-module' into develop
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 3890690c50aca896db1b2cfd95e5e592885f1ee1 Merge: 2cf7c73 ac3ffd0 Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Dec 20 05:46:44 2013 +0000 Merge branch 'fix/10446-geoip-module' into develop README | 8 +++++--- lib/bridgedb/HTTPServer.py | 25 ++++++++++++++++++------- 2 files changed, 23 insertions(+), 10 deletions(-)

1 0

[bridgedb/master] Update README to reflect pygeoip module changes.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit ac3ffd023ca19a10a51f903eaacb64d5db126373 Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Dec 20 05:41:00 2013 +0000 Update README to reflect pygeoip module changes. --- README | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README b/README index b364469..0b7bb10 100644 --- a/README +++ b/README @@ -45,8 +45,9 @@ BridgeDB requires the following OS-level dependencies: - libgpgme11-dev - OpenSSL>=1.0.1e - [SQLite3](http://www.maxmind.com/app/python) - - [MaxMind GeoIP](https://www.maxmind.com/en/geolocation_landing) (packaged - in Debian as tor-geoipdb) + - [MaxMind GeoIP](https://www.maxmind.com/en/geolocation_landing) + - libgeoip-dev + - geoip-database - [python-setuptools](https://pypi.python.org/pypi/setuptools) As well as any Python dependencies in requirements.txt. @@ -58,7 +59,8 @@ BridgeDB should work with or without a Python virtualenv. - Install Python 2.7, and other OS-level dependencies. On Debian, you can do: sudo apt-get install build-essential openssl python python-dev \ - python-setuptools sqlite3 tor-geoip libgpgme11 libgpgme11-dev + python-setuptools sqlite3 libgpgme11 libgpgme11-dev libgeoip-dev \ + geoip-database - Install Pip 1.3.1 or later. Debian has this version, but if for some reason that or a newer version isn't available, the easiest way to install a newer

1 0

[bridgedb/master] Fix ParseNetworkStatusALineTests.test_missingPrefix unittest.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 7530035f3cc625723b308633334c28f78a4c6e06 Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Dec 20 06:03:43 2013 +0000 Fix ParseNetworkStatusALineTests.test_missingPrefix unittest. --- lib/bridgedb/test/test_parse_networkstatus.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/bridgedb/test/test_parse_networkstatus.py b/lib/bridgedb/test/test_parse_networkstatus.py index ddeacdf..7b0826c 100644 --- a/lib/bridgedb/test/test_parse_networkstatus.py +++ b/lib/bridgedb/test/test_parse_networkstatus.py @@ -331,9 +331,13 @@ class ParseNetworkStatusALineTests(unittest.TestCase): self.assertTrue(field is None) def test_missingPrefix(self): + """Changed to allow a missing 'a' prefix in branch + ``hotfix/9462B-netstatus-returns-None``. + """ self.line = '%s:1234' % self.oraddr fields = networkstatus.parseALine(self.line) - self.assertAllFieldsAreNone(fields) + this(ip).should.be.a(basestring) + this(port).should.be(None) def test_IPv4(self): self.line = 'a 48.32.199.45:9001'

1 0

[bridgedb/master] Merge branch 'hotfix/9462B-netstatus-returns-None' into develop
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit bf9f61d010b983d8b8d2b2ffb6ee5fffb707d4e1 Merge: 3890690 7530035 Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Dec 20 06:05:39 2013 +0000 Merge branch 'hotfix/9462B-netstatus-returns-None' into develop lib/bridgedb/test/test_parse_networkstatus.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)

1 0

[bridgedb/master] Fix field names in ParseNetworkStatusALineTests.test_missingPrefix().
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit e6af3ad1b241d99c1089cc089b416001ed274887 Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Dec 20 06:47:24 2013 +0000 Fix field names in ParseNetworkStatusALineTests.test_missingPrefix(). --- lib/bridgedb/test/test_parse_networkstatus.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bridgedb/test/test_parse_networkstatus.py b/lib/bridgedb/test/test_parse_networkstatus.py index 7b0826c..6d78d41 100644 --- a/lib/bridgedb/test/test_parse_networkstatus.py +++ b/lib/bridgedb/test/test_parse_networkstatus.py @@ -335,7 +335,7 @@ class ParseNetworkStatusALineTests(unittest.TestCase): ``hotfix/9462B-netstatus-returns-None``. """ self.line = '%s:1234' % self.oraddr - fields = networkstatus.parseALine(self.line) + ip, port = networkstatus.parseALine(self.line) this(ip).should.be.a(basestring) this(port).should.be(None)

1 0

[bridgedb/master] Merge branch 'hotfix/9462B-netstatus-returns-None' into develop
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 21d03615a3b2401ca04e981227e7d7605d3ca995 Merge: bf9f61d e6af3ad Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Dec 20 06:49:26 2013 +0000 Merge branch 'hotfix/9462B-netstatus-returns-None' into develop lib/bridgedb/test/test_parse_networkstatus.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

1 0

[bridgedb/master] Add trial runner/report class adapter for old unittests.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 127087eef72df5f052638bf8a770728800c15fb1 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Nov 21 07:45:04 2013 +0000 Add trial runner/report class adapter for old unittests. This adds a new unittest file, `lib/bridgedb/test/test_Tests.py`, which runs the old unittests in `lib/bridgedb/Tests.py` with a unittest report adapter that is compatible with twisted.trial. It uses the underlying SynchronousTestCase wrapper for the stdlib `unittest.TestCase` (which the tests in `lib/bridgedb/Tests.py` use). * ADDS a trial runner for BridgeDB's old unittests. * FIXES #9873 --- lib/bridgedb/test/test_Tests.py | 44 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/lib/bridgedb/test/test_Tests.py b/lib/bridgedb/test/test_Tests.py new file mode 100644 index 0000000..6801565 --- /dev/null +++ b/lib/bridgedb/test/test_Tests.py @@ -0,0 +1,44 @@ +# -*- coding: utf-8 -*- +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: Isis Lovecruft 0xA3ADB67A2CDB8B35 <isis(a)torproject.org> +# please also see AUTHORS file +# :copyright: (c) 2013, Isis Lovecruft +# (c) 2007-2013, The Tor Project, Inc. +# (c) 2007-2013, all entities within the AUTHORS file +# :license: 3-Clause BSD, see LICENSE for licensing information + +"""Class wrappers to adapt BridgeDB old unittests in :mod:`bridgedb.Tests` to +be compatible with the newer :mod:`twisted.trial` unittests in this directory. +""" + +from __future__ import print_function + +import logging +import warnings + +from twisted.trial import unittest + +from bridgedb import Tests +from bridgedb.Tests import EmailBridgeDistTests +from bridgedb.Tests import IPBridgeDistTests +from bridgedb.Tests import DictStorageTests +from bridgedb.Tests import SQLStorageTests +from bridgedb.Tests import ParseDescFileTests +from bridgedb.Tests import BridgeStabilityTests + +logging.warnings.filterwarnings('ignore', module="Tests") +pyunit = __import__('unittest') + + +class TrialAdaptedOldUnittests(unittest.TestCase): + """A wrapper around :mod:`bridgedb.Tests` to produce :mod:`~twisted.trial` + compatible output. + """ + + def test_allOldUnittests(self): + testSuite = Tests.testSuite() + testResult = pyunit.TestResult() + testSuite.run(testResult, debug=True) + return unittest.PyUnitResultAdapter(testResult)

1 0

[bridgedb/master] Change `Bridges.parseORAddressLine`→`networkstatus.parseALine` in Tests.py.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit a6466077a2b4f3c6bb9a991fcee387e1d8567ab4 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat Dec 7 04:03:41 2013 +0000 Change `Bridges.parseORAddressLine`→`networkstatus.parseALine` in Tests.py. --- lib/bridgedb/Tests.py | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/lib/bridgedb/Tests.py b/lib/bridgedb/Tests.py index a761d00..8e61454 100644 --- a/lib/bridgedb/Tests.py +++ b/lib/bridgedb/Tests.py @@ -30,6 +30,9 @@ from bridgedb.Filters import filterBridgesByTransport from bridgedb.Filters import filterBridgesByNotBlockedIn from bridgedb.Stability import BridgeHistory + +from bridgedb.parse import networkstatus + from math import log def suppressWarnings(): @@ -107,8 +110,8 @@ def fakeBridge(orport=8080, running=True, stable=True, or_addresses=False, if or_addresses: for i in xrange(8): - address,portlist = bridgedb.Bridges.parseORAddressLine( - "%s:%s" % (randomIPString(),randomPortSpec())) + address,portlist = networkstatus.parseALine( + "%s:%s" % (randomIPString(), randomPortSpec())) try: portlist.add(b.or_addresses[address]) except KeyError: @@ -133,8 +136,12 @@ def fakeBridge6(orport=8080, running=True, stable=True, or_addresses=False, if or_addresses: for i in xrange(0,8): - address,portlist = bridgedb.Bridges.parseORAddressLine( - "%s:%s" % (randomIPString(),randomPortSpec())) + address, portlist = networkstatus.parseALine("a %s:%s" % ( + randomIPString(), randomPortSpec())) + + self.assertIsNotNone(address, "fakeBridge6(): Got null address!") + self.assertIsNotNone(address, "fakeBridge6(): Got null portlist!") + try: portlist.add(b.or_addresses[address]) except KeyError: @@ -286,7 +293,7 @@ class IPBridgeDistTests(unittest.TestCase): bridgeFilterRules=[filterBridgesByIP6]) bridge = random.choice(bridges) bridge_line = bridge.getConfigLine(addressClass=ipaddr.IPv6Address) - address, portlist = bridgedb.Bridges.parseORAddressLine(bridge_line) + address, portlist = networkstatus.parseALine(bridge_line) assert type(address) is ipaddr.IPv6Address assert filterBridgesByIP6(random.choice(bridges)) @@ -302,7 +309,7 @@ class IPBridgeDistTests(unittest.TestCase): bridgeFilterRules=[filterBridgesByIP4]) bridge = random.choice(bridges) bridge_line = bridge.getConfigLine(addressClass=ipaddr.IPv4Address) - address, portlist = bridgedb.Bridges.parseORAddressLine(bridge_line) + address, portlist = networkstatus.parseALine(bridge_line) assert type(address) is ipaddr.IPv4Address assert filterBridgesByIP4(random.choice(bridges)) @@ -322,10 +329,10 @@ class IPBridgeDistTests(unittest.TestCase): t = bridges.pop() assert filterBridgesByIP4(t) assert filterBridgesByIP6(t) - address, portlist = bridgedb.Bridges.parseORAddressLine( + address, portlist = networkstatus.parseALine( t.getConfigLine(addressClass=ipaddr.IPv4Address)) assert type(address) is ipaddr.IPv4Address - address, portlist = bridgedb.Bridges.parseORAddressLine( + address, portlist = networkstatus.parseALine( t.getConfigLine(addressClass=ipaddr.IPv6Address)) assert type(address) is ipaddr.IPv6Address

1 0

[bridgedb/master] Add test.deprecated module for regression testing with original code.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 0258e7adbbc9e028609f59d93994038e8311eb2f Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Dec 17 00:44:54 2013 +0000 Add test.deprecated module for regression testing with original code. * ADD `lib/bridgedb/test/deprecated.py` which contains old classes and functions (before refactoring), so that they can be `twisted.python.monkey.MonkeyPatched`ed into `bridgedb.test.test_Tests` (and perhaps other tests). This causes the unittests to function as regression tests when code is refactored, so that (hopefully) we can find regression bugs faster. In the future, we *should* be able to use `twisted.python.deprecate` to deprecate function and classes in place (i.e. without removing them from their original files). I didn't think to do this until long after these had been removed, and re-adding deprecated code back into BridgeDB's modules seems like it would be confusing. That said, it is *also* fine, in the future, to actually move deprecated code into `bridgedb.test.deprecated` (just make sure to change all references to the original in the codebase!). However, doing this *does* mean you'll need to do quite a bit more complicated monkeypatching in `bridgedb.test.test_Tests.py`, or any other future regression testsuites. --- lib/bridgedb/test/deprecated.py | 149 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+) diff --git a/lib/bridgedb/test/deprecated.py b/lib/bridgedb/test/deprecated.py new file mode 100644 index 0000000..93f28ba --- /dev/null +++ b/lib/bridgedb/test/deprecated.py @@ -0,0 +1,149 @@ +# -*- coding: utf-8 -*- +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: Isis Lovecruft 0xA3ADB67A2CDB8B35 <isis(a)torproject.org> +# please also see AUTHORS file +# :copyright: (c) 2013, Isis Lovecruft +# (c) 2007-2013, The Tor Project, Inc. +# (c) 2007-2013, all entities within the AUTHORS file +# :license: 3-Clause BSD, see LICENSE for licensing information + +"""deprecated ― functions and classes which have been removed from the +production code but are kept in order to be used in regression testing. +""" + +import ipaddr +import re + +from twisted.python import deprecate +from twisted.python.versions import Version + + +PORTSPEC_LENGTH = 16 + +re_ipv6 = re.compile("\[([a-fA-F0-9:]+)\]:(.*$)") +re_ipv4 = re.compile("((?:\d{1,3}\.?){4}):(.*$)") + +deprecate.deprecatedModuleAttribute( + Version('bridgedb', 0, 0, 1), + "Removed due to 'bridgedb.Bridges.PortList' being moved to "\ + "'bridgedb.parse.addr.PortList.", + "bridgedb.Bridges", + "PORTSPEC_LENGTH") + +deprecate.deprecatedModuleAttribute( + Version('bridgedb', 0, 0, 1), + "Attribute 'bridgedb.Bridges.re_ipv4' was removed due to "\ + "'bridgedb.Bridges.parseORAddressLine' moving to "\ + "'bridgedb.parse.networkstatus.", + "bridgedb.Bridges", + "re_ipv4") + +deprecate.deprecatedModuleAttribute( + Version('bridgedb', 0, 0, 1), + "Attribute 'bridgedb.Bridges.re_ipv6' was removed due to "\ + "'bridgedb.Bridges.parseORAddressLine' moving to "\ + "'bridgedb.parse.networkstatus.", + "bridgedb.Bridges", + "re_ipv6") + + +(a)deprecate.deprecated( + Version('bridgedb', 0, 0, 1), + replacement='bridgedb.parse.networkstatus.parseALine') +def parseORAddressLine(line): + """Deprecated :func:`bridgedb.Bridges.parseORAddressLine`, removed in + bridgedb-0.1.0, in commit 1f111e5. + + This function and the newer parsers from :mod:`bridgedb.parse.netstatus` + are alternately :meth:`~twisted.python.monkey.MonkeyPatcher.patch`ed into + the :mod:`old unittests <bridgedb.Tests>`, so that the later functions as + a suite of regression tests. + """ + address = None + portlist = None + # try regexp to discover ip version + for regex in [re_ipv4, re_ipv6]: + m = regex.match(line) + if m: + # get an address and portspec, or raise ParseError + try: + address = ipaddr.IPAddress(m.group(1)) + portlist = PortList(m.group(2)) + except (IndexError, ValueError): raise ParseORAddressError(line) + + # return a valid address, portlist or raise ParseORAddressError + if address and portlist and len(portlist): return address,portlist + raise ParseORAddressError(line) + + +(a)deprecate.deprecated( + Version('bridgedb', 0, 0, 1), + replacement='bridgedb.parse.networkstatus.NetworkstatusParsingError') +class ParseORAddressError(Exception): + def __init__(self, line=None): + msg = "Invalid or-address line" + if line: + msg += ": {0}".format(line) + Exception.__init__(self, msg) + + +(a)deprecate.deprecated( + Version('bridgedb', 0, 0, 1), + replacement='bridgedb.parse.addr.PortList') +class PortList: + """Deprecated :class:`bridgedb.Bridges.PortList`, replaced in + bridgedb-0.1.0, in commit 1f111e5, by + :class:`bridgedb.parse.addr.PortList`. + + This class and the newer class from :mod:`bridgedb.parse.addr` are + alternately :meth:`~twisted.python.monkey.MonkeyPatcher.patch`ed into the + :mod:`old unittests <bridgedb.Tests>`, so that the later functions as a + suite of regression tests. + """ + def __init__(self, *args, **kwargs): + self.ports = set() + self.add(*args) + + def _sanitycheck(self, val): + #XXX: if debug=False this is disabled. bad! + assert type(val) is int + assert(0 < val <= 65535) + + def __contains__(self, val1): + return val1 in self.ports + + def add(self, *args): + PORTSPEC_LEN = 16 + for arg in args: + try: + if type(arg) is str: + ports = set([int(p) for p in arg.split(',')][:PORTSPEC_LEN]) + [self._sanitycheck(p) for p in ports] + self.ports.update(ports) + if type(arg) is int: + self._sanitycheck(arg) + self.ports.update([arg]) + if type(arg) is PortList: + self.add(list(arg.ports)) + except AssertionError: raise ValueError + except ValueError: raise + + def __iter__(self): + return self.ports.__iter__() + + def __str__(self): + s = "" + for p in self.ports: + s += "".join(",%s"%p) + return s.lstrip(",") + + def __repr__(self): + return "PortList('%s')" % self.__str__() + + def __len__(self): + return len(self.ports) + + def __getitem__(self, x): + return list(self.ports)[x]

1 0

[bridgedb/master] Add `_trial_temp.lock` to .gitignore.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 0b29f7a461bd1dc2ee47f90ae7c3ee6f5f334c35 Author: Isis Lovecruft <isis(a)torproject.org> Date: Fri Dec 20 02:29:36 2013 +0000 Add `_trial_temp.lock` to .gitignore. --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 7bfc320..6cd8738 100644 --- a/.gitignore +++ b/.gitignore @@ -96,3 +96,4 @@ gnupghome/* # trial test run directory: _trial_temp/* +_trial_temp.lock

1 0