January 2014 - tor-commits - lists.torproject.org

[bridgedb/master] Fix Python2.6 CI failures due to changed twisted.trial.unittest.TestCase API.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit b5b5fe2e2365ecce5627f652f441a25c590ef245 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 03:37:48 2013 +0000 Fix Python2.6 CI failures due to changed twisted.trial.unittest.TestCase API. --- lib/bridgedb/test/test_persistentSaveAndLoad.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/bridgedb/test/test_persistentSaveAndLoad.py b/lib/bridgedb/test/test_persistentSaveAndLoad.py index 10467ce..73d0f34 100644 --- a/lib/bridgedb/test/test_persistentSaveAndLoad.py +++ b/lib/bridgedb/test/test_persistentSaveAndLoad.py @@ -51,7 +51,9 @@ class StateSaveAndLoadTests(unittest.TestCase): self.state.statefile = os.path.abspath('bridgedb.state') def loadedStateAssertions(self, loadedState): - self.assertIsNotNone(loadedState) + # For some reason, twisted.trial.unittest.TestCase in Python2.6 + # doesn't have an 'assertIsNotNone' attribute... + self.assertTrue(loadedState is not None) self.assertIsInstance(loadedState, persistent.State) self.assertNotIdentical(self.state, loadedState) self.assertNotEqual(self.state, loadedState)

1 0

[bridgedb/master] Fix another CI failure due to twisted.trial API changes in Python2.6.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 45fee35f74efacd5e4d21f460a06d0af07b431e4 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 03:49:07 2013 +0000 Fix another CI failure due to twisted.trial API changes in Python2.6. --- lib/bridgedb/test/test_persistentSaveAndLoad.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/bridgedb/test/test_persistentSaveAndLoad.py b/lib/bridgedb/test/test_persistentSaveAndLoad.py index 73d0f34..7919df3 100644 --- a/lib/bridgedb/test/test_persistentSaveAndLoad.py +++ b/lib/bridgedb/test/test_persistentSaveAndLoad.py @@ -57,8 +57,10 @@ class StateSaveAndLoadTests(unittest.TestCase): self.assertIsInstance(loadedState, persistent.State) self.assertNotIdentical(self.state, loadedState) self.assertNotEqual(self.state, loadedState) - self.assertItemsEqual(self.state.__dict__.keys(), - loadedState.__dict__.keys()) + # For some reason, twisted.trial.unittest.TestCase in Python2.6 + # doesn't have an 'assertItemsEqual' attribute... + self.assertEqual(self.state.__dict__.keys().sort(), + loadedState.__dict__.keys().sort()) def savedStateAssertions(self, savedStatefile=None): self.assertTrue(os.path.isfile(str(self.state.statefile)))

1 0

[bridgedb/master] Add sitecustomize.py for running parallelized coverage tests.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 00b4ed73995f52ff054be08f5c26b8796aed17aa Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 03:51:12 2013 +0000 Add sitecustomize.py for running parallelized coverage tests. --- lib/bridgedb/sitecustomize.py | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/lib/bridgedb/sitecustomize.py b/lib/bridgedb/sitecustomize.py new file mode 100644 index 0000000..264e5e1 --- /dev/null +++ b/lib/bridgedb/sitecustomize.py @@ -0,0 +1,34 @@ +# -*- coding: utf-8 -*- +"""sitecustomize ― Handles potential loading of extra code when Python starts. + +**Module Usage:** + +This is normally (this seems not to work with Twisted, for as-of-yet unknown +reasons) useful for using :mod:`coverage` to measure code execution in spawned +subprocesses in the following way: + + 1. Set the environment variable ``COVERAGE_PROCESS_START`` to the absolute + path of the coverage config file. If you are in the top-level of the + bridgedb repo, do: + + $ export COVERAGE_PROCESS_START="${PWD}/.coveragerc" + + 2. In that coverage config file, in the ``[run]`` section, set + ``parallel = True``. + + 3. Run coverage. From the top-level of the bridgedb repo, try doing: + + $ make reinstall && \ + coverage run $(which trial) ./lib/bridgedb/test/test_* && \ + coverage combine && coverage report && coverage html + +If ``COVERAGE_PROCESS_START`` is not set, this code does nothing, +``[run] parallel`` should be set to ``False``, and coverage can be run by +leaving out the ``coverage combine`` portion of the above command. + +To view the output HTML coverage data, open +``path/to/bridgedb_repo/doc/coverage_html/index.html`` in a browser. +""" + +import coverage +coverage.process_startup()

1 0

[bridgedb/master] I *really* hate Python2.6.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit e941e075d7c29eddc6803b2c5c7b98ff52610246 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 03:58:02 2013 +0000 I *really* hate Python2.6. The argparse module wasn't a builtin in Python2.6, so we have to add it to requirements.txt otherwise the CI builds fail. --- requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/requirements.txt b/requirements.txt index 3935c59..72a322c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,3 +10,4 @@ pygpgme==0.3 recaptcha==1.0rc1 recaptcha-client==1.0.6 zope.interface>=4.0.5 +argparse==1.2.1

1 0

[bridgedb/master] Add integration tests which start the BridgeDB servers through the cli scripts.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 87360c1064f268c52e94b45983e6c902b236c565 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 03:52:31 2013 +0000 Add integration tests which start the BridgeDB servers through the cli scripts. --- lib/bridgedb/test/bridgedb.conf | 55 ++++++++++++++++++ lib/bridgedb/test/test_bridgedb.py | 108 ++++++++++++++++++++++++++++++++++++ 2 files changed, 163 insertions(+) diff --git a/lib/bridgedb/test/bridgedb.conf b/lib/bridgedb/test/bridgedb.conf new file mode 100644 index 0000000..ce12096 --- /dev/null +++ b/lib/bridgedb/test/bridgedb.conf @@ -0,0 +1,55 @@ +BRIDGE_FILES = ["bridge-descriptors"] +EXTRA_INFO_FILES = ["cached-extrainfo", "cached-extrainfo.new"] +STATUS_FILE = "networkstatus-bridges" +HTTPS_CERT_FILE="cert" +HTTPS_KEY_FILE="privkey.pem" +LOGFILE = "bridgedb.log" +ASSIGNMENTS_FILE = "assignments.log" +PIDFILE = "bridgedb.pid" +DB_FILE = "bridgedist.db" +DB_LOG_FILE = "bridgedist.log" +MASTER_KEY_FILE = "secret_key" +PROXY_LIST_FILES = [] +LOGLEVEL = "DEBUG" +SAFELOGGING = True +LOGFILE_COUNT = 5 +LOGFILE_ROTATE_SIZE = 10000000 +BRIDGE_PURPOSE = "bridge" +N_IP_CLUSTERS = 4 +FORCE_PORTS = [(443, 1)] +FORCE_FLAGS = [("Stable", 1)] +HTTPS_DIST = True +HTTPS_BIND_IP = '127.0.0.1' +HTTPS_PORT = 6789 +HTTPS_N_BRIDGES_PER_ANSWER = 3 +HTTPS_INCLUDE_FINGERPRINTS = True +HTTPS_USE_IP_FROM_FORWARDED_HEADER = False +HTTP_UNENCRYPTED_BIND_IP = None +HTTP_UNENCRYPTED_PORT = None +HTTP_USE_IP_FROM_FORWARDED_HEADER = False +HTTP_N_BRIDGES_PER_ANSWER = 3 +EMAIL_DIST = True +EMAIL_FROM_ADDR = "bridges(a)torproject.org" +EMAIL_SMTP_FROM_ADDR = "bridges(a)torproject.org" +EMAIL_SMTP_HOST = "127.0.0.1" +EMAIL_SMTP_PORT = 25 +EMAIL_USERNAME = "bridges" +EMAIL_DOMAINS = ["gmail.com", "yahoo.com"] +EMAIL_DOMAIN_MAP = { "mail.google.com" : "gmail.com", + "googlemail.com" : "gmail.com"} +EMAIL_DOMAIN_RULES = { 'gmail.com' : ["ignore_dots", "dkim"], + 'yahoo.com' : ["dkim"]} +EMAIL_RESTRICT_IPS = [] +EMAIL_BIND_IP="127.0.0.1" +EMAIL_PORT=6725 +EMAIL_N_BRIDGES_PER_ANSWER=3 +EMAIL_INCLUDE_FINGERPRINTS = True +EMAIL_GPG_SIGNING_ENABLED = True +EMAIL_GPG_SIGNING_KEY = 'TESTING.subkeys.sec' +HTTPS_SHARE = 10 +EMAIL_SHARE = 10 +RESERVED_SHARE = 2 +FILE_BUCKETS = {} +RECAPTCHA_ENABLED = False +RECAPTCHA_PUB_KEY = '' +RECAPTCHA_PRIV_KEY = '' diff --git a/lib/bridgedb/test/test_bridgedb.py b/lib/bridgedb/test/test_bridgedb.py new file mode 100644 index 0000000..1c5b649 --- /dev/null +++ b/lib/bridgedb/test/test_bridgedb.py @@ -0,0 +1,108 @@ +# -*- coding: utf-8 -*- +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: Isis Lovecruft 0xA3ADB67A2CDB8B35 <isis(a)torproject.org> +# please also see AUTHORS file +# :copyright: (c) 2013, Isis Lovecruft +# (c) 2007-2013, The Tor Project, Inc. +# (c) 2007-2013, all entities within the AUTHORS file +# :license: 3-Clause BSD, see LICENSE for licensing information + +"""Unittests for the `bridgedb` commandline script.""" + +from __future__ import print_function + +import os +import shutil +import signal +import time + +from functools import wraps +from os.path import join as pjoin +from subprocess import Popen, PIPE + +from twisted.python import log +from twisted.python.procutils import which +from twisted.trial import unittest + + +def fileCheckDecorator(func): + @wraps(func) + def wrapper(self, src, dst, description): + print("Copying %s:\n %r\n\t\t↓ ↓ ↓\n %r\n" + % (str(description), src, dst)) + self.assertTrue( + os.path.isfile(src), + "Couldn't find original %s file: %r" % (str(description), src)) + func(self, src, dst, description) + self.assertTrue( + os.path.isfile(dst), + "Couldn't find new %s file: %r" % (str(description), dst)) + return wrapper + +class BridgeDBCliTest(unittest.TestCase): + """Test the `bridgedb` command.""" + + @fileCheckDecorator + def doCopyFile(self, src, dst, description=None): + shutil.copy(src, dst) + + @fileCheckDecorator + def doMoveFile(self, src, dst, description=None): + shutil.move(src, dst) + + def test_bridgedb_commands(self): + print('') + here = os.getcwd() + runDir = pjoin(here, 'rundir') + topDir = here.rstrip('_trial_temp') + scriptsDir = pjoin(topDir, 'scripts') + + # Create the lowest directory we need, and all its parents: + os.makedirs(os.path.join(runDir, 'gnupghome')) + + conf = pjoin(topDir, 'bridgedb.conf') + confMoved = pjoin(runDir, 'bridgedb.conf') + gpgFile = pjoin(topDir, 'gnupghome', 'TESTING.subkeys.sec') + gpgMoved = pjoin(runDir, 'gnupghome', 'TESTING.subkeys.sec') + certFile = pjoin(scriptsDir, 'cert') + certMoved = pjoin(runDir, 'cert') + keyFile = pjoin(scriptsDir, 'privkey.pem') + keyMoved = pjoin(runDir, 'privkey.pem') + + makeSSLCertScript = os.path.join(scriptsDir, 'make-ssl-cert') + bridgedbScript = which('bridgedb') # this returns a list + + self.doCopyFile(conf, confMoved, 'config') + self.doCopyFile(gpgFile, gpgMoved, 'GPG test key') + print("Running subcommands from directory:\n %r" % runDir) + print("Running %r..." % makeSSLCertScript) + makeSSLCertProcess = Popen(makeSSLCertScript) + makeSSLCertProcess.wait() + self.doMoveFile(certFile, certMoved, 'certificate') + self.doMoveFile(keyFile, keyMoved, 'SSL private key') + + self.assertTrue(os.path.isfile(bridgedbScript[0]), + "Couldn't find bridgedb script %r" % bridgedbScript[0]) + bridgedbScript = bridgedbScript[0] + print("Running bridgedb script %r..." % bridgedbScript) + + print("Running `bridgedb mock' to generate mock bridge descriptors...") + mockProc = Popen([bridgedbScript, 'mock', + '-n', '50', + '-r', runDir]) + mockProcCode = mockProc.wait() + #mockProcCode = os.system("%s mock -n 50 -r %s -c bridgedb.conf" + # % (bridgedbScript, runDir)) + print("`bridgedb mock' exited with status code %d" % int(mockProcCode)) + + print("Running `bridgedb' to test server startups...") + bridgedbProc = Popen([bridgedbScript, '-r', runDir]) + #bridgedbProcCode = os.system("%s -r %s -c bridgedb.conf" + # % (bridgedbScript, runDir)) + time.sleep(30) + bridgedbProc.send_signal(signal.SIGINT) + bridgedbProcCode = bridgedbProc.wait() + print("`bridgedb' exited with status code %d" % int(bridgedbProcCode)) + self.assertEqual(bridgedbProcCode, 0)

1 0

[bridgedb/master] Add unittests for the bridgedb.crypto module.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit c834082c59abe6ae6d2e065e1a5afac2d399a612 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 03:53:56 2013 +0000 Add unittests for the bridgedb.crypto module. --- lib/bridgedb/test/test_crypto.py | 57 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/lib/bridgedb/test/test_crypto.py b/lib/bridgedb/test/test_crypto.py new file mode 100644 index 0000000..2e6894e --- /dev/null +++ b/lib/bridgedb/test/test_crypto.py @@ -0,0 +1,57 @@ +# -*- coding: utf-8 -*- +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: Isis Lovecruft 0xA3ADB67A2CDB8B35 <isis(a)torproject.org> +# please also see AUTHORS file +# :copyright: (c) 2013, Isis Lovecruft +# (c) 2007-2013, The Tor Project, Inc. +# (c) 2007-2013, all entities within the AUTHORS file +# :license: 3-Clause BSD, see LICENSE for licensing information + +"""Unittests for :mod:`bridgedb.crypto`.""" + +from __future__ import print_function +from __future__ import unicode_literals + +import os + +from twisted.trial import unittest +from bridgedb import crypto + + +SEKRIT_KEY = b'v\x16Xm\xfc\x1b}\x063\x85\xaa\xa5\xf9\xad\x18\xb2P\x93\xc6k\xf9' +SEKRIT_KEY += b'\x8bI\xd9\xb8xw\xf5\xec\x1b\x7f\xa8' + + +class CryptoTest(unittest.TestCase): + + def test_getKey_nokey(self): + """Test retrieving the secret_key from an empty file.""" + filename = os.path.join(os.getcwd(), 'sekrit') + key = crypto.getKey(filename) + self.failUnlessIsInstance(key, basestring, + "key isn't a string! type=%r" % type(key)) + + def test_getKey_tmpfile(self): + """Test retrieving the secret_key from a new tmpfile.""" + filename = self.mktemp() + key = crypto.getKey(filename) + self.failUnlessIsInstance(key, basestring, + "key isn't a string! type=%r" % type(key)) + + def test_getKey_keyexists(self): + """Write the example key to a file and test reading it back.""" + filename = self.mktemp() + with open(filename, 'wb') as fh: + fh.write(SEKRIT_KEY) + fh.flush() + + key = crypto.getKey(filename) + self.failUnlessIsInstance(key, basestring, + "key isn't a string! type=%r" % type(key)) + self.assertEqual(SEKRIT_KEY, key, + """The example key and the one read from file differ! + key (in hex): %s + SEKRIT_KEY (in hex): %s""" + % (key.encode('hex'), SEKRIT_KEY.encode('hex')))

1 0

[bridgedb/master] Make the IRC Travis CI bot a bit less noisy.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 2d66c9a3c9414e99d3f8249161828bb9bdcb68ad Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 04:34:48 2013 +0000 Make the IRC Travis CI bot a bit less noisy. --- .travis.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 7a47b36..5955a13 100644 --- a/.travis.yml +++ b/.travis.yml @@ -6,7 +6,11 @@ notifications: - "irc.oftc.net#tor-bots" on_success: always on_failure: always - use_notice: true + #use_notice: true + template: + - "%{repository}#%{build_number} (%{branch} - %{commit} : %{author}): %{message}" + #- "Change view : %{compare_url}" + - "Build details : %{build_url}" email: recipients: - isis(a)torproject.org

1 0

[bridgedb/master] Fix unittest fail for test_bridgedb due to changing where make-ssl-cert puts files.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 0d4989e212b070ccfe17bb9ba907a7010f51d155 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 04:24:41 2013 +0000 Fix unittest fail for test_bridgedb due to changing where make-ssl-cert puts files. --- lib/bridgedb/test/test_bridgedb.py | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/lib/bridgedb/test/test_bridgedb.py b/lib/bridgedb/test/test_bridgedb.py index 1c5b649..a7093a6 100644 --- a/lib/bridgedb/test/test_bridgedb.py +++ b/lib/bridgedb/test/test_bridgedb.py @@ -66,9 +66,9 @@ class BridgeDBCliTest(unittest.TestCase): confMoved = pjoin(runDir, 'bridgedb.conf') gpgFile = pjoin(topDir, 'gnupghome', 'TESTING.subkeys.sec') gpgMoved = pjoin(runDir, 'gnupghome', 'TESTING.subkeys.sec') - certFile = pjoin(scriptsDir, 'cert') + certFile = pjoin(topDir, 'cert') certMoved = pjoin(runDir, 'cert') - keyFile = pjoin(scriptsDir, 'privkey.pem') + keyFile = pjoin(topDir, 'privkey.pem') keyMoved = pjoin(runDir, 'privkey.pem') makeSSLCertScript = os.path.join(scriptsDir, 'make-ssl-cert') @@ -93,14 +93,10 @@ class BridgeDBCliTest(unittest.TestCase): '-n', '50', '-r', runDir]) mockProcCode = mockProc.wait() - #mockProcCode = os.system("%s mock -n 50 -r %s -c bridgedb.conf" - # % (bridgedbScript, runDir)) print("`bridgedb mock' exited with status code %d" % int(mockProcCode)) print("Running `bridgedb' to test server startups...") bridgedbProc = Popen([bridgedbScript, '-r', runDir]) - #bridgedbProcCode = os.system("%s -r %s -c bridgedb.conf" - # % (bridgedbScript, runDir)) time.sleep(30) bridgedbProc.send_signal(signal.SIGINT) bridgedbProcCode = bridgedbProc.wait()

1 0

[bridgedb/master] Update .travis.tml to use new command syntax.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 8851ec604b8a09e7282b9d4fb1dbbf2a4f5be1a3 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat Oct 26 10:51:32 2013 +0000 Update .travis.tml to use new command syntax. --- .travis.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.travis.yml b/.travis.yml index e9e6f31..c94692a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -42,7 +42,7 @@ script: - cd run - gen_bridge_descriptors 2000 - ../scripts/make-ssl-cert - - ../scripts/executioner 60 bridgedb -c bridgedb.conf -# add this when #9199's (or some other trial-based) tests are merged: -# - bridgedb test -n 2000 -# - echo "Build successful." + - ../scripts/executioner 60 bridgedb -r . -c bridgedb.conf + # add this when #9199's (or some other trial-based) tests are merged: + #- bridgedb test -n 2000 + - echo "Build successful."

1 0

[bridgedb/master] Add placeholder file for lib/bridgedb/test package directory.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit a86f65f588a9eb9b7ab497c85cf93f05b116491e Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat Oct 26 11:30:27 2013 +0000 Add placeholder file for lib/bridgedb/test package directory. --- 0 files changed diff --git a/lib/bridgedb/test/README b/lib/bridgedb/test/README new file mode 100644 index 0000000..e69de29

1 0