August 2014 - tor-commits - lists.torproject.org

[bridgedb/develop] Add --no-use-wheel when pip installing Travis CI dependencies.
by isis＠torproject.org 13 Aug '14

13 Aug '14

commit e01aedc7ebe9cd767c87e2f0ee1ad4fd1cf4e21f Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 7 06:16:34 2014 +0000 Add --no-use-wheel when pip installing Travis CI dependencies. --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 9066fc7..e664a26 100644 --- a/.travis.yml +++ b/.travis.yml @@ -42,7 +42,7 @@ before_install: install: - sudo apt-get install -qq --no-install-suggests --no-install-recommends build-essential openssl sqlite3 libgpgme11 libgpgme11-dev python-dev python-setuptools - - pip install -q -r .travis.requirements.txt + - pip install -q --no-use-wheel -r .travis.requirements.txt - pip install -q --no-use-wheel Twisted==$TWISTED_VERSION pyOpenSSL==$PYOPENSSL_VERSION - make install

1 0

[bridgedb/develop] Move several test dependencies from .travis.yml to .travis.requirements.txt.
by isis＠torproject.org 13 Aug '14

13 Aug '14

commit 3c9b8732a5c9119837212056306deb4353d6e7bd Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 7 06:06:31 2014 +0000 Move several test dependencies from .travis.yml to .travis.requirements.txt. --- .travis.requirements.txt | 24 ++++++++++++++++++++++++ .travis.yml | 7 ------- 2 files changed, 24 insertions(+), 7 deletions(-) diff --git a/.travis.requirements.txt b/.travis.requirements.txt index e018631..428438a 100644 --- a/.travis.requirements.txt +++ b/.travis.requirements.txt @@ -1,3 +1,27 @@ +# +# This is a pip requirements.txt file for running BridgeDB's tests on Travis +# See https://travis-ci.org/ and the .travis.yml file for more info. Please +# note that there may be additional dependencies not listed here but specified +# with their version numbers in .travis.yml in order to submit test runs which +# use multiple different dependency versions. +# +#------------------------------------------------------------------------------ +# +# To run install these dependencies and run the tests, do: +# +# $ pip install -r .test.requirements.txt +# $ make coverage +# +#------------------------------------------------------------------------------ +coverage==3.7.1 +coveralls==0.4.2 +leekspin==0.1.4 +mechanize==0.2.5 +pep8==1.5.7 +# pylint must be pinned until pylint bug #203 is fixed. See +# https://bitbucket.org/logilab/pylint/issue/203/importing-namespace-packages… +pylint==1.1.0 +sure==1.2.2 Babel==0.9.6 BeautifulSoup==3.2.1 Mako==0.8.1 diff --git a/.travis.yml b/.travis.yml index 7374fa7..9066fc7 100644 --- a/.travis.yml +++ b/.travis.yml @@ -26,11 +26,6 @@ addons: - bridges.torproject.org env: - global: - - LEEKSPIN_VERSION=0.1.4 - - COVERAGE_VERSION=3.7.1 - - COVERALLS_VERSION=0.4.2 - - SURE_VERSION=1.2.2 matrix: - TWISTED_VERSION=13.2.0 PYOPENSSL_VERSION=0.13.1 @@ -48,8 +43,6 @@ before_install: install: - sudo apt-get install -qq --no-install-suggests --no-install-recommends build-essential openssl sqlite3 libgpgme11 libgpgme11-dev python-dev python-setuptools - pip install -q -r .travis.requirements.txt - - pip install -q --no-use-wheel leekspin==$LEEKSPIN_VERSION sure==$SURE_VERSION - - pip install -q --no-use-wheel coverage==$COVERAGE_VERSION coveralls==$COVERALLS_VERSION - pip install -q --no-use-wheel Twisted==$TWISTED_VERSION pyOpenSSL==$PYOPENSSL_VERSION - make install

1 0

[bridgedb/develop] Also copy gnupghome/ to run/ in .travis.yml 'before_script'.
by isis＠torproject.org 13 Aug '14

13 Aug '14

commit 8d47543523f9120f8ea0fd93612782f50fcbe941 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 7 07:52:08 2014 +0000 Also copy gnupghome/ to run/ in .travis.yml 'before_script'. --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index d8edafb..a3771b0 100644 --- a/.travis.yml +++ b/.travis.yml @@ -49,7 +49,7 @@ install: # Start a BridgeDB instance before running the tests: before_script: - mkdir run - - cp -R -t run bridgedb.conf captchas + - cp -R -t run bridgedb.conf captchas gnupghome - leekspin -n 100 - cp -t run networkstatus-bridges cached-extrainfo* bridge-descriptors - ./scripts/make-ssl-cert

1 0

[bridgedb/develop] Copy captchas/ into run/ during 'before_script' .travis.yml section.
by isis＠torproject.org 13 Aug '14

13 Aug '14

commit c6ec7e51792029a28f1c31809e35a1c6f7e47bde Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 7 06:27:56 2014 +0000 Copy captchas/ into run/ during 'before_script' .travis.yml section. --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index e664a26..d8edafb 100644 --- a/.travis.yml +++ b/.travis.yml @@ -49,7 +49,7 @@ install: # Start a BridgeDB instance before running the tests: before_script: - mkdir run - - cp -t run bridgedb.conf + - cp -R -t run bridgedb.conf captchas - leekspin -n 100 - cp -t run networkstatus-bridges cached-extrainfo* bridge-descriptors - ./scripts/make-ssl-cert

1 0

[bridgedb/develop] Split unittests in test_bridgedb.py; refactor to use already running server.
by isis＠torproject.org 13 Aug '14

13 Aug '14

commit 9873fad412081d35e508de5c8dd095a17fd921cb Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 7 07:47:05 2014 +0000 Split unittests in test_bridgedb.py; refactor to use already running server. The unittests in `lib/bridgedb/test/test_bridgedb.py` now expect a BridgeDB instance to have already been started *before* the tests run. This change is necessary because it is non-trivial to have two instances of BridgeDB running simultaneously (at least not without changing every port in the config file) and the tests in `lib/bridgedb/test/test_https.py` also require a running server. These refactored tests should raise SkipTest if they believe that there is no running BridgeDB instance to test against. For Travis CI, the BridgeDB instance is started automatically in the 'before_script' section of `travis.yml` (in the top directory of this repo). * FIXES part of #9874. --- lib/bridgedb/test/test_bridgedb.py | 167 +++++++++++++++--------------------- lib/bridgedb/test/util.py | 22 +++++ 2 files changed, 89 insertions(+), 100 deletions(-) diff --git a/lib/bridgedb/test/test_bridgedb.py b/lib/bridgedb/test/test_bridgedb.py index c7aefff..e54d324 100644 --- a/lib/bridgedb/test/test_bridgedb.py +++ b/lib/bridgedb/test/test_bridgedb.py @@ -14,121 +14,88 @@ from __future__ import print_function import os -import shutil import signal import time -from os.path import join as pjoin -from subprocess import Popen, PIPE - -from twisted.python import log -from twisted.python.procutils import which from twisted.trial import unittest +from twisted.trial.unittest import SkipTest -from bridgedb.test.util import fileCheckDecorator +from bridgedb.test.util import pidExists class BridgeDBCliTest(unittest.TestCase): """Test the `bridgedb` command.""" - @fileCheckDecorator - def doCopyFile(self, src, dst, description=None): - shutil.copy(src, dst) - - @fileCheckDecorator - def doMoveFile(self, src, dst, description=None): - shutil.move(src, dst) - - def test_bridgedb_commands(self): - print('') - here = os.getcwd() - runDir = pjoin(here, 'rundir') - topDir = here.rstrip('_trial_temp') - scriptsDir = pjoin(topDir, 'scripts') - - # Create the lowest directory we need, and all its parents: - os.makedirs(os.path.join(runDir, 'gnupghome')) - - conf = pjoin(topDir, 'bridgedb.conf') - confMoved = pjoin(runDir, 'bridgedb.conf') - gpgFile = pjoin(topDir, 'gnupghome', 'TESTING.subkeys.sec') - gpgMoved = pjoin(runDir, 'gnupghome', 'TESTING.subkeys.sec') - certFile = pjoin(topDir, 'cert') - certMoved = pjoin(runDir, 'cert') - keyFile = pjoin(topDir, 'privkey.pem') - keyMoved = pjoin(runDir, 'privkey.pem') - - makeSSLCertScript = os.path.join(scriptsDir, 'make-ssl-cert') - bridgedbScript = which('bridgedb') # this returns a list - - self.doCopyFile(conf, confMoved, 'config') - self.doCopyFile(gpgFile, gpgMoved, 'GPG test key') - print("Running subcommands from directory:\n %r" % runDir) - print("Running %r..." % makeSSLCertScript) - makeSSLCertProcess = Popen(makeSSLCertScript) - makeSSLCertProcess.wait() - self.doMoveFile(certFile, certMoved, 'certificate') - self.doMoveFile(keyFile, keyMoved, 'SSL private key') - - self.assertTrue(os.path.isfile(bridgedbScript[0]), - "Couldn't find bridgedb script %r" % bridgedbScript[0]) - bridgedbScript = bridgedbScript[0] - print("Running bridgedb script %r..." % bridgedbScript) - - os.chdir(runDir) # we have to do this to get files to end up there - print("Running `bridgedb mock' to generate mock bridge descriptors...") - mockProc = Popen([bridgedbScript, 'mock', '-n', '50']) - mockProcCode = mockProc.wait() - print("`bridgedb mock' exited with status code %d" % int(mockProcCode)) - os.chdir(here) - - # See ticket #11216, cached-extrainfo* files should not be parsed - # cumulatively. - eidesc = pjoin(runDir, 'cached-extrainfo') - eindesc = pjoin(runDir, 'cached-extrainfo.new') - self.doCopyFile(eindesc, eidesc, 'duplicated cached-extrainfo(.new)') - self.assertTrue(os.path.isfile(eidesc)) - self.assertTrue(os.path.isfile(eindesc)) - - - print("Running `bridgedb' to test server startups...") - # Sorry Windows users - devnull = open('/dev/null', 'w') - bridgedbProc = Popen([bridgedbScript, '-r', runDir], stdout=devnull) - print("Waiting 10 seconds while bridgedb loads...") - time.sleep(10) - assignments = pjoin(runDir, 'assignments.log') - self.assertTrue(os.path.isfile(assignments)) - os.unlink(assignments) - print("Sending SIGHUP, checking for assignments.log ...") - bridgedbProc.send_signal(signal.SIGHUP) - time.sleep(5) + def setUp(self): + here = os.getcwd() + topdir = here.rstrip('_trial_temp') + self.rundir = os.path.join(topdir, 'run') + self.pidfile = os.path.join(self.rundir, 'bridgedb.pid') + self.pid = self.getBridgeDBPID(self.pidfile) + self.assignmentsFile = os.path.join(self.rundir, 'assignments.log') + + def getBridgeDBPID(self, pidfile="bridgedb.pid"): + """Read the ``bridgedb.pid`` file in **rundir**, if it exists, to get + the PID. + + :param str pidfile: The path to the BridgeDB pidfile. + :rtype: int + :returns: The process ID, if available, otherwise ``0``. + """ + fh = None try: - self.assertTrue(os.path.isfile(assignments)) - except self.failureException as e: - bridgedbProc.send_signal(signal.SIGKILL) - bridgedbProcCode = bridgedbProc.wait() - print("`bridgedb' exited with status code %d" % int(bridgedbProcCode)) - raise e - print("Sending SIGUSR1, checking for bucket files...") - bridgedbProc.send_signal(signal.SIGUSR1) + fh = open(pidfile) + except (IOError, OSError) as err: + print(err) + pid = 0 + else: + pid = int(fh.read()) + + if fh: + fh.close() + + return pid + + def test_bridgedb_assignments_log(self): + """This test should only be run if a BridgeDB server has already been + started in another process. + + To see how this is done for the Travis CI tests, see the + 'before_script' section of the ``.travis.yml`` file in the top + directory of this repository. + + This test ensures that an ``assignments.log`` file is created after a + BridgeDB process was started. + """ + if not self.pid: + raise SkipTest("Can't run test: no BridgeDB process running.") + + self.assertTrue(os.path.isfile(self.assignmentsFile)) + + def test_bridgedb_SIGHUP_assignments_log(self): + """Test that BridgeDB creates a new ``assignments.log`` file after + receiving a SIGHUP. + """ + if not self.pid: + raise SkipTest("Can't run test: no BridgeDB process running.") + + os.unlink(self.assignmentsFile) + os.kill(self.pid, signal.SIGHUP) + time.sleep(5) + self.assertTrue(os.path.isfile(self.assignmentsFile)) + + def test_bridgedb_SIGUSR1_buckets(self): + """Test that BridgeDB dumps buckets appropriately after a SIGUSR1.""" + if not self.pid: + raise SkipTest("Can't run test: no BridgeDB process running.") + + os.kill(self.pid, signal.SIGUSR1) time.sleep(5) buckets = [['email', False], ['https', False], ['unallocated', False]] - for rundirfile in os.listdir(runDir): + for rundirfile in os.listdir(self.rundir): for bucket in buckets: if rundirfile.startswith(bucket[0]): bucket[1] = True break for bucket in buckets: - try: - self.assertTrue(bucket[1], "%s bucket was not dumped!" % bucket[0]) - except self.failureException as e: - bridgedbProc.send_signal(signal.SIGKILL) - bridgedbProcCode = bridgedbProc.wait() - print("`bridgedb' exited with status code %d" % int(bridgedbProcCode)) - raise e - print("Done. Killing processes.") - bridgedbProc.send_signal(signal.SIGINT) - bridgedbProcCode = bridgedbProc.wait() - print("`bridgedb' exited with status code %d" % int(bridgedbProcCode)) - self.assertEqual(bridgedbProcCode, 0) + self.assertTrue(bucket[1], "%s bucket was not dumped!" % bucket[0]) diff --git a/lib/bridgedb/test/util.py b/lib/bridgedb/test/util.py index 5538a29..1067526 100644 --- a/lib/bridgedb/test/util.py +++ b/lib/bridgedb/test/util.py @@ -65,6 +65,28 @@ def fileCheckDecorator(func): % (str(description), dst, src)) return wrapper +def pidExists(pid): + """Test if **pid** exists. + + :raises: OSError, if ``OSError.errno`` wasn't an expected errno (according + to the "ERRORS" section from ``man 2 kill``). + :rtype: bool + :returns: ``True`` if a process with **pid** exists, ``False`` otherwise. + """ + try: + os.kill(pid, 0) + except OSError as err: + if err.errno == errno.ESRCH: # ESRCH: No such process + return False + if err.errno == errno.EPERM: # EPERM: Operation not permitted + # If we're not allowed to signal the process, then there exists a + # process that we don't have permissions to access. + return True + else: + raise + else: + return True + #: Mixin class for use with :api:`~twisted.trial.unittest.TestCase`. A #: ``TestCaseMixin`` can be used to add additional methods, which should be

1 0

[bridgedb/develop] Bump leekspin version to 0.2.0.
by isis＠torproject.org 13 Aug '14

13 Aug '14

commit 4d302bd3d391fb0174914a2ce035d1dcfaa707c0 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 7 11:28:36 2014 +0000 Bump leekspin version to 0.2.0. --- .test.requirements.txt | 2 +- .travis.requirements.txt | 2 +- setup.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.test.requirements.txt b/.test.requirements.txt index df97c1a..12dfb7b 100644 --- a/.test.requirements.txt +++ b/.test.requirements.txt @@ -6,7 +6,7 @@ # $ make coverage # coverage==3.7.1 -leekspin==0.1.4 +leekspin==0.2.0 mechanize==0.2.5 pep8==1.5.7 # pylint must be pinned until pylint bug #203 is fixed. See diff --git a/.travis.requirements.txt b/.travis.requirements.txt index 428438a..feef181 100644 --- a/.travis.requirements.txt +++ b/.travis.requirements.txt @@ -15,7 +15,7 @@ #------------------------------------------------------------------------------ coverage==3.7.1 coveralls==0.4.2 -leekspin==0.1.4 +leekspin==0.2.0 mechanize==0.2.5 pep8==1.5.7 # pylint must be pinned until pylint bug #203 is fixed. See diff --git a/setup.py b/setup.py index 4654415..cc2bd32 100644 --- a/setup.py +++ b/setup.py @@ -285,7 +285,7 @@ setuptools.setup( scripts=['scripts/bridgedb'], extras_require={'test': ["sure==1.2.2", "coverage==3.7.1", - "leekspin==0.1.3"]}, + "leekspin==0.2.0"]}, zip_safe=False, cmdclass=get_cmdclass(), include_package_data=True,

1 0

[bridgedb/develop] Add '127.0.0.1' to copied bridgedb.conf in .travis.yml.
by isis＠torproject.org 13 Aug '14

13 Aug '14

commit b15e76b2d9441acf17856b888e27d1b5dd06169e Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 7 11:02:30 2014 +0000 Add '127.0.0.1' to copied bridgedb.conf in .travis.yml. This is necessary to run the tests in `lib/bridgedb/test/test_smtp.py`, and shouldn't be set on production servers. It's done with a sed command in the `.travis.yml` file to a copied bridgedb.conf so that it doesn't ever end up in a production config file. --- .travis.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.travis.yml b/.travis.yml index a3771b0..5f0aed2 100644 --- a/.travis.yml +++ b/.travis.yml @@ -50,6 +50,9 @@ install: before_script: - mkdir run - cp -R -t run bridgedb.conf captchas gnupghome + # Add '127.0.0.1' to EMAIL_DOMAINS in bridgedb.conf. This should ONLY be + # done on testing servers, never on production servers. + - sed -r -i -e "s/(EMAIL_DOMAINS)(.*)(])/\1\2\, '127.0.0.1']/" run/bridgedb.conf - leekspin -n 100 - cp -t run networkstatus-bridges cached-extrainfo* bridge-descriptors - ./scripts/make-ssl-cert

1 0

[bridgedb/develop] Merge branch 'fix/9874-https' into fix/9874-email
by isis＠torproject.org 13 Aug '14

13 Aug '14

commit 321020ef0fcbb1507a3a0f38fed4b13267dff586 Merge: b15e76b 4d302bd Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 7 12:10:44 2014 +0000 Merge branch 'fix/9874-https' into fix/9874-email .test.requirements.txt | 2 +- .travis.requirements.txt | 2 +- setup.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)

1 0

[bridgedb/develop] Sphinxify bridgedb.test.test_https module docstring.
by isis＠torproject.org 13 Aug '14

13 Aug '14

commit 7ae1282581e3d690c310fa054a7d692f8cd2fe56 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 7 09:48:22 2014 +0000 Sphinxify bridgedb.test.test_https module docstring. --- lib/bridgedb/test/test_https.py | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/lib/bridgedb/test/test_https.py b/lib/bridgedb/test/test_https.py index 8e0fc50..a154856 100644 --- a/lib/bridgedb/test/test_https.py +++ b/lib/bridgedb/test/test_https.py @@ -7,9 +7,19 @@ # (c) 2014, The Tor Project, Inc. # :license: see LICENSE for licensing information #_____________________________________________________________________________ -"""integration tests for BridgeDB . -http://wwwsearch.sourceforge.net/mechanize/ -http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html + +"""Integration tests for BridgeDB's HTTPS Distributor. + +These tests use `mechanize`_ and `BeautifulSoup`_, and require a BridgeDB +instance to have been started in a separate process. To see how a BridgeDB is +started for our CI infrastructure from a fresh clone of this repository, see +the "before_script" section of the `.travis.yml` file in the top level of this +repository. + +.. _mechanize: https://pypi.python.org/pypi/mechanize/ + http://wwwsearch.sourceforge.net/mechanize/ +.. _BeautifulSoup: + http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html """ from __future__ import print_function @@ -20,6 +30,7 @@ import mechanize HTTPS_ROOT = 'https://127.0.0.1:6789' CAPTCHA_RESPONSE = 'Tvx74Pmy' + class HTTPTests(unittest.TestCase): def setUp(self): self.br = None

1 0

[bridgedb/develop] Add file header to lib/bridgedb/test/test_https.py.
by isis＠torproject.org 13 Aug '14

13 Aug '14

commit e96ebb1b02f704a30fe63da863d2ea714d1ad1ec Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 7 09:47:43 2014 +0000 Add file header to lib/bridgedb/test/test_https.py. --- lib/bridgedb/test/test_https.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/lib/bridgedb/test/test_https.py b/lib/bridgedb/test/test_https.py index 46da832..8e0fc50 100644 --- a/lib/bridgedb/test/test_https.py +++ b/lib/bridgedb/test/test_https.py @@ -1,3 +1,12 @@ +#_____________________________________________________________________________ +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: trygve <tor-dev(a)lists.torproject.org> +# :copyright: (c) trygve +# (c) 2014, The Tor Project, Inc. +# :license: see LICENSE for licensing information +#_____________________________________________________________________________ """integration tests for BridgeDB . http://wwwsearch.sourceforge.net/mechanize/ http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html

1 0