January 2014 - tor-commits - lists.torproject.org

[bridgedb/master] Ignore state files.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit a9c48e1402b042dacfe2e6fcaa85ec8ef183f517 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sat Nov 16 11:29:18 2013 +0000 Ignore state files. --- .gitignore | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index f642ca2..47e47b1 100644 --- a/.gitignore +++ b/.gitignore @@ -21,6 +21,7 @@ MANIFEST # Installer logs pip-log.txt +pip-install.log installed-files.txt # Unit test / coverage reports @@ -84,4 +85,7 @@ run/* lib/bridgedb/_langs.py # ignore any changes to our GnuPG directory: -gnupghome/* \ No newline at end of file +gnupghome/* + +# ignore persistent state files +*bridgedb.state*

1 0

[bridgedb/master] Revert "Deepcopy the HMAC factory function's resulting function."
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit fed7c16db8b4757da757b4244dbe02b8f84a2bc2 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 00:04:47 2013 +0000 Revert "Deepcopy the HMAC factory function's resulting function." This reverts commit a49ee2a0872b474e11a5f8dd9c169b2fac1ae22e. Signed-off-by: Isis Lovecruft <isis(a)torproject.org> --- lib/bridgedb/Bridges.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/bridgedb/Bridges.py b/lib/bridgedb/Bridges.py index 579dd4f..61f3412 100644 --- a/lib/bridgedb/Bridges.py +++ b/lib/bridgedb/Bridges.py @@ -7,7 +7,6 @@ them in rings. import binascii import bisect -import copy import hmac import logging import re @@ -83,7 +82,7 @@ def get_hmac_fn(k, hex=True): If 'hex' is true, the output of the function will be hex-encoded.""" h = hmac.new(k, digestmod=DIGESTMOD) def hmac_fn(v): - h_tmp = copy.deepcopy(h) + h_tmp = h.copy() h_tmp.update(v) if hex: return h_tmp.hexdigest()

1 0

[bridgedb/master] Merge branch 'hotfix/9872-options-subcommands-relative-rundir' into develop
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit cff49b44b2a318184cea82a1017cb82673e52d7d Merge: 753d0de 99417d8 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 02:12:30 2013 +0000 Merge branch 'hotfix/9872-options-subcommands-relative-rundir' into develop lib/bridgedb/Main.py | 28 +++---- lib/bridgedb/parse/options.py | 163 ++++++++++++++++++++++++++++++++++++----- 2 files changed, 158 insertions(+), 33 deletions(-)

1 0

[bridgedb/master] Merge branch 'hotfix/9872-options-subcommands-relative-rundir' into develop
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit b1ca8168a6a3902a2a64b8e5571508f2836c4acc Merge: cff49b4 da83ef2 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 02:58:24 2013 +0000 Merge branch 'hotfix/9872-options-subcommands-relative-rundir' into develop lib/bridgedb/test/test_persistent.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

1 0

[bridgedb/master] Merge branch 'hotfix/revert-a49ee2a' into develop
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 753d0deaf9ee1cb781be97a668a6ebbd8a0d38fb Merge: a9c48e1 fed7c16 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 00:07:45 2013 +0000 Merge branch 'hotfix/revert-a49ee2a' into develop lib/bridgedb/Bridges.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)

1 0

[bridgedb/master] Fix a bug where relative rundir/config options didn't work with subCommands.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 99417d8cce924f48d63628bf8b8535726c817809 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 02:05:02 2013 +0000 Fix a bug where relative rundir/config options didn't work with subCommands. Because subclasses of bridgedb.parse.options.BaseOptions will call their own parseOptions() method, which calls their own postOptions() as well as all parent class postOptions() methods, we have to set the absolute paths of the config file and the runtime directory as globals of the bridgedb.parse.options module. Otherwise, they will get reset in the parent classes, who do not have the instance variables of the subclass which did the original commandline parsing, and thus the config file and runtime directory get reset to their default settings by the parent classes. This then raises errors when an attempt to read the config file is made in bridgedb.Main.loadConfig(), because the file is (likely) not found. * FIXES a bug introduced by the changes for #9872, pertaining to setting the runtime directory and config file paths when using options.subCommand. --- lib/bridgedb/Main.py | 28 +++---- lib/bridgedb/parse/options.py | 163 ++++++++++++++++++++++++++++++++++++----- 2 files changed, 158 insertions(+), 33 deletions(-) diff --git a/lib/bridgedb/Main.py b/lib/bridgedb/Main.py index fa2e3bf..cdf1d12 100644 --- a/lib/bridgedb/Main.py +++ b/lib/bridgedb/Main.py @@ -337,21 +337,25 @@ class ProxyCategory: def replaceProxyList(self, ipset): self.ipset = ipset -def startup(options, rundir, configFile): +def startup(options): """Parse bridges, + :type options: :class:`bridgedb.parse.options.MainOptions` + :param options: A pre-parsed options class containing any arguments and + options given in the commandline we were called with. :type state: :class:`bridgedb.persistent.State` - :param state: A persistent state object which holds options and config - changes. + :ivar state: A persistent state object which holds config changes. """ # Change to the directory where we're supposed to run. This must be done # before parsing the config file, otherwise there will need to be two # copies of the config file, one in the directory BridgeDB is started in, # and another in the directory it changes into. - os.chdir(rundir) + os.chdir(options['rundir']) + if options['verbosity'] <= 10: # Corresponds to logging.DEBUG + print("Changed to runtime directory %r" % os.getcwd()) - config = loadConfig(configFile) - config.RUN_IN_DIR = rundir + config = loadConfig(options['config']) + config.RUN_IN_DIR = options['rundir'] # Set up logging as early as possible. We cannot import from the bridgedb # package any of our modules which import :mod:`logging` and start using @@ -604,16 +608,8 @@ def run(options): configuration file, loading and parsing it, and then either starting/reloading the servers or dumping bridge assignments to files. - :type options: :class:`bridgedb.opt.MainOptions` + :type options: :class:`bridgedb.parse.options.MainOptions` :param options: A pre-parsed options class containing any arguments and options given in the commandline we were called with. """ - if not options['config']: - options.getUsage() - sys.exit(1) - else: - configFile = os.path.abspath(os.path.expanduser(options['config'])) - if options['rundir']: - rundir = os.path.abspath(os.path.expanduser(options['rundir'])) - - startup(options, rundir, configFile) + startup(options) diff --git a/lib/bridgedb/parse/options.py b/lib/bridgedb/parse/options.py index e7ec7c3..d863015 100644 --- a/lib/bridgedb/parse/options.py +++ b/lib/bridgedb/parse/options.py @@ -19,6 +19,7 @@ from __future__ import unicode_literals import sys import textwrap +import traceback import os from twisted.python import usage @@ -26,6 +27,54 @@ from twisted.python import usage from bridgedb import __version__ +#: In :meth:`BaseOptions.findRundirAndConfig`, this is set to the the +#: absolute path of the ``opts['rundir']`` setting, if given, otherwise it +#: defaults to the current directory. +_rundir = None + +#: In :meth:`BaseOptions.findRundirAndConfig`, if ``opts['config']`` is +#: given, this is set to the the absolute path of the ``opts['config']`` +#: settting relative to the ``rundir``, otherwise it defaults to +#: 'bridgedb.conf' in the current directory. +_config = None + + +def setConfig(path): + """Set the absolute path to the config file. + + See :meth:`BaseOptions.postOptions`. + + :param string path: The path to set. + """ + global _config + _config = path + +def getConfig(): + """Get the absolute path to the config file. + + :rtype: string + :returns: The path to the config file. + """ + return _config + +def setRundir(path): + """Set the absolute path to the runtime directory. + + See :meth:`BaseOptions.postOptions`. + + :param string path: The path to set. + """ + global _rundir + _rundir = path + +def getRundir(): + """Get the absolute path to the runtime directory. + + :rtype: string + :returns: The path to the config file. + """ + return _rundir + def parseOptions(): """Create the main options parser and its subcommand parsers. @@ -38,11 +87,18 @@ def parseOptions(): already parsed. """ options = MainOptions() + try: options.parseOptions() except usage.UsageError as uerr: print(uerr.message) + print(options.getUsage()) sys.exit(1) + except Exception as error: + exc, value, tb = sys.exc_info() + print("Unhandled Error: %s" % error.message) + print(traceback.format_exc(tb)) + return options @@ -53,25 +109,22 @@ class BaseOptions(usage.Options): private relays acting as bridges into the Tor network. See `bridgedb <command> --help` for addition help.""") - def opt_rundir(self, rundir): - """Change to this directory""" - if not rundir: - rundir = os.getcwdu() - else: - try: - rundir = os.path.abspath(os.path.expanduser(rundir)) - except Exception as error: - raise usage.UsageError(error.message) - if rundir and os.path.isdir(rundir): - self['rundir'] = rundir - opt_r = opt_rundir + optParameters = [ + ['config', 'c', None, + 'Configuration file [default: <rundir>/bridgedb.conf]'], + ['rundir', 'r', None, + """Change to this directory before running. [default: `os.getcwd()'] + + All other paths, if not absolute, should be relative to this path. + This includes the config file and any further files specified within + the config file. + """]] def __init__(self): """Create an options parser. All flags, parameters, and attributes of this base options parser are inherited by all child classes. """ - usage.Options.__init__(self) - self['rundir'] = os.getcwdu() + super(BaseOptions, self).__init__() self['version'] = self.opt_version self['verbosity'] = 30 @@ -90,7 +143,84 @@ class BaseOptions(usage.Options): def opt_version(self): """Display BridgeDB's version and exit.""" print("%s-%s" % (__package__, __version__)) - sys.exit() + sys.exit(0) + + @staticmethod + def findRundirAndConfigFile(rundir=None, config=None): + """Find the abspath of the config file and runtime directory, or find + suitable defaults. + + Attempts to set the absolute path of the runtime directory. If the + config path is relative, its absolute path is set relative to the + runtime directory path (unless it starts with '.' or '..', then it is + interpreted relative to the current working directory). If the path to + the config file is absolute, it is left alone. + + :type rundir: string or None + :param rundir: The user-supplied path to the runtime directory, from + the commandline options (i.e. + ``options = BaseOptions().parseOptions(); options['rundir'];``). + :type config: string or None + :param config: The user-supplied path to the config file, from the + commandline options (i.e. + ``options = BaseOptions().parseOptions(); options['config'];``). + :raises: :exc:`twisted.python.usage.UsageError` if either the runtime + directory or the config file cannot be found. + """ + gRundir = getRundir() + gConfig = getConfig() + + if gRundir is None: + if rundir is not None: + gRundir = os.path.abspath(os.path.expanduser(rundir)) + else: + gRundir = os.getcwdu() + setRundir(gRundir) + + if not os.path.isdir(gRundir): + raise usage.UsageError( + "Could not change to runtime directory: `%s'" % gRundir) + + if gConfig is None: + if config is None: + config = 'bridgedb.conf' + gConfig = config + + if not os.path.isabs(gConfig): + if gConfig.startswith('.'): # also covers '..' + gConfig = os.path.abspath(os.path.expanduser(gConfig)) + else: + gConfig = os.path.join(gRundir, gConfig) + setConfig(gConfig) + + gConfig = getConfig() + if not os.path.isfile(gConfig): + raise usage.UsageError( + "Specified config file `%s' doesn't exist!" % gConfig) + + def postOptions(self): + """Automatically called by :meth:`parseOptions`. + + Determines appropriate values for the 'config' and 'rundir' settings. + """ + super(BaseOptions, self).postOptions() + self.findRundirAndConfigFile(self['rundir'], self['config']) + + gConfig = getConfig() + gRundir = getRundir() + + if (self['rundir'] is None) and (gRundir is not None): + self['rundir'] = gRundir + + if (self['config'] is None) and (gConfig is not None): + self['config'] = gConfig + + if self['verbosity'] <= 10: + print("%s.postOptions():" % self.__class__) + print(" gCONFIG=%s" % gConfig) + print(" self['config']=%s" % self['config']) + print(" gRUNDIR=%s" % gRundir) + print(" self['rundir']=%s" % self['rundir']) class TestOptions(BaseOptions): @@ -118,6 +248,7 @@ class TestOptions(BaseOptions): """Parse any additional arguments after the options and flags.""" self['test_args'] = args + class MockOptions(BaseOptions): """Suboptions for creating necessary conditions for testing purposes.""" @@ -133,8 +264,6 @@ class MainOptions(BaseOptions): optFlags = [ ['dump-bridges', 'd', 'Dump bridges by hashring assignment into files'], ['reload', 'R', 'Reload bridge descriptors into running servers']] - optParameters = [ - ['config', 'c', 'bridgedb.conf', 'Configuration file']] subCommands = [ ['test', None, TestOptions, "Run twisted.trial tests or unittests"], ['mock', None, MockOptions, "Generate a testing environment"]]

1 0

[bridgedb/master] Fix a test_persistent unittest failing do to options rundir/config changes.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit da83ef26ef8c7734da4f6bcc039752dd6e8759d8 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 02:56:08 2013 +0000 Fix a test_persistent unittest failing do to options rundir/config changes. --- lib/bridgedb/test/test_persistent.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bridgedb/test/test_persistent.py b/lib/bridgedb/test/test_persistent.py index 34b0b25..c3c89ef 100644 --- a/lib/bridgedb/test/test_persistent.py +++ b/lib/bridgedb/test/test_persistent.py @@ -50,7 +50,7 @@ class StateTest(unittest.TestCase): exec compiled in configuration config = persistent.Conf(**configuration) - fakeArgs = ['-v', '--dump-bridges'] + fakeArgs = ['-c', os.path.join(os.getcwdu(), '..', 'bridgedb.conf')] options = MainOptions() options.parseOptions(fakeArgs)

1 0

[bridgedb/master] Return status codes from subcommands.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 933c0dedb0454acf9bd575f2dcbba3614135dc2e Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 03:07:44 2013 +0000 Return status codes from subcommands. Apparently, "unhandled" SystemExits make Travis CI angry. --- lib/bridgedb/Main.py | 11 ++++++++--- lib/bridgedb/runner.py | 7 +++++-- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/lib/bridgedb/Main.py b/lib/bridgedb/Main.py index cdf1d12..8e22719 100644 --- a/lib/bridgedb/Main.py +++ b/lib/bridgedb/Main.py @@ -584,6 +584,8 @@ def runSubcommand(options, config): # mentioned above with the EmailServer and HTTPServer. from bridgedb import runner + statuscode = 0 + if options['dump-bridges']: runner.doDumpBridges(config) @@ -591,15 +593,18 @@ def runSubcommand(options, config): logging.debug("Running BridgeDB command: '%s'" % options.subCommand) if 'descriptors' in options.subOptions: - runner.generateDescriptors(options.subOptions['descriptors'], - config.RUN_IN_DIR) + statuscode = runner.generateDescriptors( + options.subOptions['descriptors'], config.RUN_IN_DIR) if options.subCommand == 'test': if options.subOptions['trial']: runner.runTrial(options.subOptions) if options.subOptions['unittests']: runner.runTests(options.subOptions) - raise SystemExit("Subcommand '%s' finished." % options.subCommand) + + logging.info("Subcommand '%s' finished with status %s." + % (options.subCommand, statuscode)) + sys.exit(statuscode) def run(options): """This is the main entry point into BridgeDB. diff --git a/lib/bridgedb/runner.py b/lib/bridgedb/runner.py index 1f30dfb..8c641f6 100644 --- a/lib/bridgedb/runner.py +++ b/lib/bridgedb/runner.py @@ -44,6 +44,7 @@ def generateDescriptors(count=None, rundir=None): import os.path proc = None + statuscode = 0 script = 'gen_bridge_descriptors' rundir = rundir if os.path.isdir(rundir) else None count = count if count else 3 @@ -56,9 +57,11 @@ def generateDescriptors(count=None, rundir=None): if proc.returncode: print("There was an error generating bridge descriptors.", "(Returncode: %d)" % proc.returncode) + statuscode = proc.returncode else: - print("Sucessfully bridge generated descriptors.") - del subprocess + print("Sucessfully generated %s descriptors." % str(count)) + del subprocess + return statuscode def runTrial(options): """Run Twisted trial based unittests, optionally with coverage.

1 0

[bridgedb/master] Add coverage and coveralls support to the test runner.
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 0f76862f3f5dd5d38f1cbbfb1fd8749da31f440d Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 03:17:34 2013 +0000 Add coverage and coveralls support to the test runner. * FIXES the coverage part of #9865. --- .coveragerc | 33 +++++++++++++++++++++++++++++++++ .gitignore | 4 ++++ .travis.yml | 21 +++++++++++++-------- Makefile | 3 +++ README | 2 +- requirements.txt | 3 --- setup.py | 2 +- 7 files changed, 55 insertions(+), 13 deletions(-) diff --git a/.coveragerc b/.coveragerc new file mode 100644 index 0000000..f8d4732 --- /dev/null +++ b/.coveragerc @@ -0,0 +1,33 @@ +[run] +source = + bridgedb + lib/bridgedb/test/test_* +branch = True +#parallel = True +timid = True + +[report] +modules = bridgedb +omit = + */test* + */_langs* + */_version* +# Regexes for lines to exclude from report generation: +exclude_lines = + pragma: no cover + # don't complain if the code doesn't hit unimplemented sections: + raise NotImplementedError + # don't complain if non-runnable or debuging code isn't run: + if 0: + if False: + if self[.verbosity.] + if options[.verbosity.] + def __repr__ + if __name__ == .__main__.: +# Ignore source code which cannot be found: +ignore_errors = True +# Exit with status code 2 if under this percentage is covered: +fail_under = 10 + +[html] +directory = doc/coverage-html diff --git a/.gitignore b/.gitignore index 47e47b1..e1a25c3 100644 --- a/.gitignore +++ b/.gitignore @@ -89,3 +89,7 @@ gnupghome/* # ignore persistent state files *bridgedb.state* + +# ignore coverage results files: +.coverage +.coverage.* diff --git a/.travis.yml b/.travis.yml index 6dcbfd9..7a47b36 100644 --- a/.travis.yml +++ b/.travis.yml @@ -15,7 +15,10 @@ notifications: before_install: - sudo apt-get update - - sudo apt-get install --no-install-suggests --no-install-recommends tor-geoipdb build-essential python-dev python-setuptools openssl sqlite3 libgpgme11-dev + # Distro development header/library/resource/buildtime dependencies + - sudo apt-get install --no-install-suggests --no-install-recommends tor-geoipdb build-essential openssl sqlite3 libgpgme11-dev + # Distro python dependencies + - sudo apt-get install --no-install-suggests --no-install-recommends python-dev python-setuptools python: - "2.6" @@ -26,6 +29,7 @@ python: install: - pip install -r requirements.txt --use-mirrors + - pip install coverage coveralls sure --use-mirrors - make install #before_script: @@ -34,12 +38,13 @@ install: # command to run tests, e.g. python setup.py test script: - - chmod u+x scripts/make-ssl-cert - - chmod u+x scripts/executioner - - ./scripts/make-ssl-cert - bridgedb mock -n 250 - - bridgedb test - - ./scripts/executioner 180 bridgedb -c bridgedb.conf - # add this when #9199's (or some other trial-based) tests are merged: - #- bridgedb test + # Normally, we would do 'bridgedb test' here, but to run with coverage we + # have to do this instead: + - coverage run $(which trial) ./lib/bridgedb/test/test_* + - coverage report + - coverage html - echo "Build successful." + +after_success: + - coveralls diff --git a/Makefile b/Makefile index 4c49a9b..036997e 100644 --- a/Makefile +++ b/Makefile @@ -39,3 +39,6 @@ clean: -rm -rf dist -rm -rf lib/bridgedb.egg-info -rm -rf _trial_temp + +coverage: + -coverage run $(which trial) ./lib/bridgedb/test/test_* && coverage report && coverage html diff --git a/README b/README index cdacdb2..3175957 100644 --- a/README +++ b/README @@ -1,4 +1,4 @@ -## BridgeDB +## BridgeDB [![Coverage Status](https://coveralls.io/repos/isislovecruft/bridgedb/badge.png)](https://coveralls.io/r/isislovecruft/bridgedb) ----------- BridgeDB is a collection of backend servers used to distribute diff --git a/requirements.txt b/requirements.txt index eeab36f..3935c59 100644 --- a/requirements.txt +++ b/requirements.txt @@ -3,13 +3,10 @@ BeautifulSoup==3.2.1 Mako==0.8.1 MarkupSafe==0.18 Twisted==13.1.0 -coverage==3.6 https://ipaddr-py.googlecode.com/files/ipaddr-2.1.10.tar.gz#sha1=c608450b07… pyOpenSSL==0.13.1 pygeoip==0.2.7 pygpgme==0.3 recaptcha==1.0rc1 recaptcha-client==1.0.6 -sure==1.2.2 -trialcoverage==0.3.12 zope.interface>=4.0.5 diff --git a/setup.py b/setup.py index d07e51a..4359fed 100644 --- a/setup.py +++ b/setup.py @@ -281,7 +281,7 @@ setuptools.setup( scripts=['scripts/bridgedb', 'scripts/gen_bridge_descriptors'], extras_require={'test': ["sure==0.4.5", - "trialcoverage==0.3.12"]}, + "coverage==3.6"]}, zip_safe=False, cmdclass=get_cmdclass(), include_package_data=True,

1 0

[bridgedb/master] Merge branch 'hotfix/9872-return-status-codes' into develop
by isis＠torproject.org 12 Jan '14

12 Jan '14

commit 23b7eb2efb74e2e31e0aea07ab7ccd4c27e8f11f Merge: b1ca816 933c0de Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Nov 19 03:09:28 2013 +0000 Merge branch 'hotfix/9872-return-status-codes' into develop lib/bridgedb/Main.py | 11 ++++++++--- lib/bridgedb/runner.py | 7 +++++-- 2 files changed, 13 insertions(+), 5 deletions(-)

1 0