October 2012 - tor-commits - lists.torproject.org

[tor/master] Merge remote-tracking branch 'arma/bug7037' into maint-0.2.3
by nickm＠torproject.org 04 Oct '12

04 Oct '12

commit 0a3dfd0423e230bf5b29c21623480af777bb22c1 Merge: b1971d8 e50fa0d Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Oct 4 12:46:33 2012 -0400 Merge remote-tracking branch 'arma/bug7037' into maint-0.2.3 changes/bug7037 | 6 ++++++ src/or/command.c | 2 +- 2 files changed, 7 insertions(+), 1 deletions(-)

1 0

[tor/master] Refuse extra create cells with reason "resource limit"
by nickm＠torproject.org 04 Oct '12

04 Oct '12

commit e50fa0d6cb467e8f6d41995eb6d03d74e9f64e04 Author: Roger Dingledine <arma(a)torproject.org> Date: Wed Oct 3 20:17:37 2012 -0400 Refuse extra create cells with reason "resource limit" In the past we had used reason "internal", which is more vague than it needs to be. Resolves bug 7037. --- changes/bug7037 | 6 ++++++ src/or/command.c | 2 +- 2 files changed, 7 insertions(+), 1 deletions(-) diff --git a/changes/bug7037 b/changes/bug7037 new file mode 100644 index 0000000..fc3a1ad --- /dev/null +++ b/changes/bug7037 @@ -0,0 +1,6 @@ + o Minor bugfixes: + - When relays refuse a "create" cell because their queue of pending + create cells is too big (typically because their cpu can't keep up + with the arrival rate), send back reason "resource limit" rather + than reason "internal", so network measurement scripts can get a + more accurate picture. Bugfix on 0.1.1.11-alpha; fixes bug 7037. diff --git a/src/or/command.c b/src/or/command.c index abf664c..d8a409b 100644 --- a/src/or/command.c +++ b/src/or/command.c @@ -428,7 +428,7 @@ command_process_create_cell(cell_t *cell, or_connection_t *conn) log_warn(LD_GENERAL,"Failed to hand off onionskin. Closing.%s",m); tor_free(m); } - circuit_mark_for_close(TO_CIRCUIT(circ), END_CIRC_REASON_INTERNAL); + circuit_mark_for_close(TO_CIRCUIT(circ), END_CIRC_REASON_RESOURCELIMIT); return; } log_debug(LD_OR,"success: handed off onionskin.");

1 0

[tor/master] Merge remote-tracking branch 'origin/maint-0.2.3'
by nickm＠torproject.org 04 Oct '12

04 Oct '12

commit be33c3f6006c4cb06e21483c0c0460502ac2c366 Merge: d92d3f3 0a3dfd0 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Oct 4 12:49:13 2012 -0400 Merge remote-tracking branch 'origin/maint-0.2.3' changes/bug7037 | 6 ++++++ src/or/command.c | 2 +- 2 files changed, 7 insertions(+), 1 deletions(-)

1 0

[tor/maint-0.2.3] Refuse extra create cells with reason "resource limit"
by nickm＠torproject.org 04 Oct '12

04 Oct '12

commit e50fa0d6cb467e8f6d41995eb6d03d74e9f64e04 Author: Roger Dingledine <arma(a)torproject.org> Date: Wed Oct 3 20:17:37 2012 -0400 Refuse extra create cells with reason "resource limit" In the past we had used reason "internal", which is more vague than it needs to be. Resolves bug 7037. --- changes/bug7037 | 6 ++++++ src/or/command.c | 2 +- 2 files changed, 7 insertions(+), 1 deletions(-) diff --git a/changes/bug7037 b/changes/bug7037 new file mode 100644 index 0000000..fc3a1ad --- /dev/null +++ b/changes/bug7037 @@ -0,0 +1,6 @@ + o Minor bugfixes: + - When relays refuse a "create" cell because their queue of pending + create cells is too big (typically because their cpu can't keep up + with the arrival rate), send back reason "resource limit" rather + than reason "internal", so network measurement scripts can get a + more accurate picture. Bugfix on 0.1.1.11-alpha; fixes bug 7037. diff --git a/src/or/command.c b/src/or/command.c index abf664c..d8a409b 100644 --- a/src/or/command.c +++ b/src/or/command.c @@ -428,7 +428,7 @@ command_process_create_cell(cell_t *cell, or_connection_t *conn) log_warn(LD_GENERAL,"Failed to hand off onionskin. Closing.%s",m); tor_free(m); } - circuit_mark_for_close(TO_CIRCUIT(circ), END_CIRC_REASON_INTERNAL); + circuit_mark_for_close(TO_CIRCUIT(circ), END_CIRC_REASON_RESOURCELIMIT); return; } log_debug(LD_OR,"success: handed off onionskin.");

1 0

[tor/maint-0.2.3] Merge remote-tracking branch 'arma/bug7037' into maint-0.2.3
by nickm＠torproject.org 04 Oct '12

04 Oct '12

commit 0a3dfd0423e230bf5b29c21623480af777bb22c1 Merge: b1971d8 e50fa0d Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Oct 4 12:46:33 2012 -0400 Merge remote-tracking branch 'arma/bug7037' into maint-0.2.3 changes/bug7037 | 6 ++++++ src/or/command.c | 2 +- 2 files changed, 7 insertions(+), 1 deletions(-)

1 0

[tor/master] Add autoconf magic to support Bitrig
by nickm＠torproject.org 04 Oct '12

04 Oct '12

commit d92d3f33356af002892ba5754d9d36cc4504c95f Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Oct 4 10:56:33 2012 -0400 Add autoconf magic to support Bitrig Bitrig is an openbsd fork. Patch from dhill. Ticket 6982. --- changes/6982 | 3 +++ configure.ac | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/changes/6982 b/changes/6982 new file mode 100644 index 0000000..edfa066 --- /dev/null +++ b/changes/6982 @@ -0,0 +1,3 @@ + o Minor features (portability): + - Tor now builds correctly on Bitrig, an OpenBSD fork. Patch from dhill. + Ticket 6982. diff --git a/configure.ac b/configure.ac index 4db1a2c..bb7ea6b 100644 --- a/configure.ac +++ b/configure.ac @@ -743,7 +743,7 @@ if test x$transparent = xtrue ; then if test x$transparent_ok = x1 ; then AC_DEFINE(USE_TRANSPARENT, 1, "Define to enable transparent proxy support") case $host in - *-*-openbsd*) + *-*-openbsd* | *-*-bitrig*) AC_DEFINE(OPENBSD, 1, "Define to handle pf on OpenBSD properly") ;; esac else @@ -1252,7 +1252,7 @@ if test x$enable_gcc_warnings = xyes || test x$enable_gcc_warnings_advisory = xy CFLAGS="$save_CFLAGS" case $host in - *-*-openbsd*) + *-*-openbsd* | *-*-bitrig*) # Some OpenBSD versions (like 4.8) have -Wsystem-headers by default. # That's fine, except that the headers don't pass -Wredundant-decls. # Therefore, let's disable -Wsystem-headers when we're building

1 0

[ooni-probe/master] Rename new bridget to new bridget
by isis＠torproject.org 04 Oct '12

04 Oct '12

commit 0080bf6555e4b55d0391c1c2f3ad41c83428f7e9 Author: Arturo Filastò <arturo(a)filasto.net> Date: Thu Aug 23 17:54:18 2012 +0200 Rename new bridget to new bridget --- ooni/assets/bridgetests.txt | 6 +- ooni/ooniprobe.py | 14 +++--- ooni/plugins/bridget.py | 76 ------------------------------- ooni/plugins/domclass.py | 37 ++++++++++++--- ooni/plugins/new_bridget.py | 105 +++++++++++++++++++++++++++++++++++++++++++ ooni/plugoo/interface.py | 1 + ooni/plugoo/reports.py | 49 ++++++++++++++++++++ ooni/plugoo/tests.py | 1 + 8 files changed, 196 insertions(+), 93 deletions(-) diff --git a/ooni/assets/bridgetests.txt b/ooni/assets/bridgetests.txt index c837988..5519eea 100644 --- a/ooni/assets/bridgetests.txt +++ b/ooni/assets/bridgetests.txt @@ -1,3 +1,3 @@ -213.151.89.102:9001 -108.166.106.156:443 -217.150.224.213:443 +88.130.86.191:443 +195.74.237.236:9001 +127.0.0.1:9050 diff --git a/ooni/ooniprobe.py b/ooni/ooniprobe.py index 4b5c086..f80669e 100755 --- a/ooni/ooniprobe.py +++ b/ooni/ooniprobe.py @@ -76,17 +76,18 @@ def runTest(test, options, global_options, reactor=reactor): resume = 0 if not options: options = {} + if 'resume' in options: resume = options['resume'] test = test_class(options, global_options, report, reactor=reactor) + if test.tool: test.runTool() - return if test.ended: - print "Ending prematurely" - return + print "Ending test" + return None wgen = work.WorkGenerator(test, dict(options), @@ -94,6 +95,8 @@ def runTest(test, options, global_options, reactor=reactor): for x in wgen: worker.push(x) + return True + class Options(usage.Options): tests = plugoo.keys() subCommands = [] @@ -138,7 +141,6 @@ if __name__ == "__main__": config.opt_help() sys.exit(1) - runTest(config.subCommand, config.subOptions, config) - - reactor.run() + if runTest(config.subCommand, config.subOptions, config): + reactor.run() diff --git a/ooni/plugins/bridget.py b/ooni/plugins/bridget.py deleted file mode 100644 index 22ff9a0..0000000 --- a/ooni/plugins/bridget.py +++ /dev/null @@ -1,76 +0,0 @@ -""" -This is a self genrated test created by scaffolding.py. -you will need to fill it up with all your necessities. -Safe hacking :). -""" -from zope.interface import implements -from twisted.python import usage -from twisted.plugin import IPlugin -from twisted.internet import reactor - -from ooni.utils import log -from ooni.plugoo.tests import ITest, OONITest -from ooni.plugoo.assets import Asset - -class bridgetArgs(usage.Options): - optParameters = [['bridges', 'b', None, 'List of bridges to scan'], - ['relays', 'f', None, 'List of relays to scan'], - ['resume', 'r', 0, 'Resume at this index']] - -class bridgetTest(OONITest): - implements(IPlugin, ITest) - - shortName = "bridget" - description = "bridget" - requirements = None - options = bridgetArgs - blocking = False - - def experiment(self, args): - log.msg("Doing test") - # What you return here gets handed as input to control - from ooni.lib.txtorcon import TorProtocolFactory, TorConfig, TorState - from ooni.lib.txtorcon import DEFAULT_VALUE, launch_tor - def updates(prog, tag, summary): - log.msg("%d%%: %s" % (prog, summary)) - return - - def setup_failed(args): - log.msg("Setup Failed.") - report.update({'failed': args}) - return report - - def setup_complete(proto): - log.msg("Setup Complete.") - report.update({'success': args}) - return report - - config = TorConfig() - import random - config.SocksPort = random.randint(1024, 2**16) - config.ControlPort = random.randint(1024, 2**16) - - if 'bridge' in args: - config.UseBridges = 1 - config.Bridge = args['bridge'] - config.save() - print config.create_torrc() - report = {'tor_config': config.config} - log.msg("Starting Tor") - d = launch_tor(config, self.reactor, progress_updates=updates) - d.addCallback(setup_complete) - d.addErrback(setup_failed) - return d - - def load_assets(self): - assets = {} - if self.local_options: - if self.local_options['bridges']: - assets.update({'bridge': Asset(self.local_options['bridges'])}) - elif self.local_options['relays']: - assets.update({'relay': Asset(self.local_options['relay'])}) - return assets - -# We need to instantiate it otherwise getPlugins does not detect it -# XXX Find a way to load plugins without instantiating them. -bridget = bridgetTest(None, None, None) diff --git a/ooni/plugins/domclass.py b/ooni/plugins/domclass.py index cdcd508..446b9e4 100644 --- a/ooni/plugins/domclass.py +++ b/ooni/plugins/domclass.py @@ -1,4 +1,4 @@ -# +# -*- coding: utf-8 # # domclass # ******** @@ -28,6 +28,12 @@ # probability matrix B. # +try: + import numpy +except: + print "Error numpy not installed!" + +import yaml from zope.interface import implements from twisted.python import usage from twisted.plugin import IPlugin @@ -40,7 +46,7 @@ class domclassArgs(usage.Options): optParameters = [['output', 'o', None, 'Output to write'], ['file', 'f', None, 'Corpus file'], ['fileb', 'b', None, 'Corpus file'], - ['asset', 'a', None, 'URL List'], + ['urls', 'u', None, 'URL List'], ['resume', 'r', 0, 'Resume at this index']] # All HTML4 tags @@ -80,7 +86,6 @@ def compute_probability_matrix(dataset): :dataset: an array of pairs representing the parent child relationships. """ import itertools - import numpy ret = {} matrix = numpy.zeros((len(thetags) + 1, len(thetags) + 1)) @@ -155,15 +160,16 @@ class domclassTest(HTTPTest): #tool = True def runTool(self): - import yaml, numpy site_a = readDOM(filename=self.local_options['file']) site_b = readDOM(filename=self.local_options['fileb']) + a = {} a['matrix'] = compute_probability_matrix(site_a) - a['eigen'] = compute_eigenvalue(a['matrix']) + a['eigen'] = compute_eigenvalues(a['matrix']) self.result['eigenvalues'] = a['eigen'] + b = {} b['matrix'] = compute_probability_matrix(site_b) - b['eigen'] = compute_eigenvalue(b['matrix']) + b['eigen'] = compute_eigenvalues(b['matrix']) #print "A: %s" % a #print "B: %s" % b @@ -171,13 +177,15 @@ class domclassTest(HTTPTest): correlation /= numpy.linalg.norm(a['eigen'])*numpy.linalg.norm(b['eigen']) correlation = (correlation + 1)/2 print "Corelation: %s" % correlation + self.end() + return a def processResponseBody(self, data): - import yaml, numpy site_a = readDOM(data) #site_b = readDOM(self.local_options['fileb']) + a = {} a['matrix'] = compute_probability_matrix(site_a) - a['eigen'] = compute_eigenvalue(a['matrix']) + a['eigen'] = compute_eigenvalues(a['matrix']) if len(data) == 0: @@ -191,4 +199,17 @@ class domclassTest(HTTPTest): print "A: %s" % a return a['eigen'] + def load_assets(self): + if self.local_options: + if self.local_options['file']: + self.tool = True + return {} + elif self.local_options['urls']: + return {'url': Asset(self.local_options['urls'])} + else: + self.end() + return {} + else: + return {} + domclass = domclassTest(None, None, None) diff --git a/ooni/plugins/new_bridget.py b/ooni/plugins/new_bridget.py new file mode 100644 index 0000000..3e4db56 --- /dev/null +++ b/ooni/plugins/new_bridget.py @@ -0,0 +1,105 @@ +""" +This is a self genrated test created by scaffolding.py. +you will need to fill it up with all your necessities. +Safe hacking :). +""" +from exceptions import Exception +from datetime import datetime +from zope.interface import implements +from twisted.python import usage +from twisted.plugin import IPlugin +from twisted.internet import reactor, task + +from ooni.utils import log +from ooni.plugoo.tests import ITest, OONITest +from ooni.plugoo.assets import Asset + +from ooni.lib.txtorcon import TorProtocolFactory, TorConfig, TorState +from ooni.lib.txtorcon import DEFAULT_VALUE, launch_tor + +class bridgetArgs(usage.Options): + optParameters = [['bridges', 'b', None, 'List of bridges to scan'], + ['relays', 'f', None, 'List of relays to scan'], + ['resume', 'r', 0, 'Resume at this index'], + ['timeout', 't', 5, 'Timeout in seconds after which to consider a bridge not working'] + ] + +class bridgetTest(OONITest): + implements(IPlugin, ITest) + + shortName = "bridget" + description = "bridget" + requirements = None + options = bridgetArgs + blocking = False + + def experiment(self, args): + log.msg("Doing test") + last_update = datetime.now() + tor_log = [] + + def check_timeout(): + log.msg("Checking for timeout") + time_since_update = datetime.now() - last_update + if time_since_update.seconds > self.local_options['timeout']: + log.msg("Timed out when connecting to %s" % args) + l.stop() + self.result['reason'] = 'timeout' + d.errback(args) + return + + def updates(prog, tag, summary): + tor_log.append((prog, tag, summary)) + last_update = datetime.now() + log.msg("%d%%: %s" % (prog, summary)) + + def setup_failed(failure): + log.msg("Setup Failed.") + if not self.result['reason']: + self.result['reason'] = 'unknown' + self.result['input'] = args + self.result['result'] = 'failed' + self.result['tor_log'] = tor_log + return + + def setup_complete(proto): + log.msg("Setup Complete.") + self.result['input'] = args + self.result['result'] = 'success' + return + + config = TorConfig() + import random + config.SocksPort = random.randint(1024, 2**16) + config.ControlPort = random.randint(1024, 2**16) + + if 'bridge' in args: + config.UseBridges = 1 + config.Bridge = args['bridge'] + + config.save() + + print config.config + self.result['tor_config'] = config.config + log.msg("Starting Tor connecting to %s" % args['bridge']) + + l = task.LoopingCall(check_timeout) + l.start(1.0) + + d = launch_tor(config, self.reactor, control_port=config.ControlPort, progress_updates=updates) + d.addCallback(setup_complete) + d.addErrback(setup_failed) + return d + + def load_assets(self): + assets = {} + if self.local_options: + if self.local_options['bridges']: + assets.update({'bridge': Asset(self.local_options['bridges'])}) + elif self.local_options['relays']: + assets.update({'relay': Asset(self.local_options['relay'])}) + return assets + +# We need to instantiate it otherwise getPlugins does not detect it +# XXX Find a way to load plugins without instantiating them. +bridget = bridgetTest(None, None, None) diff --git a/ooni/plugoo/interface.py b/ooni/plugoo/interface.py index d00e70b..6dc83a0 100644 --- a/ooni/plugoo/interface.py +++ b/ooni/plugoo/interface.py @@ -53,3 +53,4 @@ class ITest(Interface): be created. A report will be written. """ + diff --git a/ooni/plugoo/reports.py b/ooni/plugoo/reports.py index a543151..1bfdac0 100644 --- a/ooni/plugoo/reports.py +++ b/ooni/plugoo/reports.py @@ -93,4 +93,53 @@ class Report: log.msg("Reporting to %s" % type) getattr(self, type+"_report").__call__(data) +class NewReport(object): + filename = 'report.log' + startTime = None + endTime = None + testName = None + ipAddr = None + asnAddr = None + + def _open(): + self.fp = open(self.filename, 'a+') + + @property + def header(): + pretty_date = date.pretty_date() + report_header = "# OONI Probe Report for Test %s\n" % self.testName + report_header += "# %s\n\n" % pretty_date + test_details = {'start_time': self.startTime, + 'asn': asnAddr, + 'test_name': self.testName, + 'addr': ipAddr} + report_header += yaml.dump([test_details]) + return report_header + + def create(): + """ + Create a new report by writing it's header. + """ + self.fp = open(self.filename, 'w+') + self.fp.write(self.header) + + def exists(): + """ + Returns False if the file does not exists. + """ + return os.path.exists(self.filename) + + def write(data): + """ + Write a report to the file. + + :data: python data structure to be written to report. + """ + if not self.exists(): + self.create() + else: + self._open() + yaml_encoded_data = yaml.dump([data]) + self.fp.write(yaml_encoded_data) + self.fp.close() diff --git a/ooni/plugoo/tests.py b/ooni/plugoo/tests.py index 7c190eb..5fad85e 100644 --- a/ooni/plugoo/tests.py +++ b/ooni/plugoo/tests.py @@ -99,6 +99,7 @@ class OONITest(object): self.d.addCallback(self.control, args) self.d.addCallback(self.finished) + self.d.addErrback(self.finished) return self.d def control(self, result, args):

1 0

[ooni-probe/master] Implement a working ooniclient based off of trial
by isis＠torproject.org 04 Oct '12

04 Oct '12

commit 603d5cdc27a13aef4d9d25ffbd9108fb3bdbdcac Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Sep 11 15:35:21 2012 +0000 Implement a working ooniclient based off of trial --- ooni/nettest.py | 97 +++++++++++++++++++++++++++ ooni/oonicli.py | 73 ++++----------------- ooni/plugoo/tests.py | 1 + ooni/reporter.py | 16 ++--- ooni/runner.py | 177 +++++++++++++++++++++++++++++++++++++++++++++++++- 5 files changed, 292 insertions(+), 72 deletions(-) diff --git a/ooni/nettest.py b/ooni/nettest.py new file mode 100644 index 0000000..fe8c05c --- /dev/null +++ b/ooni/nettest.py @@ -0,0 +1,97 @@ + +from twisted.python import log +from twisted.trial import unittest, itrial + +pyunit = __import__('unittest') + +def _iterateTests(testSuiteOrCase): + """ + Iterate through all of the test cases in C{testSuiteOrCase}. + """ + try: + suite = iter(testSuiteOrCase) + except TypeError: + if not testSuiteOrCase.inputs: + yield testSuiteOrCase + else: + inputs = iter(testSuiteOrCase.inputs) + print "Detected Sub shit! %s" % inputs + for input in inputs: + yield testSuiteOrCase, input + else: + for test in suite: + for subtest in _iterateTests(test): + yield subtest + + +class TestCase(unittest.TestCase): + """ + A test case represents the minimum + """ + def run(self, result, input): + """ + Run the test case, storing the results in C{result}. + + First runs C{setUp} on self, then runs the test method (defined in the + constructor), then runs C{tearDown}. As with the standard library + L{unittest.TestCase}, the return value of these methods is disregarded. + In particular, returning a L{Deferred} has no special additional + consequences. + + @param result: A L{TestResult} object. + """ + log.msg("--> %s <--" % (self.id())) + new_result = itrial.IReporter(result, None) + if new_result is None: + result = PyUnitResultAdapter(result) + else: + result = new_result + result.startTest(self) + if self.getSkip(): # don't run test methods that are marked as .skip + result.addSkip(self, self.getSkip()) + result.stopTest(self) + return + + self._passed = False + self._warnings = [] + + self._installObserver() + # All the code inside _runFixturesAndTest will be run such that warnings + # emitted by it will be collected and retrievable by flushWarnings. + unittest._collectWarnings(self._warnings.append, self._runFixturesAndTest, result) + + # Any collected warnings which the test method didn't flush get + # re-emitted so they'll be logged or show up on stdout or whatever. + for w in self.flushWarnings(): + try: + warnings.warn_explicit(**w) + except: + result.addError(self, failure.Failure()) + + result.stopTest(self) + + +class TestSuite(pyunit.TestSuite): + """ + Extend the standard library's C{TestSuite} with support for the visitor + pattern and a consistently overrideable C{run} method. + """ + + def __call__(self, result, input): + return self.run(result, input) + + + def run(self, result, input): + """ + Call C{run} on every member of the suite. + """ + # we implement this because Python 2.3 unittest defines this code + # in __call__, whereas 2.4 defines the code in run. + for test in self._tests: + if result.shouldStop: + break + print test + print "----------------" + test(result, input) + return result + diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 8ace160..199b4d4 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -22,19 +22,8 @@ from twisted.python.filepath import FilePath from twisted import plugin from twisted.python.util import spewer from twisted.python.compat import set -from twisted.trial import runner, itrial, reporter - - -# Yea, this is stupid. Leave it for for command-line compatibility for a -# while, though. -TBFORMAT_MAP = { - 'plain': 'default', - 'default': 'default', - 'emacs': 'brief', - 'brief': 'brief', - 'cgitb': 'verbose', - 'verbose': 'verbose' - } +from twisted.trial import itrial +from ooni import runner, reporter def _parseLocalVariables(line): @@ -98,26 +87,6 @@ def isTestFile(filename): and os.path.splitext(basename)[1] == ('.py')) -def _reporterAction(): - return usage.CompleteList([p.longOpt for p in - plugin.getPlugins(itrial.IReporter)]) - -class Options(usage.Options): - - optParameters = [ - ['parallelism', 'n', 10, "Specify the number of parallel tests to run"], - ['output', 'o', 'report.log', "Specify output report file"], - ['log', 'l', 'oonicli.log', "Specify output log file"] - ] - - def opt_version(self): - """ - Display OONI version and exit. - """ - print "OONI version:", __version__ - sys.exit(0) - - class Options(usage.Options, app.ReactorSelectionMixin): synopsis = """%s [options] [[file|package|module|TestCase|testmethod]...] """ % (os.path.basename(sys.argv[0]),) @@ -136,12 +105,11 @@ class Options(usage.Options, app.ReactorSelectionMixin): ["nopm", None, "don't automatically jump into debugger for " "postmorteming of exceptions"], ["dry-run", 'n', "do everything but run the tests"], - ["force-gc", None, "Have Trial run gc.collect() before and " + ["force-gc", None, "Have OONI run gc.collect() before and " "after each test case."], ["profile", None, "Run tests under the Python profiler"], ["unclean-warnings", None, "Turn dirty reactor errors into warnings"], - ["until-failure", "u", "Repeat test until it fails"], ["no-recurse", "N", "Don't recurse into packages"], ['help-reporters', None, "Help on available output plugins (reporters)"] @@ -153,13 +121,12 @@ class Options(usage.Options, app.ReactorSelectionMixin): "Run tests in random order using the specified seed"], ['temp-directory', None, '_trial_temp', 'Path to use as working directory for tests.'], - ['reporter', None, 'verbose', + ['reporter', None, 'default', 'The reporter to use for this test run. See --help-reporters for ' 'more info.']] compData = usage.Completions( optActions={"tbformat": usage.CompleteList(["plain", "emacs", "cgitb"]), - "reporter": _reporterAction, "logfile": usage.CompleteFiles(descr="log file name"), "random": usage.Completer(descr="random seed")}, extraActions=[usage.CompleteFiles( @@ -167,7 +134,7 @@ class Options(usage.Options, app.ReactorSelectionMixin): repeat=True)], ) - fallbackReporter = reporter.TreeReporter + fallbackReporter = reporter.OONIReporter tracer = None def __init__(self): @@ -208,7 +175,7 @@ class Options(usage.Options, app.ReactorSelectionMixin): # value to the test suite as a module. # # This parameter allows automated processes (like Buildbot) to pass - # a list of files to Trial with the general expectation of "these files, + # a list of files to OONI with the general expectation of "these files, # whatever they are, will get tested" if not os.path.isfile(filename): sys.stderr.write("File %r doesn't exist\n" % (filename,)) @@ -300,21 +267,11 @@ class Options(usage.Options, app.ReactorSelectionMixin): self['tests'].update(args) - def _loadReporterByName(self, name): - for p in plugin.getPlugins(itrial.IReporter): - qual = "%s.%s" % (p.module, p.klass) - if p.longOpt == name: - return reflect.namedAny(qual) - raise usage.UsageError("Only pass names of Reporter plugins to " - "--reporter. See --help-reporters for " - "more info.") - - def postOptions(self): # Only load reporters now, as opposed to any earlier, to avoid letting # application-defined plugins muck up reactor selecting by importing # t.i.reactor and causing the default to be installed. - self['reporter'] = self._loadReporterByName(self['reporter']) + self['reporter'] = reporter.OONIReporter if 'tbformat' not in self: self['tbformat'] = 'default' @@ -338,10 +295,10 @@ def _initialDebugSetup(config): def _getSuite(config): loader = _getLoader(config) recurse = not config['no-recurse'] + print "loadByNames %s" % config['tests'] return loader.loadByNames(config['tests'], recurse) - def _getLoader(config): loader = runner.TestLoader() if config['random']: @@ -349,8 +306,6 @@ def _getLoader(config): randomer.seed(config['random']) loader.sorter = lambda x : randomer.random() print 'Running tests shuffled with seed %d\n' % config['random'] - if not config['until-failure']: - loader.suiteFactory = runner.DestructiveTestSuite return loader @@ -358,10 +313,11 @@ def _getLoader(config): def _makeRunner(config): mode = None if config['debug']: - mode = runner.TrialRunner.DEBUG + mode = runner.OONIRunner.DEBUG if config['dry-run']: - mode = runner.TrialRunner.DRY_RUN - return runner.TrialRunner(config['reporter'], + mode = runner.OONIRunner.DRY_RUN + print "using %s" % config['reporter'] + return runner.OONIRunner(config['reporter'], mode=mode, profile=config['profile'], logfile=config['logfile'], @@ -384,10 +340,7 @@ def run(): _initialDebugSetup(config) trialRunner = _makeRunner(config) suite = _getSuite(config) - if config['until-failure']: - test_result = trialRunner.runUntilFailure(suite) - else: - test_result = trialRunner.run(suite) + test_result = trialRunner.run(suite) if config.tracer: sys.settrace(None) results = config.tracer.results() diff --git a/ooni/plugoo/tests.py b/ooni/plugoo/tests.py index 5fad85e..2b1e87c 100644 --- a/ooni/plugoo/tests.py +++ b/ooni/plugoo/tests.py @@ -22,6 +22,7 @@ class OONITest(object): developer to benefit from OONIs reporting system and command line argument parsing system. """ + name = "oonitest" # By default we set this to False, meaning that we don't block blocking = False reactor = None diff --git a/ooni/reporter.py b/ooni/reporter.py index 0ecf2ea..d20160f 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -1,13 +1,11 @@ from twisted.trial import reporter -class TestResult(reporter.TestResult): - """ - Accumulates the results of several ooni.nettest.TestCases. - - The output format of a TestResult is YAML and it will contain all the basic - information that a test result should contain. - """ - def __init__(self): - super(TestResult, self).__init__() +class OONIReporter(reporter.Reporter): + + def startTest(self, test, input=None): + print "Running %s" % test + print "Input %s" % input + self._input = input + super(OONIReporter, self).startTest(test) diff --git a/ooni/runner.py b/ooni/runner.py index d7caa9d..c6ad90b 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -1,7 +1,178 @@ -from twisted.trial import runner +import types +import time +import inspect +from twisted.internet import defer +from twisted.trial import unittest +from twisted.trial.runner import TrialRunner, TestLoader +from twisted.trial.runner import isPackage, isTestCase -class TestLoader(runner.TestLoader): - pass +from ooni import nettest +from ooni.plugoo import tests as oonitests + +def isLegacyTest(obj): + """ + Returns True if the test in question is written using the OONITest legacy + class. + We do this for backward compatibility of the OONIProbe API. + """ + try: + return issubclass(obj, oonitests.OONITest) + except TypeError: + return False + +def adaptLegacyTest(obj): + """ + We take a legacy OONITest class and convert it into a nettest.TestCase. + This allows backward compatibility of old OONI tests. + + XXX perhaps we could implement another extra layer that makes the even + older test cases compatible with the new OONI. + """ + class LegacyOONITest(nettest.TestCase): + pass + + +class LoggedSuite(nettest.TestSuite): + """ + Any errors logged in this suite will be reported to the L{TestResult} + object. + """ + + def run(self, result, input): + """ + Run the suite, storing all errors in C{result}. If an error is logged + while no tests are running, then it will be added as an error to + C{result}. + + @param result: A L{TestResult} object. + """ + observer = unittest._logObserver + observer._add() + super(LoggedSuite, self).run(result, input) + observer._remove() + for error in observer.getErrors(): + result.addError(TestHolder(NOT_IN_TEST), error) + observer.flushErrors() + + +class OONISuite(nettest.TestSuite): + """ + Suite to wrap around every single test in a C{trial} run. Used internally + by OONI to set up things necessary for OONI tests to work, regardless of + what context they are run in. + """ + + def __init__(self, tests=()): + suite = LoggedSuite(tests) + super(OONISuite, self).__init__([suite]) + + def _bail(self): + from twisted.internet import reactor + d = defer.Deferred() + reactor.addSystemEventTrigger('after', 'shutdown', + lambda: d.callback(None)) + reactor.fireSystemEvent('shutdown') # radix's suggestion + # As long as TestCase does crap stuff with the reactor we need to + # manually shutdown the reactor here, and that requires util.wait + # :( + # so that the shutdown event completes + nettest.TestCase('mktemp')._wait(d) + + def run(self, result, input): + try: + nettest.TestSuite.run(self, result, input) + finally: + self._bail() + + +class OONIRunner(TrialRunner): + def run(self, test): + return TrialRunner.run(self, test) + + def _runWithoutDecoration(self, test): + """ + Private helper that runs the given test but doesn't decorate it. + """ + result = self._makeResult() + # decorate the suite with reactor cleanup and log starting + # This should move out of the runner and be presumed to be + # present + suite = OONISuite([test]) + print "HERE IS THE TEST:" + print test + print "-------------" + try: + inputs = test.inputs + except: + inputs = [None] + + startTime = time.time() + if self.mode == self.DRY_RUN: + for single in nettest._iterateTests(suite): + input = None + if type(single) == type(tuple()): + single, input = single + result.startTest(single, input) + result.addSuccess(single) + result.stopTest(single) + else: + if self.mode == self.DEBUG: + # open question - should this be self.debug() instead. + debugger = self._getDebugger() + run = lambda x: debugger.runcall(suite.run, result, x) + else: + run = lambda x: suite.run(result, x) + + oldDir = self._setUpTestdir() + try: + self._setUpLogFile() + # XXX work on this better + for input in inputs: + run(input) + finally: + self._tearDownLogFile() + self._tearDownTestdir(oldDir) + + endTime = time.time() + done = getattr(result, 'done', None) + if done is None: + warnings.warn( + "%s should implement done() but doesn't. Falling back to " + "printErrors() and friends." % reflect.qual(result.__class__), + category=DeprecationWarning, stacklevel=3) + result.printErrors() + result.writeln(result.separator) + result.writeln('Ran %d tests in %.3fs', result.testsRun, + endTime - startTime) + result.write('\n') + result.printSummary() + else: + result.done() + return result + + +class TestLoader(TestLoader): + """ + Reponsible for finding the modules that can work as tests and running them. + If we detect that a certain test is written using the legacy OONI API we + will wrap it around a next gen class to make it work here too. + """ + def __init__(self): + super(TestLoader, self).__init__() + self.suiteFactory = nettest.TestSuite + + def findTestClasses(self, module): + classes = [] + for name, val in inspect.getmembers(module): + if isTestCase(val): + classes.append(val) + # This is here to allow backward compatibility with legacy OONI + # tests. + elif isLegacyTest(val): + #val = adaptLegacyTest(val) + classes.append(val) + return self.sort(classes) + #return runner.TestLoader.findTestClasses(self, module)

1 0

[ooni-probe/master] Import basic stuff from Twisted trial
by isis＠torproject.org 04 Oct '12

04 Oct '12

commit c4ad2411046deb08f17902320cb8dd21cc25f367 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Sep 11 10:38:35 2012 +0000 Import basic stuff from Twisted trial --- .gitignore | 1 + README.md | 10 + bin/ooniprobe | 16 ++- ooni/oonicli.py | 397 +++++++++++++++++++++++++++++++++++++++++++ ooni/oonitests/bridget.py | 373 ---------------------------------------- ooni/plugins/new_bridget.py | 2 +- ooni/protocols/http.py | 13 +- ooni/reporter.py | 13 ++ ooni/runner.py | 7 + 9 files changed, 448 insertions(+), 384 deletions(-) diff --git a/.gitignore b/.gitignore index 7f270bb..6b1b9ce 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ proxy-lists/italy-http-ips.txt private/* /ooni/plugins/dropin.cache oonib/oonibackend.conf +ooni/assets/* diff --git a/README.md b/README.md index b39cb2b..123d837 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,16 @@ To list the help for a specific test: python ooniprobe.py httpt --help +## Virtualenv way (Recommended) + + virtualenv2 ENV/ + source ENV/bin/activate + pip install twisted Scapy + +To install the most up to date scapy version (requires mercurial): + + pip install hg+http://hg.secdev.org/scapy + # More details diff --git a/bin/ooniprobe b/bin/ooniprobe index 5c87831..1f0c26d 100755 --- a/bin/ooniprobe +++ b/bin/ooniprobe @@ -1,5 +1,11 @@ -#!/bin/sh -ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -echo $ROOT -export PYTHONPATH=$PYTHONPATH:$ROOT -python $ROOT/ooni/ooniprobe.py $1 +#!/usr/bin/python2 +# startup script based on twisted trial +# See http://twistedmatrix.com/ +import os, sys + +sys.path[:] = map(os.path.abspath, sys.path) + +sys.path.insert(0, os.path.abspath(os.getcwd())) + +from ooni.oonicli import run +run() diff --git a/ooni/oonicli.py b/ooni/oonicli.py new file mode 100644 index 0000000..8ace160 --- /dev/null +++ b/ooni/oonicli.py @@ -0,0 +1,397 @@ +#!/usr/bin/env python +# -*- coding: UTF-8 +# +# oonicli +# ********* +# +# oonicli is the next generation ooniprober. It based off of twisted's trial +# unit testing framework. +# +# :copyright: (c) 2012 by Arturo Filastò +# :license: see LICENSE for more details. +# +# original copyright (c) by Twisted Matrix Laboratories. + + +import sys, os, random, gc, time, warnings + +from twisted.internet import defer +from twisted.application import app +from twisted.python import usage, reflect, failure +from twisted.python.filepath import FilePath +from twisted import plugin +from twisted.python.util import spewer +from twisted.python.compat import set +from twisted.trial import runner, itrial, reporter + + +# Yea, this is stupid. Leave it for for command-line compatibility for a +# while, though. +TBFORMAT_MAP = { + 'plain': 'default', + 'default': 'default', + 'emacs': 'brief', + 'brief': 'brief', + 'cgitb': 'verbose', + 'verbose': 'verbose' + } + + +def _parseLocalVariables(line): + """ + Accepts a single line in Emacs local variable declaration format and + returns a dict of all the variables {name: value}. + Raises ValueError if 'line' is in the wrong format. + + See http://www.gnu.org/software/emacs/manual/html_node/File-Variables.html + """ + paren = '-*-' + start = line.find(paren) + len(paren) + end = line.rfind(paren) + if start == -1 or end == -1: + raise ValueError("%r not a valid local variable declaration" % (line,)) + items = line[start:end].split(';') + localVars = {} + for item in items: + if len(item.strip()) == 0: + continue + split = item.split(':') + if len(split) != 2: + raise ValueError("%r contains invalid declaration %r" + % (line, item)) + localVars[split[0].strip()] = split[1].strip() + return localVars + + +def loadLocalVariables(filename): + """ + Accepts a filename and attempts to load the Emacs variable declarations + from that file, simulating what Emacs does. + + See http://www.gnu.org/software/emacs/manual/html_node/File-Variables.html + """ + f = file(filename, "r") + lines = [f.readline(), f.readline()] + f.close() + for line in lines: + try: + return _parseLocalVariables(line) + except ValueError: + pass + return {} + + +def getTestModules(filename): + testCaseVar = loadLocalVariables(filename).get('test-case-name', None) + if testCaseVar is None: + return [] + return testCaseVar.split(',') + + +def isTestFile(filename): + """ + Returns true if 'filename' looks like a file containing unit tests. + False otherwise. Doesn't care whether filename exists. + """ + basename = os.path.basename(filename) + return (basename.startswith('test_') + and os.path.splitext(basename)[1] == ('.py')) + + +def _reporterAction(): + return usage.CompleteList([p.longOpt for p in + plugin.getPlugins(itrial.IReporter)]) + +class Options(usage.Options): + + optParameters = [ + ['parallelism', 'n', 10, "Specify the number of parallel tests to run"], + ['output', 'o', 'report.log', "Specify output report file"], + ['log', 'l', 'oonicli.log', "Specify output log file"] + ] + + def opt_version(self): + """ + Display OONI version and exit. + """ + print "OONI version:", __version__ + sys.exit(0) + + +class Options(usage.Options, app.ReactorSelectionMixin): + synopsis = """%s [options] [[file|package|module|TestCase|testmethod]...] + """ % (os.path.basename(sys.argv[0]),) + + longdesc = ("ooniprobe loads and executes a suite or a set of suites of" + "network tests. These are loaded from modules, packages and" + "files listed on the command line") + + optFlags = [["help", "h"], + ["rterrors", "e", "realtime errors, print out tracebacks as " + "soon as they occur"], + ["debug", "b", "Run tests in the Python debugger. Will load " + "'.pdbrc' from current directory if it exists."], + ["debug-stacktraces", "B", "Report Deferred creation and " + "callback stack traces"], + ["nopm", None, "don't automatically jump into debugger for " + "postmorteming of exceptions"], + ["dry-run", 'n', "do everything but run the tests"], + ["force-gc", None, "Have Trial run gc.collect() before and " + "after each test case."], + ["profile", None, "Run tests under the Python profiler"], + ["unclean-warnings", None, + "Turn dirty reactor errors into warnings"], + ["until-failure", "u", "Repeat test until it fails"], + ["no-recurse", "N", "Don't recurse into packages"], + ['help-reporters', None, + "Help on available output plugins (reporters)"] + ] + + optParameters = [ + ["logfile", "l", "test.log", "log file name"], + ["random", "z", None, + "Run tests in random order using the specified seed"], + ['temp-directory', None, '_trial_temp', + 'Path to use as working directory for tests.'], + ['reporter', None, 'verbose', + 'The reporter to use for this test run. See --help-reporters for ' + 'more info.']] + + compData = usage.Completions( + optActions={"tbformat": usage.CompleteList(["plain", "emacs", "cgitb"]), + "reporter": _reporterAction, + "logfile": usage.CompleteFiles(descr="log file name"), + "random": usage.Completer(descr="random seed")}, + extraActions=[usage.CompleteFiles( + "*.py", descr="file | module | package | TestCase | testMethod", + repeat=True)], + ) + + fallbackReporter = reporter.TreeReporter + tracer = None + + def __init__(self): + self['tests'] = set() + usage.Options.__init__(self) + + + def coverdir(self): + """ + Return a L{FilePath} representing the directory into which coverage + results should be written. + """ + coverdir = 'coverage' + result = FilePath(self['temp-directory']).child(coverdir) + print "Setting coverage directory to %s." % (result.path,) + return result + + + def opt_coverage(self): + """ + Generate coverage information in the I{coverage} file in the + directory specified by the I{trial-temp} option. + """ + import trace + self.tracer = trace.Trace(count=1, trace=0) + sys.settrace(self.tracer.globaltrace) + + + def opt_testmodule(self, filename): + """ + Filename to grep for test cases (-*- test-case-name) + """ + # If the filename passed to this parameter looks like a test module + # we just add that to the test suite. + # + # If not, we inspect it for an Emacs buffer local variable called + # 'test-case-name'. If that variable is declared, we try to add its + # value to the test suite as a module. + # + # This parameter allows automated processes (like Buildbot) to pass + # a list of files to Trial with the general expectation of "these files, + # whatever they are, will get tested" + if not os.path.isfile(filename): + sys.stderr.write("File %r doesn't exist\n" % (filename,)) + return + filename = os.path.abspath(filename) + if isTestFile(filename): + self['tests'].add(filename) + else: + self['tests'].update(getTestModules(filename)) + + + def opt_spew(self): + """ + Print an insanely verbose log of everything that happens. Useful + when debugging freezes or locks in complex code. + """ + sys.settrace(spewer) + + + def opt_help_reporters(self): + synopsis = ("OONI's output can be customized using plugins called " + "Reporters. You can\nselect any of the following " + "reporters using --reporter=<foo>\n") + print synopsis + for p in plugin.getPlugins(itrial.IReporter): + print ' ', p.longOpt, '\t', p.description + print + sys.exit(0) + + + def opt_disablegc(self): + """ + Disable the garbage collector + """ + gc.disable() + + + def opt_tbformat(self, opt): + """ + Specify the format to display tracebacks with. Valid formats are + 'plain', 'emacs', and 'cgitb' which uses the nicely verbose stdlib + cgitb.text function + """ + try: + self['tbformat'] = TBFORMAT_MAP[opt] + except KeyError: + raise usage.UsageError( + "tbformat must be 'plain', 'emacs', or 'cgitb'.") + + + def opt_recursionlimit(self, arg): + """ + see sys.setrecursionlimit() + """ + try: + sys.setrecursionlimit(int(arg)) + except (TypeError, ValueError): + raise usage.UsageError( + "argument to recursionlimit must be an integer") + + + def opt_random(self, option): + try: + self['random'] = long(option) + except ValueError: + raise usage.UsageError( + "Argument to --random must be a positive integer") + else: + if self['random'] < 0: + raise usage.UsageError( + "Argument to --random must be a positive integer") + elif self['random'] == 0: + self['random'] = long(time.time() * 100) + + + def opt_without_module(self, option): + """ + Fake the lack of the specified modules, separated with commas. + """ + for module in option.split(","): + if module in sys.modules: + warnings.warn("Module '%s' already imported, " + "disabling anyway." % (module,), + category=RuntimeWarning) + sys.modules[module] = None + + + def parseArgs(self, *args): + self['tests'].update(args) + + + def _loadReporterByName(self, name): + for p in plugin.getPlugins(itrial.IReporter): + qual = "%s.%s" % (p.module, p.klass) + if p.longOpt == name: + return reflect.namedAny(qual) + raise usage.UsageError("Only pass names of Reporter plugins to " + "--reporter. See --help-reporters for " + "more info.") + + + def postOptions(self): + # Only load reporters now, as opposed to any earlier, to avoid letting + # application-defined plugins muck up reactor selecting by importing + # t.i.reactor and causing the default to be installed. + self['reporter'] = self._loadReporterByName(self['reporter']) + + if 'tbformat' not in self: + self['tbformat'] = 'default' + if self['nopm']: + if not self['debug']: + raise usage.UsageError("you must specify --debug when using " + "--nopm ") + failure.DO_POST_MORTEM = False + + + +def _initialDebugSetup(config): + # do this part of debug setup first for easy debugging of import failures + if config['debug']: + failure.startDebugMode() + if config['debug'] or config['debug-stacktraces']: + defer.setDebugging(True) + + + +def _getSuite(config): + loader = _getLoader(config) + recurse = not config['no-recurse'] + return loader.loadByNames(config['tests'], recurse) + + + +def _getLoader(config): + loader = runner.TestLoader() + if config['random']: + randomer = random.Random() + randomer.seed(config['random']) + loader.sorter = lambda x : randomer.random() + print 'Running tests shuffled with seed %d\n' % config['random'] + if not config['until-failure']: + loader.suiteFactory = runner.DestructiveTestSuite + return loader + + + +def _makeRunner(config): + mode = None + if config['debug']: + mode = runner.TrialRunner.DEBUG + if config['dry-run']: + mode = runner.TrialRunner.DRY_RUN + return runner.TrialRunner(config['reporter'], + mode=mode, + profile=config['profile'], + logfile=config['logfile'], + tracebackFormat=config['tbformat'], + realTimeErrors=config['rterrors'], + uncleanWarnings=config['unclean-warnings'], + workingDirectory=config['temp-directory'], + forceGarbageCollection=config['force-gc']) + + + +def run(): + if len(sys.argv) == 1: + sys.argv.append("--help") + config = Options() + try: + config.parseOptions() + except usage.error, ue: + raise SystemExit, "%s: %s" % (sys.argv[0], ue) + _initialDebugSetup(config) + trialRunner = _makeRunner(config) + suite = _getSuite(config) + if config['until-failure']: + test_result = trialRunner.runUntilFailure(suite) + else: + test_result = trialRunner.run(suite) + if config.tracer: + sys.settrace(None) + results = config.tracer.results() + results.write_results(show_missing=1, summary=False, + coverdir=config.coverdir().path) + sys.exit(not test_result.wasSuccessful()) + diff --git a/ooni/oonitests/bridget.py b/ooni/oonitests/bridget.py deleted file mode 100644 index a613f61..0000000 --- a/ooni/oonitests/bridget.py +++ /dev/null @@ -1,373 +0,0 @@ -# -*- coding: UTF-8 -""" - bridgeT - ******* - - an OONI test (we call them Plugoos :P) aimed - at detecting if a set of Tor bridges are working or not. - - :copyright: (c) 2012 by Arturo Filastò - :license: BSD, see LICENSE for more details. -""" -import os -import sys -import errno -import time -import random -import re -import glob -import socks -import socket -from shutil import rmtree -from subprocess import Popen, PIPE -from datetime import datetime - -import shutil -import gevent -from gevent import socket -import fcntl -from plugoo.assets import Asset -from plugoo.tests import Test -import urllib2 -import httplib -import json - -try: - from TorCtl import TorCtl -except: - print "Error TorCtl not installed!" - -__plugoo__ = "BridgeT" -__desc__ = "BridgeT, for testing Tor Bridge reachability" -ONIONOO_URL="http://85.214.195.203/summary/search/" - -class SocksiPyConnection(httplib.HTTPConnection): - def __init__(self, proxytype, proxyaddr, proxyport = None, rdns = True, - username = None, password = None, *args, **kwargs): - self.proxyargs = (proxytype, proxyaddr, proxyport, rdns, username, password) - httplib.HTTPConnection.__init__(self, *args, **kwargs) - - def connect(self): - self.sock = socks.socksocket() - self.sock.setproxy(*self.proxyargs) - if isinstance(self.timeout, float): - self.sock.settimeout(self.timeout) - self.sock.connect((self.host, self.port)) - -class SocksiPyHandler(urllib2.HTTPHandler): - def __init__(self, *args, **kwargs): - self.args = args - self.kw = kwargs - urllib2.HTTPHandler.__init__(self) - - def http_open(self, req): - def build(host, port=None, strict=None, timeout=0): - conn = SocksiPyConnection(*self.args, host=host, port=port, - strict=strict, timeout=timeout, **self.kw) - return conn - return self.do_open(build, req) - -class BridgeTAsset(Asset): - def __init__(self, file=None): - self = Asset.__init__(self, file) - -class BridgeT(Test): - # This is the timeout value after which - # we will give up - timeout = 20 - # These are the modules that should be torified - modules = [urllib2] - - def tor_greater_than(self, version): - """ - Checks if the currently installed version of Tor is greater - than the required version. - - :version The version of Tor to check against for greater than or equal - """ - fullstring = os.popen("tor --version").read().split('\n')[-2] - v_array = fullstring.split(' ')[2].split('-') - minor = v_array[1:] - v_array = v_array[0].split('.') - minor_p = version.split('-')[1:] - v_array_p = version.split('-')[0].split('.') - - for i, x in enumerate(v_array): - try: - if i > len(v_array_p): - break - - if int(x) > int(v_array_p[i]): - self.logger.debug("The Tor version is greater than %s" % version) - return True - elif int(x) == int(v_array_p[i]): - self.logger.debug("The Tor version is greater than %s" % version) - continue - else: - self.logger.debug("You run an outdated version of Tor: %s (< %s)" % (fullstring, version)) - return False - except: - self.logger.error("Error in parsing your Tor version string: %s" % fullstring) - return False - - self.logger.debug("The Tor version is equal to %s" % version) - return True - # XXX currently don't consider the minor parts of the version - # (alpha, dev, beta, etc.) - - def free_port(self, port): - s = socket.socket() - try: - s.bind(('127.0.0.1', port)) - s.close() - return True - except: - self.logger.warn("The randomly chosen port was already taken!") - s.close() - return False - - def writetorrc(self, bridge): - """ - Write the torrc file for the tor process to be used - to test the bridge. - - :bridge the bridge to be tested - """ - self.failures = [] - prange = (49152, 65535) - - # register Tor to an ephemeral port - socksport = random.randint(prange[0], prange[1]) - # Keep on trying to get a new port if the chosen one is already - # taken. - while not self.free_port(socksport): - socksport = random.randint(prange[0], prange[1]) - controlport = random.randint(prange[0], prange[1]) - while not self.free_port(controlport): - controlport = random.randint(prange[0], prange[1]) - - randomname = "tor_"+str(random.randint(0, 424242424242)) - datadir = "/tmp/" + randomname - if bridge.startswith("obfs://"): - obfsbridge = bridge.split("//")[1] - - self.logger.debug("Genearting torrc file for obfs bridge") - torrc = """SocksPort %s -UseBridges 1 -Bridge obfs2 %s -DataDirectory %s -ClientTransportPlugin obfs2 exec /usr/local/bin/obfsproxy --managed -ControlPort %s -Log info file %s -""" % (socksport, obfsbridge, datadir, controlport, os.path.join(datadir,'tor.log')) - else: - self.logger.debug("Generating torrc file for bridge") - if self.tor_greater_than('0.2.3.2'): - - torrc = """SocksPort %s -UseBridges 1 -bridge %s -DataDirectory %s -usemicrodescriptors 0 -ControlPort %s -Log info file %s -""" % (socksport, bridge, datadir, controlport, os.path.join(datadir,'tor.log')) - else: - torrc = """SocksPort %s -UseBridges 1 -bridge %s -DataDirectory %s -ControlPort %s -Log info file %s -""" % (socksport, bridge, datadir, controlport, os.path.join(datadir,'tor.log')) - - with open(randomname, "wb") as f: - f.write(torrc) - - os.mkdir(datadir) - return (randomname, datadir, controlport, socksport) - - def parsebridgeinfo(self, output): - ret = {} - fields = ['router', 'platform', 'opt', 'published', 'uptime', 'bandwidth'] - - for x in output.split("\n"): - cfield = x.split(' ') - if cfield[0] in fields: - #not sure if hellais did this on purpose, but this overwrites - #the previous entries. For ex, 'opt' has multiple entries and - #only the last value is stored - ret[cfield[0]] = ' '.join(cfield[1:]) - if cfield[1] == 'fingerprint': - ret['fingerprint'] = ''.join(cfield[2:]) - return ret - - #Can't use @torify as it doesn't support concurrency right now - def download_file(self, socksport): - opener = urllib2.build_opener(SocksiPyHandler(socks.PROXY_TYPE_SOCKS5, - '127.0.0.1', int(socksport))) - - time_start=time.time() - f = opener.open('http://38.229.72.16/bwauth.torproject.org/256k') - f.read() - time_end = time.time() - print (time_end-time_start) - return str(256/(time_end-time_start)) + " KB/s" - - def is_public(self, fp, socksport): - opener = urllib2.build_opener(SocksiPyHandler(socks.PROXY_TYPE_SOCKS5,'127.0.0.1',int(socksport))) - response = opener.open(str(ONIONOO_URL)+str(fp)) - reply = json.loads(response.read()) - if reply['bridges'] or reply['relays']: - return True - return False - - def connect(self, bridge, timeout=None): - bridgeinfo = None - bandwidth = None - public = None - if not timeout: - if self.config.tests.tor_bridges_timeout: - self.timeout = self.config.tests.tor_bridges_timeout - timeout = self.timeout - torrc, tordir, controlport, socksport = self.writetorrc(bridge) - cmd = ["tor", "-f", torrc] - - tupdate = time.time() - debugupdate = time.time() - - try: - p = Popen(cmd, stdout=PIPE) - except: - self.logger.error("Error in starting Tor (do you have tor installed?)") - - self.logger.info("Testing bridge: %s" % bridge) - while True: - o = "" - try: - o = p.stdout.read(4096) - if o: - self.logger.debug(str(o)) - if re.search("100%", o): - self.logger.info("Success in connecting to %s" % bridge) - - print "%s bridge works" % bridge - # print "%s controlport" % controlport - try: - c = TorCtl.connect('127.0.0.1', controlport) - bridgeinfo = self.parsebridgeinfo(c.get_info('dir/server/all')['dir/server/all']) - c.close() - except: - self.logger.error("Error in connecting to Tor Control port") - - # XXX disable the public checking - #public = self.is_public(bridgeinfo['fingerprint'], socksport) - #self.logger.info("Public: %s" % public) - - bandwidth = self.download_file(socksport) - self.logger.info("Bandwidth: %s" % bandwidth) - - try: - p.stdout.close() - except: - self.logger.error("Error in closing stdout FD.") - - try: - os.unlink(os.path.join(os.getcwd(), torrc)) - rmtree(tordir) - except: - self.logger.error("Error in unlinking files.") - - p.terminate() - return { - 'Time': datetime.now(), - 'Bridge': bridge, - 'Working': True, - 'Descriptor': bridgeinfo, - 'Calculated bandwidth': bandwidth, - 'Public': public - } - - if re.search("%", o): - # Keep updating the timeout if there is progress - self.logger.debug("Updating time...") - tupdate = time.time() - #print o - continue - - except IOError: - ex = sys.exc_info()[1] - if ex[0] != errno.EAGAIN: - self.logger.error("Error IOError: EAGAIN") - raise - sys.exc_clear() - print "In this exception 1" - - try: - # Set the timeout for the socket wait - ct = timeout-(time.time() - tupdate) - socket.wait_read(p.stdout.fileno(), timeout=ct) - - except: - lfh = open(os.path.join(tordir, 'tor.log'), 'r') - log = lfh.readlines() - lfh.close() - self.logger.info("%s bridge does not work (%s s timeout)" % (bridge, timeout)) - print "%s bridge does not work (%s s timeout)" % (bridge, timeout) - self.failures.append(bridge) - p.stdout.close() - os.unlink(os.path.join(os.getcwd(), torrc)) - rmtree(tordir) - p.terminate() - return { - 'Time': datetime.now(), - 'Bridge': bridge, - 'Working': False, - 'Descriptor': {}, - 'Log': log - } - - def experiment(self, *a, **kw): - # this is just a dirty hack - bridge = kw['data'] - print "Experiment" - config = self.config - - return self.connect(bridge) - - def clean(self): - for infile in glob.glob('tor_*'): - os.remove(infile) - - def print_failures(self): - if self.failures: - for item in self.failures: - print "Offline : %s" % item - else: - print "All online" - - # For logging TorCtl event msgs - #class LogHandler: - #def msg(self, severity, message): - # print "[%s] %s"%(severity, message) - -def run(ooni, assets=None): - """ - Run the test - """ - - config = ooni.config - urls = [] - - bridges = BridgeTAsset(os.path.join(config.main.assetdir, \ - config.tests.tor_bridges)) - - bridgelist = [bridges] - - bridget = BridgeT(ooni) - ooni.logger.info("Starting bridget test") - bridget.run(bridgelist) - bridget.print_failures() - bridget.clean() - ooni.logger.info("Testing completed!") - diff --git a/ooni/plugins/new_bridget.py b/ooni/plugins/new_bridget.py index 3e4db56..b26455e 100644 --- a/ooni/plugins/new_bridget.py +++ b/ooni/plugins/new_bridget.py @@ -27,7 +27,7 @@ class bridgetArgs(usage.Options): class bridgetTest(OONITest): implements(IPlugin, ITest) - shortName = "bridget" + shortName = "newbridget" description = "bridget" requirements = None options = bridgetArgs diff --git a/ooni/protocols/http.py b/ooni/protocols/http.py index 2b38f28..09bb9b9 100644 --- a/ooni/protocols/http.py +++ b/ooni/protocols/http.py @@ -84,11 +84,7 @@ class HTTPTest(OONITest): """ pass - - def experiment(self, args): - log.msg("Running experiment") - url = self.local_options['url'] if 'url' not in args else args['url'] - + def doRequest(self, url): d = self.build_request(url) def finished(data): return data @@ -97,6 +93,13 @@ class HTTPTest(OONITest): d.addCallback(finished) return d + def experiment(self, args): + log.msg("Running experiment") + url = self.local_options['url'] if 'url' not in args else args['url'] + + d = self.doRequest(url) + return d + def _cbResponse(self, response): self.response['headers'] = list(response.headers.getAllRawHeaders()) self.response['code'] = response.code diff --git a/ooni/reporter.py b/ooni/reporter.py new file mode 100644 index 0000000..0ecf2ea --- /dev/null +++ b/ooni/reporter.py @@ -0,0 +1,13 @@ +from twisted.trial import reporter + +class TestResult(reporter.TestResult): + """ + Accumulates the results of several ooni.nettest.TestCases. + + The output format of a TestResult is YAML and it will contain all the basic + information that a test result should contain. + """ + def __init__(self): + super(TestResult, self).__init__() + + diff --git a/ooni/runner.py b/ooni/runner.py new file mode 100644 index 0000000..d7caa9d --- /dev/null +++ b/ooni/runner.py @@ -0,0 +1,7 @@ +from twisted.trial import runner + + +class TestLoader(runner.TestLoader): + pass + +

1 0

[ooni-probe/master] Merge branch 'master' of ssh://git-rw.torproject.org/ooni-probe
by isis＠torproject.org 04 Oct '12

04 Oct '12

commit 2e49b7330b155ca4746896939ebe48ce67f147b3 Merge: 0080bf6 62956eb Author: Arturo Filastò <arturo(a)filasto.net> Date: Thu Sep 13 23:16:03 2012 +0000 Merge branch 'master' of ssh://git-rw.torproject.org/ooni-probe Conflicts: ooni/assets/bridgetests.txt ooni/ooniprobe.py ooni/plugins/domclass.py ooni/plugins/new_bridget.py .gitmodules | 9 + README.md | 22 ++- TODO | 26 +-- ooni/__init__.py | 2 +- ooni/assets/bridgetests.txt | 8 + ooni/example_plugins/skel.py | 4 +- ooni/lib/Makefile | 30 --- ooni/lib/__init__.py | 41 +++- ooni/lib/txscapy | 1 + ooni/lib/txscapy.py | 363 ------------------------------- ooni/lib/txtorcon | 1 + ooni/lib/txtraceroute | 1 + ooni/ooniprobe.log | 1 - ooni/ooniprobe.py | 19 ++- ooni/oonitests/dnstamper.py | 142 ++++++++----- ooni/plugins/blocking.py | 4 +- ooni/plugins/dnstamper.py | 343 ++++++++++++++++++++++++++++++ ooni/plugins/domclass.py | 3 +- ooni/plugins/httphost.py | 4 +- ooni/plugins/new_bridget.py | 484 ++++++++++++++++++++++++++++++++++-------- ooni/plugoo/reports.py | 2 +- ooni/plugoo/tests.py | 25 ++- ooni/plugoo/work.py | 2 + ooni/utils/log.py | 116 ++++++++--- 24 files changed, 1045 insertions(+), 608 deletions(-) diff --cc ooni/assets/bridgetests.txt index 5519eea,7bba841..b69fbca --- a/ooni/assets/bridgetests.txt +++ b/ooni/assets/bridgetests.txt @@@ -1,3 -1,5 +1,11 @@@ ++<<<<<<< HEAD +88.130.86.191:443 +195.74.237.236:9001 +127.0.0.1:9050 ++======= + #213.151.89.102:9001 + #108.166.106.156:443 + #217.150.224.213:443 + 85.193.252.111:443 + #68.98.41.14:9001 ++>>>>>>> 62956ebab7779c1b61ce3d6e8ac750552fd1c988 diff --cc ooni/plugins/new_bridget.py index 3e4db56,0000000..736b084 mode 100644,000000..100644 --- a/ooni/plugins/new_bridget.py +++ b/ooni/plugins/new_bridget.py @@@ -1,105 -1,0 +1,421 @@@ - """ - This is a self genrated test created by scaffolding.py. - you will need to fill it up with all your necessities. - Safe hacking :). - """ - from exceptions import Exception - from datetime import datetime - from zope.interface import implements - from twisted.python import usage - from twisted.plugin import IPlugin - from twisted.internet import reactor, task - - from ooni.utils import log - from ooni.plugoo.tests import ITest, OONITest - from ooni.plugoo.assets import Asset - - from ooni.lib.txtorcon import TorProtocolFactory, TorConfig, TorState - from ooni.lib.txtorcon import DEFAULT_VALUE, launch_tor - - class bridgetArgs(usage.Options): - optParameters = [['bridges', 'b', None, 'List of bridges to scan'], - ['relays', 'f', None, 'List of relays to scan'], - ['resume', 'r', 0, 'Resume at this index'], - ['timeout', 't', 5, 'Timeout in seconds after which to consider a bridge not working'] - ] - - class bridgetTest(OONITest): ++#!/usr/bin/env python ++# -*- encoding: utf-8 -*- ++# ++# +-----------+ ++# | BRIDGET | ++# | +----------------------------------------------+ ++# +--------| Use a slave Tor process to test making a Tor | ++# | connection to a list of bridges or relays. | ++# +----------------------------------------------+ ++# ++# :authors: Arturo Filasto, Isis Lovecruft ++# :licence: see included LICENSE ++# :version: 0.1.0-alpha ++ ++from __future__ import with_statement ++from zope.interface import implements ++from twisted.python import usage ++from twisted.plugin import IPlugin ++from twisted.internet import defer, error, reactor ++ ++import random ++import sys ++ ++try: ++ from ooni.lib.txtorcon import CircuitListenerMixin, IStreamAttacher ++except: ++ print "BridgeT requires txtorcon: https://github.com/meejah/txtorcon.git" ++ print "Your copy of OONI should have it included, if you're seeing this" ++ print "message, please file a bug report." ++ log.msg ("Bridget: Unable to import from ooni.lib.txtorcon") ++ ++from ooni.utils import log ++from ooni.plugoo.tests import ITest, OONITest ++from ooni.plugoo.assets import Asset ++ ++ ++class BridgetArgs(usage.Options): ++ ++ def portCheck(number): ++ number = int(number) ++ if number not in range(1024, 65535): ++ raise ValueError("Port out of range") ++ portCheck.coerceDoc = "Ports must be between 1024 and 65535" ++ ++ optParameters = [ ++ ['bridges', 'b', None, ++ 'List of bridges to scan <IP>:<ORport>'], ++ ['relays', 'f', None, ++ 'List of relays to scan <IP>'], ++ ['socks', 's', 9049, portCheck, ++ 'Tor SocksPort to use'], ++ ['control', 'c', 9052, portCheck, ++ 'Tor ControlPort to use'], ++ ['tor-path', 'p', None, ++ 'Path to the Tor binary to use'], ++ ['data-dir', 'd', None, ++ 'Tor DataDirectory to use'], ++ ['transport', 't', None, ++ 'Tor ClientTransportPlugin'], ++ ['resume', 'r', 0, ++ 'Resume at this index']] ++ optFlags = [['random', 'x', 'Randomize control and socks ports']] ++ ++ def postOptions(self): ++ ## We can't test pluggable transports without bridges ++ if self['transport'] and not self['bridges']: ++ e = "Pluggable transport requires the bridges option" ++ raise usage.UsageError, e ++ ## We can't use random and static port simultaneously ++ if self['socks'] and self['control']: ++ if self['random']: ++ e = "Unable to use random and specific ports simultaneously" ++ raise usageError, e ++ ++class CustomCircuit(CircuitListenerMixin): ++ implements(IStreamAttacher) ++ ++ from txtorcon.interface import IRouterContainer, ICircuitContainer ++ ++ def __init__(self, state): ++ self.state = state ++ self.waiting_circuits = [] ++ ++ def waiting_on(self, circuit): ++ for (circid, d) in self.waiting_circuits: ++ if circuit.id == circid: ++ return true ++ return False ++ ++ def circuit_extend(self, circuit, router): ++ "ICircuitListener" ++ if circuit.purpose != 'GENERAL': ++ return ++ if self.waiting_on(circuit): ++ log.msg("Circuit %d (%s)" % (circuit.id, router.id_hex)) ++ ++ def circuit_built(self, circuit): ++ "ICircuitListener" ++ if circuit.purpose != 'GENERAL': ++ return ++ ++ log.msg("Circuit %s built ..." % circuit.id) ++ log.msg("Full path of %s: %s" % (circuit.id, circuit.path)) ++ ++ for (circid, d) in self.waiting_circuits: ++ if circid == circuit.id: ++ self.waiting_circuits.remove(circid, d) ++ d.callback(circuit) ++ ++ def circuit_failed(self, circuit, reason): ++ if self.waiting_on(circuit): ++ log.msg("A circuit we requested %s failed for reason %s" % ++ (circuit.id, reason)) ++ circid, d = None, None ++ for x in self.waiting_circuits: ++ if x[0] == circuit.id: ++ circid, d, stream_cc = x ++ if d is None: ++ raise Exception("Expected to find circuit.") ++ ++ self.waiting_circuits.remove((circid, d)) ++ log.msg("Trying to build a circuit for %s" % circid) ++ self.request_circuit_build(d) ++ ++ def check_circuit_route(self, circuit, router): ++ if router in circuit.path: ++ #router.update() ## XXX can i use without args? no. ++ TorInfo.dump(self) ++ ++ def request_circuit_build(self, deferred): ++ entries = self.state.entry_guards.value() ++ relays = self.state.routers.values() ++ ++ log.msg("We have these nodes listed as entry guards:") ++ log.msg("%s" % entries) ++ log.msg("We have these nodes listed as relays:") ++ log.msg("%s" % relays) ++ ++ path = [random.choice(entries), ++ random.choice(relays), ++ random.choice(relays)] ++ ++ log.msg("Requesting a circuit: %s" ++ % '-->'.join(map(lambda x: x.location.countrycode, path))) ++ ++ class AppendWaiting: ++ def __init__(self, attacher, deferred): ++ self.attacher = attacher ++ self.d = deferred ++ ++ def __call__(self, circuit): ++ """ ++ Return from build_circuit is a Circuit, however, we want to ++ wait until it is built before we can issue an attach on it and ++ callback to the Deferred we issue here. ++ """ ++ log.msg("Circuit %s is in progress ..." % circuit.id) ++ self.attacher.waiting_circuits.append((circuit.id, self.d)) ++ ++ return self.state.build_circuit(path).addCallback(AppendWaiting(self, deferred_to_callback)).addErrback(log.err) ++ ++class BridgetAsset(Asset): ++ """ ++ Class for parsing bridge assets so that they can be commented out. ++ """ ++ def __init__(self, file=None): ++ self = Asset.__init__(self, file) ++ ++ def parse_line(self, line): ++ if line.startswith('#'): ++ return ++ else: ++ return line.replace('\n','') ++ ++class BridgetTest(OONITest): ++ """ ++ XXX fill me in ++ ++ :ivar config: ++ An :class:`ooni.lib.txtorcon.TorConfig` instance. ++ :ivar relay_list: ++ A list of all provided relays to test. We have to do this because ++ txtorcon.TorState().entry_guards won't build a custom circuit if the ++ first hop isn't in the torrc's EntryNodes. ++ :ivar bridge_list: ++ A list of all provided bridges to test. ++ :ivar socks_port: ++ Integer for Tor's SocksPort. ++ :ivar control_port: ++ Integer for Tor's ControlPort. ++ :ivar plug_transport: ++ String defining the Tor's ClientTransportPlugin, for testing ++ a bridge's pluggable transport functionality. ++ :ivar tor_binary: ++ Path to the Tor binary to use, e.g. \'/usr/sbin/tor\' ++ """ + implements(IPlugin, ITest) + + shortName = "bridget" - description = "bridget" ++ description = "Use a Tor process to test connecting to bridges and relays" + requirements = None - options = bridgetArgs ++ options = BridgetArgs + blocking = False + - def experiment(self, args): - log.msg("Doing test") - last_update = datetime.now() - tor_log = [] - - def check_timeout(): - log.msg("Checking for timeout") - time_since_update = datetime.now() - last_update - if time_since_update.seconds > self.local_options['timeout']: - log.msg("Timed out when connecting to %s" % args) - l.stop() - self.result['reason'] = 'timeout' - d.errback(args) - return ++ def initialize(self): ++ """ ++ Extra initialization steps. We only want one child Tor process ++ running, so we need to deal with the creation of TorConfig() only ++ once, before the experiment runs. ++ """ ++ self.socks_port = 9049 ++ self.control_port = 9052 ++ self.tor_binary = '/usr/sbin/tor' ++ self.data_directory = None + - def updates(prog, tag, summary): - tor_log.append((prog, tag, summary)) - last_update = datetime.now() - log.msg("%d%%: %s" % (prog, summary)) ++ if self.local_options: ++ try: ++ from ooni.lib.txtorcon import TorConfig ++ except: ++ e = "Could not import TorConfig class from txtorcon!" ++ raise ImportError, e + - def setup_failed(failure): - log.msg("Setup Failed.") - if not self.result['reason']: - self.result['reason'] = 'unknown' - self.result['input'] = args - self.result['result'] = 'failed' - self.result['tor_log'] = tor_log - return ++ options = self.local_options ++ self.config = TorConfig() + - def setup_complete(proto): - log.msg("Setup Complete.") - self.result['input'] = args - self.result['result'] = 'success' - return ++ ## Don't run the experiment if we don't have anything to test ++ if not options['bridges'] and not options['relays']: ++ self.suicide = True ++ ++ if options['bridges']: ++ self.config.UseBridges = 1 + - config = TorConfig() - import random - config.SocksPort = random.randint(1024, 2**16) - config.ControlPort = random.randint(1024, 2**16) ++ if options['relays']: ++ ## Stupid hack for testing only relays: ++ ## Tor doesn't use EntryNodes when UseBridges is enabled, but ++ ## config.state.entry_guards needs to include the first hop to ++ ## build a custom circuit. ++ self.config.EntryNodes = ','.join(relay_list) + - if 'bridge' in args: - config.UseBridges = 1 - config.Bridge = args['bridge'] ++ if options['socks']: ++ self.socks_port = options['socks'] + - config.save() ++ if options['control']: ++ self.control_port = options['control'] + - print config.config - self.result['tor_config'] = config.config - log.msg("Starting Tor connecting to %s" % args['bridge']) ++ if options['random']: ++ log.msg("Using randomized ControlPort and SocksPort ...") ++ self.socks_port = random.randint(1024, 2**16) ++ self.control_port = random.randint(1024, 2**16) + - l = task.LoopingCall(check_timeout) - l.start(1.0) ++ if options['tor-path']: ++ self.tor_binary = options['tor-path'] + - d = launch_tor(config, self.reactor, control_port=config.ControlPort, progress_updates=updates) - d.addCallback(setup_complete) - d.addErrback(setup_failed) - return d ++ if options['data-dir']: ++ self.config.DataDirectory = options['data-dir'] ++ ++ if options['transport']: ++ ## ClientTransportPlugin transport socks4|socks5 IP:PORT ++ ## ClientTransportPlugin transport exec path-to-binary [options] ++ if not options['bridges']: ++ e = "You must use the bridge option to test a transport." ++ raise usage.UsageError("%s" % e) ++ ++ log.msg("Using pluggable transport ...") ++ ## XXX fixme there's got to be a better way to check the exec ++ assert type(options['transport']) is str ++ self.config.ClientTransportPlugin = options['transport'] ++ ++ self.config.SocksPort = self.socks_port ++ self.config.ControlPort = self.control_port ++ self.config.save() + + def load_assets(self): - assets = {} ++ """ ++ Load bridges and/or relays from files given in user options. Bridges ++ should be given in the form IP:ORport. We don't want to load these as ++ assets, because it's inefficient to start a Tor process for each one. ++ """ ++ assets = {} ++ self.bridge_list = [] ++ self.relay_list = [] ++ ++ ## XXX fix me ++ ## we should probably find a more memory nice way to load addresses, ++ ## in case the files are really large + if self.local_options: + if self.local_options['bridges']: - assets.update({'bridge': Asset(self.local_options['bridges'])}) - elif self.local_options['relays']: - assets.update({'relay': Asset(self.local_options['relay'])}) ++ log.msg("Loading bridge information from %s ..." ++ % self.local_options['bridges']) ++ with open(self.local_options['bridges']) as bridge_file: ++ for line in bridge_file.readlines(): ++ if line.startswith('#'): ++ continue ++ else: ++ self.bridge_list.append(line.replace('\n','')) ++ assets.update({'bridges': self.bridge_list}) ++ ++ if self.local_options['relays']: ++ log.msg("Loading relay information from %s ..." ++ % self.local_options['relays']) ++ with open(options['relays']) as relay_file: ++ for line in relay_file.readlines(): ++ if line.startswith('#'): ++ continue ++ else: ++ self.relay_list.append(line.replace('\n','')) ++ assets.update({'relays': self.relay_list}) + return assets + - # We need to instantiate it otherwise getPlugins does not detect it - # XXX Find a way to load plugins without instantiating them. - bridget = bridgetTest(None, None, None) ++ def experiment(self, args): ++ """ ++ XXX fill me in ++ ++ :param args: ++ The :class:`ooni.plugoo.asset.Asset <Asset>` line currently being ++ used. ++ :meth launch_tor: ++ Returns a Deferred which callbacks with a ++ :class:`ooni.lib.txtorcon.torproto.TorProcessProtocol ++ <TorProcessProtocol>` connected to the fully-bootstrapped Tor; ++ this has a :class:`ooni.lib.txtorcon.torcontol.TorControlProtocol ++ <TorControlProtocol>` instance as .protocol. ++ """ ++ from ooni.lib.txtorcon import TorProtocolFactory, TorConfig, TorState ++ from ooni.lib.txtorcon import DEFAULT_VALUE, launch_tor ++ ++ def bootstrap(ctrl): ++ """ ++ Launch a Tor process with the TorConfig instance returned from ++ initialize(). ++ """ ++ conf = TorConfig(ctrl) ++ conf.post_bootstrap.addCallback(setup_done).addErrback(setup_fail) ++ log.msg("Tor process connected, bootstrapping ...") ++ ++ def reconf_controller(conf, bridge): ++ ## if bridges and relays, use one bridge then build a circuit ++ ## from three relays ++ conf.Bridge = bridge ++ ## XXX do we need a SIGHUP to restart? ++ ++ ## XXX see txtorcon.TorControlProtocol.add_event_listener we ++ ## may not need full CustomCircuit class ++ ++ ## if bridges only, try one bridge at a time, but don't build ++ ## circuits, just return ++ ## if relays only, build circuits from relays ++ ++ def reconf_fail(args): ++ log.msg("Reconfiguring Tor config with args %s failed" % args) ++ reactor.stop() ++ ++ def setup_fail(args): ++ log.msg("Setup Failed.") ++ report.update({'failed': args}) ++ reactor.stop() ++ ++ def setup_done(proto): ++ log.msg("Setup Complete: %s" % proto) ++ state = TorState(proto.tor_protocol) ++ state.post_bootstrap.addCallback(state_complete).addErrback(setup_fail) ++ report.update({'success': args}) ++ ++ def updates(prog, tag, summary): ++ log.msg("%d%%: %s" % (prog, summary)) ++ ++ if len(args) == 0: ++ log.msg("Bridget needs lists of bridges and/or relays to test!") ++ log.msg("Exiting ...") ++ d = sys.exit() ++ return d ++ ++ else: ++ if len(self.bridge_list) >= 1: ++ for bridge in self.bridge_list: ++ try: ++ print "BRIDGE IS %s" % bridge ++ reconf_controller(self.config, bridge) ++ except: ++ reconf_fail(bridge) ++ ++ log.msg("Bridget: initiating test ... ") ++ log.msg("Using the following as our torrc:\n%s" ++ % self.config.create_torrc()) ++ report = {'tor_config': self.config.config} ++ log.msg("Starting Tor ...") ++ ++ ## :return: a Deferred which callbacks with a TorProcessProtocol ++ ## connected to the fully-bootstrapped Tor; this has a ++ ## txtorcon.TorControlProtocol instance as .protocol. ++ d = launch_tor(self.config, ++ reactor, ++ progress_updates=updates, ++ tor_binary=self.tor_binary) ++ d.addCallback(bootstrap, self.config) ++ d.addErrback(setup_fail) ++ ## now build circuits ++ ++ #print "Tor process ID: %s" % d.transport.pid ++ return d ++ ++## So that getPlugins() can register the Test: ++bridget = BridgetTest(None, None, None) ++ ++## ISIS' NOTES ++## ----------- ++## self.config.save() only needs to be called if Tor is already running. ++## ++## need to add transport type to torrc Bridge line: ++## Bridge <transport> IP:ORPort <fingerprint> ++## ++## TODO: ++## o add option for any kwarg=arg self.config setting ++## o cleanup documentation ++## o check if bridges are public relays ++## o take bridge_desc file as input, also be able to give same ++## format as output ++## o change the stupid name ++## ++## FIX: ++## data directory is not found, or permissions aren't right

1 0