- tor-commits - lists.torproject.org

[flashproxy/master] Increase version to 0.2.
by dcf＠torproject.org 10 Jul '12

10 Jul '12

commit 2169199df9718642a7c8acb1f6562eb5a9b201d3 Author: David Fifield <david(a)bamsoftware.com> Date: Tue Jul 10 10:58:04 2012 -0700 Increase version to 0.2. --- Makefile | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Makefile b/Makefile index cbe5344..a4fcd18 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ PREFIX = /usr/local BINDIR = $(PREFIX)/bin -VERSION = 0.1 +VERSION = 0.2 CLIENT_DIST_FILES = flashproxy-client.py flashproxy-reg-http.py README torrc

1 0

[flashproxy/master] Use an iframe attribute called frameborder not frameBorder.
by dcf＠torproject.org 10 Jul '12

10 Jul '12

commit 154bcc1c9dc34c657448498a708a8097c870f856 Author: David Fifield <david(a)bamsoftware.com> Date: Tue Jul 10 09:47:03 2012 -0700 Use an iframe attribute called frameborder not frameBorder. I got a notice from Joseph Long that our iframe code didn't validate as XHTML because of the frameBorder element. (XHTML 1.0 doesn't even have the iframe element, but XHTML 1.0 Transitional and XHTML 1.0 Frameset do.) It turns out that the attribute name must be all lower-case: http://www.w3.org/TR/xhtml1/#h-4.2 http://www.w3.org/TR/1999/xhtml-modularization-19990406/DTD/doc/xhtml1-f.el… I think I originally had the capital B because it's required by some versions of IE: http://stackoverflow.com/questions/65034/remove-border-from-iframe/65052#65… Apparently setting the CSS "border" property to 0 also has problems with some versions of IE: http://stackoverflow.com/questions/65034/remove-border-from-iframe/8437926#… --- README | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/README b/README index e096a3f..b8b2f5c 100644 --- a/README +++ b/README @@ -126,7 +126,7 @@ Add this line to torrc to make the relay use the external proxy: Paste in this HTML where you want the badge to appear: -<iframe src="//crypto.stanford.edu/flashproxy/embed.html" width="80" height="15" frameBorder="0" scrolling="no"></iframe> +<iframe src="//crypto.stanford.edu/flashproxy/embed.html" width="80" height="15" frameborder="0" scrolling="no"></iframe> == For developers

1 0

[stem/master] Implement convenience method for handling SAVECONF requests
by atagar＠torproject.org 10 Jul '12

10 Jul '12

commit e68adb7fdd465e1780bc4ee1243e0c226add77d9 Author: Ravi Chandra Padmala <neenaoffline(a)gmail.com> Date: Wed Jun 27 18:34:15 2012 +0530 Implement convenience method for handling SAVECONF requests This also changes the Exception structure a bit. I have added an InsatisfiableRequest class which will be raised when a valid request couldn't be satisfied by Tor. I've also added a super class for non-socket non-protocol errors called OperationFailed. Any error that was raised by an error response returned by Tor (i.e., one with an error code & a message) should subclass this. The exceptions now look like ControllerError - Base exception raised when using the controller. |- ProtocolError - Malformed socket data. |- OperationFailed - Tor was unable to successfully complete the operation. | |- UnsatisfiableRequest - Tor was unable to satisfy a valid request. | +- InvalidRequest - Invalid request. | +- InvalidArguments - Invalid request parameters. +- SocketError - Communication with the socket failed. +- SocketClosed - Socket has been shut down. --- stem/control.py | 20 ++++++++++++++++++++ stem/socket.py | 28 ++++++++++++++++++++-------- test/integ/control/controller.py | 20 ++++++++++++++++++++ 3 files changed, 60 insertions(+), 8 deletions(-) diff --git a/stem/control.py b/stem/control.py index 51804cc..e2bf282 100644 --- a/stem/control.py +++ b/stem/control.py @@ -20,6 +20,7 @@ interacting at a higher level. |- reset_conf - reverts configuration options to their default values |- set_options - sets or resets the values of multiple configuration options |- load_conf - loads configuration information as if it was in the torrc + |- save_conf - saves configuration information to the torrc |- get_version - convenience method to get tor version |- authenticate - convenience method to authenticate the controller +- protocolinfo - convenience method to get the protocol info @@ -777,6 +778,25 @@ class Controller(BaseController): raise stem.socket.InvalidRequest(response.code, response.message) elif not response.is_ok(): raise stem.socket.ProtocolError("+LOADCONF Received unexpected response\n%s" % str(response)) + + def save_conf(self): + """ + Saves the current configuration options into the active torrc file. + + :raises: + :class:`stem.socket.ControllerError` if the call fails + :class:`stem.socket.OperationFailed` if the client is unable to save the configuration file + """ + + response = self.msg("SAVECONF") + stem.response.convert("SINGLELINE", response) + + if response.is_ok(): + return True + elif response.code == "551": + raise stem.socket.OperationFailed(response.code, response.message) + else: + raise stem.socket.ProtocolError("SAVECONF returned unexpected response code") def _case_insensitive_lookup(entries, key, default = UNDEFINED): """ diff --git a/stem/socket.py b/stem/socket.py index 7d40056..5492e30 100644 --- a/stem/socket.py +++ b/stem/socket.py @@ -28,8 +28,10 @@ as instances of the :class:`stem.response.ControlMessage` class. ControllerError - Base exception raised when using the controller. |- ProtocolError - Malformed socket data. - |- InvalidRequest - Invalid request. - | +- InvalidArguments - Invalid request parameters. + |- OperationFailed - Tor was unable to successfully complete the operation. + | |- UnsatisfiableRequest - Tor was unable to satisfy a valid request. + | +- InvalidRequest - Invalid request. + | +- InvalidArguments - Invalid request parameters. +- SocketError - Communication with the socket failed. +- SocketClosed - Socket has been shut down. """ @@ -550,9 +552,9 @@ class ControllerError(Exception): class ProtocolError(ControllerError): "Malformed content from the control socket." -class InvalidRequest(ControllerError): +class OperationFailed(ControllerError): """ - Base Exception class for invalid requests + Base exception class for failed operations that return an error code :var str code: error code returned by Tor :var str message: error message returned by Tor or a human readable error message @@ -560,21 +562,31 @@ class InvalidRequest(ControllerError): def __init__(self, code = None, message = None): """ - Initializes an InvalidRequest object. + Initializes an OperationFailed object. :param str code: error code returned by Tor :param str message: error message returned by Tor or a human readable error message - :returns: object of InvalidRequest class + :returns: object of OperationFailed class """ - super(InvalidRequest, self).__init__() + super(ControllerError, self).__init__() self.code = code self.message = message +class UnsatisfiableRequest(OperationFailed): + """ + Exception raised if Tor was unable to process our request. + """ + +class InvalidRequest(OperationFailed): + """ + Exception raised when the request was invalid or malformed. + """ + class InvalidArguments(InvalidRequest): """ - Exception class for invalid requests which contain invalid arguments. + Exception class for requests which had invalid arguments. :var str code: error code returned by Tor :var str message: error message returned by Tor or a human readable error message diff --git a/test/integ/control/controller.py b/test/integ/control/controller.py index 857799d..54f2626 100644 --- a/test/integ/control/controller.py +++ b/test/integ/control/controller.py @@ -296,4 +296,24 @@ class TestController(unittest.TestCase): finally: # reload original valid config controller.load_conf(oldconf) + + def test_saveconf(self): + + if test.runner.require_control(self): return + + runner = test.runner.get_runner() + + # only testing for success, since we need to run out of disk space to test + # for failure + with runner.get_tor_controller() as controller: + oldconf = runner.get_torrc_contents() + + try: + controller.set_conf("ContactInfo", "confsaved") + controller.save_conf() + with file(runner.get_torrc_path()) as torrcfile: + self.assertTrue("\nContactInfo confsaved\n" in torrcfile.read()) + finally: + controller.load_conf(oldconf) + controller.save_conf()

1 0

[ooni-probe/master] Write some docstrings and minor refactor
by art＠torproject.org 10 Jul '12

10 Jul '12

commit 3f9714d942656c552ad71e8172c83d55016a8c73 Author: Arturo Filastò <art(a)torproject.org> Date: Tue Jul 10 07:09:35 2012 +0200 Write some docstrings and minor refactor --- ooni/plugins/captiveportal.py | 6 +--- ooni/plugoo/tests.py | 44 +++++++++++++++++++++++++++++++++++++++++ ooni/plugoo/work.py | 27 +++++++++++++++++++++++++ 3 files changed, 73 insertions(+), 4 deletions(-) diff --git a/ooni/plugins/captiveportal.py b/ooni/plugins/captiveportal.py index 9b7bee3..1f13431 100644 --- a/ooni/plugins/captiveportal.py +++ b/ooni/plugins/captiveportal.py @@ -93,14 +93,12 @@ class CaptivePortal(OONITest): if fuzzy: pattern = re.compile(control_result) match = pattern.search(response_content) + log.msg("Fuzzy HTTP content comparison for experiment URL") + log.msg("'%s'" % experimental_url) if not match: - log.msg("Fuzzy HTTP content comparison for experiment URL") - log.msg("'%s'" % experimental_url) log.msg("does not match!") return False, response_code, response_headers else: - log.msg("Fuzzy HTTP content comparison of experiment URL") - log.msg("'%s'" % experimental_url) log.msg("and the expected control result yielded a match.") return True, response_code, response_headers else: diff --git a/ooni/plugoo/tests.py b/ooni/plugoo/tests.py index a3e9150..47bb135 100644 --- a/ooni/plugoo/tests.py +++ b/ooni/plugoo/tests.py @@ -15,11 +15,21 @@ from ooni.plugoo.interface import ITest class OONITest(object): + """ + This is the base class for writing OONI Tests. + + It should be used in conjunction with the ITest Interface. It allows the + developer to benefit from OONIs reporting system and command line argument + parsing system. + """ + # By default we set this to False, meaning that we don't block blocking = False def __init__(self, local_options, global_options, report, ooninet=None, reactor=None): + # These are the options that are read through the tests suboptions self.local_options = local_options + # These are the options global to all of OONI self.global_options = global_options self.assets = self.load_assets() self.report = report @@ -29,6 +39,10 @@ class OONITest(object): self.result = {} def initialize(self): + """ + Override this method if you are interested in having some extra + behavior when your test class is instantiated. + """ pass def load_assets(self): @@ -43,6 +57,10 @@ class OONITest(object): self.assets) def finished(self, control): + """ + The Test has finished running, we must now calculate the test runtime + and add all time data to the report. + """ #self.ooninet.report(result) self.end_time = date.now() result = self.result @@ -55,6 +73,13 @@ class OONITest(object): return result def _do_experiment(self, args): + """ + A wrapper around the launch of experiment. + If we are running a blocking test experiment will be run in a thread if + not we expect it to return a Deferred. + + @param args: the asset line(s) that we are working on. + """ if self.blocking: self.d = threads.deferToThread(self.experiment, args) else: @@ -65,6 +90,13 @@ class OONITest(object): return self.d def control(self, result, args): + """ + Run the control. + + @param result: what was returned by experiment. + + @param args: the asset(s) lines that we are working on. + """ log.msg("Doing control") if self.blocking: @@ -77,11 +109,23 @@ class OONITest(object): return d def experiment(self, args): + """ + Run the experiment. This sample implementation returns a deferred, + making it a non-blocking test. + + @param args: the asset(s) lines that we are working on. + """ log.msg("Doing experiment") d = defer.Deferred() return d def startTest(self, args): + """ + This method is invoked by the worker to start the test with one line of + the asset file. + + @param args: the asset(s) lines that we are working on. + """ self.start_time = date.now() log.msg("Starting test %s" % self.__class__) return self._do_experiment(args) diff --git a/ooni/plugoo/work.py b/ooni/plugoo/work.py index fc338d7..a8b6595 100644 --- a/ooni/plugoo/work.py +++ b/ooni/plugoo/work.py @@ -25,11 +25,23 @@ class Worker(object): runs them concurrently. """ def __init__(self, maxconcurrent=10): + """ + @param maxconcurrent: how many test instances should be run + concurrently. + """ self.maxconcurrent = maxconcurrent self._running = 0 self._queued = [] def _run(self, r): + """ + Check if we should start another test because we are below maximum + concurrency. + + This function is called every time a test finishes running. + + @param r: the return value of a previous test. + """ self._running -= 1 if self._running < self.maxconcurrent and self._queued: workunit, d = self._queued.pop(0) @@ -38,6 +50,7 @@ class Worker(object): actuald = test.startTest(asset).addBoth(self._run) if isinstance(r, failure.Failure): + # XXX probably we should be doing something to retry test running r.trap() if self._running == 0 and not self._queued: @@ -46,6 +59,15 @@ class Worker(object): return r def push(self, workunit): + """ + Add a test to the test queue and run it if we are not maxed out on + concurrency. + + @param workunit: a tuple containing the (asset, test, idx), where asset + is the line of the asset(s) we are working on, test + is an instantiated test and idx is the index we are + currently at. + """ if self._running < self.maxconcurrent: asset, test, idx = workunit self._running += 1 @@ -83,6 +105,11 @@ class WorkGenerator(object): return self def skip(self, start): + """ + Skip the first x number of lines of the asset. + + @param start: int how many items we should skip. + """ for j in xrange(0, start-1): for i in xrange(0, self.size): self.assetGenerator.next()

1 0

[ooni-probe/master] Minor refactoring of CaptivePortal tests
by art＠torproject.org 10 Jul '12

10 Jul '12

commit d5be93592dd12d7aea584cbc9bc26c55def4e271 Author: Arturo Filastò <art(a)torproject.org> Date: Tue Jul 10 07:24:04 2012 +0200 Minor refactoring of CaptivePortal tests (Flat is better than nested.) --- ooni/plugins/captiveportal.py | 73 +++++++++++++++++++--------------------- 1 files changed, 35 insertions(+), 38 deletions(-) diff --git a/ooni/plugins/captiveportal.py b/ooni/plugins/captiveportal.py index 1f13431..4ed00b3 100644 --- a/ooni/plugins/captiveportal.py +++ b/ooni/plugins/captiveportal.py @@ -89,45 +89,41 @@ class CaptivePortal(OONITest): response, response_headers = self.http_fetch(experimental_url, headers) response_content = response.read() response_code = response.code - if response_content is not None: - if fuzzy: - pattern = re.compile(control_result) - match = pattern.search(response_content) - log.msg("Fuzzy HTTP content comparison for experiment URL") - log.msg("'%s'" % experimental_url) - if not match: - log.msg("does not match!") - return False, response_code, response_headers - else: - log.msg("and the expected control result yielded a match.") - return True, response_code, response_headers + if response_content is None: + log.warn("HTTP connection appears to have failed.") + return False, False, False + + if fuzzy: + pattern = re.compile(control_result) + match = pattern.search(response_content) + log.msg("Fuzzy HTTP content comparison for experiment URL") + log.msg("'%s'" % experimental_url) + if not match: + log.msg("does not match!") + return False, response_code, response_headers else: - if str(response_content) != str(control_result): - log.msg("HTTP content comparison of experiment URL") - log.msg("'%s'" % experimental_url) - log.msg("and the expected control result do not match.") - return False, response_code, response_headers - else: - return True, response_code, response_headers + log.msg("and the expected control result yielded a match.") + return True, response_code, response_headers else: - log.warn("HTTP connection appears to have failed.") - return False, False, False + if str(response_content) != str(control_result): + log.msg("HTTP content comparison of experiment URL") + log.msg("'%s'" % experimental_url) + log.msg("and the expected control result do not match.") + return False, response_code, response_headers + else: + return True, response_code, response_headers def http_status_code_match(self, experiment_code, control_code): """ Compare two HTTP status codes, returns True if they match. """ - if int(experiment_code) != int(control_code): - return False - return True + return int(experiment_code) == int(control_code) def http_status_code_no_match(self, experiment_code, control_code): """ Compare two HTTP status codes, returns True if they do not match. """ - if self.http_status_code_match(experiment_code, control_code): - return False - return True + return int(experiment_code) != int(control_code) def dns_resolve(self, hostname, nameserver=None): """ @@ -163,9 +159,10 @@ class CaptivePortal(OONITest): log.msg("DNS resolution for %s returned NXDOMAIN" % hn) response.append('NXDOMAIN') finally: - if answer: - for addr in answer: - response.append(addr.address) + if not answer: + return response + for addr in answer: + response.append(addr.address) return response def dns_resolve_match(self, experiment_hostname, control_address): @@ -176,17 +173,17 @@ class CaptivePortal(OONITest): returns False and experiment_address. """ experiment_address = self.dns_resolve(experiment_hostname) - if experiment_address: - if len(set(experiment_address) & set([control_address])) > 0: - return True, experiment_address - else: - log.msg("DNS comparison of control '%s' does not" % control_address) - log.msg("match experiment response '%s'" % experiment_address) - return False, experiment_address - else: + if not experiment_address: log.debug("dns_resolve() for %s failed" % experiment_hostname) return None, experiment_address + if len(set(experiment_address) & set([control_address])) > 0: + return True, experiment_address + else: + log.msg("DNS comparison of control '%s' does not" % control_address) + log.msg("match experiment response '%s'" % experiment_address) + return False, experiment_address + def get_auth_nameservers(self, hostname): """ Many CPs set a nameserver to be used. Let's query that

1 0

[ooni-probe/master] Write unit tests for all the OONI plugins
by art＠torproject.org 10 Jul '12

10 Jul '12

commit 6aa7ca883a500eb2b909b21386d03f9bd29e5384 Author: Arturo Filastò <art(a)torproject.org> Date: Tue Jul 10 09:50:33 2012 +0200 Write unit tests for all the OONI plugins * Refactor some code --- ooni/log.py | 5 ++++- ooni/ooniprobe.py | 31 +++++++++++++++++++++---------- ooni/plugins/blocking.py | 7 +++++-- ooni/plugins/tcpconnect.py | 3 +-- ooni/plugoo/tests.py | 12 +++--------- ooni/protocols/http.py | 5 +---- tests/assets/ipports.txt | 2 ++ tests/assets/urllist.txt | 3 +++ tests/test_plugins.py | 40 ++++++++++++++++++++++++++++++++++++++++ tests/test_tests.py | 17 +++++++++-------- 10 files changed, 89 insertions(+), 36 deletions(-) diff --git a/ooni/log.py b/ooni/log.py index f1b7839..3dc52dd 100644 --- a/ooni/log.py +++ b/ooni/log.py @@ -34,7 +34,10 @@ def start(logfile=None, loglevel=None, logstdout=True): msg("Started OONI") def msg(message, level=INFO, **kw): - log.msg(message, **kw) + log.msg(message, logLevel=level, **kw) def err(message, **kw): log.err(message, **kw) + +def debug(message, **kw): + log.msg(message, logLevel=DEBUG, **kw) diff --git a/ooni/ooniprobe.py b/ooni/ooniprobe.py index 3eb70f6..22299ac 100755 --- a/ooni/ooniprobe.py +++ b/ooni/ooniprobe.py @@ -55,8 +55,17 @@ def retrieve_plugoo(): plugoo = retrieve_plugoo() -def runTest(test, options, global_options): +def runTest(test, options, global_options, reactor=None): + """ + Run an OONI probe test by name. + + @param test: a string specifying the test name as specified inside of + shortName. + @param options: the local options to be passed to the test. + + @param global_options: the global options for OONI + """ parallelism = int(global_options['parallelism']) worker = work.Worker(parallelism) test_class = plugoo[test].__class__ @@ -64,6 +73,8 @@ def runTest(test, options, global_options): log.start(global_options['log'], 1) resume = 0 + if not options: + options = {} if 'resume' in options: resume = options['resume'] @@ -74,7 +85,6 @@ def runTest(test, options, global_options): for x in wgen: worker.push(x) - reactor.run() class Options(usage.Options): tests = plugoo.keys() @@ -111,14 +121,15 @@ class Options(usage.Options): return getlogo() + '\n' + self.getSynopsis() + '\n' + \ self.getUsage(width=None).replace("Commands:", "Tests:") -config = Options() -config.parseOptions() - -if not config.subCommand: - print "Error! No Test Specified." - config.opt_help() - sys.exit(1) +if __name__ == "__main__": + config = Options() + config.parseOptions() -runTest(config.subCommand, config.subOptions, config) + if not config.subCommand: + print "Error! No Test Specified." + config.opt_help() + sys.exit(1) + runTest(config.subCommand, config.subOptions, config) + reactor.run() diff --git a/ooni/plugins/blocking.py b/ooni/plugins/blocking.py index 6ac3c91..6ddb5a4 100644 --- a/ooni/plugins/blocking.py +++ b/ooni/plugins/blocking.py @@ -27,8 +27,11 @@ class BlockingTest(OONITest): def experiment(self, args): import urllib - url = 'https://torproject.org/' if not 'asset' in args else args['asset'] - req = urllib.urlopen(url) + url = 'http://torproject.org/' if not 'asset' in args else args['asset'] + try: + req = urllib.urlopen(url) + except: + return {'error': 'Connection failed!'} return {'page': req.readlines()} diff --git a/ooni/plugins/tcpconnect.py b/ooni/plugins/tcpconnect.py index 27df08d..7c04994 100644 --- a/ooni/plugins/tcpconnect.py +++ b/ooni/plugins/tcpconnect.py @@ -6,7 +6,6 @@ Safe hacking :). from zope.interface import implements from twisted.python import usage from twisted.plugin import IPlugin -from twisted.internet import reactor from twisted.internet.protocol import Factory, Protocol from twisted.internet.endpoints import TCP4ClientEndpoint @@ -48,7 +47,7 @@ class tcpconnectTest(OONITest): return {'result': False, 'target': [host, port]} # What you return here gets handed as input to control - point = TCP4ClientEndpoint(reactor, host, int(port)) + point = TCP4ClientEndpoint(self.reactor, host, int(port)) d = point.connect(DummyFactory()) d.addCallback(gotProtocol) d.addErrback(gotError) diff --git a/ooni/plugoo/tests.py b/ooni/plugoo/tests.py index 47bb135..42294d6 100644 --- a/ooni/plugoo/tests.py +++ b/ooni/plugoo/tests.py @@ -79,6 +79,8 @@ class OONITest(object): not we expect it to return a Deferred. @param args: the asset line(s) that we are working on. + + returns a deferred. """ if self.blocking: self.d = threads.deferToThread(self.experiment, args) @@ -98,15 +100,7 @@ class OONITest(object): @param args: the asset(s) lines that we are working on. """ log.msg("Doing control") - - if self.blocking: - return result - - def end(cb): - return result - d = defer.Deferred() - d.addCallback(end) - return d + return result def experiment(self, args): """ diff --git a/ooni/protocols/http.py b/ooni/protocols/http.py index 835735b..1dd2261 100644 --- a/ooni/protocols/http.py +++ b/ooni/protocols/http.py @@ -41,13 +41,10 @@ class HTTPTest(OONITest): """ randomize_ua = True - def initialize(self, reactor=None): + def initialize(self): from twisted.web.client import Agent import yaml - if not self.reactor: - from twisted.internet import reactor - self.reactor = reactor self.agent = Agent(self.reactor) self.request = {} self.response = {} diff --git a/tests/assets/ipports.txt b/tests/assets/ipports.txt new file mode 100644 index 0000000..ade757f --- /dev/null +++ b/tests/assets/ipports.txt @@ -0,0 +1,2 @@ +127.0.0.1:80 +8.8.8.8:53 diff --git a/tests/assets/urllist.txt b/tests/assets/urllist.txt new file mode 100644 index 0000000..dc38e22 --- /dev/null +++ b/tests/assets/urllist.txt @@ -0,0 +1,3 @@ +http://google.com/ +http://127.0.0.1/ +http://dio.it/ diff --git a/tests/test_plugins.py b/tests/test_plugins.py new file mode 100644 index 0000000..26bee51 --- /dev/null +++ b/tests/test_plugins.py @@ -0,0 +1,40 @@ +from twisted.internet import defer, reactor +from twisted.trial import unittest + +from ooni.ooniprobe import retrieve_plugoo, runTest, Options +from ooni.plugoo import work, tests + +def asset_file(filename): + import os + file_dir = os.path.normpath(os.path.join(__file__, '..')) + return os.path.join(file_dir, 'assets', filename) + +class PluginsTestCase(unittest.TestCase): + def test_plugin_blocking(self): + suboptions = {'asset': asset_file('urllist.txt')} + runTest('blocking', suboptions, Options(), reactor) + return + + def test_plugin_tcpconnect(self): + suboptions = {'asset': asset_file('ipports.txt')} + runTest('tcpconnect', suboptions, Options(), reactor) + return + + + def test_plugin_captivep(self): + runTest('blocking', None, Options(), reactor) + return + + + def test_plugin_httphost(self): + suboptions = {'asset': asset_file('urllist.txt')} + runTest('httphost', suboptions, Options(), reactor) + return + + + def test_plugin_httpt(self): + suboptions = {'urls': asset_file('urllist.txt')} + runTest('httpt', suboptions, Options(), reactor) + return + + diff --git a/tests/test_tests.py b/tests/test_tests.py index bed5922..d0cc88c 100644 --- a/tests/test_tests.py +++ b/tests/test_tests.py @@ -13,9 +13,7 @@ class TestsTestCase(unittest.TestCase): self.errbackResults = None def _callback(self, *args, **kw): - print "BBB" - print args, kw - print "CCCC" + #print args, kw self.callbackResults = args, kw def _errback(self, *args, **kw): @@ -31,14 +29,17 @@ class TestsTestCase(unittest.TestCase): class DummyTest(tests.OONITest): blocking = False def experiment(self, args): - def cb(aaa): + def bla(a): + print a return test_dict - d = defer.Deferred() - d.addCallback(cb) - d.callback(None) - return d + d2 = defer.Deferred() + d2.addCallback(bla) + from twisted.internet import reactor + reactor.callLater(0.1, d2.callback, None) + return d2 test = DummyTest(None, None, self.dummyreport) yield test.startTest(None).addCallback(self._callback) self.assertEqual(self.callbackResults[0][0]['control'], test_dict) + return

1 0

[ooni-probe/master] Code refactoring, restructure directory tree.
by art＠torproject.org 10 Jul '12

10 Jul '12

commit a129f69242588fdc679eb33cfcc38288b09d07e5 Author: Arturo Filastò <art(a)torproject.org> Date: Tue Jul 10 10:07:47 2012 +0200 Code refactoring, restructure directory tree. --- ooni/config.py | 53 --------- ooni/date.py | 22 ---- ooni/example_plugins/examplescapy.py | 2 +- ooni/example_plugins/skel.py | 2 +- ooni/log.py | 43 ------- ooni/logo.py | 214 ---------------------------------- ooni/ooniprobe.py | 11 +- ooni/plugins/bridget.py | 2 +- ooni/plugins/captiveportal.py | 2 +- ooni/plugins/chinatrigger.py | 2 +- ooni/plugins/httpt.py | 2 +- ooni/plugins/tcpconnect.py | 2 +- ooni/plugoo/nodes.py | 2 +- ooni/plugoo/reports.py | 2 +- ooni/plugoo/tests.py | 4 +- ooni/protocols/http.py | 2 +- ooni/protocols/scapy.py | 2 +- ooni/scaffolding.py | 4 +- ooni/utils.py | 146 ----------------------- ooni/utils/__init__.py | 146 +++++++++++++++++++++++ ooni/utils/config.py | 53 +++++++++ ooni/utils/date.py | 22 ++++ ooni/utils/log.py | 43 +++++++ ooni/utils/logo.py | 214 ++++++++++++++++++++++++++++++++++ 24 files changed, 499 insertions(+), 498 deletions(-) diff --git a/ooni/config.py b/ooni/config.py deleted file mode 100644 index 42a14f6..0000000 --- a/ooni/config.py +++ /dev/null @@ -1,53 +0,0 @@ -import ConfigParser -from utils import Storage - -class Config(Storage): - """ - A Storage-like class which loads and store each attribute into a portable - conf file. - """ - def __init__(self, section, cfgfile="ooni-probe.conf"): - super(Config, self).__init__() - - self._cfgfile = cfgfile - # setting up confgiparser - self._cfgparser = ConfigParser.ConfigParser() - self._cfgparser.read([self._cfgfile]) - self._section = section - - def __getattr__(self, name): - if name.startswith('_'): - return self.__dict__.get(name, None) - - try: - value = self._cfgparser.get(self._section, name) - if value.isdigit(): - return int(value) - elif value.lower() in ('true', 'false'): - return value.lower() == 'true' - else: - return value - except ConfigParser.NoOptionError: - return '' # if option doesn't exists return an empty string - - def __setattr__(self, name, value): - # keep an open port with private attributes - if name.startswith('_'): - self.__dict__[name] = value - return - - try: - # XXX: Automagically discover variable type - self._cfgparser.set(self._section, name, value) - except ConfigParser.NoOptionError: - raise NameError(name) - - def commit(self): - """ - Commit changes in config file. - """ - cfgfile = open(self._cfgfile, 'w') - try: - self._cfgparser.write(cfgfile) - finally: - cfgfile.close() diff --git a/ooni/date.py b/ooni/date.py deleted file mode 100644 index 59ec1f8..0000000 --- a/ooni/date.py +++ /dev/null @@ -1,22 +0,0 @@ -from ooni.lib.rfc3339 import rfc3339 -from datetime import datetime - -class odate(datetime): - def __str__(self): - return rfc3339(self) - - def __repr__(self): - return "'%s'" % rfc3339(self) - - def from_rfc(self, datestr): - pass - -def now(): - return odate.utcnow() - -def pretty_date(): - cur_time = datetime.utcnow() - d_format = "%d %B %Y %H:%M:%S" - pretty = cur_time.strftime(d_format) - return pretty - diff --git a/ooni/example_plugins/examplescapy.py b/ooni/example_plugins/examplescapy.py index aa6d81b..144afee 100644 --- a/ooni/example_plugins/examplescapy.py +++ b/ooni/example_plugins/examplescapy.py @@ -5,7 +5,7 @@ from twisted.plugin import IPlugin from twisted.internet import protocol, defer from ooni.plugoo.tests import ITest, OONITest from ooni.plugoo.assets import Asset -from ooni import log +from ooni.utils import log from ooni.protocols.scapy import ScapyTest from ooni.lib.txscapy import txsr, txsend diff --git a/ooni/example_plugins/skel.py b/ooni/example_plugins/skel.py index de95b48..52c7d4f 100644 --- a/ooni/example_plugins/skel.py +++ b/ooni/example_plugins/skel.py @@ -2,7 +2,7 @@ from zope.interface import implements from twisted.python import usage from twisted.plugin import IPlugin from plugoo.tests import ITest, OONITest -from ooni import log +from ooni.utils import log class SkelArgs(usage.Options): optParameters = [['asset', 'a', None, 'Asset file'], diff --git a/ooni/log.py b/ooni/log.py deleted file mode 100644 index 3dc52dd..0000000 --- a/ooni/log.py +++ /dev/null @@ -1,43 +0,0 @@ -""" -OONI logging facility. -""" -import sys -import logging -import warnings - -from twisted.python import log - -# Logging levels -DEBUG = logging.DEBUG -INFO = logging.INFO -WARNING = logging.WARNING -ERROR = logging.ERROR -CRITICAL = logging.CRITICAL - -def _get_log_level(level): - if not level: - return INFO - else: - return level - -def start(logfile=None, loglevel=None, logstdout=True): - if log.defaultObserver: - print "%s" % logstdout - loglevel = _get_log_level(loglevel) - file = open(logfile, 'a') if logfile else sys.stderr - observer = log.FileLogObserver(file) - if logstdout: - log.startLogging(sys.stdout) - else: - log.startLogging() - log.addObserver(observer.emit) - msg("Started OONI") - -def msg(message, level=INFO, **kw): - log.msg(message, logLevel=level, **kw) - -def err(message, **kw): - log.err(message, **kw) - -def debug(message, **kw): - log.msg(message, logLevel=DEBUG, **kw) diff --git a/ooni/logo.py b/ooni/logo.py deleted file mode 100644 index 8924462..0000000 --- a/ooni/logo.py +++ /dev/null @@ -1,214 +0,0 @@ -import random -def getlogo(): - logo = [] - logo.append(""" - _____ _____ __ _ _____ _____ ______ _____ ______ _______ - | | | | | \ | | |_____] |_____/ | | |_____] |______ - |_____| |_____| | \_| __|__ | | \_ |_____| |_____] |______""") - - logo.append(""" - _ _ - (_) | | - ___ ___ _ __ _ _ __ _ __ ___ | |__ ___ - / _ \ / _ \| '_ \| | '_ \| '__/ _ \| '_ \ / _ \\ -| (_) | (_) | | | | | |_) | | | (_) | |_) | __/ - \___/ \___/|_| |_|_| .__/|_| \___/|_.__/ \___| - | | - |_|""") - - - logo.append(""" - .__ ___. - ____ ____ ____ |__|_____________ ____\_ |__ ____ - / _ \ / _ \ / \| |\____ \_ __ \/ _ \| __ \_/ __ \\ -( <_> | <_> ) | \ || |_> > | \( <_> ) \_\ \ ___/ - \____/ \____/|___| /__|| __/|__| \____/|___ /\___ > - \/ |__| \/ \/""") - - - logo.append(""" - __ __ __ __ __ __ ___ -/ \ / \ |\ | | |__) |__) / \ |__) |__ -\__/ \__/ | \| | | | \ \__/ |__) |___ """) - - - logo.append(""" - _ _ ._ o ._ ._ _ |_ _ - (_) (_) | | | |_) | (_) |_) (/_ - | -""") - - logo.append(""" - _______ _______ __ _ ___ _______ ______ _______ _______ _______ -| || || | | || | | || _ | | || _ || | -| _ || _ || |_| || | | _ || | || | _ || |_| || ___| -| | | || | | || || | | |_| || |_|| | | | || || |___ -| |_| || |_| || _ || | | ___|| __ || |_| || _ | | ___| -| || || | | || | | | | | ||| || |_| || |___ -|_______||_______||_| |__||___| |___| |___| |||_______||_______||_______| -""") - - - logo.append(""" - _ _ - ___ ___ _ __ (_)_ __ _ __ ___ | |__ ___ - / _ \ / _ \| '_ \| | '_ \| '__/ _ \| '_ \ / _ \\ -| (_) | (_) | | | | | |_) | | | (_) | |_) | __/ - \___/ \___/|_| |_|_| .__/|_| \___/|_.__/ \___| - |_| -""") - - logo.append(""" - o - o O - O - o -.oOo. .oOo. 'OoOo. O .oOo. `OoOo. .oOo. OoOo. .oOo. -O o O o o O o O o o O o O o OooO' -o O o O O o O o O O o O o O O -`OoO' `OoO' o O o' oOoO' o `OoO' `OoO' `OoO' - O - o' -""") - - - logo.append(""" - _____ _____ ____ _ ____ _____ _____ _____ ______ ______ - / \/ \| \ | || || || | / \| >| ___| - | || || \| || || _|| \ | || < | ___| - \_____/\_____/|__/\____||____||___| |__|\__\\\\_____/|______>|______| -""") - - logo.append(""" - _ _ - ___ ___ ___|_|___ ___ ___| |_ ___ -| . | . | | | . | _| . | . | -_| -|___|___|_|_|_| _|_| |___|___|___| - |_| -""") - - logo.append(""" - _ _ - (_) | | - ___ ___ ____ _ ____ ____ ___ | |__ _____ - / _ \ / _ \| _ \| | _ \ / ___) _ \| _ \| ___ | -| |_| | |_| | | | | | |_| | | | |_| | |_) ) ____| - \___/ \___/|_| |_|_| __/|_| \___/|____/|_____) - |_| -""") - logo.append(""" - _ __ - ____ ____ ____ (_)____ _________ / /_ ___ - / __ \/ __ \/ __ \/ // __ \/ ___/ __ \/ __ \/ _ \\ -/ /_/ / /_/ / / / / // /_/ / / / /_/ / /_/ / __/ -\____/\____/_/ /_/_// .___/_/ \____/_.___/\___/ - /_/ -""") - logo.append(""" - _) | - _ \ _ \ \ | _ \ _| _ \ _ \ -_) - \___/ \___/ _| _| _| .__/ _| \___/ _.__/ \___| - _| -""") - logo.append(""" - _ _ - ___ ___ _ __ (_)_ __ _ __ ___ | |__ ___ - / _ \ / _ \| '_ \| | '_ \| '__/ _ \| '_ \ / _ \\ - | (_) | (_) | | | | | |_) | | | (_) | |_) | __/ - \___/ \___/|_| |_|_| .__/|_| \___/|_.__/ \___| - |_| -""") - logo.append(""" - .-. ( ) - .--. .--. ___ .-. ( __) .-.. ___ .-. .--. | |.-. .--. - / \ / \ ( ) \ (''") / \ ( ) \ / \ | / \ / \\ -| .-. ; | .-. ; | .-. . | | ' .-, ; | ' .-. ; | .-. ; | .-. | | .-. ; -| | | | | | | | | | | | | | | | . | | / (___) | | | | | | | | | | | | -| | | | | | | | | | | | | | | | | | | | | | | | | | | | | |/ | -| | | | | | | | | | | | | | | | | | | | | | | | | | | | | ' _.' -| ' | | | ' | | | | | | | | | | ' | | | | ' | | | ' | | | .'.-. -' `-' / ' `-' / | | | | | | | `-' ' | | ' `-' / ' `-' ; ' `-' / - `.__.' `.__.' (___)(___) (___) | \__.' (___) `.__.' `.__. `.__.' - | | - (___) -""") - logo.append(""" - o | -,---.,---.,---..,---.,---.,---.|---.,---. -| || || ||| || | || ||---' -`---'`---'` '`|---'` `---'`---'`---' - | -""") - logo.append(""" - ________________________________________ -/ OONI-PROBE -- WE FIGHT CENSORSHIP LIKE \\ -\ DRAGONS EAT KNIGHTS! / - ---------------------------------------- - \ ^ /^ - \ / \ // \\ - \ |\___/| / \// .\\ - \ /O O \__ / // | \ \ *----* - / / \/_/ // | \ \ \ | - @___@` \/_ // | \ \ \/\ \\ - 0/0/| \/_ // | \ \ \ \\ - 0/0/0/0/| \/// | \ \ | | - 0/0/0/0/0/_|_ / ( // | \ _\ | / - 0/0/0/0/0/0/`/,_ _ _/ ) ; -. | _ _\.-~ / / - ,-} _ *-.|.-~-. .~ ~ - \ \__/ `/\ / ~-. _ .-~ / - \____(oo) *. } { / - ( (--) .----~-.\ \-` .~ - //__\\\\ \__ Ack! ///.----..< \ _ -~ - // \\\\ ///-._ _ _ _ _ _ _{^ - - - - ~ -""") - logo.append(""" - _________________ -| OONI-PROBE | -| DON'T CENSOR ME | -|_________________| -\ . . - \ / `. .' " - \ .---. < > < > .---. - \ | \ \ - ~ ~ - / / | - _____ ..-~ ~-..-~ - | (A)| \~~~\.' `./~~~/ - --------- \__/ \__/ - .' O \ / / \ " - (_____, `._.' | } \/~~~/ - `----. / } | / \__/ - `-. | / | / `. ,~~| - ~-.__| /_ - ~ ^| /- _ `..-' - | / | / ~-. `-. _ _ _ - |_____| |_____| ~ - . _ _ _ _ _> -""") - logo.append(""" - _________________ - ,/~~, | | - /`` \ /| OONI-PROBE... | - `( O^O ) / | | - `\_-_/` | ...DOES THIS | - ~~~ | EXCITE YOU? | - __| |__ |_________________| - | | - | af | - | | | | - | | | \\ - | | |\ \\ - | | | \ \\ - | | | \ \\ - | |___| ((( - (((|/*\| - | | | - | | | - | | | - | | | - | | | - | | | - | | | - | | | - | | | - | | | - _| | |_ - (___|___) -""") - return random.choice(logo) diff --git a/ooni/ooniprobe.py b/ooni/ooniprobe.py index 22299ac..261eb74 100755 --- a/ooni/ooniprobe.py +++ b/ooni/ooniprobe.py @@ -17,19 +17,20 @@ # import sys +from pprint import pprint from twisted.python import usage -from twisted.plugin import getPlugins from twisted.internet import reactor +from twisted.plugin import getPlugins +from zope.interface.verify import verifyObject from zope.interface.exceptions import BrokenImplementation from zope.interface.exceptions import BrokenMethodImplementation -from zope.interface.verify import verifyObject -from pprint import pprint from ooni.plugoo import tests, work, assets, reports -from ooni.logo import getlogo -from ooni import plugins, log +from ooni.utils.logo import getlogo +from ooni.utils import log +from ooni import plugins __version__ = "0.0.1-prealpha" diff --git a/ooni/plugins/bridget.py b/ooni/plugins/bridget.py index 0614d5d..fa3cb67 100644 --- a/ooni/plugins/bridget.py +++ b/ooni/plugins/bridget.py @@ -8,7 +8,7 @@ from twisted.python import usage from twisted.plugin import IPlugin from twisted.internet import reactor -from ooni import log +from ooni.utils import log from ooni.plugoo.tests import ITest, OONITest from ooni.plugoo.assets import Asset diff --git a/ooni/plugins/captiveportal.py b/ooni/plugins/captiveportal.py index 4ed00b3..b48f70b 100644 --- a/ooni/plugins/captiveportal.py +++ b/ooni/plugins/captiveportal.py @@ -25,7 +25,7 @@ from twisted.plugin import IPlugin from ooni.plugoo.assets import Asset from ooni.plugoo.tests import ITest, OONITest from ooni.protocols import http -from ooni import log +from ooni.utils import log try: from dns import resolver diff --git a/ooni/plugins/chinatrigger.py b/ooni/plugins/chinatrigger.py index 4f2dc8c..538ef0b 100644 --- a/ooni/plugins/chinatrigger.py +++ b/ooni/plugins/chinatrigger.py @@ -9,7 +9,7 @@ from twisted.plugin import IPlugin from twisted.internet import protocol, defer from ooni.plugoo.tests import ITest, OONITest from ooni.plugoo.assets import Asset -from ooni import log +from ooni.utils import log from ooni.protocols.scapy import ScapyTest from ooni.lib.txscapy import txsr, txsend diff --git a/ooni/plugins/httpt.py b/ooni/plugins/httpt.py index d0ede0f..4113aef 100644 --- a/ooni/plugins/httpt.py +++ b/ooni/plugins/httpt.py @@ -9,7 +9,7 @@ from twisted.plugin import IPlugin from ooni.plugoo.tests import ITest, OONITest from ooni.plugoo.assets import Asset from ooni.protocols import http -from ooni import log +from ooni.utils import log class httptArgs(usage.Options): optParameters = [['urls', 'f', None, 'Urls file'], diff --git a/ooni/plugins/tcpconnect.py b/ooni/plugins/tcpconnect.py index 7c04994..db3d969 100644 --- a/ooni/plugins/tcpconnect.py +++ b/ooni/plugins/tcpconnect.py @@ -11,7 +11,7 @@ from twisted.internet.endpoints import TCP4ClientEndpoint from ooni.plugoo.tests import ITest, OONITest from ooni.plugoo.assets import Asset -from ooni import log +from ooni.utils import log class tcpconnectArgs(usage.Options): optParameters = [['asset', 'a', None, 'File containing IP:PORT combinations, one per line.'], diff --git a/ooni/plugoo/nodes.py b/ooni/plugoo/nodes.py index 6cdba65..0d01348 100644 --- a/ooni/plugoo/nodes.py +++ b/ooni/plugoo/nodes.py @@ -99,7 +99,7 @@ class PlanetLab(CodeExecNode): def __init__(self, address, auth_creds, ooni): self.auth_creds = auth_creds - self.config = ooni.config + self.config = ooni.utils.config self.logger = ooni.logger self.name = "PlanetLab" diff --git a/ooni/plugoo/reports.py b/ooni/plugoo/reports.py index a963f1c..3a9c5d2 100644 --- a/ooni/plugoo/reports.py +++ b/ooni/plugoo/reports.py @@ -4,7 +4,7 @@ import os import yaml import itertools -from ooni import log, date +from ooni.utils import log, date class Report: """This is the ooni-probe reporting mechanism. It allows diff --git a/ooni/plugoo/tests.py b/ooni/plugoo/tests.py index 42294d6..a99a144 100644 --- a/ooni/plugoo/tests.py +++ b/ooni/plugoo/tests.py @@ -7,8 +7,8 @@ import itertools from twisted.internet import reactor, defer, threads from twisted.python import failure -from ooni import log -from ooni import date +from ooni.utils import log +from ooni.utils import date from ooni.plugoo import assets, work from ooni.plugoo.reports import Report from ooni.plugoo.interface import ITest diff --git a/ooni/protocols/http.py b/ooni/protocols/http.py index 1dd2261..100db88 100644 --- a/ooni/protocols/http.py +++ b/ooni/protocols/http.py @@ -5,7 +5,7 @@ from twisted.plugin import IPlugin from twisted.internet import protocol, defer from ooni.plugoo.tests import ITest, OONITest from ooni.plugoo.assets import Asset -from ooni import log +from ooni.utils import log useragents = [("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6", "Firefox 2.0, Windows XP"), ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)", "Internet Explorer 7, Windows Vista"), diff --git a/ooni/protocols/scapy.py b/ooni/protocols/scapy.py index bacc163..4166146 100644 --- a/ooni/protocols/scapy.py +++ b/ooni/protocols/scapy.py @@ -5,7 +5,7 @@ from twisted.plugin import IPlugin from twisted.internet import protocol, defer from ooni.plugoo.tests import ITest, OONITest from ooni.plugoo.assets import Asset -from ooni import log +from ooni.utils import log from ooni.lib.txscapy import txsr, txsend diff --git a/ooni/scaffolding.py b/ooni/scaffolding.py index b4c5645..0a832aa 100755 --- a/ooni/scaffolding.py +++ b/ooni/scaffolding.py @@ -5,7 +5,7 @@ This script should be used for creating the scaffolding for a test. """ import os import sys -from ooni import log +from ooni.utils import log test_template = """\"\"\" This is a self genrated test created by scaffolding.py. @@ -17,7 +17,7 @@ from twisted.python import usage from twisted.plugin import IPlugin from ooni.plugoo.tests import ITest, OONITest from ooni.plugoo.assets import Asset -from ooni import log +from ooni.utils import log class %(testShortname)sArgs(usage.Options): optParameters = [['asset', 'a', None, 'Asset file'], diff --git a/ooni/utils.py b/ooni/utils.py deleted file mode 100644 index e9c55f1..0000000 --- a/ooni/utils.py +++ /dev/null @@ -1,146 +0,0 @@ -""" - -""" - -import imp -import logging -try: - import yaml -except: - print "Error in importing YAML" - -class Storage(dict): - """ - A Storage object is like a dictionary except `obj.foo` can be used - in addition to `obj['foo']`. - - >>> o = Storage(a=1) - >>> o.a - 1 - >>> o['a'] - 1 - >>> o.a = 2 - >>> o['a'] - 2 - >>> del o.a - >>> o.a - None - """ - - def __getattr__(self, key): - try: - return self[key] - except KeyError, k: - return None - - def __setattr__(self, key, value): - self[key] = value - - def __delattr__(self, key): - try: - del self[key] - except KeyError, k: - raise AttributeError, k - - def __repr__(self): - return '<Storage ' + dict.__repr__(self) + '>' - - def __getstate__(self): - return dict(self) - - def __setstate__(self, value): - for (k, v) in value.items(): - self[k] = v - - -def get_logger(config): - loglevel = getattr(logging, config.loglevel.upper()) - logging.basicConfig(level=loglevel, - format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', - filename=config.logfile, - filemode='w') - - console = logging.StreamHandler() - console.setLevel(getattr(logging, config.consoleloglevel.upper())) - # Set the console logger to a different format - formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s') - console.setFormatter(formatter) - logging.getLogger('').addHandler(console) - - return logging.getLogger('ooniprobe') - -def parse_asset(asset): - parsed = Storage() - try: - with open(asset, 'r') as f: - for line in f.readlines(): - # XXX This should be rewritten, if the values contain - # #: they will be rewritten with blank. - # should not be an issue but this is not a very good parser - if line.startswith("#:"): - n = line.split(' ')[0].replace('#:','') - v = line.replace('#:'+n+' ', '').strip() - if n in ('tests', 'files'): - parsed[n] = v.split(",") - else: - parsed[n] = v - - elif line.startswith("#"): - continue - else: - break - finally: - if not parsed.name: - parsed.name = asset - if not parsed.files: - parsed.files = asset - return parsed - -def import_test(name, config): - if name.endswith(".py"): - test = Storage() - test_name = name.split(".")[0] - fp, pathname, description = imp.find_module(test_name, - [config.main.testdir]) - module = imp.load_module(name, fp, pathname, description) - - try: - test.name = module.__name__ - test.desc = module.__desc__ - test.module = module - except: - test.name = test_name - test.desc = "" - test.module = module - - return test_name, test - - return None, None - -class Log(): - """ - This is a class necessary for parsing YAML log files. - It is required because pyYaml has a bug in parsing - log format YAML files. - """ - def __init__(self, file=None): - if file: - self.fh = open(file) - - def __iter__(self): - return self - - def next(self): - lines = [] - try: - line = self.fh.readline() - if not line: - raise StopIteration - while not line.startswith("---"): - lines.append(line) - line = self.fh.readline() - return lines - except: - raise StopIteration - - diff --git a/ooni/utils/__init__.py b/ooni/utils/__init__.py new file mode 100644 index 0000000..e9c55f1 --- /dev/null +++ b/ooni/utils/__init__.py @@ -0,0 +1,146 @@ +""" + +""" + +import imp +import logging +try: + import yaml +except: + print "Error in importing YAML" + +class Storage(dict): + """ + A Storage object is like a dictionary except `obj.foo` can be used + in addition to `obj['foo']`. + + >>> o = Storage(a=1) + >>> o.a + 1 + >>> o['a'] + 1 + >>> o.a = 2 + >>> o['a'] + 2 + >>> del o.a + >>> o.a + None + """ + + def __getattr__(self, key): + try: + return self[key] + except KeyError, k: + return None + + def __setattr__(self, key, value): + self[key] = value + + def __delattr__(self, key): + try: + del self[key] + except KeyError, k: + raise AttributeError, k + + def __repr__(self): + return '<Storage ' + dict.__repr__(self) + '>' + + def __getstate__(self): + return dict(self) + + def __setstate__(self, value): + for (k, v) in value.items(): + self[k] = v + + +def get_logger(config): + loglevel = getattr(logging, config.loglevel.upper()) + logging.basicConfig(level=loglevel, + format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', + filename=config.logfile, + filemode='w') + + console = logging.StreamHandler() + console.setLevel(getattr(logging, config.consoleloglevel.upper())) + # Set the console logger to a different format + formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s') + console.setFormatter(formatter) + logging.getLogger('').addHandler(console) + + return logging.getLogger('ooniprobe') + +def parse_asset(asset): + parsed = Storage() + try: + with open(asset, 'r') as f: + for line in f.readlines(): + # XXX This should be rewritten, if the values contain + # #: they will be rewritten with blank. + # should not be an issue but this is not a very good parser + if line.startswith("#:"): + n = line.split(' ')[0].replace('#:','') + v = line.replace('#:'+n+' ', '').strip() + if n in ('tests', 'files'): + parsed[n] = v.split(",") + else: + parsed[n] = v + + elif line.startswith("#"): + continue + else: + break + finally: + if not parsed.name: + parsed.name = asset + if not parsed.files: + parsed.files = asset + return parsed + +def import_test(name, config): + if name.endswith(".py"): + test = Storage() + test_name = name.split(".")[0] + fp, pathname, description = imp.find_module(test_name, + [config.main.testdir]) + module = imp.load_module(name, fp, pathname, description) + + try: + test.name = module.__name__ + test.desc = module.__desc__ + test.module = module + except: + test.name = test_name + test.desc = "" + test.module = module + + return test_name, test + + return None, None + +class Log(): + """ + This is a class necessary for parsing YAML log files. + It is required because pyYaml has a bug in parsing + log format YAML files. + """ + def __init__(self, file=None): + if file: + self.fh = open(file) + + def __iter__(self): + return self + + def next(self): + lines = [] + try: + line = self.fh.readline() + if not line: + raise StopIteration + while not line.startswith("---"): + lines.append(line) + line = self.fh.readline() + return lines + except: + raise StopIteration + + diff --git a/ooni/utils/config.py b/ooni/utils/config.py new file mode 100644 index 0000000..42a14f6 --- /dev/null +++ b/ooni/utils/config.py @@ -0,0 +1,53 @@ +import ConfigParser +from utils import Storage + +class Config(Storage): + """ + A Storage-like class which loads and store each attribute into a portable + conf file. + """ + def __init__(self, section, cfgfile="ooni-probe.conf"): + super(Config, self).__init__() + + self._cfgfile = cfgfile + # setting up confgiparser + self._cfgparser = ConfigParser.ConfigParser() + self._cfgparser.read([self._cfgfile]) + self._section = section + + def __getattr__(self, name): + if name.startswith('_'): + return self.__dict__.get(name, None) + + try: + value = self._cfgparser.get(self._section, name) + if value.isdigit(): + return int(value) + elif value.lower() in ('true', 'false'): + return value.lower() == 'true' + else: + return value + except ConfigParser.NoOptionError: + return '' # if option doesn't exists return an empty string + + def __setattr__(self, name, value): + # keep an open port with private attributes + if name.startswith('_'): + self.__dict__[name] = value + return + + try: + # XXX: Automagically discover variable type + self._cfgparser.set(self._section, name, value) + except ConfigParser.NoOptionError: + raise NameError(name) + + def commit(self): + """ + Commit changes in config file. + """ + cfgfile = open(self._cfgfile, 'w') + try: + self._cfgparser.write(cfgfile) + finally: + cfgfile.close() diff --git a/ooni/utils/date.py b/ooni/utils/date.py new file mode 100644 index 0000000..59ec1f8 --- /dev/null +++ b/ooni/utils/date.py @@ -0,0 +1,22 @@ +from ooni.lib.rfc3339 import rfc3339 +from datetime import datetime + +class odate(datetime): + def __str__(self): + return rfc3339(self) + + def __repr__(self): + return "'%s'" % rfc3339(self) + + def from_rfc(self, datestr): + pass + +def now(): + return odate.utcnow() + +def pretty_date(): + cur_time = datetime.utcnow() + d_format = "%d %B %Y %H:%M:%S" + pretty = cur_time.strftime(d_format) + return pretty + diff --git a/ooni/utils/log.py b/ooni/utils/log.py new file mode 100644 index 0000000..3dc52dd --- /dev/null +++ b/ooni/utils/log.py @@ -0,0 +1,43 @@ +""" +OONI logging facility. +""" +import sys +import logging +import warnings + +from twisted.python import log + +# Logging levels +DEBUG = logging.DEBUG +INFO = logging.INFO +WARNING = logging.WARNING +ERROR = logging.ERROR +CRITICAL = logging.CRITICAL + +def _get_log_level(level): + if not level: + return INFO + else: + return level + +def start(logfile=None, loglevel=None, logstdout=True): + if log.defaultObserver: + print "%s" % logstdout + loglevel = _get_log_level(loglevel) + file = open(logfile, 'a') if logfile else sys.stderr + observer = log.FileLogObserver(file) + if logstdout: + log.startLogging(sys.stdout) + else: + log.startLogging() + log.addObserver(observer.emit) + msg("Started OONI") + +def msg(message, level=INFO, **kw): + log.msg(message, logLevel=level, **kw) + +def err(message, **kw): + log.err(message, **kw) + +def debug(message, **kw): + log.msg(message, logLevel=DEBUG, **kw) diff --git a/ooni/utils/logo.py b/ooni/utils/logo.py new file mode 100644 index 0000000..8924462 --- /dev/null +++ b/ooni/utils/logo.py @@ -0,0 +1,214 @@ +import random +def getlogo(): + logo = [] + logo.append(""" + _____ _____ __ _ _____ _____ ______ _____ ______ _______ + | | | | | \ | | |_____] |_____/ | | |_____] |______ + |_____| |_____| | \_| __|__ | | \_ |_____| |_____] |______""") + + logo.append(""" + _ _ + (_) | | + ___ ___ _ __ _ _ __ _ __ ___ | |__ ___ + / _ \ / _ \| '_ \| | '_ \| '__/ _ \| '_ \ / _ \\ +| (_) | (_) | | | | | |_) | | | (_) | |_) | __/ + \___/ \___/|_| |_|_| .__/|_| \___/|_.__/ \___| + | | + |_|""") + + + logo.append(""" + .__ ___. + ____ ____ ____ |__|_____________ ____\_ |__ ____ + / _ \ / _ \ / \| |\____ \_ __ \/ _ \| __ \_/ __ \\ +( <_> | <_> ) | \ || |_> > | \( <_> ) \_\ \ ___/ + \____/ \____/|___| /__|| __/|__| \____/|___ /\___ > + \/ |__| \/ \/""") + + + logo.append(""" + __ __ __ __ __ __ ___ +/ \ / \ |\ | | |__) |__) / \ |__) |__ +\__/ \__/ | \| | | | \ \__/ |__) |___ """) + + + logo.append(""" + _ _ ._ o ._ ._ _ |_ _ + (_) (_) | | | |_) | (_) |_) (/_ + | +""") + + logo.append(""" + _______ _______ __ _ ___ _______ ______ _______ _______ _______ +| || || | | || | | || _ | | || _ || | +| _ || _ || |_| || | | _ || | || | _ || |_| || ___| +| | | || | | || || | | |_| || |_|| | | | || || |___ +| |_| || |_| || _ || | | ___|| __ || |_| || _ | | ___| +| || || | | || | | | | | ||| || |_| || |___ +|_______||_______||_| |__||___| |___| |___| |||_______||_______||_______| +""") + + + logo.append(""" + _ _ + ___ ___ _ __ (_)_ __ _ __ ___ | |__ ___ + / _ \ / _ \| '_ \| | '_ \| '__/ _ \| '_ \ / _ \\ +| (_) | (_) | | | | | |_) | | | (_) | |_) | __/ + \___/ \___/|_| |_|_| .__/|_| \___/|_.__/ \___| + |_| +""") + + logo.append(""" + o + o O + O + o +.oOo. .oOo. 'OoOo. O .oOo. `OoOo. .oOo. OoOo. .oOo. +O o O o o O o O o o O o O o OooO' +o O o O O o O o O O o O o O O +`OoO' `OoO' o O o' oOoO' o `OoO' `OoO' `OoO' + O + o' +""") + + + logo.append(""" + _____ _____ ____ _ ____ _____ _____ _____ ______ ______ + / \/ \| \ | || || || | / \| >| ___| + | || || \| || || _|| \ | || < | ___| + \_____/\_____/|__/\____||____||___| |__|\__\\\\_____/|______>|______| +""") + + logo.append(""" + _ _ + ___ ___ ___|_|___ ___ ___| |_ ___ +| . | . | | | . | _| . | . | -_| +|___|___|_|_|_| _|_| |___|___|___| + |_| +""") + + logo.append(""" + _ _ + (_) | | + ___ ___ ____ _ ____ ____ ___ | |__ _____ + / _ \ / _ \| _ \| | _ \ / ___) _ \| _ \| ___ | +| |_| | |_| | | | | | |_| | | | |_| | |_) ) ____| + \___/ \___/|_| |_|_| __/|_| \___/|____/|_____) + |_| +""") + logo.append(""" + _ __ + ____ ____ ____ (_)____ _________ / /_ ___ + / __ \/ __ \/ __ \/ // __ \/ ___/ __ \/ __ \/ _ \\ +/ /_/ / /_/ / / / / // /_/ / / / /_/ / /_/ / __/ +\____/\____/_/ /_/_// .___/_/ \____/_.___/\___/ + /_/ +""") + logo.append(""" + _) | + _ \ _ \ \ | _ \ _| _ \ _ \ -_) + \___/ \___/ _| _| _| .__/ _| \___/ _.__/ \___| + _| +""") + logo.append(""" + _ _ + ___ ___ _ __ (_)_ __ _ __ ___ | |__ ___ + / _ \ / _ \| '_ \| | '_ \| '__/ _ \| '_ \ / _ \\ + | (_) | (_) | | | | | |_) | | | (_) | |_) | __/ + \___/ \___/|_| |_|_| .__/|_| \___/|_.__/ \___| + |_| +""") + logo.append(""" + .-. ( ) + .--. .--. ___ .-. ( __) .-.. ___ .-. .--. | |.-. .--. + / \ / \ ( ) \ (''") / \ ( ) \ / \ | / \ / \\ +| .-. ; | .-. ; | .-. . | | ' .-, ; | ' .-. ; | .-. ; | .-. | | .-. ; +| | | | | | | | | | | | | | | | . | | / (___) | | | | | | | | | | | | +| | | | | | | | | | | | | | | | | | | | | | | | | | | | | |/ | +| | | | | | | | | | | | | | | | | | | | | | | | | | | | | ' _.' +| ' | | | ' | | | | | | | | | | ' | | | | ' | | | ' | | | .'.-. +' `-' / ' `-' / | | | | | | | `-' ' | | ' `-' / ' `-' ; ' `-' / + `.__.' `.__.' (___)(___) (___) | \__.' (___) `.__.' `.__. `.__.' + | | + (___) +""") + logo.append(""" + o | +,---.,---.,---..,---.,---.,---.|---.,---. +| || || ||| || | || ||---' +`---'`---'` '`|---'` `---'`---'`---' + | +""") + logo.append(""" + ________________________________________ +/ OONI-PROBE -- WE FIGHT CENSORSHIP LIKE \\ +\ DRAGONS EAT KNIGHTS! / + ---------------------------------------- + \ ^ /^ + \ / \ // \\ + \ |\___/| / \// .\\ + \ /O O \__ / // | \ \ *----* + / / \/_/ // | \ \ \ | + @___@` \/_ // | \ \ \/\ \\ + 0/0/| \/_ // | \ \ \ \\ + 0/0/0/0/| \/// | \ \ | | + 0/0/0/0/0/_|_ / ( // | \ _\ | / + 0/0/0/0/0/0/`/,_ _ _/ ) ; -. | _ _\.-~ / / + ,-} _ *-.|.-~-. .~ ~ + \ \__/ `/\ / ~-. _ .-~ / + \____(oo) *. } { / + ( (--) .----~-.\ \-` .~ + //__\\\\ \__ Ack! ///.----..< \ _ -~ + // \\\\ ///-._ _ _ _ _ _ _{^ - - - - ~ +""") + logo.append(""" + _________________ +| OONI-PROBE | +| DON'T CENSOR ME | +|_________________| +\ . . + \ / `. .' " + \ .---. < > < > .---. + \ | \ \ - ~ ~ - / / | + _____ ..-~ ~-..-~ + | (A)| \~~~\.' `./~~~/ + --------- \__/ \__/ + .' O \ / / \ " + (_____, `._.' | } \/~~~/ + `----. / } | / \__/ + `-. | / | / `. ,~~| + ~-.__| /_ - ~ ^| /- _ `..-' + | / | / ~-. `-. _ _ _ + |_____| |_____| ~ - . _ _ _ _ _> +""") + logo.append(""" + _________________ + ,/~~, | | + /`` \ /| OONI-PROBE... | + `( O^O ) / | | + `\_-_/` | ...DOES THIS | + ~~~ | EXCITE YOU? | + __| |__ |_________________| + | | + | af | + | | | | + | | | \\ + | | |\ \\ + | | | \ \\ + | | | \ \\ + | |___| ((( + (((|/*\| + | | | + | | | + | | | + | | | + | | | + | | | + | | | + | | | + | | | + | | | + _| | |_ + (___|___) +""") + return random.choice(logo)

1 0

[stem/master] Only consulting major descriptor versions
by atagar＠torproject.org 10 Jul '12

10 Jul '12

commit d52e934c24c9d327e277eb60bfee060697cb43ec Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Jul 9 11:52:56 2012 -0700 Only consulting major descriptor versions Minor descriptor version bumps are backward compatable changes. We might not recognize new additions, but it won't break us either (they'll just be unrecognized). Pointed out by Karsten on... https://trac.torproject.org/6257 --- stem/descriptor/__init__.py | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/stem/descriptor/__init__.py b/stem/descriptor/__init__.py index f3b40c5..bb6f874 100644 --- a/stem/descriptor/__init__.py +++ b/stem/descriptor/__init__.py @@ -81,13 +81,13 @@ def parse_file(path, descriptor_file): desc_type, major_version, minor_version = metrics_header_match.groups() major_version, minor_version = int(major_version), int(minor_version) - if desc_type == "server-descriptor" and major_version == 1 and minor_version == 0: + if desc_type == "server-descriptor" and major_version == 1: desc = stem.descriptor.server_descriptor.RelayDescriptor(descriptor_file.read()) - elif desc_type == "bridge-server-descriptor" and major_version == 1 and minor_version == 0: + elif desc_type == "bridge-server-descriptor" and major_version == 1: desc = stem.descriptor.server_descriptor.BridgeDescriptor(descriptor_file.read()) - elif desc_type == "extra-info" and major_version == 1 and minor_version == 0: + elif desc_type == "extra-info" and major_version == 1: desc = stem.descriptor.extrainfo_descriptor.RelayExtraInfoDescriptor(descriptor_file.read()) - elif desc_type == "bridge-extra-info" and major_version == 1 and minor_version in (0, 1): + elif desc_type == "bridge-extra-info" and major_version == 1: # version 1.1 introduced a 'transport' field... # https://trac.torproject.org/6257

1 0

[stem/master] Supporting multiple extra-info transport lines
by atagar＠torproject.org 10 Jul '12

10 Jul '12

commit a0f980c996587ef112379891339d463bda3d487a Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Jul 9 12:00:08 2012 -0700 Supporting multiple extra-info transport lines The 'transport' lines being added to extra-info descriptors can appear any number of times, so storing them as a list. These options are weird in that they have an "address:port [arglist]" which never appears in the wild since they both only appear on bridge descriptors and are scrubbed from bridge descripotrs. --- stem/descriptor/extrainfo_descriptor.py | 11 +++++++++-- test/unit/descriptor/extrainfo_descriptor.py | 2 +- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/stem/descriptor/extrainfo_descriptor.py b/stem/descriptor/extrainfo_descriptor.py index b1c7b09..366279c 100644 --- a/stem/descriptor/extrainfo_descriptor.py +++ b/stem/descriptor/extrainfo_descriptor.py @@ -743,7 +743,7 @@ class BridgeExtraInfoDescriptor(ExtraInfoDescriptor): """ Bridge extra-info descriptor (`specification <https://metrics.torproject.org/formats.html#bridgedesc>`_) - :var str transport: transport method recognized by the bridge (ex. obfs3) + :var list transport: transport method recognized by the bridge (ex. obfs3) """ def __init__(self, raw_contents, validate = True): @@ -764,7 +764,14 @@ class BridgeExtraInfoDescriptor(ExtraInfoDescriptor): line = "%s %s" % (keyword, value) # original line if keyword == "transport": - self.transport = value + # "transport" transportname address:port [arglist] + # Everything after the transportname is scrubbed in published bridge + # descriptors, so we'll never see it in practice. + + if self.transport is None: + self.transport = [] + + self.transport.append(value) del entries["transport"] elif keyword == "router-digest": if validate and not stem.util.tor_tools.is_hex_digits(value, 40): diff --git a/test/unit/descriptor/extrainfo_descriptor.py b/test/unit/descriptor/extrainfo_descriptor.py index e9b06a0..a3590ee 100644 --- a/test/unit/descriptor/extrainfo_descriptor.py +++ b/test/unit/descriptor/extrainfo_descriptor.py @@ -529,7 +529,7 @@ class TestExtraInfoDescriptor(unittest.TestCase): desc_text = _make_descriptor({"transport": "obfs3"}, is_bridge = True) desc = BridgeExtraInfoDescriptor(desc_text) - self.assertEquals("obfs3", desc.transport) + self.assertEquals(["obfs3"], desc.transport) self.assertEquals([], desc.get_unrecognized_lines()) def _expect_invalid_attr(self, desc_text, attr = None, expected_value = None):

1 0

[stem/master] Allowing extra-info dirreq-v*-share to be above 100%
by atagar＠torproject.org 10 Jul '12

10 Jul '12

commit 611e75dbeb5a81a655ca0f8ec1b4495850c3459a Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Jul 10 08:22:55 2012 -0700 Allowing extra-info dirreq-v*-share to be above 100% Spotted an extra-info descriptor with dirreq-v2-share and dirreq-v3-share values over 100%... https://lists.torproject.org/pipermail/tor-dev/2012-June/003679.html Karsten suggested simply removing the restriction on this field's upper bound since it's soon going away. --- stem/descriptor/extrainfo_descriptor.py | 7 +++++-- test/unit/descriptor/extrainfo_descriptor.py | 1 - 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/stem/descriptor/extrainfo_descriptor.py b/stem/descriptor/extrainfo_descriptor.py index 366279c..be2a6dc 100644 --- a/stem/descriptor/extrainfo_descriptor.py +++ b/stem/descriptor/extrainfo_descriptor.py @@ -488,8 +488,11 @@ class ExtraInfoDescriptor(stem.descriptor.Descriptor): if not value.endswith("%"): raise ValueError() percentage = float(value[:-1]) / 100 - if validate and (percentage > 1 or percentage < 0): - raise ValueError() + # Bug lets these be above 100%, however they're soon going away... + # https://lists.torproject.org/pipermail/tor-dev/2012-June/003679.html + + if validate and percentage < 0: + raise ValueError("Negative percentage value: %s" % line) if keyword == "dirreq-v2-share": self.dir_v2_share = percentage diff --git a/test/unit/descriptor/extrainfo_descriptor.py b/test/unit/descriptor/extrainfo_descriptor.py index a3590ee..c40ac94 100644 --- a/test/unit/descriptor/extrainfo_descriptor.py +++ b/test/unit/descriptor/extrainfo_descriptor.py @@ -305,7 +305,6 @@ class TestExtraInfoDescriptor(unittest.TestCase): ("", None), (" ", None), ("100", None), - ("100.1%", 1.001), ("-5%", -0.05), )

1 0