tor-commits June 2013

tor-commits@lists.torproject.org

19 participants
1571 discussions

[ooni-probe/develop] Update setup.py, requirements and Vagrant file
by isis＠torproject.org 26 Jun '13

26 Jun '13

commit 3f233bfc9284bfada1ea60fb83072ccdf8eb7e29 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Jun 19 23:58:07 2013 +0100 Update setup.py, requirements and Vagrant file --- Vagrantfile | 5 ++--- requirements.txt | 1 + setup.py | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Vagrantfile b/Vagrantfile index 90002aa..897838f 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -142,9 +142,8 @@ pip install -r requirements.txt echo "Installing ooniprobe" python setup.py install -echo "Fetching all inputs" -cd /data/ooniprobe/inputs -make lists +cd /usr/share/ooni/ +make geoip SCRIPT diff --git a/requirements.txt b/requirements.txt index 2dcab52..b335716 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,6 +10,7 @@ txtorcon>=0.7 txsocksx>=0.0.2 Pyrex>=0.9.8.6 parsley>=1.1 +cyclone # Taken from the versions required by twisted 13.0.0 (latest) transaction>=1.1.1 diff --git a/setup.py b/setup.py index 5f221d0..3c42218 100644 --- a/setup.py +++ b/setup.py @@ -38,7 +38,7 @@ for root, dirs, file_names in os.walk('data/'): for file_name in file_names: if not file_name.endswith('.pyc'): files.append(pj(root, file_name)) - data_files.append([pj(usr_share_path, root), files]) + data_files.append([pj(usr_share_path, root.replace('data/', '')), files]) with open('requirements.txt') as f: for line in f:

1 0

[ooni-probe/develop] Add step to install virtualbox
by isis＠torproject.org 26 Jun '13

26 Jun '13

commit ede4751899994fde449b47691dd20ff1f0d42604 Author: Arturo Filastò <hellais(a)gmail.com> Date: Mon Jun 24 16:47:33 2013 +0300 Add step to install virtualbox --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6da4680..bf1506e 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ network data that will assist you with your endeavors! ## Getting started with ooniprobe is easy (with Vagrant) -0) [Install Vagrant](http://downloads.vagrantup.com/) +0) [Install Vagrant](http://downloads.vagrantup.com/) and [Install Virtualbox](https://www.virtualbox.org/wiki/Downloads) 0.1)

1 0

[ooni-probe/develop] update links to nettests which has been moved to data.
by isis＠torproject.org 26 Jun '13

26 Jun '13

commit 0a2c19197c23d0b7187c379d518b25459d083f78 Author: Johannes Fürmann <johannes(a)weltraumpflege.org> Date: Mon Jun 24 19:38:55 2013 +0300 update links to nettests which has been moved to data. --- docs/source/index.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 43ad1fe..07737b9 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -39,9 +39,9 @@ Core ooniprobe Tests -------------------- The source for `Content blocking tests -<https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/nettests/blocking>`_ +<https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/data/nettests/block…>`_ and `Traffic Manipulation tests -<https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/nettests/blocking>`_ +<https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/data/nettests/block…>`_ can be found in the nettests/blocking and nettests/manipulation directories respectively. @@ -77,17 +77,17 @@ being experimented with. You can find these in: * `nettests/experimental - <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/nettests/experiment…>`_ + <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/data/nettests/exper…>`_ Tests that don't do a measurement but are useful for scanning can be found in: * `nettests/scanning - <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/nettests/scanning>`_ + <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/data/nettests/scann…>`_ Tests that involve running third party tools may be found in: * `nettests/third_party - <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/nettests/third_party>`_ + <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/data/nettests/third…>`_ oonib *****

1 0

[ooni-probe/develop] Merge pull request #124 from waaaaargh/master
by isis＠torproject.org 26 Jun '13

26 Jun '13

commit 117ee8ef4aa675505e2856900dadb6c6fd3fa265 Merge: ede4751 7695773 Author: Arturo Filastò <hellais(a)gmail.com> Date: Mon Jun 24 16:19:30 2013 -0700 Merge pull request #124 from waaaaargh/master Update links to nettest tree in documentation docs/source/index.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)

1 0

[ooni-probe/develop] update link names as well
by isis＠torproject.org 26 Jun '13

26 Jun '13

commit 76957738e81661e9728a5a2bc50db0678defdeaa Author: Johannes Fürmann <johannes(a)weltraumpflege.org> Date: Mon Jun 24 18:40:40 2013 +0200 update link names as well --- docs/source/index.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 07737b9..b74799c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -76,17 +76,17 @@ being experimented with. You can find these in: - * `nettests/experimental + * `data/nettests/experimental <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/data/nettests/exper…>`_ Tests that don't do a measurement but are useful for scanning can be found in: - * `nettests/scanning + * `data/nettests/scanning <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/data/nettests/scann…>`_ Tests that involve running third party tools may be found in: - * `nettests/third_party + * `data/nettests/third_party <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/data/nettests/third…>`_ oonib

1 0

[ooni-probe/develop] Fix director unittest
by isis＠torproject.org 26 Jun '13

26 Jun '13

commit 1ac4e4fd8f13b8e358034d3b69174fdc19b02543 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Feb 27 17:11:22 2013 +0100 Fix director unittest --- tests/test_director.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tests/test_director.py b/tests/test_director.py index 484cb21..a9dfbe8 100644 --- a/tests/test_director.py +++ b/tests/test_director.py @@ -42,9 +42,11 @@ class TestDirector(unittest.TestCase): pass def test_start_net_test(self): - net_test_loader = NetTestLoader(net_test_string, dummyArgs) - net_test_loader.checkOptions() - d = self.director.startNetTest('', net_test_loader, self.reporters) + ntl = NetTestLoader(dummyArgs) + ntl.loadNetTestString(net_test_string) + + ntl.checkOptions() + d = self.director.startNetTest('', ntl, self.reporters) @d.addCallback def done(result):

1 0

[ooni-probe/develop] Refactoring of NetTestLoader
by isis＠torproject.org 26 Jun '13

26 Jun '13

commit 8bead060681d86cdfa43f35b1b57567e51816aa6 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Feb 27 17:05:39 2013 +0100 Refactoring of NetTestLoader * Make it clear that calling one of those methods can be extremely dangerous * Kill a bug spotted thanks to unittesting --- ooni/nettest.py | 86 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 46 insertions(+), 40 deletions(-) diff --git a/ooni/nettest.py b/ooni/nettest.py index 6273d34..6323989 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -23,7 +23,10 @@ class NetTestLoader(object): def __init__(self, options): self.options = options - self.testCases = self.loadNetTest(options['test']) + if 'test_file' in options: + self.loadNetTestFile(options['test_file']) + elif 'test_string' in options: + self.loadNetTestString(options['test_string']) @property def testDetails(self): @@ -109,7 +112,45 @@ class NetTestLoader(object): assert usage_options == test_class.usageOptions return usage_options - def loadNetTest(self, net_test_file): + def loadNetTestString(self, net_test_string): + """ + Load NetTest from a string. + WARNING input to this function *MUST* be sanitized and *NEVER* be + untrusted. + Failure to do so will result in code exec. + + net_test_string: + + a string that contains the net test to be run. + """ + net_test_file_object = StringIO(net_test_string) + + ns = {} + test_cases = [] + exec net_test_file_object.read() in ns + for item in ns.itervalues(): + test_cases.extend(self._get_test_methods(item)) + + if not test_cases: + raise NoTestCasesFound + + self.setupTestCases(test_cases) + + def loadNetTestFile(self, net_test_file): + """ + Load NetTest from a file. + """ + test_cases = [] + module = filenameToModule(net_test_file) + for __, item in getmembers(module): + test_cases.extend(self._get_test_methods(item)) + + if not test_cases: + raise NoTestCasesFound + + self.setupTestCases(test_cases) + + def setupTestCases(self, test_cases): """ Creates all the necessary test_cases (a list of tuples containing the NetTestCase (test_class, test_method)) @@ -130,25 +171,10 @@ class NetTestLoader(object): is either a file path or a file like object that will be used to generate the test_cases. """ - test_cases = None - try: - if os.path.isfile(net_test_file): - test_cases = self._loadNetTestFile(net_test_file) - else: - net_test_file = StringIO(net_test_file) - raise TypeError("not a file path") - - except TypeError: - if hasattr(net_test_file, 'read'): - test_cases = self._loadNetTestFromFileObject(net_test_file) - - if not test_cases: - raise NoTestCasesFound - test_class, _ = test_cases[0] self.testVersion = test_class.version self.testName = test_class.name.lower().replace(' ','_') - return test_cases + self.testCases = test_cases def checkOptions(self): """ @@ -160,7 +186,8 @@ class NetTestLoader(object): for klass in test_classes: options = self.usageOptions() - options.parseOptions(self.options['subargs']) + options.parseOptions(self.options) + if options: klass.localOptions = options @@ -175,27 +202,6 @@ class NetTestLoader(object): inputs = [None] klass.inputs = inputs - def _loadNetTestFromFileObject(self, net_test_string): - """ - Load NetTest from a string - """ - ns = {} - test_cases = [] - exec net_test_string.read() in ns - for item in ns.itervalues(): - test_cases.extend(self._get_test_methods(item)) - return test_cases - - def _loadNetTestFile(self, net_test_file): - """ - Load NetTest from a file - """ - test_cases = [] - module = filenameToModule(net_test_file) - for __, item in getmembers(module): - test_cases.extend(self._get_test_methods(item)) - return test_cases - def _get_test_methods(self, item): """ Look for test_ methods in subclasses of NetTestCase

1 0

[ooni-probe/develop] Merge branch 'master' into task_manager
by isis＠torproject.org 26 Jun '13

26 Jun '13

commit d3ab36b744b6e8e9862fd14ffae586d1b1a8e32f Merge: 1710ca5 6f16d8c Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Feb 27 16:36:44 2013 +0100 Merge branch 'master' into task_manager * master: Order matters for little ol pip requirements.txt | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)

1 0

[ooni-probe/develop] Fix unittests for nettest
by isis＠torproject.org 26 Jun '13

26 Jun '13

commit 04d5fc5d157b7e5ac47230ee018fb382756f85dc Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Feb 27 17:05:27 2013 +0100 Fix unittests for nettest --- tests/test_nettest.py | 62 +++++++++++++++++++++++++++++-------------------- 1 file changed, 37 insertions(+), 25 deletions(-) diff --git a/tests/test_nettest.py b/tests/test_nettest.py index 5645dbf..78240d5 100644 --- a/tests/test_nettest.py +++ b/tests/test_nettest.py @@ -61,7 +61,7 @@ class DummyTestCase(NetTestCase): self.report['foo'] = 'foo' """ -net_test_with_required_option = """ +net_test_string_with_required_option = """ from twisted.python import usage from ooni.nettest import NetTestCase @@ -124,8 +124,9 @@ class TestNetTest(unittest.TestCase): f.write(net_test_string) f.close() - ntl = NetTestLoader(net_test_file, - dummyArgs) + ntl = NetTestLoader(dummyArgs) + ntl.loadNetTestFile(net_test_file) + self.verifyMethods(ntl.testCases) os.unlink(net_test_file) @@ -134,8 +135,9 @@ class TestNetTest(unittest.TestCase): Given a file like object verify that the net test cases are properly generated. """ - ntl = NetTestLoader(net_test_string, - dummyArgs) + ntl = NetTestLoader(dummyArgs) + ntl.loadNetTestString(net_test_string) + self.verifyMethods(ntl.testCases) def test_load_net_test_from_StringIO(self): @@ -143,13 +145,15 @@ class TestNetTest(unittest.TestCase): Given a file like object verify that the net test cases are properly generated. """ - ntl = NetTestLoader(StringIO(net_test_string), - dummyArgs) + ntl = NetTestLoader(dummyArgs) + ntl.loadNetTestString(net_test_string) + self.verifyMethods(ntl.testCases) def test_load_with_option(self): - ntl = NetTestLoader(StringIO(net_test_string), - dummyArgs) + ntl = NetTestLoader(dummyArgs) + ntl.loadNetTestString(net_test_string) + self.assertIsInstance(ntl, NetTestLoader) for test_klass, test_meth in ntl.testCases: for option in dummyOptions.keys(): @@ -157,27 +161,32 @@ class TestNetTest(unittest.TestCase): def test_load_with_invalid_option(self): try: - ntl = NetTestLoader(StringIO(net_test_string), dummyInvalidArgs) + ntl = NetTestLoader(dummyInvalidArgs) + ntl.loadNetTestString(net_test_string) + ntl.checkOptions() raise Exception except UsageError: pass def test_load_with_required_option(self): - net_test = NetTestLoader(StringIO(net_test_with_required_option), - dummyArgsWithRequiredOptions) - self.assertIsInstance(net_test, NetTestLoader) + ntl = NetTestLoader(dummyArgsWithRequiredOptions) + ntl.loadNetTestString(net_test_string_with_required_option) + + self.assertIsInstance(ntl, NetTestLoader) def test_load_with_missing_required_option(self): try: - net_test = NetTestLoader(StringIO(net_test_with_required_option), - dummyArgs) + ntl = NetTestLoader(dummyArgs) + ntl.loadNetTestString(net_test_string_with_required_option) + except MissingRequiredOption: pass def test_net_test_inputs(self): - ntl = NetTestLoader(StringIO(net_test_string_with_file), - dummyArgsWithFile) + ntl = NetTestLoader(dummyArgsWithFile) + ntl.loadNetTestString(net_test_string_with_file) + ntl.checkOptions() # XXX: if you use the same test_class twice you will have consumed all @@ -189,17 +198,18 @@ class TestNetTest(unittest.TestCase): self.assertEqual(len(list(test_class.inputs)), 10) def test_setup_local_options_in_test_cases(self): - ntl = NetTestLoader(StringIO(net_test_string), - dummyArgs) + ntl = NetTestLoader(dummyArgs) + ntl.loadNetTestString(net_test_string) + ntl.checkOptions() for test_class, test_method in ntl.testCases: self.assertEqual(test_class.localOptions, dummyOptions) def test_generate_measurements_size(self): + ntl = NetTestLoader(dummyArgsWithFile) + ntl.loadNetTestString(net_test_string_with_file) - ntl = NetTestLoader(StringIO(net_test_string_with_file), - dummyArgsWithFile) ntl.checkOptions() net_test = NetTest(ntl, None) @@ -207,8 +217,9 @@ class TestNetTest(unittest.TestCase): self.assertEqual(len(measurements), 20) def test_net_test_completed_callback(self): - ntl = NetTestLoader(StringIO(net_test_string_with_file), - dummyArgsWithFile) + ntl = NetTestLoader(dummyArgsWithFile) + ntl.loadNetTestString(net_test_string_with_file) + ntl.checkOptions() director = Director() @@ -224,8 +235,9 @@ class TestNetTest(unittest.TestCase): def test_require_root_succeed(self): #XXX: will require root to run - ntl = NetTestLoader(StringIO(net_test_root_required), - dummyArgs) + ntl = NetTestLoader(dummyArgs) + ntl.loadNetTestString(net_test_root_required) + for test_class, method in ntl.testCases: self.assertTrue(test_class.requiresRoot)

1 0

[ooni-probe/develop] Rename private option to test_file to separate it from test string.
by isis＠torproject.org 26 Jun '13

26 Jun '13

commit d1f41883c8d0187e6360b2a2b9075f0cb7eb7a37 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Feb 27 17:08:43 2013 +0100 Rename private option to test_file to separate it from test string. --- ooni/oonicli.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 2fd4688..243b90c 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -68,7 +68,7 @@ class Options(usage.Options): if self['testdeck']: return try: - self['test'] = args[0] + self['test_file'] = args[0] self['subargs'] = args[1:] except: raise usage.UsageError("No test filename specified!")

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits June 2013