April 2016 - tor-commits - lists.torproject.org

[ooni-probe/master] Merge pull request #459 from TheTorProject/doc/fix-test-helpers-doc
by art＠torproject.org 29 Apr '16

29 Apr '16

commit a9d48f3efae4d29d616fdc5dfbcf318a760a6048 Merge: 1fac7cd b6c84cd Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Feb 5 13:57:00 2016 +0100 Merge pull request #459 from TheTorProject/doc/fix-test-helpers-doc Doc/fix test helpers doc docs/source/writing_test_helpers.rst | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)

1 0

[ooni-probe/master] Fix misleading error message
by art＠torproject.org 29 Apr '16

29 Apr '16

commit be4cc909dcb620f5e1788ad5cf8549d14d0918f6 Author: Simone Basso <bassosimone(a)gmail.com> Date: Wed Nov 4 16:02:32 2015 +0100 Fix misleading error message --- bin/oonireport | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/oonireport b/bin/oonireport index 1335a83..6a79f4c 100755 --- a/bin/oonireport +++ b/bin/oonireport @@ -14,7 +14,7 @@ def failed(failure): r = failure.trap(exceptions.SystemExit, Exception) if r != exceptions.SystemExit: - log.err("Failed to run oonideckgen") + log.err("Failed to run oonireport") log.exception(failure) exitCode = 127 else:

1 0

[ooni-probe/master] Fix: Typo correction
by art＠torproject.org 29 Apr '16

29 Apr '16

commit b6c84cd03ce1a4ea1bfc2e07bcad1574e298cd7a Author: anadahz <kojgelo(a)inbox.com> Date: Thu Feb 4 11:03:38 2016 -0300 Fix: Typo correction --- docs/source/writing_test_helpers.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/source/writing_test_helpers.rst b/docs/source/writing_test_helpers.rst index 70d5e21..1926589 100644 --- a/docs/source/writing_test_helpers.rst +++ b/docs/source/writing_test_helpers.rst @@ -1,7 +1,7 @@ Writing Test Helpers ======== -OONI test helpers are used by OONI nettests to perform their meausrements. They can be used either to establish a ground truth or to exchange information with the probe to determine if some form of network manipulation is happening in the network path between the probe and the backend. +OONI test helpers are used by OONI nettests to perform their measurements. They can be used either to establish a ground truth or to exchange information with the probe to determine if some form of network manipulation is happening in the network path between the probe and the backend. Writing a Censorship Directionality Test Helper -------------------------- @@ -17,7 +17,7 @@ Creating the test helper ooni-backend keeps all the test-helpers in the `oonib/testhelpers directory <https://github.com/TheTorProject/ooni-backend/tree/master/oonib/testhelpers>`_ Each individual test helper is a twisted service. Most of the current test-helpers consists of a twisted Factory and a twisted Protocol defined in the test helpers directory and a `stock Twisted Server <https://twistedmatrix.com/documents/current/api/twisted.application.interne…>`_ that is defined in the backend code. We will follow this model in the tutorial. -Because of how simple this example test-helper is the job of our test-helper factory is merely to deploy a single instance of our protocol each time it's buildProtocol method is called. Because we have our factory inheret from the base `Factory object <https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.5.0/twisted…>`_ we merely have to define its ``protocol`` property to point to our protocol.:: +Because of how simple this example test-helper is the job of our test-helper factory is merely to deploy a single instance of our protocol each time it's buildProtocol method is called. Because we have our factory inhered from the base `Factory object <https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.5.0/twisted…>`_ we merely have to define its ``protocol`` property to point to our protocol.:: class TCPDirectionalityTestHelper(Factory): """ @@ -44,7 +44,7 @@ The protocol for this helper needs to do two things. First, upon receiving encod self.transport.write(original_string) -In order to make this test-helper slightly more flexible we will be allowing the backend to determine the encoding within their config file. To this end we will have to retreive the encoding from the config file.:: +In order to make this test-helper slightly more flexible we will be allowing the backend to determine the encoding within their config file. To this end we will have to retrieve the encoding from the config file.:: def dataReceived(self, data): @@ -70,9 +70,9 @@ Adding the helper to the config file ooni-backend uses a config file located at `/etc/oonibackend.conf <https://github.com/TheTorProject/ooni-backend/blob/master/oonib.conf.example>`_. This file contains a `section where each test-helper can be configured. <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_. -The test-helper will need to be given a unique identifyer so that it can be called from the config file. In this example we use ``tcp-directionality`` as our identifyer. +The test-helper will need to be given a unique identifier so that it can be called from the config file. In this example we use ``tcp-directionality`` as our identifier. -For a helper to be used in the ooni-backend it needs to be given an identifyer so that it can be called from the config file:: +For a helper to be used in the ooni-backend it needs to be given an identifier so that it can be called from the config file:: tcp-directionality: encoding: rot13 @@ -91,7 +91,7 @@ The OONI test-helper system is a collection of `Twisted services <https://twiste **NOTE:** In this example I have placed the original service in the existing tcp_helpers file. If you created your own file for your test-helper you will have to make sure that you import that file at the top of `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`_. -OONI uses a `Multi Service <https://twistedmatrix.com/documents/current/api/twisted.application.service…>`_ which allows them to combine all the OONI test-helpers and the report-collector into a singlular service for easier management. The next step for creating our test-helper is to add it to the ooni-backend `multiService <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_:: +OONI uses a `Multi Service <https://twistedmatrix.com/documents/current/api/twisted.application.service…>`_ which allows them to combine all the OONI test-helpers and the report-collector into a singular service for easier management. The next step for creating our test-helper is to add it to the ooni-backend `multiService <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_:: # Add the helper as a child of the backends multi-service test-helper multiService.addService(tcp_directionality_helper) @@ -101,7 +101,7 @@ Finally, we need to start our service.:: # Start the test-helpers service tcp_directionality_helper.startService() -In order for our test-helper to be managed using the backend config file we will need to modify this code to check the config file for a test-helper that uses the identifyer we selected earlier. For the directionality helper we check to see if our test-helper had its port specified in the config file to determine if it should be run. I also added a default encoding in case +In order for our test-helper to be managed using the backend config file we will need to modify this code to check the config file for a test-helper that uses the identifier we selected earlier. For the directionality helper we check to see if our test-helper had its port specified in the config file to determine if it should be run. I also added a default encoding in case This snippet contains the final code that would be inserted into `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`_.:: @@ -128,7 +128,7 @@ This snippet contains the final code that would be inserted into `oonibackend.py Requiring the helper in a test ------------- -If you are creating tests that rely on custom test-hepers you will want to make sure that you do not get innacturate results because your test-helper being missing in the ooni-backend you are testing against. You can specify required test-helpers within a ooni-probe test by setting its ``requiredTestHelpers`` property. In this example we have made our test helper require the tcp-directionality test that we created above.:: +If you are creating tests that rely on custom test-helpers you will want to make sure that you do not get inaccurate results because your test-helper being missing in the ooni-backend you are testing against. You can specify required test-helpers within a ooni-probe test by setting its ``requiredTestHelpers`` property. In this example we have made our test helper require the tcp-directionality test that we created above.:: class MyDirectionalityTest(nettest.NetTestCase): """ An example test."""

1 0

[ooni-probe/master] Refactoring of the ORG queue
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 814b2879843a04450f1822dbdd42909906e876df Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Feb 5 13:33:27 2016 +0100 Refactoring of the ORG queue * Do not start tor if we already have a tor state * Fix failure to import GeoIP --- ooni/director.py | 2 +- ooni/geoip.py | 1 + ooni/oonicli.py | 166 ++++++++++++++++++++++--------------------------------- 3 files changed, 67 insertions(+), 102 deletions(-) diff --git a/ooni/director.py b/ooni/director.py index 9bac49d..f633048 100644 --- a/ooni/director.py +++ b/ooni/director.py @@ -130,7 +130,7 @@ class Director(object): if start_tor: if check_incoherences: yield config.check_tor() - if config.advanced.start_tor: + if config.advanced.start_tor and config.tor_state is None: yield self.startTor() elif config.tor.control_port and config.tor_state is None: log.msg("Connecting to Tor Control Port...") diff --git a/ooni/geoip.py b/ooni/geoip.py index cb66675..11800b6 100644 --- a/ooni/geoip.py +++ b/ooni/geoip.py @@ -1,3 +1,4 @@ +from __future__ import absolute_import import re import os import random diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 47aacfb..fc5eca0 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -226,8 +226,8 @@ def setupCollector(global_options, net_test_loader): return collector -def createDeck(global_options,url=None,filename=None): - log.msg("Creating deck for: %s" %(url or filename,) ) +def createDeck(global_options, url=None, filename=None): + log.msg("Creating deck for: %s" % (url or filename,)) deck = Deck(no_collector=global_options['no-collector']) deck.bouncer = global_options['bouncer'] @@ -239,9 +239,9 @@ def createDeck(global_options,url=None,filename=None): log.debug("No test deck detected") test_file = nettest_to_path(global_options['test_file'], True) if url is not None: - args = ('-u',url) + args = ('-u', url) elif filename is not None: - args = ('-f',filename) + args = ('-f', filename) else: args = tuple() if any(global_options['subargs']): @@ -270,48 +270,13 @@ def createDeck(global_options,url=None,filename=None): sys.exit(5) return deck -def runWithDirector(logging=True, start_tor=True, check_incoherences=True): - """ - Instance the director, parse command line options and start an ooniprobe - test! - """ - - global_options = setupGlobalOptions(logging, start_tor, check_incoherences) - - director = Director() - if global_options['list']: - print "# Installed nettests" - for net_test_id, net_test in director.getNetTests().items(): - print "* %s (%s/%s)" % (net_test['name'], - net_test['category'], - net_test['id']) - print " %s" % net_test['description'] - - sys.exit(0) - - elif global_options['printdeck']: - del global_options['printdeck'] - print "# Copy and paste the lines below into a test deck to run the specified test with the specified arguments" - print yaml.safe_dump([{'options': global_options}]).strip() - - sys.exit(0) - - if global_options.get('annotations') is not None: - annotations = setupAnnotations(global_options) - - if global_options['no-collector']: - log.msg("Not reporting using a collector") - global_options['collector'] = None - start_tor = False - else: - start_tor = True - - if global_options['collector']: - start_tor |= True - deck = createDeck(global_options) +def runTestWithDirector(director, global_options, url=None, filename=None, + start_tor=True, check_incoherences=True): + deck = createDeck(global_options, url=url, filename=filename) start_tor |= deck.requiresTor + d = director.start(start_tor=start_tor, check_incoherences=check_incoherences) @@ -341,18 +306,59 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): test_details['annotations'] = global_options['annotations'] director.startNetTest(net_test_loader, - global_options['reportfile'], - collector) + global_options['reportfile'], + collector) return director.allTestsDone - d.addCallback(setup_nettest) d.addCallback(post_director_start) d.addErrback(director_startup_handled_failures) d.addErrback(director_startup_other_failures) return d - +def runWithDirector(logging=True, start_tor=True, check_incoherences=True): + """ + Instance the director, parse command line options and start an ooniprobe + test! + """ + + global_options = setupGlobalOptions(logging, start_tor, check_incoherences) + + director = Director() + if global_options['list']: + print "# Installed nettests" + for net_test_id, net_test in director.getNetTests().items(): + print "* %s (%s/%s)" % (net_test['name'], + net_test['category'], + net_test['id']) + print " %s" % net_test['description'] + + sys.exit(0) + + elif global_options['printdeck']: + del global_options['printdeck'] + print "# Copy and paste the lines below into a test deck to run the specified test with the specified arguments" + print yaml.safe_dump([{'options': global_options}]).strip() + + sys.exit(0) + + if global_options.get('annotations') is not None: + annotations = setupAnnotations(global_options) + + if global_options['no-collector']: + log.msg("Not reporting using a collector") + global_options['collector'] = None + start_tor = False + else: + start_tor = True + + if global_options['collector']: + start_tor |= True + + return runTestWithDirector(director=director, + global_options=global_options, + check_incoherences=check_incoherences) + # this variant version of runWithDirector splits the process in two, # allowing a single director instance to be reused with multiple decks. @@ -373,7 +379,6 @@ def runWithDaemonDirector(logging=True, start_tor=True, check_incoherences=True) sys.exit(7) - global_options = setupGlobalOptions(logging, start_tor, check_incoherences) director = Director() @@ -388,80 +393,40 @@ def runWithDaemonDirector(logging=True, start_tor=True, check_incoherences=True) else: start_tor = True - - def run_test(global_options, url=None, filename=None): - assert url is not None or filename is not None - - deck = createDeck(global_options, url=url, filename=filename) - - d = director.start(start_tor=True, - check_incoherences=check_incoherences) - - def setup_nettest(_): - try: - return deck.setup() - except errors.UnableToLoadDeckInput as error: - return defer.failure.Failure(error) - - - - # Wait until director has started up (including bootstrapping Tor) - # before adding tests - def post_director_start(_): - for net_test_loader in deck.netTestLoaders: - # Decks can specify different collectors - # for each net test, so that each NetTest - # may be paired with a test_helper and its collector - # However, a user can override this behavior by - # specifying a collector from the command-line (-c). - # If a collector is not specified in the deck, or the - # deck is a singleton, the default collector set in - # ooniprobe.conf will be used - - collector = setupCollector(global_options, net_test_loader) - - test_details = net_test_loader.testDetails - test_details['annotations'] = global_options['annotations'] - - director.startNetTest(net_test_loader, - global_options['reportfile'], - collector) - return director.allTestsDone - - d.addCallback(setup_nettest) - d.addCallback(post_director_start) - d.addErrback(director_startup_handled_failures) - d.addErrback(director_startup_other_failures) - return d - finished = defer.Deferred() @defer.inlineCallbacks - def readmsg(_,channel, queue_object, consumer_tag, counter): + def readmsg(_, channel, queue_object, consumer_tag, counter): # Wait for a message and decode it. if counter >= lifetime: + log.msg("Counter") queue_object.close(LifetimeExceeded()) yield channel.basic_cancel(consumer_tag=consumer_tag) finished.callback(None) else: log.msg("Waiting for message") - + try: ch, method, properties, body = yield queue_object.get() log.msg("Got message") data = json.loads(body) counter += 1 - log.msg("Received %d/%d: %s" %(counter, lifetime, data['url'],)) + log.msg("Received %d/%d: %s" % (counter, lifetime, data['url'],)) # acknowledge the message ch.basic_ack(delivery_tag=method.delivery_tag) - d = run_test(global_options, url=data['url'].encode('utf8')) + d = runTestWithDirector(director=director, + global_options=global_options, + url=data['url'].encode('utf8'), + check_incoherences=check_incoherences) # When the test has been completed, go back to waiting for a message. d.addCallback(readmsg, channel, queue_object, consumer_tag, counter+1) except exceptions.AMQPError,v: + log.msg("Error") + log.exception(v) finished.errback(v) @@ -474,7 +439,7 @@ def runWithDaemonDirector(logging=True, start_tor=True, check_incoherences=True) queue_object, consumer_tag = yield channel.basic_consume( queue=name, no_ack=False) - readmsg(None,channel,queue_object,consumer_tag, 0) + readmsg(None, channel, queue_object, consumer_tag, 0) @@ -484,10 +449,9 @@ def runWithDaemonDirector(logging=True, start_tor=True, check_incoherences=True) urlargs = dict(urlparse.parse_qsl(urlp.query)) # random lifetime requests counter - lifetime = random.randint(820,1032) + lifetime = random.randint(820, 1032) # AMQP connection details are sent through the cmdline parameter '-Q' - creds = pika.PlainCredentials(urlp.username or 'guest', urlp.password or 'guest') parameters = pika.ConnectionParameters(urlp.hostname,

1 0

[ooni-probe/master] Consistency fix: lowercase ooni backend/probe words
by art＠torproject.org 29 Apr '16

29 Apr '16

commit d35a9c5b552a985d70bbad021ec9727e4e6378df Author: anadahz <kojgelo(a)inbox.com> Date: Thu Feb 4 10:45:18 2016 -0300 Consistency fix: lowercase ooni backend/probe words --- docs/source/writing_test_helpers.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/source/writing_test_helpers.rst b/docs/source/writing_test_helpers.rst index a8601cc..70d5e21 100644 --- a/docs/source/writing_test_helpers.rst +++ b/docs/source/writing_test_helpers.rst @@ -6,16 +6,16 @@ OONI test helpers are used by OONI nettests to perform their meausrements. They Writing a Censorship Directionality Test Helper -------------------------- -Our goal is to write an OONI test helper that helps an OONI-probe client determine the directionality of keyword censorship it has detected. To do this our helper will receive "encoded" data from an OONI-probe client, decode that data into a text string, and send the OONI-probe client a confirmation packet and a second packet containing the decoded string. +Our goal is to write an OONI test helper that helps an ooni-probe client determine the directionality of keyword censorship it has detected. To do this our helper will receive "encoded" data from an OONI-probe client, decode that data into a text string, and send the OONI-probe client a confirmation packet and a second packet containing the decoded string. -The OONI-backend code-base has many concise examples of test-helpers that you can build off to create your own. For this tutorial I used the `TCP echo test-helper <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_ as my guide. +The ooni-backend code-base has many concise examples of test-helpers that you can build off to create your own. For this tutorial I used the `TCP echo test-helper <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_ as my guide. Following this tutorial requires basic knowledge of event-driven programming (specifically 'Twisted'). You will be more than ready to build and implement a test-helper after reading though one or two `tutorials online. <http://krondo.com/?page_id=1327>`_ Creating the test helper -------------- -OONI-backend keeps all the test-helpers in the `oonib/testhelpers directory <https://github.com/TheTorProject/ooni-backend/tree/master/oonib/testhelpers>`_ Each individual test helper is a twisted service. Most of the current test-helpers consists of a twisted Factory and a twisted Protocol defined in the test helpers directory and a `stock Twisted Server <https://twistedmatrix.com/documents/current/api/twisted.application.interne…>`_ that is defined in the backend code. We will follow this model in the tutorial. +ooni-backend keeps all the test-helpers in the `oonib/testhelpers directory <https://github.com/TheTorProject/ooni-backend/tree/master/oonib/testhelpers>`_ Each individual test helper is a twisted service. Most of the current test-helpers consists of a twisted Factory and a twisted Protocol defined in the test helpers directory and a `stock Twisted Server <https://twistedmatrix.com/documents/current/api/twisted.application.interne…>`_ that is defined in the backend code. We will follow this model in the tutorial. Because of how simple this example test-helper is the job of our test-helper factory is merely to deploy a single instance of our protocol each time it's buildProtocol method is called. Because we have our factory inheret from the base `Factory object <https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.5.0/twisted…>`_ we merely have to define its ``protocol`` property to point to our protocol.:: @@ -26,7 +26,7 @@ Because of how simple this example test-helper is the job of our test-helper fac protocol = TCPDirectionalityTestProtocol -The protocol for this helper needs to do two things. First, upon receiving encoded data it needs to send the OONI-probe client back confirmation that the data has been received. Second, it needs to send the decoded data back to the OONI-probe client. Because we are extending the `Protocol object <https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.5.0/twisted…>`_ we can rewrite its ``dataReceived`` method which is called and passed data whenever it is received.:: +The protocol for this helper needs to do two things. First, upon receiving encoded data it needs to send the ooni-probe client back confirmation that the data has been received. Second, it needs to send the decoded data back to the OONI-probe client. Because we are extending the `Protocol object <https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.5.0/twisted…>`_ we can rewrite its ``dataReceived`` method which is called and passed data whenever it is received.:: class TCPDirectionalityTestProtocol(Protocol): @@ -68,7 +68,7 @@ With this added we have completed the base of simple test-helper and now just ha Adding the helper to the config file ------ -OONI-backend uses a config file located at `/etc/oonibackend.conf <https://github.com/TheTorProject/ooni-backend/blob/master/oonib.conf.example>`_. This file contains a `section where each test-helper can be configured. <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_. +ooni-backend uses a config file located at `/etc/oonibackend.conf <https://github.com/TheTorProject/ooni-backend/blob/master/oonib.conf.example>`_. This file contains a `section where each test-helper can be configured. <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_. The test-helper will need to be given a unique identifyer so that it can be called from the config file. In this example we use ``tcp-directionality`` as our identifyer. @@ -91,7 +91,7 @@ The OONI test-helper system is a collection of `Twisted services <https://twiste **NOTE:** In this example I have placed the original service in the existing tcp_helpers file. If you created your own file for your test-helper you will have to make sure that you import that file at the top of `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`_. -OONI uses a `Multi Service <https://twistedmatrix.com/documents/current/api/twisted.application.service…>`_ which allows them to combine all the OONI test-helpers and the report-collector into a singlular service for easier management. The next step for creating our test-helper is to add it to the OONI-backend `multiService <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_:: +OONI uses a `Multi Service <https://twistedmatrix.com/documents/current/api/twisted.application.service…>`_ which allows them to combine all the OONI test-helpers and the report-collector into a singlular service for easier management. The next step for creating our test-helper is to add it to the ooni-backend `multiService <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`_:: # Add the helper as a child of the backends multi-service test-helper multiService.addService(tcp_directionality_helper) @@ -128,7 +128,7 @@ This snippet contains the final code that would be inserted into `oonibackend.py Requiring the helper in a test ------------- -If you are creating tests that rely on custom test-hepers you will want to make sure that you do not get innacturate results because your test-helper being missing in the OONI-backend you are testing against. You can specify required test-helpers within a OONI-probe test by setting its ``requiredTestHelpers`` property. In this example we have made our test helper require the tcp-directionality test that we created above.:: +If you are creating tests that rely on custom test-hepers you will want to make sure that you do not get innacturate results because your test-helper being missing in the ooni-backend you are testing against. You can specify required test-helpers within a ooni-probe test by setting its ``requiredTestHelpers`` property. In this example we have made our test helper require the tcp-directionality test that we created above.:: class MyDirectionalityTest(nettest.NetTestCase): """ An example test."""

1 0

[ooni-probe/master] Remove accidental change to Vagrantfile
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 6f437a0937884a597386c81ab5ca9c80122d8201 Author: Daniel Ramsay <daniel(a)dretzq.org.uk> Date: Tue Nov 3 09:08:54 2015 +0000 Remove accidental change to Vagrantfile --- Vagrantfile | 2 -- 1 file changed, 2 deletions(-) diff --git a/Vagrantfile b/Vagrantfile index a06dc41..16a6ba0 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -12,8 +12,6 @@ Vagrant.configure("2") do |config| # Place the ooni-backend source code in ../ooni-backend to sync it with the vagrant instance config.vm.synced_folder "../ooni-backend", "/backend" - config.vm.network :private_network, ip: "192.168.38.20" - end $script = <<SCRIPT

1 0

[ooni-probe/master] Fix typos in Readme.md
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 305bef050c6be3e34acb48c39cb1fe6a3cae96c7 Author: Prayag Verma <prayag.verma(a)gmail.com> Date: Tue Feb 2 19:05:58 2016 +0530 Fix typos in Readme.md Found some spelling mistakes while reading Readme.md `depedencies` > `dependencies` `distributation` > `distribution` `Projec` > `Project` --- README.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.rst b/README.rst index a1b6386..59c7afa 100644 --- a/README.rst +++ b/README.rst @@ -127,7 +127,7 @@ You can install ooniprobe on OSX if you have installed homebrew (http://mxcl.git Unix systems (with pip) ----------------------- -Make sure you have installed the following depedencies: +Make sure you have installed the following dependencies: * build-essential * python (>=2.7) @@ -330,7 +330,7 @@ Download Obfsproxy: https://www.torproject.org/projects/obfsproxy.html.en Setting capabilities on your virtualenv python binary ===================================================== -If your distributation supports capabilities you can avoid needing to run OONI as root:: +If your distribution supports capabilities you can avoid needing to run OONI as root:: setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python @@ -339,7 +339,7 @@ If your distributation supports capabilities you can avoid needing to run OONI a Reporting bugs ============== -You can report bugs and issues you find with ooni-probe on The Tor Projec issue +You can report bugs and issues you find with ooni-probe on The Tor Project issue tracker filing them under the "Ooni" component: https://trac.torproject.org/projects/tor/newticket?component=Ooni. You can either register an account or use the group account "cypherpunks" with

1 0

[ooni-probe/master] deal with existing, but empty, reporting.yml
by art＠torproject.org 29 Apr '16

29 Apr '16

commit da614dfd225ddf493b6d039111539cce7acf95eb Author: Simone Basso <bassosimone(a)gmail.com> Date: Thu Nov 5 14:47:13 2015 +0100 deal with existing, but empty, reporting.yml Spotted while helping @anadahz to debug lepidopter. --- ooni/reporter.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ooni/reporter.py b/ooni/reporter.py index 5e702bb..a62cb6b 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -404,6 +404,8 @@ class OONIBReportLog(object): def get_report_log(self): with open(self.file_name) as f: report_log = yaml.safe_load(f) + if not report_log: + report_log = {} # consumers expect dictionary structure return report_log @property

1 0

[ooni-probe/master] Make bin/ooniprobe-dev portable
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 5cbf718350a7aacab0d2a273424f3ccd33c1f9e3 Author: Simone Basso <bassosimone(a)gmail.com> Date: Fri Oct 9 09:38:42 2015 +0200 Make bin/ooniprobe-dev portable - use a portable recipe to get the absolute path of the toplevel directory where ooni-probe sources are located [1] - make sure we exit with a meaningful error message if it is not possible to get the toplevel directory name - while there put quotes around $@ since according to [2] this is the best way to forward shell script arguments .. [1] http://serverfault.com/questions/40144 .. [2] http://stackoverflow.com/questions/255898/how-to-iterate-over-arguments-in-… --- bin/ooniprobe-dev | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/bin/ooniprobe-dev b/bin/ooniprobe-dev index b85d3a3..7eaaf40 100755 --- a/bin/ooniprobe-dev +++ b/bin/ooniprobe-dev @@ -2,5 +2,9 @@ # Developer script for running ooniprobe directly from the repository. # We don't automatically add "$PWD" to PYTHONPATH as that is a security risk # when run as /usr/bin/ooniprobe on an end user's computer. -cd "$(realpath "$(dirname "$0")")/.." -PYTHONPATH="$PWD" exec ./bin/ooniprobe $@ +ROOTDIR=$(cd $(dirname $(dirname $0)) && pwd -P) +if [ $? -ne 0 ]; then + echo "$0: cannot determine toplevel directory" 1>&2 + exit 1 +fi +PYTHONPATH="$ROOTDIR" exec ./bin/ooniprobe "$@"

1 0

[ooni-probe/master] Add Bridge Reachability Test documentation
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 9617f31aa5dede2b49460cf4ae9e70139c7f61db Author: anadahz <kojgelo(a)inbox.com> Date: Tue Oct 13 06:03:57 2015 +0200 Add Bridge Reachability Test documentation --- docs/source/tests/bridge_reachability.rst | 129 ++++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) diff --git a/docs/source/tests/bridge_reachability.rst b/docs/source/tests/bridge_reachability.rst new file mode 100644 index 0000000..3f8999b --- /dev/null +++ b/docs/source/tests/bridge_reachability.rst @@ -0,0 +1,129 @@ +Details +======= + +*Test Name*: Bridge Reachability Test + +*Current version*: 0.1.2 + +*NetTest*: HTTP Requests (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/nettests/block…) + +*Test Helper*: None + +*Test Type*: Content Blocking + +*Requires Root*: No + +Description +=========== + +This test detects which Tor bridges are working from the given network vantage +point and which ones are not. For every bridge specified as input a new tor +instance is started and it is instructed to connect to Tor using that bridge. +It will try to bootstrap to 100% and either succeed or timeout after a default +time of 120 seconds. + +How to run the test +=================== + +`ooniprobe blocking/bridge_reachability.py -f <input_file>` + +*input file* a list of TransportType IP:ORPort bridges (ex. obfs3 +198.51.100.45:9045) or IP:ORPort (ex. 203.0.113.27:1828) to test reachability +for. + +Optional test options +===================== + +*-t* Specify the timeout after which to consider the Tor bootstrapping process +to have failed. The default is 120 seconds. + +Sample report +============= + +`ooniprobe blocking/bridge_reachability.py` + +:: + +########################################### +# OONI Probe Report for bridge_reachability (0.1) +# Thu Aug 7 16:10:50 2014 +########################################### +--- +input_hashes: [595c8ad9b63ff9f142eca6296021990df6367ab03b3c4eb1c69a8747f0cd41a1] +options: [-f, bridges.txt] +probe_asn: AS3269 +probe_cc: IT +probe_city: null +probe_ip: 127.0.0.1 +software_name: ooniprobe +software_version: 1.0.2 +start_time: 1407420650.560137 +test_name: bridge_reachability +test_version: '0.1' +... +--- +{bridge_address: '169.229.59.74:31493', error: null, input: 'obfs3 169.229.59.74:31493 + AF9F66B7B04F8FF6F32D455F05135250A16543C9', success: true, timeout: 120, tor_log: null, + tor_progress: 100, tor_progress_summary: Done, tor_progress_tag: done, tor_version: 0.2.4.20, + transport_name: obfs3} +... +--- +{bridge_address: '169.229.59.75:46328', error: null, input: 'obfs3 169.229.59.75:46328 + AF9F66B7B04F8FF6F32D455F05135250A16543C9', success: true, timeout: 120, tor_log: null, + tor_progress: 100, tor_progress_summary: Done, tor_progress_tag: done, tor_version: 0.2.4.20, + transport_name: obfs3} +... +--- +{bridge_address: '208.79.90.242:35658', error: null, input: 'obfs3 208.79.90.242:35658 + BA61757846841D64A83EA2514C766CB92F1FB41F', success: true, timeout: 120, tor_log: null, + tor_progress: 100, tor_progress_summary: Done, tor_progress_tag: done, tor_version: 0.2.4.20, + transport_name: obfs3} +... +--- +{bridge_address: '83.212.100.216:47870', error: null, input: 'obfs2 83.212.100.216:47870 + 1F01A7BB60F49FC96E0850A6BAD6D076DFEFAF80', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs2} +... +--- +{bridge_address: '83.212.96.182:46602', error: null, input: 'obfs2 83.212.96.182:46602 + 6F058CBEF888EB20D1DEB9886909F1E812245D41', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs2} +... +--- +{bridge_address: '70.182.182.109:54542', error: null, input: 'obfs2 70.182.182.109:54542 + 94C9E691688FAFDEC701A0788BD15BE8AD34ED35', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs2} +... +--- +{bridge_address: '128.31.0.34:1051', error: null, input: 'obfs2 128.31.0.34:1051 CA7434F14A898C7D3427B8295A7F83446BC7F496', + success: false, timeout: 120, tor_log: '', tor_progress: 0, tor_progress_summary: null, + tor_progress_tag: null, tor_version: 0.2.4.20, transport_name: obfs2} +... +--- +{bridge_address: '83.212.101.2:45235', error: null, input: 'obfs2 83.212.101.2:45235 + 2ADFE7AA8D272C520D1FBFBF4E413F3A1B26313D', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs2} +... +--- +{bridge_address: '83.212.101.2:42782', error: null, input: 'obfs3 83.212.101.2:42782 + 2ADFE7AA8D272C520D1FBFBF4E413F3A1B26313D', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs3} +... +--- +{bridge_address: '83.212.101.2:443', error: null, input: 'obfs3 83.212.101.2:443 2ADFE7AA8D272C520D1FBFBF4E413F3A1B26313D', + success: false, timeout: 120, tor_log: '', tor_progress: 0, tor_progress_summary: null, + tor_progress_tag: null, tor_version: 0.2.4.20, transport_name: obfs3} +... +--- +{bridge_address: '209.141.36.236:45496', error: null, input: 'obfs3 209.141.36.236:45496 + 58D91C3A631F910F32E18A55441D5A0463BA66E2', success: false, timeout: 120, tor_log: '', + tor_progress: 0, tor_progress_summary: null, tor_progress_tag: null, tor_version: 0.2.4.20, + transport_name: obfs3} +... + ... +

1 0