April 2016 - tor-commits - lists.torproject.org

[ooni-probe/master] change again command report structure for ProcessTest
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 6e1cc4bb698d361154bf20bc82e44a86754b7c93 Author: juga0 <juga> Date: Thu Dec 24 13:28:38 2015 +0000 change again command report structure for ProcessTest * Change report structure as commented in https://github.com/TheTorProject/ooni-probe/pull/445#issuecomment-166981539 --- ooni/templates/process.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/ooni/templates/process.py b/ooni/templates/process.py index b7dafdd..82457dc 100644 --- a/ooni/templates/process.py +++ b/ooni/templates/process.py @@ -103,10 +103,14 @@ class ProcessTest(NetTestCase): def processEnded(self, result, command): log.debug("Finished %s: %s" % (command, result)) - self.report['command_name'] = ' '.join(command) - self.report['command_stdout'] = result['stdout'] - self.report['command_stderr'] = result['stderr'] - self.report['command_exit_reason'] = result['exit_reason'] + if not isinstance(self.report.get('commands'), list): + self.report['commands'] = [] + self.report['commands'].append({ + 'command_name': ' '.join(command), + 'command_stdout': result['stdout'], + 'command_stderr': result['stderr'], + 'command_exit_reason': result['exit_reason'], + }) return result def run(self, command, finished=None, env={}, path=None, usePTY=0):

1 0

[ooni-probe/master] fixed identation
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 848b4dd9b03b3648686948529040af6fc6ba604d Author: juga0 <juga> Date: Tue Dec 22 21:22:55 2015 +0000 fixed identation --- ooni/nettests/third_party/psiphon.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ooni/nettests/third_party/psiphon.py b/ooni/nettests/third_party/psiphon.py index 94dfa51..17c83c6 100644 --- a/ooni/nettests/third_party/psiphon.py +++ b/ooni/nettests/third_party/psiphon.py @@ -88,7 +88,7 @@ connect(False) log.debug('PsiphonTest.test_psiphon') self.report['success'] = None - self.report['psiphon_installed'] = None + self.report['psiphon_installed'] = None if not os.path.exists(self.psiphonpath): log.err('psiphon path does not exists, is it installed?') self.report['success'] = False

1 0

[ooni-probe/master] add variables for paths, change variable names
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 5a14541113a59787ed83bc4808c4a1bf83ca0c1e Author: juga0 <juga> Date: Fri Dec 25 19:54:31 2015 +0000 add variables for paths, change variable names --- scripts/psiphon_install.sh | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/scripts/psiphon_install.sh b/scripts/psiphon_install.sh index 12086c8..7624c12 100755 --- a/scripts/psiphon_install.sh +++ b/scripts/psiphon_install.sh @@ -2,12 +2,14 @@ set -e -PSIPHON_PATH=$HOME -PSIPHON_PYCLIENT_PATH=$PSIPHON_PATH/psiphon-circumvention-system/pyclient -PSIPHON_REPO_URL=https://bitbucket.org/psiphon/psiphon-circumvention-system -OONI_VIRTUALENV_PATH=$HOME/.virtualenvs/ooniprobe +PSIPHON_HOME_PATH=$HOME +PSIPHON_PYCLIENT_PATH=$PSIPHON_HOME_PATH/psiphon-circumvention-system/pyclient +PSIPHON_SSH_PATH=$PSIPHON_HOME_PATH/psiphon-circumvention-system/Server/3rdParty/openssh-5.9p1 +PSIPHON_REPO_URL=https://bitbucket.org/psiphon/psiphon-circumvention-system#af438ec2c16c +VIRTUALENVS_PATH=$HOME/.virtualenvs +OONI_VIRTUALENV_PATH=$VIRTUALENVS_PATH/ooniprobe -mkdir -p $PSIPHON_PATH +mkdir -p $PSIPHON_HOME_PATH command_exists() { command -v "$@" > /dev/null 2>&1 @@ -40,7 +42,7 @@ if ! command_exists hg; then fi echo "[D] mercurial installed" -cd $PSIPHON_PATH +cd $PSIPHON_HOME_PATH if [ ! -d "psiphon-circumvention-system" ]; then echo "[D] cloning psiphon repository" hg clone $PSIPHON_REPO_URL @@ -51,10 +53,10 @@ echo "[D] psiphon repository cloned" # optional, compile their ssh if [ ! -f "$PSIPHON_PYCLIENT_PATH/ssh" ]; then echo "[D] compiling psiphon ssh" - cd psiphon-circumvention-system/Server/3rdParty/openssh-5.9p1/ + cd $PSIPHON_SSH_PATH ./configure make - mv ssh ../../../pyclient/pyclient/ + mv ssh $PSIPHON_PYCLIENT_PATH make clean echo "[D] psiphon ssh compiled" fi

1 0

[ooni-probe/master] initialize success report variable
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 9aa1a96f7fa54860cbd1c80387b042ea5c39502d Author: juga0 <juga> Date: Tue Dec 22 21:08:32 2015 +0000 initialize success report variable --- ooni/nettests/third_party/psiphon.py | 1 + 1 file changed, 1 insertion(+) diff --git a/ooni/nettests/third_party/psiphon.py b/ooni/nettests/third_party/psiphon.py index fb912a9..1e85c78 100644 --- a/ooni/nettests/third_party/psiphon.py +++ b/ooni/nettests/third_party/psiphon.py @@ -88,6 +88,7 @@ connect(False) log.debug('PsiphonTest.test_psiphon') self.report['success'] = None + self.report['psiphon_installed'] = None if not os.path.exists(self.psiphonpath): log.err('psiphon path does not exists, is it installed?') self.report['psiphon_installed'] = False

1 0

[ooni-probe/master] Handle and log errors when OpenVPN fails to start
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 46be7badf8d509b6315e282e69245f5b7fa11910 Author: srvetus <srvetus(a)users.noreply.github.com> Date: Thu Nov 19 12:23:38 2015 +0100 Handle and log errors when OpenVPN fails to start When OpenVPN encouters a configuration error it will exit the process with error code 1 before beginning to bootstrap. This commit monkeypatches `processDirector.processExited` to log these pre-bootstrap OpenVPN configuration failures. This commit also replaces `handleAllFailures()` with `failureToString()` for generating the report content. `handleAllFailures()` causes difficult to debug errors. It silently raises an exception from `failure.trap()` if an unknown exception type is passed to it. Any raised Exception will causes the remaining error logging to be skipped making debugging difficult. Ooni should produce some sort of log message if an Exception is raised in `handleAllFailures()`. --- ooni/nettests/third_party/openvpn.py | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/ooni/nettests/third_party/openvpn.py b/ooni/nettests/third_party/openvpn.py index b681222..028bf17 100644 --- a/ooni/nettests/third_party/openvpn.py +++ b/ooni/nettests/third_party/openvpn.py @@ -3,7 +3,7 @@ from twisted.python import usage from twisted.web.client import Agent, readBody from ooni.templates.process import ProcessTest from ooni.utils import log -from ooni.errors import handleAllFailures +from ooni.errors import handleAllFailures, failureToString import distutils.spawn import re @@ -32,7 +32,7 @@ class OpenVPNTest(ProcessTest): name = "OpenVPN Client Test" description = "Connects to an OpenVPN server and does a HTTP GET for the specified URL" author = "srvetus " - version = "0.0.1" + version = "0.0.2" timeout = 20 usageOptions = UsageOptions requiredOptions = ['url', 'openvpn-config'] @@ -57,6 +57,16 @@ class OpenVPNTest(ProcessTest): self.processDirector.transport.signalProcess('TERM') self.exited = True + def processExited(self, reason): + """Monkeypatch processExited to log failure if the process ends + unexpectedly before OpenVPN bootstraps. + """ + log.debug("Exited %s" % handleAllFailures(reason)) + + # Process exited before OpenVPN bootstrapped. Add failure to report + if not self.bootstrapped.called: + self.bootstrapped.errback(Exception("openvpn_exited_unexpectedly")) + def handleRead(self, stdout=None, stderr=None): """handleRead is called with each chunk of data from stdout and stderr @@ -88,8 +98,8 @@ class OpenVPNTest(ProcessTest): self.report['success'] = True def addFailureToReport(failure): - log.debug("failed: %s" % handleAllFailures(failure)) - self.report['failure'] = handleAllFailures(failure) + log.debug("Failed: %s" % failureToString(failure)) + self.report['failure'] = failureToString(failure) self.report['success'] = False def doRequest(noreason): @@ -106,6 +116,9 @@ class OpenVPNTest(ProcessTest): log.debug("Spawning OpenVPN") self.d = self.run(self.command) + # Monkeypatch processExited to log when OpenVPN exits early + self.processDirector.processExited = self.processExited + # Try to make a request when the OpenVPN connection successfully bootstraps self.bootstrapped.addCallback(doRequest)

1 0

[ooni-probe/master] OpenVPN connection circumvention test
by art＠torproject.org 29 Apr '16

29 Apr '16

commit ed92669d1677ed0d180a358ef3f8d9603902154f Author: srvetus <srvetus(a)users.noreply.github.com> Date: Thu Sep 24 01:43:14 2015 +0200 OpenVPN connection circumvention test This test attempts to connect to an OpenVPN server specified in an OpenVPN config file. A HTTP request is sent if the VPN connection is successful. This basic test can be expanded to parse more of the output from the OpenVPN client. The output could provide more information about why a connection might be failing. --- ooni/nettests/third_party/openvpn.py | 117 +++++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) diff --git a/ooni/nettests/third_party/openvpn.py b/ooni/nettests/third_party/openvpn.py new file mode 100644 index 0000000..b681222 --- /dev/null +++ b/ooni/nettests/third_party/openvpn.py @@ -0,0 +1,117 @@ +from twisted.internet import defer, reactor +from twisted.python import usage +from twisted.web.client import Agent, readBody +from ooni.templates.process import ProcessTest +from ooni.utils import log +from ooni.errors import handleAllFailures + +import distutils.spawn +import re + + +class UsageOptions(usage.Options): + optParameters = [ + ['url', 'u', None, 'Specify a single URL on the OpenVPN subnet to test.'], + ['openvpn-config', 'c', None, 'Specify an OpenVPN configuration file.'], + ] + + +class OpenVPNTest(ProcessTest): + + """ + This class tests OpenVPN connections. + + test_openvpn_circumvent + Starts an OpenVPN client on Linux and determines + if it connects successfully to an OpenVPN server. + Then, it make a HTTP request for http://google.com + and records the response body or failure string. + + """ + + name = "OpenVPN Client Test" + description = "Connects to an OpenVPN server and does a HTTP GET for the specified URL" + author = "srvetus " + version = "0.0.1" + timeout = 20 + usageOptions = UsageOptions + requiredOptions = ['url', 'openvpn-config'] + requiresRoot = True + + def setUp(self): + self.bootstrapped = defer.Deferred() + self.command = [distutils.spawn.find_executable("openvpn")] + self.exited = False + self.url = self.localOptions.get('url') + + if self.localOptions.get('openvpn-config'): + openvpn_config = self.localOptions.get('openvpn-config') + self.command.extend(['--config', openvpn_config]) + + def stop(self, reason=None): + """Stop the running OpenVPN process and close the connection""" + if not self.exited: + self.processDirector.close() + + # OpenVPN needs to be sent SIGTERM to end cleanly + self.processDirector.transport.signalProcess('TERM') + self.exited = True + + def handleRead(self, stdout=None, stderr=None): + """handleRead is called with each chunk of data from stdout and stderr + + stdout only contains the latest data chunk, self.processDirector.stdout + contains the combined stdout data. + """ + + # Read OpenVPN output until bootstrapping succeeds or fails + if not self.bootstrapped.called: + + # TODO: Determine other OpenVPN messages which indicate connection failure + if re.search(r'connect to .* failed', self.processDirector.stdout): + log.debug("OpenVPN connection failed") + + # Bootstrapping failed + self.bootstrapped.errback(Exception("openvpn_connection_failed")) + + # Check if OpenVPN has bootstrapped and connected successfully + elif "Initialization Sequence Completed" in self.processDirector.stdout: + log.debug("OpenVPN connection successful") + self.processDirector.cancelTimer() + self.bootstrapped.callback("bootstrapped") + + def test_openvpn_circumvent(self): + + def addResultToReport(result): + log.debug("request_successful") + self.report['body'] = result + self.report['success'] = True + + def addFailureToReport(failure): + log.debug("failed: %s" % handleAllFailures(failure)) + self.report['failure'] = handleAllFailures(failure) + self.report['success'] = False + + def doRequest(noreason): + """Make a HTTP request over initialized VPN connection""" + agent = Agent(reactor) + + log.debug("Doing HTTP request to the OpenVPN subnet: %s" % self.url) + request = agent.request("GET", self.url) + request.addCallback(readBody) + request.addCallback(addResultToReport) + request.addErrback(addFailureToReport) + return request + + log.debug("Spawning OpenVPN") + self.d = self.run(self.command) + + # Try to make a request when the OpenVPN connection successfully bootstraps + self.bootstrapped.addCallback(doRequest) + + # Fire failure if OpenVPN connection fails + self.bootstrapped.addErrback(addFailureToReport) + + # Close OpenVPN after each successful or failed test + self.bootstrapped.addBoth(self.stop) + return self.d

1 0

[ooni-probe/master] Edited various bits of the documentation page & moved "using ooniprobe" towards the beginning
by art＠torproject.org 29 Apr '16

29 Apr '16

commit e42401329816f87457d65b8159189f3d2b4ede16 Author: Maria Xynou <maria.x(a)espiv.net> Date: Sat Jan 30 13:09:53 2016 +0100 Edited various bits of the documentation page & moved "using ooniprobe" towards the beginning --- docs/source/index.rst | 96 +++++++++++++++++++++------------------------------ 1 file changed, 40 insertions(+), 56 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 1eabdee..13f7a8f 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,29 +3,19 @@ OONI: Open Observatory of Network Interference ============================================== -OONI, the Open Observatory of Network Interference, is a global observation -network which aims is to collect high quality data using open methodologies, -using Free and Open Source Software (FL/OSS) to share observations and data -about the various types, methods, and amounts of network tampering in the -world. +The Open Observatory of Network Interference (OONI) is a global observation +network which collects high quality data, using open methodologies and Free and Open Source Software (FL/OSS), to share observations about the various types, methods, and amounts of network tampering in the world. "The Net interprets censorship as damage and routes around it." - John Gilmore; TIME magazine (6 December 1993) +Are you interested in testing your network for signs of surveillance and censorship? Do you want to collect data so that you and others can better understand your network? If so, please read this document and we hope that ooniprobe will help you gather network data that will assist you with your endeavors! + ooniprobe --------- -ooniprobe is the first program that users run to probe their network and to -collect data for the OONI project. Are you interested in testing your network -for signs of surveillance and censorship? Do you want to collect data to share -with others, so that you and others may better understand your network? If so, -please read this document and we hope ooniprobe will help you to gather -network data that will assist you with your endeavors! - -ooniprobe allows the user to select what test should be run and what backend -should be used for storing the test report and/or assisting them in the running -of the test. +ooniprobe is a program that users can run to probe their network and to collect network measurement data. ooniprobe also allows users to choose what test should be run and what backend should be used for storing the test report and for assisting them in running the test. ooniprobe tests are divided into two categories: **Traffic Manipulation** and **Content Blocking**. @@ -35,23 +25,28 @@ tampering with the internet traffic between the probe and a remote test helper backend. As such they usually require the selection of a oonib backend component for running the test. -**Content Blocking** are aimed at enumerating the kind of content that is -blocked from the probes network point of view. As such they usually require to -have specified an input list for running the test. +**Content Blocking** tests aim to enumerate the kind of content that is +blocked from the probes network point of view. As such they usually require the specification of an input list for running the test. Read this before running ooniprobe! ----------------------------------- -Running ooniprobe is a potentially risky activity. This greatly depends on the -jurisdiction in which you are in and which test you are running. It is -technically possible for a person observing your internet connection to be -aware of the fact that you are running ooniprobe. This means that if running -network measurement tests is something considered to be illegal in your country -then you could be spotted. +Running ooniprobe can potentially be a risky activity. This largely depends on the laws and jurisdiction of the country you are running ooniprobe from, as well as on the type of test you are running. Technically, it is possible for a person observing your internet connection to be aware of the fact that you are running ooniprobe. This means that if running network measurement tests is something that is considered to be illegal in your country then you could be spotted. + +Furthermore, ooniprobe takes no precautions to protect the install target machine from forensics analysis. If the fact that you have installed or used ooniprobe is a liability for you, please be aware of this risk. + +Using ooniprobe +=============== + +**Net test** is a set of measurements to assess what kind of internet censorship is occurring. + +**Decks** are collections of ooniprobe nettests with some associated inputs. + +**Collector** is a service used to report the results of measurements. + +**Test helper** is a service used by a probe for successfully performing its measurements. -Furthermore, ooniprobe takes no precautions to protect the install target machine -from forensics analysis. If the fact that you have installed or used ooni -probe is a liability for you, please be aware of this risk. +**Bouncer** is a service used to discover the addresses of test helpers and collectors. OONI in 5 minutes ================= @@ -91,8 +86,7 @@ This should generate the home and allow you to run oonideckgen. The output from the last command will tell you how to run ooniprobe to perform the measurement. -If you would like to contribute measurements to OONI daily you can also add -this to your crontab:: +If you would like to contribute measurements to OONI daily you can also add this to your crontab:: @daily ooniprobe $THE_OONI_COMMAND @@ -185,19 +179,6 @@ ooniprobe will be installed in ``/ooni``. ooniprobe blocking/http_requests -f /ooni/example_inputs/alexa-top-1k.txt -Using ooniprobe -=============== - -**Net test** is a set of measurements to assess what kind of internet censorship is occurring. - -**Decks** are collections of ooniprobe nettests with some associated inputs. - -**Collector** is a service used to report the results of measurements. - -**Test helper** is a service used by a probe for successfully performing its measurements. - -**Bouncer** is a service used to discover the addresses of test helpers and collectors. - Configuring ooniprobe --------------------- @@ -217,10 +198,10 @@ This can be done with:: ooniresources --update-inputs -If you get a permission error, you may have to run the command as root or +If you get a permission error, you might have to run the command as root or change the ooniprobe data directory inside of `ooniprobe.conf`. -On some platforms, for example debian contrib, you will not get all the geoip +On some platforms, such as debian contrib, you will not get all the geoip related files needed. In that case it is possible to manually download them with ``ooniresources``:: @@ -229,7 +210,7 @@ with ``ooniresources``:: Generating decks ---------------- -You can generate decks for your country thanks to the oonideckgen command. +You can generate decks for your country with the oonideckgen command. If you wish, for example, to generate a deck to be run in the country of Italy, you can do so (be sure to have updated the input resources first) by running:: @@ -238,7 +219,8 @@ you can do so (be sure to have updated the input resources first) by running:: You will now have in your home a folder called `deck-it`, containing the ooni deck (ends with .deck) and the inputs. -Note: that you should not move the `deck-*` directory once it has been + +**Note:** You should not move the `deck-*` directory once it has been generated as the paths to the inputs referenced by the test in the deck are absolute. If you want your deck to live in another directory you must regenerated it. @@ -327,6 +309,7 @@ obfsproxy is installed). The tor.bridges option in ooniprobe.conf sets the path to a file that should contain a set of "bridge" lines (of the same format as used in torrc, and as returned by https://bridges.torproject.org) If obfsproxy bridges are to be used, the path to the obfsproxy binary must be configured. + See option advanced.obfsproxy_binary, in ooniprobe.conf. (Optional) Install obfsproxy @@ -371,7 +354,7 @@ https://github.com/TheTorProject/ooni-probe Read this article to learn how to create a pull request on github (https://help.github.com/articles/creating-a-pull-request) -If you prefer not to use github (or don't have an account), you may also submit +If you prefer not to use github (or don't have an account), you can also submit patches as attachments to tickets. Be sure to format the patch (given that you are working on a feature branch @@ -441,7 +424,7 @@ Tests that don't do a measurement but are useful for scanning can be found in: * `ooni/nettests/scanning <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/ooni/nettests/scann…>`_ -Tests that involve running third party tools may be found in: +Tests that involve running third party tools can be found in: * `ooni/nettests/third_party <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/ooni/nettests/third…>`_ @@ -449,7 +432,7 @@ Tests that involve running third party tools may be found in: ooni-backend ------------ -This is the server side component of ooniprobe. It will store that data +This is the server side component of ooniprobe. It will store data collected from ooniprobes and it will run a series of Test Helpers that assist `Traffic Manipulation Tests`_ in performing their measurements. @@ -476,7 +459,7 @@ Threat Model Our adversary is capable of doing country wide network surveillance and manipulation of network traffic. -The goals of our adversary are: +The goals of our adversary are to: * Restrict access to certain content, while not degrading overall quality of the network @@ -489,14 +472,15 @@ More specifc to the running of network filtering detection tests: 2. Fool people running such tests into believing that the network is unrestricted -*Note* that while 2) => 1) it is not true that 1) => 2) as the identification of +**Note:** While 2) => 1) it is not true that 1) => 2) as the identification of such actors does not necessarily have to happen in real time. -While our intention is to minimize the risk of users running OONI probe to be -identified, this comes with a tradeoff in accuracy. It is therefore necessary in -certain tests to trade-off fingerprintability in favour of tests accuracy. -This is why we divide tests based on what risk the user running it can face, -allowing the user to freely choose what threat model they wish to adere to. +While our intention is to minimize the risk of users running OONI probe being +identified, this comes with a trade-off in accuracy. It is therefore necessary in +certain tests to trade-off fingerprintability in favour of test accuracy. + +This is why we divide tests based on what risk the users running them can face, +allowing users to freely choose what threat model they wish to adere to. More developer documentation

1 0

[ooni-probe/master] Added a simple tutorial on how to write test helpers.
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 2b1bfac9b7a9d52af9f36f25d19430b8935fac4c Author: seamus tuohy <s2e(a)seamustuohy.com> Date: Mon Jan 18 14:24:59 2016 -0500 Added a simple tutorial on how to write test helpers. --- docs/source/writing_test_helpers.rst | 137 +++++++++++++++++++++++++++++++++++ 1 file changed, 137 insertions(+) diff --git a/docs/source/writing_test_helpers.rst b/docs/source/writing_test_helpers.rst new file mode 100644 index 0000000..2206115 --- /dev/null +++ b/docs/source/writing_test_helpers.rst @@ -0,0 +1,137 @@ +Writing Test Helpers +======== + +OONI test helpers are... + +Writing a Censorship Directionality Test Helper +-------------------------- + +Our goal is to write an OONI test helper that helps an OONI-probe client determine the directionality of keyword censorship it has detected. To do this our helper will receive "encoded" data from an OONI-probe client, decode that data into a text string, and send the OONI-probe client a confirmation packet and a second packet containing the decoded string. + +The OONI-backend code-base has many concise examples of test-helpers that you can build off to create your own. For this tutorial I used the `TCP echo test-helper <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>` as my guide. + +Following this tutorial requires basic knowledge of event-driven programming (specifically 'Twisted'). You will be more than ready to build and implement a test-helper after reading though one or two `tutorials online. <http://krondo.com/?page_id=1327>` + +Creating the test helper +-------------- + +OONI-backend keeps all the test-helpers in the `oonib/testhelpers directory <https://github.com/TheTorProject/ooni-backend/tree/master/oonib/testhelpers>` Each individual test helper is a twisted service. Most of the current test-helpers consists of a twisted Factory and a twisted Protocol defined in the test helpers directory and a `stock Twisted Server <https://twistedmatrix.com/documents/current/api/twisted.application.interne…>` that is defined in the backend code. We will follow this model in the tutorial. + +Because of how simple this example test-helper is the job of our test-helper factory is merely to deploy a single instance of our protocol each time it's buildProtocol method is called. Because we have our factory inheret from the base `Factory object <https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.5.0/twisted…>` we merely have to define its ``protocol`` property to point to our protocol.:: + + class TCPDirectionalityTestHelper(Factory): + """ + A test helper for checking for directionality of censorship + """ + protocol = TCPDirectionalityTestProtocol + + +The protocol for this helper needs to do two things. First, upon receiving encoded data it needs to send the OONI-probe client back confirmation that the data has been received. Second, it needs to send the decoded data back to the OONI-probe client. Because we are extending the `Protocol object <https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.5.0/twisted…>` we can rewrite its ``dataReceived`` method which is called and passed data whenever it is received.:: + + + class TCPDirectionalityTestProtocol(Protocol): + """Takes encoded packet data, decodes it, and then sends it back. + + This protocol sends two packets in response to an encoded packet. + It first sends a confirmation packet, and then follows with a packet + containing the decoded data requested from the test. + """ + def dataReceived(self, data): + # send back receipt of the packet + self.transport.write(data) + # send back the decoded term to test against. + original_string = data.decode("rot13").decode("unicode-escape").encode("utf-8") + self.transport.write(original_string) + + +In order to make this test-helper slightly more flexible we will be allowing the backend to determine the encoding within their config file. To this end we will have to retreive the encoding from the config file.:: + + + def dataReceived(self, data): + # send back receipt of the packet + self.transport.write(data) + + # Get the encoding from the config or fallback to rot13 + if config.helpers['tcp-directionality'].encoding: + tcp_dir_encoding = config.helpers['tcp-directionality'].encoding + else: + tcp_dir_encoding = "rot13" + + # send back the decoded term to test against. + original_string = data.decode("rot13").decode("unicode-escape").encode("utf-8") + self.transport.write(original_string) + + +With this added we have completed the base of simple test-helper and now just have to integrate it into the rest of the backend. + + +Adding the helper to the config file +------ + +OONI-backend uses a config file located at `/etc/oonibackend.conf <https://github.com/TheTorProject/ooni-backend/blob/master/oonib.conf.example>`. This file contains a `section where each test-helper can be configured. <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`. + +The test-helper will need to be given a unique identifyer so that it can be called from the config file. In this example we use ``tcp-directionality`` as our identifyer. + +For a helper to be used in the ooni-backend it needs to be given an identifyer so that it can be called from the config file.:: + + tcp-echo: + encoding: rot13 + port: 57009 + +Adding the helper to the backend +------ + +For a helper to be integrated into the ooni-backend it needs to be added to the initialization scripts contained within `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`. + +The OONI test-helper system is a collection of `Twisted services <https://twistedmatrix.com/documents/current/core/howto/application.html>`. For our test-helper we will need to define a service that will run our test-helper factory.:: + + # Create the service that will run our test-helpers factory. + tcp_directionality_helper = internet.TCPServer(int(port), + tcp_helpers.TCPDirectionalityTestHelper()) + +**NOTE:** In this example I have placed the original service in the existing tcp_helpers file. If you created your own file for your test-helper you will have to make sure that you import that file at the top of `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`. + +OONI uses a `Multi Service <https://twistedmatrix.com/documents/current/api/twisted.application.service…>` which allows them to combine all the OONI test-helpers and the report-collector into a singlular service for easier management. The next step for creating our test-helper is to add it to the OONI-backend `multiService <https://github.com/TheTorProject/ooni-backend/blob/479a1bb154037b834292ccc4…>`:: + + # Add the helper as a child of the backends multi-service test-helper + multiService.addService(tcp_directionality_helper) + +Finally, we need to start our service.:: + + # Start the test-helpers service + tcp_directionality_helper.startService() + +In order for our test-helper to be managed using the backend config file we will need to modify this code to check the config file for a test-helper that uses the identifyer we selected earlier. For the directionality helper we check to see if our test-helper had its port specified in the config file to determine if it should be run. I also added a default encoding in case + +This snippet contains the final code that would be inserted into `oonibackend.py <https://github.com/TheTorProject/ooni-backend/blob/master/oonib/oonibackend…>`.:: + + # Check to see if our test-helper was defined in the config + if config.helpers['tcp-directionality'].port: + print "Starting TCP directionality helper on %s" % config.helpers['tcp-directionality'].port + + # Check for encoding in our config file and set default if missing + if config.helpers['tcp-directionality'].encoding: + tcp_dir_encoding = config.helpers['tcp-directionality'].encoding + else: + tcp_dir_encoding = "rot13" + # Get & set the port and encoding from our config file + tcp_directionality_helper = internet.TCPServer(int(config.helpers['tcp-directionality'].port), + tcp_helpers.TCPDirectionalityTestHelper(tcp_dir_encoding)) + + # Add the helper as a child of the backends multi-service test-helper + multiService.addService(tcp_directionality_helper) + + # Start the test-helpers service + tcp_directionality_helper.startService() + + +Requiring the helper in a test +------------- + +If you are creating tests that rely on custom test-hepers you will want to make sure that you do not get innacturate results because your test-helper being missing in the OONI-backend you are testing against. You can specify required test-helpers within a OONI-probe test by setting its ``requiredTestHelpers`` property. In this example we have made our test helper require the tcp-directionality test that we created above.:: + + class MyDirectionalityTest(nettest.NetTestCase): + """ An example test.""" + + requiredTestHelpers = {'backend': 'tcp-directionality'} + ...

1 0

[ooni-probe/master] psiphon.py: prepare for merge with master
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 03cec435f36d43fdc0e47c0067548b92c41d55d5 Author: Simone Basso <bassosimone(a)gmail.com> Date: Sun Nov 15 15:58:54 2015 +0100 psiphon.py: prepare for merge with master - for now check.torprokect.org will do - for now it's OK to keep psiphon in the home directory - couple of cosmetic changes --- ooni/nettests/third_party/psiphon.py | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/ooni/nettests/third_party/psiphon.py b/ooni/nettests/third_party/psiphon.py index 9fec2fd..637a505 100644 --- a/ooni/nettests/third_party/psiphon.py +++ b/ooni/nettests/third_party/psiphon.py @@ -59,22 +59,18 @@ class PsiphonTest(httpt.HTTPTest, process.ProcessTest): if self.localOptions['url']: self.url = self.localOptions['url'] else: - # FIXME: use http://google.com? - # self.url = 'https://wtfismyip.com/text' self.url = 'https://check.torproject.org' if self.localOptions['psiphonpath']: self.psiphonpath = self.localOptions['psiphonpath'] else: - # FIXME: search for pyclient path instead of assuming is in the - # home? - # psiphon is not installable and to run it manually, it has to be + # Psiphon is not installable and to run it manually, it has to be # run from the psiphon directory, so it wouldn't make sense to - # nstall it in the PATH - from os import path, getenv + # install it in the PATH. For now, we assume that Psiphon sources + # are in the user's home directory. + from os import path, getenv self.psiphonpath = path.join( - getenv('HOME'), - 'psiphon-circumvention-system/pyclient') + getenv('HOME'), 'psiphon-circumvention-system/pyclient') log.debug('psiphon path: %s' % self.psiphonpath) # psi_client.py can not be run directly because the paths in the @@ -92,7 +88,7 @@ connect(False) def handleRead(self, stdout, stderr): if 'Press Ctrl-C to terminate.' in self.processDirector.stdout: if not self.bootstrapped.called: - log.debug("PsiphonTest.test_psiphon: calling bootstrapped.callback") + log.debug("PsiphonTest: calling bootstrapped.callback") self.bootstrapped.callback(None) def test_psiphon(self): @@ -114,9 +110,9 @@ connect(False) # full with some block size and therefore the test would # terminate with error finished = self.run(self.command, - env=dict(PYTHONPATH=self.psiphonpath), - path=self.psiphonpath, - usePTY=1) + env=dict(PYTHONPATH=self.psiphonpath), + path=self.psiphonpath, + usePTY=1) def callDoRequest(_): return self.doRequest(self.url) @@ -130,6 +126,3 @@ connect(False) self.bootstrapped.addBoth(cleanup) return self.bootstrapped - - -

1 0

[ooni-probe/master] psiphon: fix bug that psiphon hangs ooniprobe -ng
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 85ae20219c13a19e26896d867ea121fdf24ccc0e Author: Simone Basso <bassosimone(a)gmail.com> Date: Sun Nov 15 16:58:37 2015 +0100 psiphon: fix bug that psiphon hangs ooniprobe -ng Marked my fix with XXX because I'm not sure whether this is the proper approach, or whether it would be better to work on a patch to teach ooniprobe's core to deal with that case. For now, and given that a psiphon MVP must be delivered as soon as possible, this fix will do. --- ooni/nettests/third_party/psiphon.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/ooni/nettests/third_party/psiphon.py b/ooni/nettests/third_party/psiphon.py index 637a505..543e856 100644 --- a/ooni/nettests/third_party/psiphon.py +++ b/ooni/nettests/third_party/psiphon.py @@ -95,12 +95,18 @@ connect(False) log.debug('PsiphonTest.test_psiphon') if not os.path.exists(self.psiphonpath): - log.debug('psiphon path does not exists, is it installed?') + log.err('psiphon path does not exists, is it installed?') self.report['success'] = False self.report['psiphon_installed'] = False log.debug("Adding %s to report" % self.report) - #FIXME: this completes the test but ooni doesn't stop running, why? - return defer.succeed(None) + # XXX: the original code written by juga0 readed + # > return defer.succeed(None) + # but this caused `ooniprobe -ng` to hang forever, so I + # rewrote the code to return a deferred and simulate calling + # its callback method, to trigger an event. + # -sbs + reactor.callLater(0.0, self.bootstrapped.callback, None) + return self.bootstrapped self.report['psiphon_installed'] = True log.debug("Adding %s to report" % self.report)

1 0