January 2016 - tor-commits - lists.torproject.org

[ooni-probe/master] Merge pull request #445 from TheTorProject/fix/psiphon
by art＠torproject.org 30 Jan '16

30 Jan '16

commit f222a9d68cfd845c3bbb0c66b0e20a0667f78530 Merge: 91f4a18 c70886a Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Jan 26 17:25:50 2016 +0100 Merge pull request #445 from TheTorProject/fix/psiphon Fix to make the psiphon test run ooni/nettests/third_party/psiphon.py | 44 +++++++++++++++++++++++++----------- ooni/templates/process.py | 14 +++++++----- 2 files changed, 39 insertions(+), 19 deletions(-)

1 0

[ooni-probe/master] Edited various bits of the documentation page & moved "using ooniprobe" towards the beginning
by art＠torproject.org 30 Jan '16

30 Jan '16

commit c8afae85af648942607faa2a3315533de80176fa Author: Maria Xynou <maria.x(a)espiv.net> Date: Sat Jan 30 13:09:53 2016 +0100 Edited various bits of the documentation page & moved "using ooniprobe" towards the beginning --- docs/source/index.rst | 96 +++++++++++++++++++++------------------------------ 1 file changed, 40 insertions(+), 56 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 1eabdee..13f7a8f 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,29 +3,19 @@ OONI: Open Observatory of Network Interference ============================================== -OONI, the Open Observatory of Network Interference, is a global observation -network which aims is to collect high quality data using open methodologies, -using Free and Open Source Software (FL/OSS) to share observations and data -about the various types, methods, and amounts of network tampering in the -world. +The Open Observatory of Network Interference (OONI) is a global observation +network which collects high quality data, using open methodologies and Free and Open Source Software (FL/OSS), to share observations about the various types, methods, and amounts of network tampering in the world. "The Net interprets censorship as damage and routes around it." - John Gilmore; TIME magazine (6 December 1993) +Are you interested in testing your network for signs of surveillance and censorship? Do you want to collect data so that you and others can better understand your network? If so, please read this document and we hope that ooniprobe will help you gather network data that will assist you with your endeavors! + ooniprobe --------- -ooniprobe is the first program that users run to probe their network and to -collect data for the OONI project. Are you interested in testing your network -for signs of surveillance and censorship? Do you want to collect data to share -with others, so that you and others may better understand your network? If so, -please read this document and we hope ooniprobe will help you to gather -network data that will assist you with your endeavors! - -ooniprobe allows the user to select what test should be run and what backend -should be used for storing the test report and/or assisting them in the running -of the test. +ooniprobe is a program that users can run to probe their network and to collect network measurement data. ooniprobe also allows users to choose what test should be run and what backend should be used for storing the test report and for assisting them in running the test. ooniprobe tests are divided into two categories: **Traffic Manipulation** and **Content Blocking**. @@ -35,23 +25,28 @@ tampering with the internet traffic between the probe and a remote test helper backend. As such they usually require the selection of a oonib backend component for running the test. -**Content Blocking** are aimed at enumerating the kind of content that is -blocked from the probes network point of view. As such they usually require to -have specified an input list for running the test. +**Content Blocking** tests aim to enumerate the kind of content that is +blocked from the probes network point of view. As such they usually require the specification of an input list for running the test. Read this before running ooniprobe! ----------------------------------- -Running ooniprobe is a potentially risky activity. This greatly depends on the -jurisdiction in which you are in and which test you are running. It is -technically possible for a person observing your internet connection to be -aware of the fact that you are running ooniprobe. This means that if running -network measurement tests is something considered to be illegal in your country -then you could be spotted. +Running ooniprobe can potentially be a risky activity. This largely depends on the laws and jurisdiction of the country you are running ooniprobe from, as well as on the type of test you are running. Technically, it is possible for a person observing your internet connection to be aware of the fact that you are running ooniprobe. This means that if running network measurement tests is something that is considered to be illegal in your country then you could be spotted. + +Furthermore, ooniprobe takes no precautions to protect the install target machine from forensics analysis. If the fact that you have installed or used ooniprobe is a liability for you, please be aware of this risk. + +Using ooniprobe +=============== + +**Net test** is a set of measurements to assess what kind of internet censorship is occurring. + +**Decks** are collections of ooniprobe nettests with some associated inputs. + +**Collector** is a service used to report the results of measurements. + +**Test helper** is a service used by a probe for successfully performing its measurements. -Furthermore, ooniprobe takes no precautions to protect the install target machine -from forensics analysis. If the fact that you have installed or used ooni -probe is a liability for you, please be aware of this risk. +**Bouncer** is a service used to discover the addresses of test helpers and collectors. OONI in 5 minutes ================= @@ -91,8 +86,7 @@ This should generate the home and allow you to run oonideckgen. The output from the last command will tell you how to run ooniprobe to perform the measurement. -If you would like to contribute measurements to OONI daily you can also add -this to your crontab:: +If you would like to contribute measurements to OONI daily you can also add this to your crontab:: @daily ooniprobe $THE_OONI_COMMAND @@ -185,19 +179,6 @@ ooniprobe will be installed in ``/ooni``. ooniprobe blocking/http_requests -f /ooni/example_inputs/alexa-top-1k.txt -Using ooniprobe -=============== - -**Net test** is a set of measurements to assess what kind of internet censorship is occurring. - -**Decks** are collections of ooniprobe nettests with some associated inputs. - -**Collector** is a service used to report the results of measurements. - -**Test helper** is a service used by a probe for successfully performing its measurements. - -**Bouncer** is a service used to discover the addresses of test helpers and collectors. - Configuring ooniprobe --------------------- @@ -217,10 +198,10 @@ This can be done with:: ooniresources --update-inputs -If you get a permission error, you may have to run the command as root or +If you get a permission error, you might have to run the command as root or change the ooniprobe data directory inside of `ooniprobe.conf`. -On some platforms, for example debian contrib, you will not get all the geoip +On some platforms, such as debian contrib, you will not get all the geoip related files needed. In that case it is possible to manually download them with ``ooniresources``:: @@ -229,7 +210,7 @@ with ``ooniresources``:: Generating decks ---------------- -You can generate decks for your country thanks to the oonideckgen command. +You can generate decks for your country with the oonideckgen command. If you wish, for example, to generate a deck to be run in the country of Italy, you can do so (be sure to have updated the input resources first) by running:: @@ -238,7 +219,8 @@ you can do so (be sure to have updated the input resources first) by running:: You will now have in your home a folder called `deck-it`, containing the ooni deck (ends with .deck) and the inputs. -Note: that you should not move the `deck-*` directory once it has been + +**Note:** You should not move the `deck-*` directory once it has been generated as the paths to the inputs referenced by the test in the deck are absolute. If you want your deck to live in another directory you must regenerated it. @@ -327,6 +309,7 @@ obfsproxy is installed). The tor.bridges option in ooniprobe.conf sets the path to a file that should contain a set of "bridge" lines (of the same format as used in torrc, and as returned by https://bridges.torproject.org) If obfsproxy bridges are to be used, the path to the obfsproxy binary must be configured. + See option advanced.obfsproxy_binary, in ooniprobe.conf. (Optional) Install obfsproxy @@ -371,7 +354,7 @@ https://github.com/TheTorProject/ooni-probe Read this article to learn how to create a pull request on github (https://help.github.com/articles/creating-a-pull-request) -If you prefer not to use github (or don't have an account), you may also submit +If you prefer not to use github (or don't have an account), you can also submit patches as attachments to tickets. Be sure to format the patch (given that you are working on a feature branch @@ -441,7 +424,7 @@ Tests that don't do a measurement but are useful for scanning can be found in: * `ooni/nettests/scanning <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/ooni/nettests/scann…>`_ -Tests that involve running third party tools may be found in: +Tests that involve running third party tools can be found in: * `ooni/nettests/third_party <https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/ooni/nettests/third…>`_ @@ -449,7 +432,7 @@ Tests that involve running third party tools may be found in: ooni-backend ------------ -This is the server side component of ooniprobe. It will store that data +This is the server side component of ooniprobe. It will store data collected from ooniprobes and it will run a series of Test Helpers that assist `Traffic Manipulation Tests`_ in performing their measurements. @@ -476,7 +459,7 @@ Threat Model Our adversary is capable of doing country wide network surveillance and manipulation of network traffic. -The goals of our adversary are: +The goals of our adversary are to: * Restrict access to certain content, while not degrading overall quality of the network @@ -489,14 +472,15 @@ More specifc to the running of network filtering detection tests: 2. Fool people running such tests into believing that the network is unrestricted -*Note* that while 2) => 1) it is not true that 1) => 2) as the identification of +**Note:** While 2) => 1) it is not true that 1) => 2) as the identification of such actors does not necessarily have to happen in real time. -While our intention is to minimize the risk of users running OONI probe to be -identified, this comes with a tradeoff in accuracy. It is therefore necessary in -certain tests to trade-off fingerprintability in favour of tests accuracy. -This is why we divide tests based on what risk the user running it can face, -allowing the user to freely choose what threat model they wish to adere to. +While our intention is to minimize the risk of users running OONI probe being +identified, this comes with a trade-off in accuracy. It is therefore necessary in +certain tests to trade-off fingerprintability in favour of test accuracy. + +This is why we divide tests based on what risk the users running them can face, +allowing users to freely choose what threat model they wish to adere to. More developer documentation

1 0

[ooni-probe/master] OpenVPN connection circumvention test
by art＠torproject.org 30 Jan '16

30 Jan '16

commit 716913e40e6737b49263d43fbdcaf98effdf8d4d Author: srvetus <srvetus(a)users.noreply.github.com> Date: Thu Sep 24 01:43:14 2015 +0200 OpenVPN connection circumvention test This test attempts to connect to an OpenVPN server specified in an OpenVPN config file. A HTTP request is sent if the VPN connection is successful. This basic test can be expanded to parse more of the output from the OpenVPN client. The output could provide more information about why a connection might be failing. --- ooni/nettests/third_party/openvpn.py | 117 +++++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) diff --git a/ooni/nettests/third_party/openvpn.py b/ooni/nettests/third_party/openvpn.py new file mode 100644 index 0000000..b681222 --- /dev/null +++ b/ooni/nettests/third_party/openvpn.py @@ -0,0 +1,117 @@ +from twisted.internet import defer, reactor +from twisted.python import usage +from twisted.web.client import Agent, readBody +from ooni.templates.process import ProcessTest +from ooni.utils import log +from ooni.errors import handleAllFailures + +import distutils.spawn +import re + + +class UsageOptions(usage.Options): + optParameters = [ + ['url', 'u', None, 'Specify a single URL on the OpenVPN subnet to test.'], + ['openvpn-config', 'c', None, 'Specify an OpenVPN configuration file.'], + ] + + +class OpenVPNTest(ProcessTest): + + """ + This class tests OpenVPN connections. + + test_openvpn_circumvent + Starts an OpenVPN client on Linux and determines + if it connects successfully to an OpenVPN server. + Then, it make a HTTP request for http://google.com + and records the response body or failure string. + + """ + + name = "OpenVPN Client Test" + description = "Connects to an OpenVPN server and does a HTTP GET for the specified URL" + author = "srvetus " + version = "0.0.1" + timeout = 20 + usageOptions = UsageOptions + requiredOptions = ['url', 'openvpn-config'] + requiresRoot = True + + def setUp(self): + self.bootstrapped = defer.Deferred() + self.command = [distutils.spawn.find_executable("openvpn")] + self.exited = False + self.url = self.localOptions.get('url') + + if self.localOptions.get('openvpn-config'): + openvpn_config = self.localOptions.get('openvpn-config') + self.command.extend(['--config', openvpn_config]) + + def stop(self, reason=None): + """Stop the running OpenVPN process and close the connection""" + if not self.exited: + self.processDirector.close() + + # OpenVPN needs to be sent SIGTERM to end cleanly + self.processDirector.transport.signalProcess('TERM') + self.exited = True + + def handleRead(self, stdout=None, stderr=None): + """handleRead is called with each chunk of data from stdout and stderr + + stdout only contains the latest data chunk, self.processDirector.stdout + contains the combined stdout data. + """ + + # Read OpenVPN output until bootstrapping succeeds or fails + if not self.bootstrapped.called: + + # TODO: Determine other OpenVPN messages which indicate connection failure + if re.search(r'connect to .* failed', self.processDirector.stdout): + log.debug("OpenVPN connection failed") + + # Bootstrapping failed + self.bootstrapped.errback(Exception("openvpn_connection_failed")) + + # Check if OpenVPN has bootstrapped and connected successfully + elif "Initialization Sequence Completed" in self.processDirector.stdout: + log.debug("OpenVPN connection successful") + self.processDirector.cancelTimer() + self.bootstrapped.callback("bootstrapped") + + def test_openvpn_circumvent(self): + + def addResultToReport(result): + log.debug("request_successful") + self.report['body'] = result + self.report['success'] = True + + def addFailureToReport(failure): + log.debug("failed: %s" % handleAllFailures(failure)) + self.report['failure'] = handleAllFailures(failure) + self.report['success'] = False + + def doRequest(noreason): + """Make a HTTP request over initialized VPN connection""" + agent = Agent(reactor) + + log.debug("Doing HTTP request to the OpenVPN subnet: %s" % self.url) + request = agent.request("GET", self.url) + request.addCallback(readBody) + request.addCallback(addResultToReport) + request.addErrback(addFailureToReport) + return request + + log.debug("Spawning OpenVPN") + self.d = self.run(self.command) + + # Try to make a request when the OpenVPN connection successfully bootstraps + self.bootstrapped.addCallback(doRequest) + + # Fire failure if OpenVPN connection fails + self.bootstrapped.addErrback(addFailureToReport) + + # Close OpenVPN after each successful or failed test + self.bootstrapped.addBoth(self.stop) + return self.d

1 0

[ooni-probe/master] Merge pull request #440 from TheTorProject/hotfix/disable-twisted-15.5.0
by art＠torproject.org 30 Jan '16

30 Jan '16

commit a52ab0267a777d9a0f1ecddd3b5124ef7a69b9a1 Merge: 0c67cab 0e40771 Author: Simone Basso <bassosimone(a)gmail.com> Date: Wed Dec 2 14:10:51 2015 +0100 Merge pull request #440 from TheTorProject/hotfix/disable-twisted-15.5.0 Avoid installing Twisted 15.5 to avoid bugs with txtorcon requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

1 0

[ooni-probe/master] Handle and log errors when OpenVPN fails to start
by art＠torproject.org 30 Jan '16

30 Jan '16

commit 4606063140b98d4cd36d1bd2bbe97ffb7ee1e8df Author: srvetus <srvetus(a)users.noreply.github.com> Date: Thu Nov 19 12:23:38 2015 +0100 Handle and log errors when OpenVPN fails to start When OpenVPN encouters a configuration error it will exit the process with error code 1 before beginning to bootstrap. This commit monkeypatches `processDirector.processExited` to log these pre-bootstrap OpenVPN configuration failures. This commit also replaces `handleAllFailures()` with `failureToString()` for generating the report content. `handleAllFailures()` causes difficult to debug errors. It silently raises an exception from `failure.trap()` if an unknown exception type is passed to it. Any raised Exception will causes the remaining error logging to be skipped making debugging difficult. Ooni should produce some sort of log message if an Exception is raised in `handleAllFailures()`. --- ooni/nettests/third_party/openvpn.py | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/ooni/nettests/third_party/openvpn.py b/ooni/nettests/third_party/openvpn.py index b681222..028bf17 100644 --- a/ooni/nettests/third_party/openvpn.py +++ b/ooni/nettests/third_party/openvpn.py @@ -3,7 +3,7 @@ from twisted.python import usage from twisted.web.client import Agent, readBody from ooni.templates.process import ProcessTest from ooni.utils import log -from ooni.errors import handleAllFailures +from ooni.errors import handleAllFailures, failureToString import distutils.spawn import re @@ -32,7 +32,7 @@ class OpenVPNTest(ProcessTest): name = "OpenVPN Client Test" description = "Connects to an OpenVPN server and does a HTTP GET for the specified URL" author = "srvetus " - version = "0.0.1" + version = "0.0.2" timeout = 20 usageOptions = UsageOptions requiredOptions = ['url', 'openvpn-config'] @@ -57,6 +57,16 @@ class OpenVPNTest(ProcessTest): self.processDirector.transport.signalProcess('TERM') self.exited = True + def processExited(self, reason): + """Monkeypatch processExited to log failure if the process ends + unexpectedly before OpenVPN bootstraps. + """ + log.debug("Exited %s" % handleAllFailures(reason)) + + # Process exited before OpenVPN bootstrapped. Add failure to report + if not self.bootstrapped.called: + self.bootstrapped.errback(Exception("openvpn_exited_unexpectedly")) + def handleRead(self, stdout=None, stderr=None): """handleRead is called with each chunk of data from stdout and stderr @@ -88,8 +98,8 @@ class OpenVPNTest(ProcessTest): self.report['success'] = True def addFailureToReport(failure): - log.debug("failed: %s" % handleAllFailures(failure)) - self.report['failure'] = handleAllFailures(failure) + log.debug("Failed: %s" % failureToString(failure)) + self.report['failure'] = failureToString(failure) self.report['success'] = False def doRequest(noreason): @@ -106,6 +116,9 @@ class OpenVPNTest(ProcessTest): log.debug("Spawning OpenVPN") self.d = self.run(self.command) + # Monkeypatch processExited to log when OpenVPN exits early + self.processDirector.processExited = self.processExited + # Try to make a request when the OpenVPN connection successfully bootstraps self.bootstrapped.addCallback(doRequest)

1 0

[ooni-probe/master] Avoid installing Twisted 15.5 to avoid bugs with txtorcon
by art＠torproject.org 30 Jan '16

30 Jan '16

commit 0e4077143fad28a88c3fd242ccff4dcdf1367e86 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Dec 2 12:40:08 2015 +0100 Avoid installing Twisted 15.5 to avoid bugs with txtorcon --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 8b8d5ca..b052adb 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ PyYAML>=3.10 -Twisted>=12.2.0 +Twisted>=12.2.0,<15.5.0 ipaddr>=2.1.10 pyOpenSSL>=0.13 geoip

1 0

[ooni-probe/master] Merge pull request #441 from TheTorProject/fix/typo
by art＠torproject.org 30 Jan '16

30 Jan '16

commit 6a86f0bcb7762c1f5b1363c4b658731103584a2e Merge: a52ab02 e57d8f8 Author: Joe Landers <joe(a)joelanders.net> Date: Wed Dec 2 15:42:13 2015 +0100 Merge pull request #441 from TheTorProject/fix/typo Fix typo in docstring spotted by @armadev ooni/tests/test_errors.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

1 0

[ooni-probe/master] Fix typo in docstring spotted by @armadev
by art＠torproject.org 30 Jan '16

30 Jan '16

commit e57d8f885c2be591bcfa7b5f337495cdb2e6ce64 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Dec 2 15:27:22 2015 +0100 Fix typo in docstring spotted by @armadev --- ooni/tests/test_errors.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ooni/tests/test_errors.py b/ooni/tests/test_errors.py index 7baa1ea..fa1e001 100644 --- a/ooni/tests/test_errors.py +++ b/ooni/tests/test_errors.py @@ -9,7 +9,7 @@ class TestErrors(unittest.TestCase): Verify that more specific Failures are caught first by handleAllFailures() and failureToString(). - Fails if a subclass is listed after it's parent Failure. + Fails if a subclass is listed after its parent Failure. """ # Check each Failure against all subsequent failures

1 0

[ooni-probe/master] Merge pull request #443 from TheTorProject/fix/twisted15
by art＠torproject.org 30 Jan '16

30 Jan '16

commit b792e1e340ba3c318f44b7177cec489092b63e26 Merge: 6a86f0b 649228f Author: Simone Basso <bassosimone(a)gmail.com> Date: Thu Dec 3 14:39:50 2015 +0100 Merge pull request #443 from TheTorProject/fix/twisted15 We no longer need to avoid twisted 15.5.0 requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

1 0

[ooni-probe/master] Fix to make the psiphon test run
by art＠torproject.org 30 Jan '16

30 Jan '16

commit c8e624436c1d8e0160116d80c5e6eb8ad3d4abc6 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Dec 9 17:34:51 2015 +0100 Fix to make the psiphon test run * Remove unneeded imports --- ooni/nettests/third_party/psiphon.py | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/ooni/nettests/third_party/psiphon.py b/ooni/nettests/third_party/psiphon.py index 543e856..fb912a9 100644 --- a/ooni/nettests/third_party/psiphon.py +++ b/ooni/nettests/third_party/psiphon.py @@ -1,19 +1,12 @@ import tempfile -import stat import os import sys from twisted.internet import defer, reactor -from twisted.internet.endpoints import TCP4ClientEndpoint -from twisted.web.client import readBody from twisted.python import usage -from txsocksx.http import SOCKS5Agent - -from ooni.errors import handleAllFailures, TaskTimedOut from ooni.utils import log from ooni.templates import process, httpt -from ooni.templates.process import ProcessTest class UsageOptions(usage.Options): @@ -94,9 +87,9 @@ connect(False) def test_psiphon(self): log.debug('PsiphonTest.test_psiphon') + self.report['success'] = None if not os.path.exists(self.psiphonpath): log.err('psiphon path does not exists, is it installed?') - self.report['success'] = False self.report['psiphon_installed'] = False log.debug("Adding %s to report" % self.report) # XXX: the original code written by juga0 readed @@ -116,7 +109,8 @@ connect(False) # full with some block size and therefore the test would # terminate with error finished = self.run(self.command, - env=dict(PYTHONPATH=self.psiphonpath), + env=dict(PYTHONPATH=os.path.join(self.psiphonpath, + 'pyclient')), path=self.psiphonpath, usePTY=1)

1 0