August 2014 - tor-commits - lists.torproject.org

[ooni-probe/master] Improve REAMDE formatting.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit d79e3d28a4cf2c55735a52940fd5879f48e63ef3 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 11:54:01 2014 +0200 Improve REAMDE formatting. --- README.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 654acc3..b1977b1 100644 --- a/README.rst +++ b/README.rst @@ -7,6 +7,8 @@ ooniprobe: a network interference detection tool .. image:: https://coveralls.io/repos/TheTorProject/ooni-probe/badge.png :target: https://coveralls.io/r/TheTorProject/ooni-probe +___________________________________________________________________________ + .. image:: https://ooni.torproject.org/theme/img/ooni-logo.png :target: https:://ooni.torproject.org/ @@ -16,8 +18,9 @@ using Free and Open Source Software (FL/OSS) to share observations and data about the various types, methods, and amounts of network tampering in the world. + "The Net interprets censorship as damage and routes around it." - - John Gilmore; TIME magazine (6 December 1993) + - John Gilmore; TIME magazine (6 December 1993) ooniprobe is the first program that users run to probe their network and to

1 0

[ooni-probe/master] Update README
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 9f4b1e31c6fa78a048788a2fe0b0df8b8c04ebc8 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 11:48:24 2014 +0200 Update README --- README.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/README.rst b/README.rst index d8fe11e..4abbe68 100644 --- a/README.rst +++ b/README.rst @@ -1,18 +1,22 @@ +ooniprobe: a network interference detection tool +================================================ + .. image:: https://travis-ci.org/TheTorProject/ooni-probe.png?branch=master :target: https://travis-ci.org/TheTorProject/ooni-probe .. image:: https://coveralls.io/repos/TheTorProject/ooni-probe/badge.png :target: https://coveralls.io/r/TheTorProject/ooni-probe - "The Net interprets censorship as damage and routes around it." - - John Gilmore; TIME magazine (6 December 1993) - OONI, the Open Observatory of Network Interference, is a global observation network which aims is to collect high quality data using open methodologies, using Free and Open Source Software (FL/OSS) to share observations and data about the various types, methods, and amounts of network tampering in the world. + "The Net interprets censorship as damage and routes around it." + - John Gilmore; TIME magazine (6 December 1993) + + ooniprobe is the first program that users run to probe their network and to collect data for the OONI project. Are you interested in testing your network for signs of surveillance and censorship? Do you want to collect data to share

1 0

[ooni-probe/master] Fix Changelog reading in setup.py
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 2ea71126f0f9ddedade4a40c253dcb78aa879bd3 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 11:55:29 2014 +0200 Fix Changelog reading in setup.py --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index e7fa493..6d8e335 100644 --- a/setup.py +++ b/setup.py @@ -97,7 +97,7 @@ with open('requirements.txt') as f: with open('README.rst') as f: readme = f.read() -with open('Changelog.rst') as f: +with open('ChangeLog.rst') as f: changelog = f.read() setup(

1 0

[ooni-probe/master] Fix rst formatting.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 0fd5458c0392b81347460242b2ccbb2d85fa5532 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 12:00:31 2014 +0200 Fix rst formatting. --- README.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index b1977b1..a8bad75 100644 --- a/README.rst +++ b/README.rst @@ -123,7 +123,8 @@ Using ooniprobe **Bouncer** is a service used to discover the addresses of test helpers and collectors. -### Configuring ooniprobe +Configuring ooniprobe +--------------------- You may edit the configuration for ooniprobe by editing the configuration file found inside of ``~/.ooni/ooniprobe.conf``.

1 0

[ooni-probe/master] Add back OONI logo
by art＠torproject.org 19 Aug '14

19 Aug '14

commit b245789da9c7af20c490343484af2c7974dc3df4 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 11:49:31 2014 +0200 Add back OONI logo --- README.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.rst b/README.rst index 4abbe68..654acc3 100644 --- a/README.rst +++ b/README.rst @@ -7,6 +7,9 @@ ooniprobe: a network interference detection tool .. image:: https://coveralls.io/repos/TheTorProject/ooni-probe/badge.png :target: https://coveralls.io/r/TheTorProject/ooni-probe +.. image:: https://ooni.torproject.org/theme/img/ooni-logo.png + :target: https:://ooni.torproject.org/ + OONI, the Open Observatory of Network Interference, is a global observation network which aims is to collect high quality data using open methodologies, using Free and Open Source Software (FL/OSS) to share observations and data

1 0

[ooni-probe/master] Fix multi protocol traceroute test.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 9cc8aada84063cc3f426e855f34efba1997de90f Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 12:48:20 2014 +0200 Fix multi protocol traceroute test. Sharing the MTraceroute() instance amongst all the test methods leads to problems. --- ooni/nettests/manipulation/traceroute.py | 68 +++++++++++++++--------------- 1 file changed, 35 insertions(+), 33 deletions(-) diff --git a/ooni/nettests/manipulation/traceroute.py b/ooni/nettests/manipulation/traceroute.py index a561281..b9267b0 100644 --- a/ooni/nettests/manipulation/traceroute.py +++ b/ooni/nettests/manipulation/traceroute.py @@ -1,9 +1,11 @@ # -*- encoding: utf-8 -*- +from twisted.internet import defer from twisted.python import usage from ooni.templates import scapyt +from ooni.utils import log from ooni.utils.txscapy import MPTraceroute from ooni.settings import config @@ -37,50 +39,50 @@ class Traceroute(scapyt.BaseScapyTest): version = "0.3" def setUp(self): - self.st = MPTraceroute() - if self.localOptions['maxttl']: - self.st.ttl_max = int(self.localOptions['maxttl']) - if self.localOptions['dstport']: - self.st.dst_ports = [int(self.localOptions['dstport'])] - if self.localOptions['interval']: - self.st.interval = float(self.localOptions['interval']) - - config.scapyFactory.registerProtocol(self.st) - self.report['test_tcp_traceroute'] = dict( [('hops_%d' % d, []) for d in self.dst_ports]) self.report['test_udp_traceroute'] = dict( [('hops_%d' % d, []) for d in self.dst_ports]) self.report['test_icmp_traceroute'] = {'hops': []} + @defer.inlineCallbacks + def run_traceroute(self, protocol): + st = MPTraceroute() + if self.localOptions['maxttl']: + st.ttl_max = int(self.localOptions['maxttl']) + if self.localOptions['dstport']: + st.dst_ports = [int(self.localOptions['dstport'])] + if self.localOptions['interval']: + st.interval = float(self.localOptions['interval']) + log.msg("Running %s traceroute towards %s" % (protocol, + self.localOptions['backend'])) + log.msg("This will take about %s seconds" % st.timeout) + config.scapyFactory.registerProtocol(st) + traceroute = getattr(st, protocol + 'Traceroute') + yield traceroute(self.localOptions['backend']) + st.stopListening() + st.matchResponses() + for packet in st.sent_packets: + self.report['sent_packets'].append(packet) + for packet in st.matched_packets.values(): + self.report['answered_packets'].extend(packet) + + for ttl in xrange(st.ttl_min, st.ttl_max): + matchedPackets = filter( + lambda x: x.ttl == ttl, + st.matched_packets.keys()) + for packet in matchedPackets: + for response in st.matched_packets[packet]: + self.addToReport(packet, response) + def test_icmp_traceroute(self): - return self.st.ICMPTraceroute(self.localOptions['backend']) + return self.run_traceroute('ICMP') def test_tcp_traceroute(self): - return self.st.TCPTraceroute(self.localOptions['backend']) + return self.run_traceroute('TCP') def test_udp_traceroute(self): - return self.st.UDPTraceroute(self.localOptions['backend']) - - def postProcessor(self, measurements): - # should be called after all deferreds have calledback - self.st.stopListening() - self.st.matchResponses() - - if measurements[0][1].result == self.st: - for packet in self.st.sent_packets: - self.report['sent_packets'].append(packet) - for packet in self.st.matched_packets.values(): - self.report['answered_packets'].extend(packet) - - for ttl in xrange(self.st.ttl_min, self.st.ttl_max): - matchedPackets = filter( - lambda x: x.ttl == ttl, - self.st.matched_packets.keys()) - for packet in matchedPackets: - for response in self.st.matched_packets[packet]: - self.addToReport(packet, response) - return self.report + return self.run_traceroute('UDP') def addToReport(self, packet, response): if packet.proto == 1:

1 0

[ooni-probe/master] Also check if the attributes exist
by art＠torproject.org 19 Aug '14

19 Aug '14

commit cd1bc4b81ac41451a4d82d153dbdd888ad466cf1 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 13:09:47 2014 +0200 Also check if the attributes exist --- ooni/nettests/blocking/http_requests.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/ooni/nettests/blocking/http_requests.py b/ooni/nettests/blocking/http_requests.py index d37489e..997fb4e 100644 --- a/ooni/nettests/blocking/http_requests.py +++ b/ooni/nettests/blocking/http_requests.py @@ -131,10 +131,12 @@ class HTTPRequestsTest(httpt.HTTPTest): control = measurement.result if experiment and control: - if experiment.body and control.body: + if hasattr(experiment, 'body') and hasattr(control, 'body') \ + and experiment.body and control.body: self.compare_body_lengths(len(control.body), - len(experiment.body)) - if experiment.headers and control.headers: + len(experiment.body)) + if hasattr(experiment, 'headers') and hasattr(control, 'headers') \ + and experiment.headers and control.headers: self.compare_headers(control.headers, - experiment.headers) + experiment.headers) return self.report

1 0

[ooni-probe/master] Add debugging code for pyobfsproxy and fteproxy
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 02c41274662f707417443c88af3b2ccde8f2f6bd Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 13:29:37 2014 +0200 Add debugging code for pyobfsproxy and fteproxy --- ooni/nettests/blocking/bridge_reachability.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ooni/nettests/blocking/bridge_reachability.py b/ooni/nettests/blocking/bridge_reachability.py index 77dfce9..7f10f0c 100644 --- a/ooni/nettests/blocking/bridge_reachability.py +++ b/ooni/nettests/blocking/bridge_reachability.py @@ -132,6 +132,7 @@ class BridgeReachability(nettest.NetTestCase): if transport_name and transport_name == 'fte' and self.fteproxy_bin: config.ClientTransportPlugin = "%s exec %s --managed" % ( transport_name, self.fteproxy_bin) + log.debug("Using fte from %s" % self.fteproxy_bin) self.report['transport_name'] = transport_name self.report['bridge_address'] = self.bridge.split(' ')[1] elif transport_name and transport_name == 'fte'\ @@ -142,6 +143,7 @@ class BridgeReachability(nettest.NetTestCase): elif transport_name and self.pyobfsproxy_bin: config.ClientTransportPlugin = "%s exec %s managed" % ( transport_name, self.pyobfsproxy_bin) + log.debug("Using pyobfsproxy from %s" % self.pyobfsproxy_bin) self.report['transport_name'] = transport_name self.report['bridge_address'] = self.bridge.split(' ')[1] elif transport_name and not self.pyobfsproxy_bin:

1 0

[ooni-probe/master] Add report package to setup.py
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 5dcf9b0ab1ff5f4b50b4b3bd7e5fe4199e91f7e5 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 13:11:51 2014 +0200 Add report package to setup.py --- setup.py | 1 + 1 file changed, 1 insertion(+) diff --git a/setup.py b/setup.py index 6d8e335..b19cb47 100644 --- a/setup.py +++ b/setup.py @@ -80,6 +80,7 @@ packages=[ 'ooni.nettests.scanning', 'ooni.nettests.blocking', 'ooni.nettests.third_party', + 'ooni.report', 'ooni.templates', 'ooni.tests', 'ooni.utils'

1 0

[ooni-probe/master] Also add obfsproxy version to the bridge reachability test report.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit b67df141e2d7cff6a7a318407cd779801a82c7c8 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 13:48:06 2014 +0200 Also add obfsproxy version to the bridge reachability test report. Raise exception if obfsproxy is older than 0.2 --- ooni/nettests/blocking/bridge_reachability.py | 9 ++++++- ooni/oonicli.py | 2 ++ ooni/utils/onion.py | 31 +++++++++++++++++++++---- 3 files changed, 36 insertions(+), 6 deletions(-) diff --git a/ooni/nettests/blocking/bridge_reachability.py b/ooni/nettests/blocking/bridge_reachability.py index 7f10f0c..f4d5bf1 100644 --- a/ooni/nettests/blocking/bridge_reachability.py +++ b/ooni/nettests/blocking/bridge_reachability.py @@ -64,12 +64,13 @@ class BridgeReachability(nettest.NetTestCase): self.report['tor_progress_tag'] = None self.report['tor_progress_summary'] = None self.report['tor_log'] = None + self.report['obfsproxy_version'] = str(onion.obfsproxy_details['version']) self.report['bridge_address'] = None self.bridge = self.input if self.input.startswith('Bridge'): self.bridge = self.input.replace('Bridge ', '') - self.pyobfsproxy_bin = find_executable('obfsproxy') + self.pyobfsproxy_bin = onion.obfsproxy_details['binary'] self.fteproxy_bin = find_executable('fteproxy') def postProcessor(self, measurements): @@ -143,6 +144,12 @@ class BridgeReachability(nettest.NetTestCase): elif transport_name and self.pyobfsproxy_bin: config.ClientTransportPlugin = "%s exec %s managed" % ( transport_name, self.pyobfsproxy_bin) + if onion.OBFSProxyVersion('0.2') > onion.obfsproxy_details['version']: + log.err( + "The obfsproxy version you are using appears to be outdated." + ) + self.report['error'] = 'old-obfsproxy' + return log.debug("Using pyobfsproxy from %s" % self.pyobfsproxy_bin) self.report['transport_name'] = transport_name self.report['bridge_address'] = self.bridge.split(' ')[1] diff --git a/ooni/oonicli.py b/ooni/oonicli.py index f8cf849..6505584 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -201,6 +201,8 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): print e.net_test_loader.usageOptions().getUsage() sys.exit(4) except Exception as e: + if config.advanced.debug: + log.exception(e) log.err(e) sys.exit(5) diff --git a/ooni/utils/onion.py b/ooni/utils/onion.py index cb490b3..09e104c 100644 --- a/ooni/utils/onion.py +++ b/ooni/utils/onion.py @@ -1,5 +1,6 @@ import string import subprocess +from distutils.spawn import find_executable from distutils.version import LooseVersion from txtorcon.util import find_tor_binary as tx_find_tor_binary @@ -11,28 +12,43 @@ class TorVersion(LooseVersion): pass +class OBFSProxyVersion(LooseVersion): + pass + + def find_tor_binary(): if config.advanced.tor_binary: return config.advanced.tor_binary return tx_find_tor_binary() -def tor_version(): - tor_binary = find_tor_binary() - if not tor_binary: +def executable_version(binary, strip=lambda x: x): + if not binary: return None try: - proc = subprocess.Popen((tor_binary, '--version'), + proc = subprocess.Popen((binary, '--version'), stdout=subprocess.PIPE, stderr=subprocess.PIPE) except OSError: pass else: stdout, _ = proc.communicate() if proc.poll() == 0 and stdout != '': - return TorVersion(stdout.strip().split(' ')[2]) + version = stdout.strip() + return LooseVersion(strip(version)) return None +def tor_version(): + version = executable_version(find_tor_binary(), + lambda x: x.split(' ')[2]) + return TorVersion(str(version)) + + +def obfsproxy_version(): + version = executable_version(find_executable('obfsproxy')) + return OBFSProxyVersion(str(version)) + + def transport_name(address): """ If the address of the bridge starts with a valid c identifier then @@ -53,3 +69,8 @@ tor_details = { 'binary': find_tor_binary(), 'version': tor_version() } + +obfsproxy_details = { + 'binary': find_executable('obfsproxy'), + 'version': obfsproxy_version() +}

1 0