April 2016 - tor-commits - lists.torproject.org

[ooni-probe/master] [bugfix] Critical bug that sometimes lead to incorrect body content
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 0bfb58cd9d2647e6defa3bcdf0dbd7a9e2e16189 Author: Arturo Filastò <arturo(a)filasto.net> Date: Thu Apr 14 22:44:47 2016 +0200 [bugfix] Critical bug that sometimes lead to incorrect body content --- ooni/utils/trueheaders.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ooni/utils/trueheaders.py b/ooni/utils/trueheaders.py index b7671b5..d5aa206 100644 --- a/ooni/utils/trueheaders.py +++ b/ooni/utils/trueheaders.py @@ -103,7 +103,7 @@ class HTTPClientParser(_newclient.HTTPClientParser): self._partialHeader = None def headerReceived(self, name, value): - if self.isConnectionControlHeader(name): + if self.isConnectionControlHeader(name.lower()): headers = self.connHeaders else: headers = self.headers

1 0

[ooni-probe/master] Explicitly test twisted version 15.5.0
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 52bb445c881d111b19c3a8f365b435dc8bb889b8 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Feb 5 20:54:34 2016 +0100 Explicitly test twisted version 15.5.0 --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index acda409..e3033f6 100644 --- a/.travis.yml +++ b/.travis.yml @@ -10,6 +10,7 @@ env: - TWISTED=Twisted==15.2.1 - TWISTED=Twisted==15.3.0 - TWISTED=Twisted==15.4.0 + - TWISTED=Twisted==15.5.0 - TWISTED=Twisted before_install: - sudo apt-get update

1 0

[ooni-probe/master] Update changelog for version 1.4.0
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 2f8a8a3cfcbab9c019f613d2588b281549788f39 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Apr 22 12:52:08 2016 +0200 Update changelog for version 1.4.0 --- ChangeLog.rst | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/ChangeLog.rst b/ChangeLog.rst index 79cb883..47851e1 100644 --- a/ChangeLog.rst +++ b/ChangeLog.rst @@ -1,6 +1,22 @@ Changelog ========= +v1.4.0 (XXX, XX Apr 2016) +------------------------- + +* Support for reporting using JSON + +* Support for running ooniprobe with a message queue providing URLs to test + +* Psiphon censorship circumvention test + +* OpenVPN censorship circumvention test + +* Support for disabling reporting to disk + +* Improvements to HTTP response body decoding (includes fix that lead to empty + bodies being misrepresented) + v1.3.2 (Fri, 20 Nov 2015) -------------------------

1 0

[ooni-probe/master] Merge pull request #456 from TheTorProject/feature/json_reports
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 5252b00b29f8c9570ecb9533f4577c0a5b69f06f Merge: a0fc5e6 dd58c40 Author: Simone Basso <bassosimone(a)gmail.com> Date: Mon Apr 4 18:41:52 2016 +0200 Merge pull request #456 from TheTorProject/feature/json_reports Feature/json reports .travis.yml | 4 - ooni/deckgen/processors/citizenlab_test_lists.py | 5 + ooni/director.py | 11 +- ooni/errors.py | 3 + ooni/nettest.py | 17 +-- ooni/nettests/blocking/dns_consistency.py | 29 +++-- ooni/nettests/blocking/http_requests.py | 11 +- ooni/nettests/experimental/daphne.py | 121 ++++++++++++++++++ ooni/nettests/manipulation/captiveportal.py | 2 +- ooni/nettests/manipulation/daphne.py | 121 ------------------ ooni/nettests/manipulation/http_host.py | 4 +- .../manipulation/http_invalid_request_line.py | 7 +- ooni/oonibclient.py | 3 - ooni/oonicli.py | 50 ++++---- ooni/otime.py | 79 +----------- ooni/report/tool.py | 4 +- ooni/reporter.py | 136 ++++++++++++++------- ooni/tasks.py | 5 +- ooni/templates/dnst.py | 107 +++++++++++----- ooni/templates/httpt.py | 50 ++++++-- ooni/templates/scapyt.py | 12 +- ooni/templates/tcpt.py | 1 + ooni/tests/test_director.py | 6 +- ooni/tests/test_oonicli.py | 17 +-- ooni/tests/test_otime.py | 14 --- ooni/tests/test_reporter.py | 23 +++- ooni/tests/test_templates.py | 44 ++++++- ooni/tests/test_trueheaders.py | 12 +- ooni/tests/test_utils.py | 10 +- ooni/utils/__init__.py | 13 +- ooni/utils/log.py | 2 +- ooni/utils/trueheaders.py | 2 +- requirements.txt | 2 +- 33 files changed, 526 insertions(+), 401 deletions(-)

1 0

[ooni-probe/master] fix the dependency on pyasn1 (#470)
by art＠torproject.org 29 Apr '16

29 Apr '16

commit f0b6332693cd7ea149d2b43780e133d9489a1379 Author: Veggie Monster <vmonmoonshine(a)gmail.com> Date: Tue Apr 26 09:44:51 2016 -0400 fix the dependency on pyasn1 (#470) --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 0143b58..28e5f78 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,7 +10,7 @@ scapy-real>=2.2.0-dev pypcap>=1.1 service-identity pydumbnet -pyasn1 +pyasn1>=0.1.8 pyasn1-modules characteristic zope.interface

1 0

[ooni-probe/master] Drop dependencies that are for sub-modules of ooni
by art＠torproject.org 29 Apr '16

29 Apr '16

commit f098d7048e9394d0ad2f624be153b8d802ab4e61 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Apr 8 16:10:48 2016 +0200 Drop dependencies that are for sub-modules of ooni --- requirements.txt | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/requirements.txt b/requirements.txt index 28e5f78..fc6f629 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,4 @@ +pyasn1>=0.1.8 PyYAML>=3.10 Twisted>=13.2.0 ipaddr>=2.1.10 @@ -5,13 +6,8 @@ pyOpenSSL>=0.13 geoip txtorcon>=0.7 txsocksx>=0.0.2 -parsley>=1.1 scapy-real>=2.2.0-dev pypcap>=1.1 service-identity pydumbnet -pyasn1>=0.1.8 -pyasn1-modules -characteristic zope.interface -cryptography

1 0

[ooni-probe/master] Add support for downloading resources from mirror on github.
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 6a6e0ac21e9ee20db47df37a4fc9d4fb4e3be794 Author: Arturo Filastò <arturo(a)filasto.net> Date: Sat Apr 23 15:20:02 2016 +0200 Add support for downloading resources from mirror on github. * Download resources when they are missing from oonideckgen * Remove dependency on GeoLiteCity database (16MB is just too much) --- ooni/deckgen/cli.py | 32 +++++++-- ooni/deckgen/processors/citizenlab_test_lists.py | 6 +- ooni/geoip.py | 11 --- ooni/resources/__init__.py | 59 +--------------- ooni/resources/cli.py | 37 ++++------ ooni/resources/update.py | 86 ++++++++++++++---------- ooni/tests/test_geoip.py | 1 - 7 files changed, 93 insertions(+), 139 deletions(-) diff --git a/ooni/deckgen/cli.py b/ooni/deckgen/cli.py index 1f618b7..b1a6b87 100644 --- a/ooni/deckgen/cli.py +++ b/ooni/deckgen/cli.py @@ -13,8 +13,9 @@ from ooni.geoip import ProbeIP from ooni.settings import config from ooni.deckgen import __version__ -from ooni.resources import inputs - +from ooni.deckgen.processors import citizenlab_test_lists +from ooni.deckgen.processors import namebench_dns_servers +from ooni.resources.update import download_resources class Options(usage.Options): synopsis = """%s [options] @@ -70,11 +71,9 @@ class Deck(object): def generate_deck(options): - dns_servers_processor = inputs['namebench-dns-servers.csv']['processor'] - url_lists_processor = inputs['citizenlab-test-lists.zip']['processor'] try: - url_list_country = url_lists_processor.generate_country_input( + url_list_country = citizenlab_test_lists.generate_country_input( options['country-code'], options['output'] ) @@ -84,10 +83,10 @@ def generate_deck(options): print "We will just use the global one." url_list_country = None - url_list_global = url_lists_processor.generate_global_input( + url_list_global = citizenlab_test_lists.generate_global_input( options['output'] ) - dns_servers = dns_servers_processor.generate_country_input( + dns_servers = namebench_dns_servers.generate_country_input( options['country-code'], options['output'] ) @@ -124,6 +123,20 @@ def get_user_country_code(): yield probe_ip.lookup() defer.returnValue(probe_ip.geodata['countrycode']) +def resources_up_to_date(): + if config.get_data_file_path("GeoIP/GeoIP.dat") is None: + return False + + if config.get_data_file_path("resources/" + "namebench-dns-servers.csv") is None: + return False + + if config.get_data_file_path("resources/" + "citizenlab-test-lists/" + "global.csv") is None: + return False + + return True @defer.inlineCallbacks def run(): @@ -135,6 +148,11 @@ def run(): print options sys.exit(1) + if not resources_up_to_date(): + print("Resources for running ooniprobe are not up to date.") + print("Will update them now.") + yield download_resources() + if not options['output']: options['output'] = os.getcwd() diff --git a/ooni/deckgen/processors/citizenlab_test_lists.py b/ooni/deckgen/processors/citizenlab_test_lists.py index 086b550..8a660cf 100644 --- a/ooni/deckgen/processors/citizenlab_test_lists.py +++ b/ooni/deckgen/processors/citizenlab_test_lists.py @@ -28,10 +28,9 @@ def generate_country_input(country_code, dst): input_list = config.get_data_file_path("resources/" "citizenlab-test-lists/" - "test-lists-master/lists/" + country_code + ".csv") - if not os.path.exists(input_list): + if not input_list: raise Exception("Could not find list for country %s" % country_code) load_input(input_list, filename) @@ -44,12 +43,11 @@ def generate_global_input(dst): input_list = config.get_data_file_path("resources/" "citizenlab-test-lists/" - "test-lists-master/lists/" "global.csv") if not input_list: print("Could not find the global input list") - print("Perhaps you should run ooniresources --update-inputs") + print("Perhaps you should run ooniresources") raise Exception("Could not find the global input list") load_input(input_list, filename) diff --git a/ooni/geoip.py b/ooni/geoip.py index 11800b6..83f3648 100644 --- a/ooni/geoip.py +++ b/ooni/geoip.py @@ -31,7 +31,6 @@ class GeoIPDataFilesNotFound(Exception): def IPToLocation(ipaddr): from ooni.settings import config - city_file = config.get_data_file_path('GeoIP/GeoLiteCity.dat') country_file = config.get_data_file_path('GeoIP/GeoIP.dat') asn_file = config.get_data_file_path('GeoIP/GeoIPASNum.dat') @@ -51,12 +50,6 @@ def IPToLocation(ipaddr): error() try: - city_dat = GeoIP(city_file) - location['city'] = city_dat.record_by_addr(ipaddr)['city'] - except: - error() - - try: asn_dat = GeoIP(asn_file) location['asn'] = asn_dat.org_by_addr(ipaddr).split(' ')[0] except: @@ -75,10 +68,6 @@ def database_version(): 'GeoIPASNum': { 'sha256': None, 'timestamp': None - }, - 'GeoLiteCity': { - 'sha256': None, - 'timestamp': None } } diff --git a/ooni/resources/__init__.py b/ooni/resources/__init__.py index 94cd36e..8b7e63b 100644 --- a/ooni/resources/__init__.py +++ b/ooni/resources/__init__.py @@ -1,59 +1,6 @@ -import os - -from ooni.settings import config -from ooni.utils import unzip, gunzip - -from ooni.deckgen.processors import citizenlab_test_lists -from ooni.deckgen.processors import namebench_dns_servers +from ooni import __version__ as ooniprobe_version __version__ = "0.1.0" -if os.access(config.var_lib_path, os.W_OK): - resources_directory = os.path.join(config.var_lib_path, - "resources") - geoip_directory = os.path.join(config.var_lib_path, - "GeoIP") -else: - resources_directory = os.path.join(config.ooni_home, - "resources") - geoip_directory = os.path.join(config.ooni_home, - "GeoIP") - -inputs = { - "namebench-dns-servers.csv": { - "url": "https://namebench.googlecode.com/svn/trunk/config/servers.csv", - "action": None, - "action_args": [], - "processor": namebench_dns_servers, - }, - "citizenlab-test-lists.zip": { - "url": "https://github.com/citizenlab/test-lists/archive/master.zip", - "action": unzip, - "action_args": [resources_directory], - "processor": citizenlab_test_lists - } -} - -geoip = { - "GeoLiteCity.dat.gz": { - "url": "https://geolite.maxmind.com/download/" - "geoip/database/GeoLiteCity.dat.gz", - "action": gunzip, - "action_args": [geoip_directory], - "processor": None - }, - "GeoIPASNum.dat.gz": { - "url": "https://geolite.maxmind.com/download/" - "geoip/database/asnum/GeoIPASNum.dat.gz", - "action": gunzip, - "action_args": [geoip_directory], - "processor": None - }, - "GeoIP.dat.gz": { - "url": "https://geolite.maxmind.com/" - "download/geoip/database/GeoLiteCountry/GeoIP.dat.gz", - "action": gunzip, - "action_args": [geoip_directory], - "processor": None - } -} +ooni_resources_url = ("https://github.com/TheTorProject/ooni-probe/releases" + "/download/v{}/ooni-resources.tar.gz").format(ooniprobe_version) diff --git a/ooni/resources/cli.py b/ooni/resources/cli.py index cab9700..0d7832a 100644 --- a/ooni/resources/cli.py +++ b/ooni/resources/cli.py @@ -3,18 +3,23 @@ import sys from twisted.internet import defer from twisted.python import usage -from ooni.utils import log - from ooni.resources import __version__ from ooni.resources import update class Options(usage.Options): - synopsis = """%s""" % sys.argv[0] + synopsis = """%s + This is used to update the resources required to run oonideckgen and + ooniprobe. + You just run this script with no arguments and it will update the + resources. + """ % sys.argv[0] optFlags = [ - ["update-inputs", None, "Update the resources needed for inputs."], - ["update-geoip", None, "Update the geoip related resources."] + ["update-inputs", None, "(deprecated) update the resources needed for " + "inputs."], + ["update-geoip", None, "(deprecated) Update the geoip related " + "resources."] ] optParameters = [] @@ -33,23 +38,7 @@ def run(): print "%s: Try --help for usage details." % (sys.argv[0]) sys.exit(1) - if not any(options.values()): - print("%s: no command specified" % sys.argv[0]) - print options - sys.exit(1) + if options['update-inputs'] or options['update-geoip']: + print("WARNING: Passing command line arguments is deprecated") - if options['update-inputs']: - print "Downloading inputs" - try: - yield update.download_inputs() - except Exception as exc: - log.err("failed to download geoip files") - log.exception(exc) - - if options['update-geoip']: - print "Downloading geoip files" - try: - yield update.download_geoip() - except Exception as exc: - log.err("failed to download geoip files") - log.exception(exc) + yield update.download_resources() diff --git a/ooni/resources/update.py b/ooni/resources/update.py index 7a949c9..46bbdf8 100644 --- a/ooni/resources/update.py +++ b/ooni/resources/update.py @@ -1,43 +1,57 @@ import os +import tarfile +import tempfile +from twisted.python.filepath import FilePath from twisted.internet import defer from twisted.web.client import downloadPage -from ooni.resources import inputs, geoip, resources_directory -from ooni.utils import unzip, gunzip - +from ooni.settings import config +from ooni.resources import ooni_resources_url @defer.inlineCallbacks -def download_resource(resources): - for filename, resource in resources.items(): - print "Downloading %s" % filename - - if resource["action"] in [unzip, gunzip] and resource["action_args"]: - dirname = resource["action_args"][0] - filename = os.path.join(dirname, filename) - else: - filename = os.path.join(resources_directory, filename) - if not os.path.exists(filename): - directory = os.path.dirname(filename) - if not os.path.isdir(directory): - os.makedirs(directory) - f = open(filename, 'w') - f.close() - elif not os.path.isfile(filename): - print "[!] %s must be a file." % filename - defer.returnValue(False) - yield downloadPage(resource['url'], filename) - - if resource['action'] is not None: - yield defer.maybeDeferred(resource['action'], - filename, - *resource['action_args']) - print "%s written." % filename - - -def download_inputs(): - return download_resource(inputs) - - -def download_geoip(): - return download_resource(geoip) +def download_resources(): + if os.access(config.var_lib_path, os.W_OK): + dst_directory = FilePath(config.var_lib_path) + else: + dst_directory = FilePath(config.ooni_home) + + print("Downloading {} to {}".format(ooni_resources_url, + dst_directory.path)) + tmp_download_directory = FilePath(tempfile.mkdtemp()) + tmp_download_filename = tmp_download_directory.temporarySibling() + + + try: + yield downloadPage(ooni_resources_url, tmp_download_filename.path) + ooni_resources_tar_gz = tarfile.open(tmp_download_filename.path) + ooni_resources_tar_gz.extractall(tmp_download_directory.path) + + if not tmp_download_directory.child('GeoIP').exists(): + raise Exception("Could not find GeoIP data files in downloaded " + "tar.") + + if not tmp_download_directory.child('resources').exists(): + raise Exception("Could not find resources data files in " + "downloaded tar.") + + geoip_dir = dst_directory.child('GeoIP') + resources_dir = dst_directory.child('resources') + + if geoip_dir.exists(): + geoip_dir.remove() + tmp_download_directory.child('GeoIP').moveTo(geoip_dir) + + if resources_dir.exists(): + resources_dir.remove() + tmp_download_directory.child('resources').moveTo(resources_dir) + + print("Written GeoIP files to {}".format(geoip_dir.path)) + print("Written resources files to {}".format(resources_dir.path)) + + except Exception as exc: + print("Failed to download resources!") + raise exc + + finally: + tmp_download_directory.remove() diff --git a/ooni/tests/test_geoip.py b/ooni/tests/test_geoip.py index 5e8b67b..66ba13e 100644 --- a/ooni/tests/test_geoip.py +++ b/ooni/tests/test_geoip.py @@ -26,7 +26,6 @@ class TestGeoIP(bases.ConfigTestCase): version = geoip.database_version() assert 'GeoIP' in version.keys() assert 'GeoIPASNum' in version.keys() - assert 'GeoLiteCity' in version.keys() assert len(version['GeoIP']['sha256']) == 64 assert isinstance(version['GeoIP']['timestamp'], float)

1 0

[ooni-probe/master] Remove hacks module and patch reduce_ex
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 55e55a55a0365c3e4958482e315add4f0090243a Author: Arturo Filastò <arturo(a)filasto.net> Date: Sat Apr 23 19:03:49 2016 +0200 Remove hacks module and patch reduce_ex * Delete benchmarks txt file --- bin/ooniprobe | 7 ------ bin/ooniprobequeue | 5 ----- ooni/kit/benchmarks.txt | 35 ----------------------------- ooni/utils/hacks.py | 60 ------------------------------------------------- 4 files changed, 107 deletions(-) diff --git a/bin/ooniprobe b/bin/ooniprobe index 31da086..4832d91 100755 --- a/bin/ooniprobe +++ b/bin/ooniprobe @@ -1,13 +1,6 @@ #!/usr/bin/env python -import copy_reg from twisted.internet import reactor -# This is a hack to overcome a bug in python -from ooni.utils.hacks import patched_reduce_ex -copy_reg._reduce_ex = patched_reduce_ex - -# from ooni.oonicli import run -# run() from ooni.oonicli import runWithDirector d = runWithDirector() @d.addBoth diff --git a/bin/ooniprobequeue b/bin/ooniprobequeue index f81f156..88359e0 100755 --- a/bin/ooniprobequeue +++ b/bin/ooniprobequeue @@ -1,12 +1,7 @@ #!/usr/bin/env python import sys -import copy_reg from twisted.internet import reactor -# This is a hack to overcome a bug in python -from ooni.utils.hacks import patched_reduce_ex -copy_reg._reduce_ex = patched_reduce_ex - code=0 from ooni.oonicli import runWithDaemonDirector diff --git a/ooni/kit/benchmarks.txt b/ooni/kit/benchmarks.txt deleted file mode 100644 index c2070e5..0000000 --- a/ooni/kit/benchmarks.txt +++ /dev/null @@ -1,35 +0,0 @@ -Read file B -done in 0.74356508255 -Read file A -done in 0.94336104393 -Computing prob matrix -done in 0.0432229042053 -Computing eigenvalues -done in 0.00188422203064 -Corelation: 0.999999079331 - - -Read file B -done in 0.946599960327 -Read file A -done in 0.909075975418 -Computing prob matrix -done in 0.0541450977325 -Computing eigenvalues -done in 0.00259518623352 -Computing prob matrix B -done in 0.060467004776 -Computing eigen B -done in 0.00102496147156 -Computing correlation -done in 0.000158071517944 -Corelation: 0.999999079331 - -this was with: -683 filea.txt -678 fileb.txt - -diff file* | wc -l -283 - - diff --git a/ooni/utils/hacks.py b/ooni/utils/hacks.py deleted file mode 100644 index 7ec3d0f..0000000 --- a/ooni/utils/hacks.py +++ /dev/null @@ -1,60 +0,0 @@ -# When some software has issues and we need to fix it in a -# hackish way, we put it in here. This one day will be empty. - -import copy_reg -from twisted.web.client import SchemeNotSupported - -from txsocksx.http import SOCKS5Agent as SOCKS5AgentOriginal - - -def patched_reduce_ex(self, proto): - """ - This is a hack to overcome a bug in one of pythons core functions. It is - located inside of copy_reg and is called _reduce_ex. - - Some background on the issue can be found here: - - http://stackoverflow.com/questions/569754/how-to-tell-for-which-object-attr… - http://stackoverflow.com/questions/2049849/why-cant-i-pickle-this-object - - There was also an open bug on the pyyaml trac repo, but it got closed because - they could not reproduce. - http://pyyaml.org/ticket/190 - - It turned out to be easier to patch the python core library than to monkey - patch yaml. - - XXX see if there is a better way. sigh... - """ - _HEAPTYPE = 1 << 9 - assert proto < 2 - for base in self.__class__.__mro__: - if hasattr(base, '__flags__') and not base.__flags__ & _HEAPTYPE: - break - else: - base = object # not really reachable - if base is object: - state = None - elif base is int: - state = None - else: - if base is self.__class__: - raise TypeError("can't pickle %s objects" % base.__name__) - state = base(self) - args = (self.__class__, base, state) - try: - getstate = self.__getstate__ - except AttributeError: - if getattr(self, "__slots__", None): - raise TypeError("a class that defines __slots__ without " - "defining __getstate__ cannot be pickled") - try: - dict = self.__dict__ - except AttributeError: - dict = None - else: - dict = getstate() - if dict: - return copy_reg._reconstructor, args, dict - else: - return copy_reg._reconstructor, args

1 0

[ooni-probe/master] Print a helpful message when run with no arguments (inciting to run oonideckgen)
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 96dad3f578e0797aee4992fc79b25ea8af2b9414 Author: Arturo Filastò <arturo(a)filasto.net> Date: Sat Apr 23 15:48:09 2016 +0200 Print a helpful message when run with no arguments (inciting to run oonideckgen) * Optimise a bit the importing of modules --- ooni/oonicli.py | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 63312b5..5bc2124 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -7,20 +7,11 @@ import random import urlparse from twisted.python import usage -from twisted.python.util import spewer -from twisted.internet import defer, reactor, protocol +from twisted.internet import defer from ooni import errors, __version__ - from ooni.settings import config -from ooni.director import Director -from ooni.deck import Deck, nettest_to_path -from ooni.nettest import NetTestLoader - from ooni.utils import log -from ooni.utils.net import hasRawSocketPermission - - class LifetimeExceeded(Exception): pass @@ -30,7 +21,7 @@ class Options(usage.Options): longdesc = ("ooniprobe loads and executes a suite or a set of suites of" " network tests. These are loaded from modules, packages and" - " files listed on the command line") + " files listed on the command line.") optFlags = [["help", "h"], ["resume", "r"], @@ -71,11 +62,21 @@ class Options(usage.Options): self['test'] = None usage.Options.__init__(self) + def getUsage(self, width=None): + return super(Options, self).getUsage(width) + """ +To get started you may want to run: + +$ oonideckgen + +This will tell you how to run ooniprobe :) +""" + def opt_spew(self): """ Print an insanely verbose log of everything that happens. Useful when debugging freezes or locks in complex code. """ + from twisted.python.util import spewer sys.settrace(spewer) def opt_version(self): @@ -186,6 +187,7 @@ def setupGlobalOptions(logging, start_tor, check_incoherences): log.start(global_options['logfile']) if config.privacy.includepcap: + from ooni.utils.net import hasRawSocketPermission if hasRawSocketPermission(): from ooni.utils.txscapy import ScapyFactory config.scapyFactory = ScapyFactory(config.advanced.interface) @@ -223,6 +225,9 @@ def setupCollector(global_options, collector_address): def createDeck(global_options, url=None): + from ooni.nettest import NetTestLoader + from ooni.deck import Deck, nettest_to_path + if url: log.msg("Creating deck for: %s" % (url)) @@ -331,9 +336,9 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): Instance the director, parse command line options and start an ooniprobe test! """ - global_options = setupGlobalOptions(logging, start_tor, check_incoherences) + from ooni.director import Director director = Director() if global_options['list']: print "# Installed nettests" @@ -379,7 +384,8 @@ def runWithDaemonDirector(logging=True, start_tor=True, check_incoherences=True) Instance the director, parse command line options and start an ooniprobe test! """ - + from twisted.internet import reactor, protocol + from ooni.director import Director try: import pika from pika import exceptions

1 0

[ooni-probe/master] Detect when the control response contains a blockpage from cloudflare
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 210ec234e1d60be752b44e1b2136b69659de6eec Author: Arturo Filastò <arturo(a)filasto.net> Date: Sun Apr 24 21:31:52 2016 +0200 Detect when the control response contains a blockpage from cloudflare --- ooni/nettests/blocking/http_requests.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/ooni/nettests/blocking/http_requests.py b/ooni/nettests/blocking/http_requests.py index abc7efc..24d95cc 100644 --- a/ooni/nettests/blocking/http_requests.py +++ b/ooni/nettests/blocking/http_requests.py @@ -35,10 +35,10 @@ class HTTPRequestsTest(httpt.HTTPTest): lengths match. """ name = "HTTP Requests" - description = "Performs a HTTP GET request over Tor and one over the " \ - "local network and compares the two results." + description = ("Performs a HTTP GET request over Tor and one over the " + "local network and compares the two results.") author = "Arturo Filastò" - version = "0.2.4" + version = "0.2.5" usageOptions = UsageOptions @@ -78,6 +78,7 @@ class HTTPRequestsTest(httpt.HTTPTest): self.report['factor'] = float(self.factor) self.report['headers_diff'] = None self.report['headers_match'] = None + self.report['control_cloudflare'] = None self.headers = {'User-Agent': [random.choice(userAgents)]} @@ -145,6 +146,11 @@ class HTTPRequestsTest(httpt.HTTPTest): if experiment and control: if hasattr(experiment, 'body') and hasattr(control, 'body') \ and experiment.body and control.body: + self.report['control_cloudflare'] = False + if 'Attention Required! | CloudFlare' in control.body: + log.msg("The control body contains a blockpage from " + "cloudflare. This will skew our results.") + self.report['control_cloudflare'] = True self.compare_body_lengths(len(control.body), len(experiment.body)) if hasattr(experiment, 'headers') and hasattr(control, 'headers') \

1 0