April 2016 - tor-commits - lists.torproject.org

[ooni-probe/master] Remove references to the geoip_data_dir since it's not respected
by art＠torproject.org 29 Apr '16

29 Apr '16

commit c79a087b5926e36962290ce1fb76a4301bddd49c Author: Arturo Filastò <arturo(a)filasto.net> Date: Sat Apr 23 19:04:14 2016 +0200 Remove references to the geoip_data_dir since it's not respected --- data/ooniprobe.conf.sample | 1 - ooni/deckgen/processors/namebench_dns_servers.py | 3 +-- ooni/geoip.py | 6 +++--- ooni/oonicli.py | 1 + ooni/tests/test_oonicli.py | 1 - 5 files changed, 5 insertions(+), 7 deletions(-) diff --git a/data/ooniprobe.conf.sample b/data/ooniprobe.conf.sample index 7aaacd7..5f3f17d 100644 --- a/data/ooniprobe.conf.sample +++ b/data/ooniprobe.conf.sample @@ -23,7 +23,6 @@ reports: pcap: null collector: null advanced: - geoip_data_dir: /usr/share/GeoIP debug: false # enable if auto detection fails #tor_binary: /usr/sbin/tor diff --git a/ooni/deckgen/processors/namebench_dns_servers.py b/ooni/deckgen/processors/namebench_dns_servers.py index f35ca93..9855611 100644 --- a/ooni/deckgen/processors/namebench_dns_servers.py +++ b/ooni/deckgen/processors/namebench_dns_servers.py @@ -8,7 +8,6 @@ from ooni.settings import config class GeoIPDB(object): _borg = {} country = None - asn = None def __init__(self): self.__dict__ = self._borg @@ -25,7 +24,7 @@ class GeoIPDB(object): def generate_country_input(country_code, dst): csv_file = config.get_data_file_path("resources/" - "namebench-dns-servers.csv") + "namebench-dns-servers.csv") filename = os.path.join(dst, "dns-server-%s.txt" % country_code) fw = open(filename, "w") diff --git a/ooni/geoip.py b/ooni/geoip.py index 83f3648..6179844 100644 --- a/ooni/geoip.py +++ b/ooni/geoip.py @@ -37,9 +37,9 @@ def IPToLocation(ipaddr): location = {'city': None, 'countrycode': 'ZZ', 'asn': 'AS0'} def error(): - log.err("Could not find GeoIP data file in %s." - "Try running ooniresources --update-geoip or" - " edit your ooniprobe.conf" % config.advanced.geoip_data_dir) + log.err("Could not find GeoIP data file in data directories." + "Try running ooniresources or" + " edit your ooniprobe.conf") try: country_dat = GeoIP(country_file) diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 5bc2124..8e7d1f1 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -169,6 +169,7 @@ def director_startup_other_failures(failure): def setupGlobalOptions(logging, start_tor, check_incoherences): global_options = parseOptions() + config.global_options = global_options config.set_paths() config.initialize_ooni_home() diff --git a/ooni/tests/test_oonicli.py b/ooni/tests/test_oonicli.py index c57205c..c3b179a 100644 --- a/ooni/tests/test_oonicli.py +++ b/ooni/tests/test_oonicli.py @@ -39,7 +39,6 @@ reports: pcap: null collector: null advanced: - geoip_data_dir: /usr/share/GeoIP debug: false interface: auto start_tor: false

1 0

[ooni-probe/master] Use debian/jessie instead of precise
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 223ba4803e17b25bc4d1a54bafc5b51896a2c14c Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Apr 26 15:45:22 2016 +0200 Use debian/jessie instead of precise * Make the setup of ooni-backend conditional and do it in a another machine --- Vagrantfile | 85 ++++++++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 61 insertions(+), 24 deletions(-) diff --git a/Vagrantfile b/Vagrantfile index 16a6ba0..e61bf68 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -1,22 +1,8 @@ # -*- mode: ruby -*- # vi: set ft=ruby : -Vagrant.configure("2") do |config| - # All Vagrant configuration is done here. The most common configuration - # options are documented and commented below. For a complete reference, - # please see the online documentation at vagrantup.com. - config.vm.box = "precise32" - config.vm.box_url = "http://files.vagrantup.com/precise32.box" - - config.vm.synced_folder ".", "/ooni" - # Place the ooni-backend source code in ../ooni-backend to sync it with the vagrant instance - config.vm.synced_folder "../ooni-backend", "/backend" - -end - -$script = <<SCRIPT - -echo "deb http://deb.torproject.org/torproject.org precise main" >> /etc/apt/sources.list +$setup_ooniprobe = <<SCRIPT +echo "deb http://deb.torproject.org/torproject.org jessie main" >> /etc/apt/sources.list # Install deb.torproject.org key gpg --keyserver keys.gnupg.net --recv 886DDD89 @@ -27,18 +13,69 @@ apt-get update apt-get install -y tor deb.torproject.org-keyring # Setup for sniffer subsystem -apt-get install -y build-essential libdumbnet-dev python-dumbnet python-pypcap libpcap-dev python-dev python-pip libgeoip-dev libffi-dev -cd /ooni +apt-get install -y build-essential libdumbnet-dev libpcap-dev libgeoip-dev libffi-dev python-dev python-pip +cd /data/ooni-probe python setup.py install -echo "Login using 'vagrant ssh', and don't forget to run ooniprobe as root." -echo "First run: 'sudo ooniprobe -i /usr/share/ooni/decks/fast.deck'" + +echo "Now:" +echo "1. vagrant ssh probe" +echo "2. oonideckgen" +echo "3. ooniprobe -i deck-*/*.deck" SCRIPT -# TODO: -# Somehow, ooniprobe is not capable to connect to tor by default. My current -# workaround is to kill tor, and set "start_tor: true" in /root/.ooni/ooniprobe.conf +$setup_oonibackend = <<SCRIPT +echo "Installing Tor..." + +echo "deb http://deb.torproject.org/torproject.org jessie main" >> /etc/apt/sources.list + +gpg --keyserver keys.gnupg.net --recv 886DDD89 +gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | apt-key add - + +apt-get update + +apt-get -y install deb.torproject.org-keyring tor tor-geoipdb + +apt-get -y install curl python-pip python-dev libffi-dev + +cd /data/ooni-backend + +echo "Generating SSL keys" + +openssl genrsa -out private.key 4096 +openssl req -new -key private.key -out server.csr -subj '/CN=www.example.com/O=Example/C=AU' + +openssl x509 -req -days 365 -in server.csr -signkey private.key -out certificate.crt + +cp oonib.conf.example /etc/oonibackend.conf + +echo "Installing ooni-backend" +python setup.py install + +echo "Now:" +echo "1. vagrant ssh" +echo "2. vi /etc/oonibackend.conf # possibly" +echo "3. cd /data/ooni-backend" +echo "4. sudo ./bin/oonib -c /etc/oonibackend.conf" + +SCRIPT Vagrant.configure("2") do |config| - config.vm.provision :shell, :inline => $script + # Use Debian jessie with the vboxfs contrib add-on + config.vm.box = "debian/contrib-jessie64" + + config.vm.define "probe" do |probe| + probe.vm.synced_folder ".", "/data/ooni-probe" + probe.vm.provision :shell, :inline => $setup_ooniprobe + end + + # If we find a ooni-backend directory in ../ooni-backend we configure a + # ooni-backend as well. + if File.directory?("../ooni-backend") + config.vm.define "backend" do |backend| + backend.vm.synced_folder "../ooni-backend", "/data/ooni-backend" + backend.vm.provision :shell, :inline => $setup_oonibackend + end + end + end

1 0

[ooni-probe/master] Improve readability of the ooniprobe -s command
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 33e86a4a4ae96053812685c9f663db261703e449 Author: Arturo Filastò <arturo(a)filasto.net> Date: Sun Apr 24 19:13:45 2016 +0200 Improve readability of the ooniprobe -s command * Remove various pieces of dead code --- data/ooniprobe.conf.sample | 2 -- ooni/director.py | 2 +- ooni/geoip.py | 20 -------------------- ooni/nettests/third_party/psiphon.py | 4 ++-- ooni/oonicli.py | 30 +++++++++++++++++++++++------- ooni/tests/test_oonicli.py | 1 - ooni/utils/log.py | 2 -- 7 files changed, 26 insertions(+), 35 deletions(-) diff --git a/data/ooniprobe.conf.sample b/data/ooniprobe.conf.sample index 5f3f17d..edc3f86 100644 --- a/data/ooniprobe.conf.sample +++ b/data/ooniprobe.conf.sample @@ -12,8 +12,6 @@ privacy: includeasn: true # Should we include the country as reported by GeoIP in the report? includecountry: true - # Should we include the city as reported by GeoIP in the report? - includecity: false # Should we collect a full packet capture on the client? includepcap: false reports: diff --git a/ooni/director.py b/ooni/director.py index ab8da65..a60b91e 100644 --- a/ooni/director.py +++ b/ooni/director.py @@ -61,7 +61,7 @@ class Director(object): _scheduledTests = 0 # Only list NetTests belonging to these categories - categories = ['blocking', 'manipulation'] + categories = ['blocking', 'manipulation', 'third_party'] def __init__(self): self.activeNetTests = [] diff --git a/ooni/geoip.py b/ooni/geoip.py index 6179844..417c7d5 100644 --- a/ooni/geoip.py +++ b/ooni/geoip.py @@ -171,8 +171,6 @@ class ProbeIP(object): self.geodata['ip'] = self.address if not config.privacy.includeasn: self.geodata['asn'] = 'AS0' - if not config.privacy.includecity: - self.geodata['city'] = None if not config.privacy.includecountry: self.geodata['countrycode'] = 'ZZ' if not config.privacy.includeip: @@ -194,16 +192,6 @@ class ProbeIP(object): log.msg("Unable to lookup the probe IP via Tor.") try: - yield self.askTraceroute() - log.msg("Found your IP via Traceroute %s" % self.address) - self.resolveGeodata() - defer.returnValue(self.address) - except errors.InsufficientPrivileges: - log.debug("Cannot determine the probe IP address with a traceroute, becase of insufficient priviledges") - except: - log.msg("Unable to lookup the probe IP via traceroute") - - try: yield self.askGeoIPService() log.msg("Found your IP via a GeoIP service: %s" % self.address) self.resolveGeodata() @@ -230,14 +218,6 @@ class ProbeIP(object): if not self.address: raise errors.ProbeIPUnknown - def askTraceroute(self): - """ - Perform a UDP traceroute to determine the probes IP address. - """ - if not hasRawSocketPermission(): - raise errors.InsufficientPrivileges - raise NotImplemented - def askTor(self): """ Obtain the probes IP address by asking the Tor Control port via GET INFO diff --git a/ooni/nettests/third_party/psiphon.py b/ooni/nettests/third_party/psiphon.py index bfdec70..2633f60 100644 --- a/ooni/nettests/third_party/psiphon.py +++ b/ooni/nettests/third_party/psiphon.py @@ -30,8 +30,8 @@ class PsiphonTest(httpt.HTTPTest, process.ProcessTest): """ name = "Psiphon Test" - description = "Bootstraps Psiphon and \ - does a HTTP GET for the specified URL" + description = ("Bootstraps Psiphon and" + "does a HTTP GET for the specified URL") author = "juga" version = "0.0.1" timeout = 20 diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 8e7d1f1..3be64a9 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -4,6 +4,7 @@ import os import json import yaml import random +import textwrap import urlparse from twisted.python import usage @@ -24,7 +25,6 @@ class Options(usage.Options): " files listed on the command line.") optFlags = [["help", "h"], - ["resume", "r"], ["no-collector", "n", "disable writing to collector"], ["no-yamloo", "N", "disable writing to YAML file"], ["no-geoip", "g"], @@ -342,12 +342,28 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): from ooni.director import Director director = Director() if global_options['list']: - print "# Installed nettests" - for net_test_id, net_test in director.getNetTests().items(): - print "* %s (%s/%s)" % (net_test['name'], - net_test['category'], - net_test['id']) - print " %s" % net_test['description'] + net_tests = [net_test for net_test in director.getNetTests().items()] + print "" + print "Installed nettests" + print "==================" + for net_test_id, net_test in net_tests: + optList = [] + for name, details in net_test['arguments'].items(): + optList.append({'long': name, 'doc': details['description']}) + + desc = ('\n' + + net_test['name'] + + '\n' + + '-'*len(net_test['name']) + + '\n' + + '\n'.join(textwrap.wrap(net_test['description'], 80)) + + '\n\n' + + '$ ooniprobe {}/{}'.format(net_test['category'], + net_test['id']) + + '\n\n' + + ''.join(usage.docMakeChunks(optList)) + ) + print desc sys.exit(0) diff --git a/ooni/tests/test_oonicli.py b/ooni/tests/test_oonicli.py index c3b179a..172b9cd 100644 --- a/ooni/tests/test_oonicli.py +++ b/ooni/tests/test_oonicli.py @@ -33,7 +33,6 @@ privacy: includeip: false includeasn: true includecountry: true - includecity: false includepcap: true reports: pcap: null diff --git a/ooni/utils/log.py b/ooni/utils/log.py index d3036f0..f36c0ff 100644 --- a/ooni/utils/log.py +++ b/ooni/utils/log.py @@ -52,8 +52,6 @@ class OONILogger(object): def start(self, logfile=None, application_name="ooniprobe"): from ooni.settings import config - daily_logfile = None - if not logfile: logfile = os.path.expanduser(config.basic.logfile)

1 0

[ooni-probe/master] Add --collector option to oonideckgen
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 88cfa90a7b76f2c0ff6cc3253459fce4d0790cd2 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Apr 26 16:33:43 2016 +0200 Add --collector option to oonideckgen * Remove unused variable in ooniprobe * Add description to requirements.txt * Remove printing of probe IP --- Vagrantfile | 3 ++- ooni/deckgen/cli.py | 49 +++++++++++++++++++++++++++---------------------- ooni/geoip.py | 5 ++--- ooni/oonicli.py | 1 - requirements.txt | 4 ++++ 5 files changed, 35 insertions(+), 27 deletions(-) diff --git a/Vagrantfile b/Vagrantfile index 3eeb54c..59b2751 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -65,7 +65,9 @@ Vagrant.configure("2") do |config| backend.vm.synced_folder "../ooni-backend", "/data/ooni-backend" backend.vm.provision :shell, :inline => $setup_oonibackend end + end + if File.directory?("../ooni-backend") config.vm.provision "shell", inline: <<-EOF echo "To run oonibackend:" echo "1. vagrant ssh backend" @@ -73,7 +75,6 @@ Vagrant.configure("2") do |config| echo "3. cd /data/ooni-backend" echo "4. sudo ./bin/oonib -c /etc/oonibackend.conf" EOF - end config.vm.provision "shell", inline: <<-EOF diff --git a/ooni/deckgen/cli.py b/ooni/deckgen/cli.py index 664842e..c2ebfd8 100644 --- a/ooni/deckgen/cli.py +++ b/ooni/deckgen/cli.py @@ -22,15 +22,13 @@ class Options(usage.Options): """ % sys.argv[0] optParameters = [ - ["country-code", "c", - None, + ["country-code", "c", None, "Specify the two letter country code for which we should " - "generate the deck." - ], - ["output", "o", - None, - "Specify the directory where to write output." - ] + "generate the deck."], + ["collector", None, None, "Specify a custom collector to use when " + "submitting reports"], + ["output", "o", None, + "Specify the directory where to write output."] ] def opt_version(self): @@ -41,33 +39,40 @@ class Options(usage.Options): class Deck(object): _base_entry = { "options": { + "test_file": None, + "subargs": [], + "annotations": None, + "collector": None, - "help": 0, - "logfile": None, - "no-default-reporter": 0, - "parallelism": None, - "pcapfile": None, + # XXX setting this is currently not supported + "bouncer": None, + "reportfile": None, - "resume": 0, - "testdeck": None + + "no-collector": 0, + "no-geoip": 0, + "no-yamloo": 0, + "verbose": 0 } } - def __init__(self): - self.deck = [] + def __init__(self, collector=None): + self.deck_entries = [] + self.collector = collector def add_test(self, test_file, subargs=[]): deck_entry = copy.deepcopy(self._base_entry) + deck_entry['options']['collector'] = self.collector deck_entry['options']['test_file'] = test_file deck_entry['options']['subargs'] = subargs - self.deck.append(deck_entry) + self.deck_entries.append(deck_entry) def pprint(self): - print yaml.safe_dump(self.deck) + print yaml.safe_dump(self.deck_entries) def write_to_file(self, filename): with open(filename, "w+") as f: - f.write(yaml.safe_dump(self.deck)) + f.write(yaml.safe_dump(self.deck_entries)) def generate_deck(options): @@ -89,11 +94,11 @@ def generate_deck(options): options['output'] ) - deck = Deck() + deck = Deck(collector=options['collector']) deck.add_test('manipulation/http_invalid_request_line') deck.add_test('manipulation/http_header_field_manipulation') - if url_list_country is not None: + if url_list_country is not None: deck.add_test('blocking/http_requests', ['-f', url_list_country]) deck.add_test('blocking/http_requests', ['-f', url_list_global]) diff --git a/ooni/geoip.py b/ooni/geoip.py index 417c7d5..dcc799e 100644 --- a/ooni/geoip.py +++ b/ooni/geoip.py @@ -6,7 +6,6 @@ import random from hashlib import sha256 from twisted.web import client, http_headers -from ooni.utils.net import hasRawSocketPermission client._HTTP11ClientFactory.noisy = False @@ -183,7 +182,7 @@ class ProbeIP(object): else: try: yield self.askTor() - log.msg("Found your IP via Tor %s" % self.address) + log.msg("Found your IP via Tor") self.resolveGeodata() defer.returnValue(self.address) except errors.TorStateNotFound: @@ -193,7 +192,7 @@ class ProbeIP(object): try: yield self.askGeoIPService() - log.msg("Found your IP via a GeoIP service: %s" % self.address) + log.msg("Found your IP via a GeoIP service") self.resolveGeodata() defer.returnValue(self.address) except Exception: diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 3be64a9..4da81bc 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -59,7 +59,6 @@ class Options(usage.Options): tracer = None def __init__(self): - self['test'] = None usage.Options.__init__(self) def getUsage(self, width=None): diff --git a/requirements.txt b/requirements.txt index fc6f629..229504b 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,8 @@ +# These are dependencies of ooniprobe depedencies that require specific +# versions. pyasn1>=0.1.8 + +# These are direct dependencies of ooniprobe. PyYAML>=3.10 Twisted>=13.2.0 ipaddr>=2.1.10

1 0

[ooni-probe/master] Run ooniresources without any command line options in the setup.py
by art＠torproject.org 29 Apr '16

29 Apr '16

commit a4244fcb6eeb402a606ab99fec6ac1a15e0d8960 Author: Arturo Filastò <arturo(a)filasto.net> Date: Sat Apr 23 20:44:37 2016 +0200 Run ooniresources without any command line options in the setup.py * Print some progress when running oonireport * Use print function instead of old print --- ooni/report/tool.py | 23 ++++++++++++++--------- setup.py | 2 +- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/ooni/report/tool.py b/ooni/report/tool.py index 5946c42..5c7bcb2 100644 --- a/ooni/report/tool.py +++ b/ooni/report/tool.py @@ -1,4 +1,6 @@ +from __future__ import print_function import yaml +import sys from twisted.internet import defer @@ -49,9 +51,12 @@ def upload(report_file, collector=None, bouncer=None): report_id = yield oonib_reporter.createReport() report.header['report_id'] = report_id yield oonib_report_log.created(report_file, collector, report_id) + log.msg("Writing report entries") for entry in report: yield oonib_reporter.writeReportEntry(entry) - log.msg("Closing report.") + sys.stdout.write('.') + sys.stdout.flush() + log.msg("Closing report") yield oonib_reporter.finish() yield oonib_report_log.closed(report_file) @@ -68,24 +73,24 @@ def upload_all(collector=None, bouncer=None): def print_report(report_file, value): - print "* %s" % report_file - print " %s" % value['created_at'] + print("* %s" % report_file) + print(" %s" % value['created_at']) def status(): oonib_report_log = OONIBReportLog() - print "Reports to be uploaded" - print "----------------------" + print("Reports to be uploaded") + print("----------------------") for report_file, value in oonib_report_log.reports_to_upload: print_report(report_file, value) - print "Reports in progress" - print "-------------------" + print("Reports in progress") + print("-------------------") for report_file, value in oonib_report_log.reports_in_progress: print_report(report_file, value) - print "Incomplete reports" - print "------------------" + print("Incomplete reports") + print("------------------") for report_file, value in oonib_report_log.reports_incomplete: print_report(report_file, value) diff --git a/setup.py b/setup.py index 388de5f..1fd9a7b 100644 --- a/setup.py +++ b/setup.py @@ -148,7 +148,7 @@ class install(_st_install): def ooniresources(self): ooniresources = find_executable("ooniresources") from subprocess import Popen - process = Popen([ooniresources, '--update-inputs', '--update-geoip'], + process = Popen([ooniresources], stdout=sys.stdout.fileno(), stderr=sys.stderr.fileno()) process.wait()

1 0

[ooni-probe/master] Update documentation to reflect changes in oonideckgen
by art＠torproject.org 29 Apr '16

29 Apr '16

commit b2dbd382c030c80cf09fcefe52ce5a1116bf0159 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Apr 26 21:42:26 2016 +0200 Update documentation to reflect changes in oonideckgen --- README.rst | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/README.rst b/README.rst index 384f71b..f137b88 100644 --- a/README.rst +++ b/README.rst @@ -58,7 +58,6 @@ To install it from the current master run:: Then run:: mkdir my_decks - sudo ooniresources --update-inputs --update-geoip oonideckgen -o my_decks/ **BUG** Note: @@ -174,17 +173,11 @@ To generate decks you will have to update the input resources of ooniprobe. This can be done with:: - ooniresources --update-inputs + ooniresources If you get a permission error, you may have to run the command as root or change the ooniprobe data directory inside of `ooniprobe.conf`. -On some platforms, for example debian contrib, you will not get all the geoip -related files needed. In that case it is possible to manually download them -with ``ooniresources``:: - - ooniresources --update-geoip - Generating decks ---------------- @@ -195,7 +188,7 @@ you can do so (be sure to have updated the input resources first) by running:: oonideckgen --country-code IT --output ~/ -You will now have in your home a folder called `deck-it`, containing the ooni +You will now have in your home a folder called `deck-YOUR_COUNTRY_CODE`, containing the ooni deck (ends with .deck) and the inputs. Note: that you should not move the `deck-*` directory once it has been generated as the paths to the inputs referenced by the test in the deck are

1 0

[ooni-probe/master] Print messages about how to use the boxes after they have both been deployed.
by art＠torproject.org 29 Apr '16

29 Apr '16

commit bc416be172786b7e58e5ad2d5da4ed09f2c452d3 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Apr 26 16:09:08 2016 +0200 Print messages about how to use the boxes after they have both been deployed. --- Vagrantfile | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/Vagrantfile b/Vagrantfile index e61bf68..3eeb54c 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -17,11 +17,6 @@ apt-get install -y build-essential libdumbnet-dev libpcap-dev libgeoip-dev libff cd /data/ooni-probe python setup.py install -echo "Now:" -echo "1. vagrant ssh probe" -echo "2. oonideckgen" -echo "3. ooniprobe -i deck-*/*.deck" - SCRIPT $setup_oonibackend = <<SCRIPT @@ -52,12 +47,6 @@ cp oonib.conf.example /etc/oonibackend.conf echo "Installing ooni-backend" python setup.py install -echo "Now:" -echo "1. vagrant ssh" -echo "2. vi /etc/oonibackend.conf # possibly" -echo "3. cd /data/ooni-backend" -echo "4. sudo ./bin/oonib -c /etc/oonibackend.conf" - SCRIPT Vagrant.configure("2") do |config| @@ -76,6 +65,22 @@ Vagrant.configure("2") do |config| backend.vm.synced_folder "../ooni-backend", "/data/ooni-backend" backend.vm.provision :shell, :inline => $setup_oonibackend end + + config.vm.provision "shell", inline: <<-EOF + echo "To run oonibackend:" + echo "1. vagrant ssh backend" + echo "2. vi /etc/oonibackend.conf # possibly" + echo "3. cd /data/ooni-backend" + echo "4. sudo ./bin/oonib -c /etc/oonibackend.conf" + EOF + end + config.vm.provision "shell", inline: <<-EOF + echo "To run ooniprobe:" + echo "1. vagrant ssh probe" + echo "2. oonideckgen" + echo "3. ooniprobe -i deck-*/*.deck" + EOF + end

1 0

[ooni-probe/master] Change the order in which decks get generated by default
by art＠torproject.org 29 Apr '16

29 Apr '16

commit 6d49b3fbbbebf824c106f2257d6c774839cd6703 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Apr 26 15:46:45 2016 +0200 Change the order in which decks get generated by default * Give priority to country specific lists * Give priority to faster tests * Run dns consistency for last --- ooni/deckgen/cli.py | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/ooni/deckgen/cli.py b/ooni/deckgen/cli.py index b1a6b87..664842e 100644 --- a/ooni/deckgen/cli.py +++ b/ooni/deckgen/cli.py @@ -71,17 +71,15 @@ class Deck(object): def generate_deck(options): - + url_list_country = None try: url_list_country = citizenlab_test_lists.generate_country_input( options['country-code'], options['output'] ) - except Exception: print "Could not generate country specific url list" print "We will just use the global one." - url_list_country = None url_list_global = citizenlab_test_lists.generate_global_input( options['output'] @@ -92,19 +90,19 @@ def generate_deck(options): ) deck = Deck() - # deck.add_test('manipulation/http_host', ['-f', 'somefile.txt']) + deck.add_test('manipulation/http_invalid_request_line') + deck.add_test('manipulation/http_header_field_manipulation') + + if url_list_country is not None: + deck.add_test('blocking/http_requests', ['-f', url_list_country]) deck.add_test('blocking/http_requests', ['-f', url_list_global]) - deck.add_test('blocking/dns_consistency', - ['-f', url_list_global, '-T', dns_servers]) if url_list_country is not None: deck.add_test('blocking/dns_consistency', ['-f', url_list_country, '-T', dns_servers]) - deck.add_test('blocking/http_requests', ['-f', url_list_country]) + deck.add_test('blocking/dns_consistency', + ['-f', url_list_global, '-T', dns_servers]) - deck.add_test('manipulation/http_invalid_request_line') - deck.add_test('manipulation/http_header_field_manipulation') - # deck.add_test('manipulation/traceroute') if config.advanced.debug: deck.pprint() deck_filename = os.path.join(options['output'],

1 0

[ooni-probe/master] Update manpages for ooniprobe, ooniresources, oonideckgen and oonireport.
by art＠torproject.org 29 Apr '16

29 Apr '16

commit a6897988fb96bae8816dd1bd7973a6e1781757f1 Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Apr 26 21:40:32 2016 +0200 Update manpages for ooniprobe, ooniresources, oonideckgen and oonireport. * Update versions of oonideckgen and ooniresources * Clean up the help text of ooniprobe options --- data/oonideckgen.1 | 7 +++++- data/ooniprobe.1 | 18 ++++++++++++---- data/oonireport.1 | 4 ++-- data/ooniresources.1 | 22 ++++++++----------- ooni/deckgen/__init__.py | 2 +- ooni/oonicli.py | 54 +++++++++++++++++++++++++++------------------- ooni/resources/__init__.py | 2 +- 7 files changed, 65 insertions(+), 44 deletions(-) diff --git a/data/oonideckgen.1 b/data/oonideckgen.1 index f808165..e2c8fed 100644 --- a/data/oonideckgen.1 +++ b/data/oonideckgen.1 @@ -35,6 +35,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] .B oonideckgen .RB [ --version ] .RB [ --help ] +.RB [ --collector +.IR httpo://collector-address.onion ] .RB [ \-c .IR country-code] .RB [ \-o @@ -56,6 +58,9 @@ Display this help and exit. .BR \-\^c " or " \-\-country-code Specify the country code for which a test deck should be geneated .TP +.BR \-\-collector +Specify the address of the collector to use for submitting reports. +.TP .BR \-\^o " or " \-\-output Specify in which directory the deck should be written. .TP @@ -81,5 +86,5 @@ In order to run deckgen you must have first have run ooniresources --update-inpu .SH AUTHOR The Tor Project .SH COPYRIGHT -2014, The Tor Project +2016, The Tor Project . diff --git a/data/ooniprobe.1 b/data/ooniprobe.1 index cf3f593..5fa47b4 100644 --- a/data/ooniprobe.1 +++ b/data/ooniprobe.1 @@ -1,6 +1,5 @@ -.\" Man page generated from reStructuredText. . -.TH "ooniprobe" "1" "April 29, 2014" "1.0.2" "ooniprobe" +.TH "ooniprobe" "1" "April 26, 2016" "1.4.0" "ooniprobe" .SH NAME ooniprobe - a network censorship measurement tool. . @@ -33,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] .SH SYNOPSIS .B ooniprobe -.RB [ \-hnsp ] +.RB [ \-hnspgN ] .RB [ --version ] .RB [ --spew ] .RB [ \-o @@ -52,6 +51,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] .IR configfile ] .RB [ \-d .IR datadir ] +.RB [ \-a +.IR annotations] .I "test_name" .SH DESCRIPTION @@ -69,6 +70,12 @@ Display this help and exit. .BR \-\^n " or " \-\-no\-collector Disable reporting the net test results to an oonib collector. .TP +.BR \-\^N " or " \-\-no\-yamloo +Disable writing report file to disk. +.TP +.BR \-\^g " or " \-\-no\-geoip +Disable performing geoip lookup. +.TP .BR \-\^s " or " \-\-list List all of the available net tests. .TP @@ -103,6 +110,9 @@ Specify a path to the ooniprobe configuration file. .BR \-\^d " or " \-\-datadir Specify a path to the ooniprobe data directory .TP +.BR \-\^a " or " \-\-annotations +Annotate the report with a key:value[, key:value] format. +.TP .BR \-\-spew Print an insanely verbose log of everything that happens. Useful when debugging freezes or locks in complex code. @@ -1096,5 +1106,5 @@ require for a researcher to assess what is the output of a test. .SH AUTHOR The Tor Project .SH COPYRIGHT -2014, The Tor Project +2016, The Tor Project . diff --git a/data/oonireport.1 b/data/oonireport.1 index 7910e67..8a91009 100644 --- a/data/oonireport.1 +++ b/data/oonireport.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH "oonireport" "1" "June 26, 2014" "1.0.2" "oonireport" +.TH "oonireport" "1" "April 26, 2016" "0.1.0" "oonireport" .SH NAME oonireport - tool for managing ooniprobe reports. . @@ -96,5 +96,5 @@ of the collector did not support the specified test. .SH AUTHOR The Tor Project .SH COPYRIGHT -2014, The Tor Project +2016, The Tor Project . diff --git a/data/ooniresources.1 b/data/ooniresources.1 index 5cd66cf..1e16822 100644 --- a/data/ooniresources.1 +++ b/data/ooniresources.1 @@ -1,8 +1,8 @@ .\" Man page generated from reStructuredText. . -.TH "ooniresources" "1" "October 1, 2014" "1.1.4" "ooniresources" +.TH "ooniresources" "1" "April 26, 2016" "0.2.0" "ooniresources" .SH NAME -ooniresources - tool for generating ooniprobe test decks +ooniresources - tool for updating resources for OONI tests . .nr rst2man-indent-level 0 . @@ -35,14 +35,15 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] .B ooniresources .RB [ --version ] .RB [ --help ] -.RB [ --update-inputs ] -.RB [ --update-geoip ] .SH DESCRIPTION .sp ooniresources will update the resources that are needed for the functioning of ooniprobe and related tools. +By invoking this command with no arguments the inputs and GeoIP data files +shall be updated. + .SH OPTIONS .TP @@ -50,11 +51,11 @@ ooniprobe and related tools. Display this help and exit. .TP .BR \-\-update\-inputs -This will update the inputs that are used for generating test decks with +(DEPRECATED) This will update the inputs that are used for generating test decks with oonideckgen. .TP .BR \-\-update\-geoip -This will update the maxmind geoip databases that are used for detecting the +(DEPRECATED) This will update the maxmind geoip databases that are used for detecting the probes country and ASN. .TP @@ -63,16 +64,11 @@ probes country and ASN. This tools will update the resources needed for running ooniprobe and oonideckgen. You don't need to run it very often, but it's just sufficient to run it once -every month of so as the data usually does not update or change that often. -If you are seeing some errors related to the geoip subsystem not having some -Maxmind databases, you probably want to run: -ooniresources --update-geoip. -If on the other hand you are interested in using oonideckgen, you should run: -ooniresources --update-inputs. +every month or so as the data usually does not update or change that often. .sp .SH AUTHOR The Tor Project .SH COPYRIGHT -2014, The Tor Project +2016, The Tor Project . diff --git a/ooni/deckgen/__init__.py b/ooni/deckgen/__init__.py index 3dc1f76..d3ec452 100644 --- a/ooni/deckgen/__init__.py +++ b/ooni/deckgen/__init__.py @@ -1 +1 @@ -__version__ = "0.1.0" +__version__ = "0.2.0" diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 4da81bc..09936b6 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -25,31 +25,41 @@ class Options(usage.Options): " files listed on the command line.") optFlags = [["help", "h"], - ["no-collector", "n", "disable writing to collector"], - ["no-yamloo", "N", "disable writing to YAML file"], - ["no-geoip", "g"], - ["list", "s"], - ["printdeck", "p"], - ["verbose", "v"] + ["no-collector", "n", "Disable writing to collector"], + ["no-yamloo", "N", "Disable writing to YAML file"], + ["no-geoip", "g", "Disable geoip lookup on start"], + ["list", "s", "List the currently installed ooniprobe " + "nettests"], + ["printdeck", "p", "Print the equivalent deck for the " + "provided command"], + ["verbose", "v", "Show more verbose information"] ] optParameters = [ - ["reportfile", "o", None, "report file name"], - ["testdeck", "i", None, - "Specify as input a test deck: a yaml file containing the tests to run and their arguments"], - ["collector", "c", None, - "Address of the collector of test results. This option should not be used, but you should always use a bouncer."], - ["bouncer", "b", 'httpo://nkvphnp3p6agi5qq.onion', - "Address of the bouncer for test helpers."], - ["logfile", "l", None, "log file name"], - ["pcapfile", "O", None, "pcap file name"], - ["configfile", "f", None, - "Specify a path to the ooniprobe configuration file"], - ["datadir", "d", None, - "Specify a path to the ooniprobe data directory"], - ["annotations", "a", None, - "Annotate the report with a key:value[, key:value] format."], - ["queue", "Q", None, "AMQP Queue URL amqp://user:pass@host:port/vhost/queue"]] + ["reportfile", "o", None, "Specify the report file name to write to."], + ["testdeck", "i", None, "Specify as input a test deck: a yaml file " + "containing the tests to run and their " + "arguments."], + ["collector", "c", None, "Specify the address of the collector for " + "test results. In most cases a user will " + "prefer to specify a bouncer over this."], + ["bouncer", "b", 'httpo://nkvphnp3p6agi5qq.onion', "Specify the " + "bouncer used to " + "obtain the " + "address of the " + "collector and " + "test helpers."], + ["logfile", "l", None, "Write to this logs to this filename."], + ["pcapfile", "O", None, "Write a PCAP of the ooniprobe session to " + "this filename."], + ["configfile", "f", None, "Specify a path to the ooniprobe " + "configuration file."], + ["datadir", "d", None, "Specify a path to the ooniprobe data " + "directory."], + ["annotations", "a", None, "Annotate the report with a key:value[, " + "key:value] format."], + ["queue", "Q", None, "AMQP Queue URL amqp://user:pass@host:port/vhost/queue"] + ] compData = usage.Completions( extraActions=[usage.CompleteFiles( diff --git a/ooni/resources/__init__.py b/ooni/resources/__init__.py index 8b7e63b..4b163bb 100644 --- a/ooni/resources/__init__.py +++ b/ooni/resources/__init__.py @@ -1,6 +1,6 @@ from ooni import __version__ as ooniprobe_version -__version__ = "0.1.0" +__version__ = "0.2.0" ooni_resources_url = ("https://github.com/TheTorProject/ooni-probe/releases" "/download/v{}/ooni-resources.tar.gz").format(ooniprobe_version)

1 0

[ooni-probe/master] Remove obsolete install method on Debian/Ubuntu
by art＠torproject.org 29 Apr '16

29 Apr '16

commit f73a26aa1c2d62a2e2ebe0c335ac29da4d4c4702 Author: anadahz <kojgelo(a)inbox.com> Date: Fri Mar 11 00:12:38 2016 -0300 Remove obsolete install method on Debian/Ubuntu Currently the ooniprobe packages offered in Debian and Ubuntu are old, unavailable in some releases or very old version. Removing obsolete install instructions until the ooniprobe packages in the repositories are available and up-to-date --- README.rst | 32 -------------------------------- 1 file changed, 32 deletions(-) diff --git a/README.rst b/README.rst index 59c7afa..6234b62 100644 --- a/README.rst +++ b/README.rst @@ -47,15 +47,6 @@ probe is a liability for you, please be aware of this risk. OONI in 5 minutes ================= -On debian testing or unstable:: - - sudo apt-get install ooniprobe - -If you are running debian stable you can get it from backports via:: - - sudo sh -c 'echo "deb http://http.debian.net/debian jessie-backports main" >> /etc/apt/sources.list' - sudo apt-get update && sudo apt-get install ooniprobe - On unix systems:: sudo pip install ooniprobe @@ -94,29 +85,6 @@ Run this command to automatically update your crontab:: Installation ============ -Debian based systems --------------------- - -If you are running Debian testing or Debian unstable you can install ooniprobe -simply with:: - - apt-get install ooniprobe - -If you are running Debian stable you can get it from backports via:: - - sudo sh -c 'echo "deb http://http.debian.net/debian wheezy-backports main" >> /etc/apt/sources.list' - sudo apt-get update && sudo apt-get install ooniprobe - -If you are running Ubuntu 14.04 LTS, 14.10 or 15.04 you can install it from the PPA -(https://launchpad.net/~irl/+archive/ubuntu/ooni/):: - - sudo add-apt-repository ppa:irl/ooni - sudo apt-get update && sudo apt-get install ooniprobe - -You will be warned that the packages are unauthenticated. This is due to the -PPA not being signed and is normal behaviour. If you would prefer to verify the -integrity of the package, use our private Debian repository below. - Mac OS X --------

1 0