January 2014 - tor-commits - lists.torproject.org

[ooni-probe/master] Update the debian packaging related files
by art＠torproject.org 13 Jan '14

13 Jan '14

commit 3d2a7598b1b99035584d90e7821945b81bf729a7 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Jan 12 18:22:40 2014 +0100 Update the debian packaging related files --- debian/changelog | 2 +- debian/control | 20 ++++++++++++++++---- debian/rules | 2 ++ requirements.txt | 4 +--- 4 files changed, 20 insertions(+), 8 deletions(-) diff --git a/debian/changelog b/debian/changelog index 05b6f6d..7d66611 100644 --- a/debian/changelog +++ b/debian/changelog @@ -2,4 +2,4 @@ ooniprobe (1.0.0-rc5-1) UNRELEASED; urgency=low * Initial release. (Closes: #XXXXXX) - -- Open Observatory of Network Interference <ooni-dev(a)torproject.org> Fri, 3 Jan 2014 11:14:46 +0200 + -- OONI Developers <ooni-dev(a)lists.torproject.org> Fri, 3 Jan 2014 11:14:46 +0200 diff --git a/debian/control b/debian/control index 60f42cb..b5fcaec 100644 --- a/debian/control +++ b/debian/control @@ -4,11 +4,10 @@ Section: net Priority: extra Build-Depends: debhelper (>= 8), python-all (>= 2.6.6-3~), - python-dnspython (>= 1.10.0), + python-dnspython, python-docutils (>= 0.9.1), - python-geoip (>= 0.2.5), python-ipaddr (>= 2.1.10), - python-openssl (>= 0.13), + python-openssl, python-pyrex, python-scapy (>= 2.2), python-setuptools, @@ -27,7 +26,20 @@ Package: ooniprobe Architecture: all Depends: ${misc:Depends}, ${python:Depends}, tcpdump, - tor + tor, + python-all (>= 2.6.6-3~), + python-dnspython, + python-geoip (>= 0.2.5), + python-ipaddr (>= 2.1.10), + python-openssl, + python-pyrex, + python-scapy (>= 2.2), + python-setuptools, + python-twisted (>= 12.2.0), + python-txsocksx, + python-txtorcon (>= 0.7), + python-yaml (>= 3.10), + python-zope.interface (>= 3.6), Suggests: obfsproxy Description: probe for the Open Observatory of Network Interference (OONI) OONI, the Open Observatory of Network Interference, is a global observation diff --git a/debian/rules b/debian/rules index 579a1b1..9f48e79 100755 --- a/debian/rules +++ b/debian/rules @@ -2,3 +2,5 @@ %: dh $@ --buildsystem=python_distutils --with=python2 +override_dh_python2: + dh_python2 --no-guessing-deps diff --git a/requirements.txt b/requirements.txt index 638b33b..ca46ca9 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,9 +4,7 @@ argparse>=1.2.1 docutils>=0.9.1 ipaddr>=2.1.10 pyOpenSSL>=0.13 -# XXX fix this once the following issue is resolved: https://github.com/appliedsec/pygeoip/issues/56 -https://github.com/appliedsec/pygeoip/archive/master.zip#egg=pygeoip-0.3.1 -pygeoip>=0.3.1 +geoip txtorcon>=0.7 txsocksx>=0.0.2 Pyrex>=0.9.8.6

1 0

[ooni-probe/master] Install the geoip data files to a different directory
by art＠torproject.org 13 Jan '14

13 Jan '14

commit f0edbbd8e0a2af12b51c24de5d4455deec174623 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Jan 12 18:40:12 2014 +0100 Install the geoip data files to a different directory --- data/ooniprobe.conf.sample | 2 +- setup.py | 5 ++--- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/data/ooniprobe.conf.sample b/data/ooniprobe.conf.sample index 6fa7caf..4ce1e19 100644 --- a/data/ooniprobe.conf.sample +++ b/data/ooniprobe.conf.sample @@ -21,7 +21,7 @@ reports: pcap: null collector: 'httpo://nkvphnp3p6agi5qq.onion' advanced: - geoip_data_dir: /usr/share/GeoIP + geoip_data_dir: /usr/share/ooni/geoip debug: false # enable if auto detection fails #tor_binary: /usr/sbin/tor diff --git a/setup.py b/setup.py index 5f97f42..0082365 100644 --- a/setup.py +++ b/setup.py @@ -68,14 +68,13 @@ if hasattr(sys, 'real_prefix'): )] else: data_files = [( - '/usr/share/GeoIP/', + '/usr/share/ooni/geoip/', [ + 'data/GeoIP.dat', 'data/GeoIPASNum.dat', 'data/GeoLiteCity.dat' ] )] - if not os.path.exists('/usr/share/GeoIP/GeoIP.dat'): - data_files[0][1].append('data/GeoIP.dat') for root, dirs, file_names in os.walk('data/'): files = []

1 0

[ooni-probe/master] Make the defaults paths relative to the ooni home directory.
by art＠torproject.org 13 Jan '14

13 Jan '14

commit 575ecd79e8428e413d4952e19f618be6b8c6ff13 Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Jan 13 12:53:20 2014 +0100 Make the defaults paths relative to the ooni home directory. --- data/ooniprobe.conf.sample | 2 +- ooni/deck.py | 7 ++----- ooni/oonibclient.py | 1 - ooni/settings.py | 6 ++++++ ooni/utils/log.py | 2 +- 5 files changed, 10 insertions(+), 8 deletions(-) diff --git a/data/ooniprobe.conf.sample b/data/ooniprobe.conf.sample index 4ce1e19..3a1b6f4 100644 --- a/data/ooniprobe.conf.sample +++ b/data/ooniprobe.conf.sample @@ -4,7 +4,7 @@ basic: # Where OONIProbe should be writing it's log file - logfile: /var/log/ooniprobe.log + logfile: ~/.ooni/ooniprobe.log privacy: # Should we include the IP address of the probe in the report? includeip: false diff --git a/ooni/deck.py b/ooni/deck.py index 23bb9a1..faddfa4 100644 --- a/ooni/deck.py +++ b/ooni/deck.py @@ -16,9 +16,7 @@ from hashlib import sha256 class InputFile(object): def __init__(self, input_hash): self.id = input_hash - cached_input_dir = os.path.join(config.advanced.data_dir, - 'inputs') - cache_path = os.path.join(cached_input_dir, input_hash) + cache_path = os.path.join(config.inputs_directory, input_hash) self.cached_file = cache_path self.cached_descriptor = cache_path + '.desc' @@ -96,8 +94,7 @@ class Deck(InputFile): @property def cached_file(self): - cached_deck_dir = os.path.join(config.advanced.data_dir, 'decks') - return os.path.join(cached_deck_dir, self.deckHash) + return os.path.join(config.decks_directory, self.deckHash) @property def cached_descriptor(self): diff --git a/ooni/oonibclient.py b/ooni/oonibclient.py index 5735155..1fa2e5b 100644 --- a/ooni/oonibclient.py +++ b/ooni/oonibclient.py @@ -75,7 +75,6 @@ class OONIBClient(object): # we have reached the retry count. if attempts < self.retries: log.err("Lookup failed. Retrying.") - log.exception(err) attempts += 1 perform_request(attempts) else: diff --git a/ooni/settings.py b/ooni/settings.py index 7d8a7bd..de24a76 100644 --- a/ooni/settings.py +++ b/ooni/settings.py @@ -1,4 +1,5 @@ import os +import sys import yaml from shutil import copyfile from os.path import abspath, expanduser @@ -31,12 +32,16 @@ class OConfig(object): self.data_directory = abspath(expanduser(self.global_options['datadir'])) elif self.advanced.get('data_dir'): self.data_directory = self.advanced['data_dir'] + elif hasattr(sys, 'real_prefix'): + self.data_directory = os.path.abspath(os.path.join(sys.prefix, 'share', 'ooni')) else: self.data_directory = '/usr/share/ooni/' + self.nettest_directory = abspath(os.path.join(__file__, '..', 'nettests')) self.ooni_home = os.path.join(expanduser('~'), '.ooni') self.inputs_directory = os.path.join(self.ooni_home, 'inputs') + self.decks_directory = os.path.join(self.ooni_home, 'decks') self.reports_directory = os.path.join(self.ooni_home, 'reports') if self.global_options.get('configfile'): @@ -51,6 +56,7 @@ class OConfig(object): print "Creating it in '%s'." % self.ooni_home os.mkdir(self.ooni_home) os.mkdir(self.inputs_directory) + os.mkdir(self.decks_directory) if not os.path.isdir(self.reports_directory): os.mkdir(self.reports_directory) diff --git a/ooni/utils/log.py b/ooni/utils/log.py index 036a4b4..e0ad5d9 100644 --- a/ooni/utils/log.py +++ b/ooni/utils/log.py @@ -28,7 +28,7 @@ def start(logfile=None, application_name="ooniprobe"): daily_logfile = None if not logfile: - logfile = config.basic.logfile + logfile = os.path.expanduser(config.basic.logfile) log_folder = os.path.dirname(logfile) log_filename = os.path.basename(logfile)

1 0

[ooni-probe/master] Add two other required dependencies.
by art＠torproject.org 13 Jan '14

13 Jan '14

commit b3d40ac1471d2439e9d93a0f3849b45b13882a5e Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Jan 12 18:49:54 2014 +0100 Add two other required dependencies. --- debian/control | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/debian/control b/debian/control index b5fcaec..ba7281b 100644 --- a/debian/control +++ b/debian/control @@ -5,10 +5,11 @@ Priority: extra Build-Depends: debhelper (>= 8), python-all (>= 2.6.6-3~), python-dnspython, + python-dumbnet, python-docutils (>= 0.9.1), python-ipaddr (>= 2.1.10), python-openssl, - python-pyrex, + python-pypcap, python-scapy (>= 2.2), python-setuptools, python-twisted (>= 12.2.0), @@ -29,10 +30,11 @@ Depends: ${misc:Depends}, ${python:Depends}, tor, python-all (>= 2.6.6-3~), python-dnspython, + python-dumbnet, python-geoip (>= 0.2.5), python-ipaddr (>= 2.1.10), python-openssl, - python-pyrex, + python-pypcap, python-scapy (>= 2.2), python-setuptools, python-twisted (>= 12.2.0),

1 0

[ooni-probe/master] Use the system Tor in the debian package
by art＠torproject.org 13 Jan '14

13 Jan '14

commit e60ca5934b256f46ad47b3c2dce736bc075c9cf8 Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Jan 13 13:35:35 2014 +0100 Use the system Tor in the debian package --- scripts/build-ooniprobe.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/scripts/build-ooniprobe.sh b/scripts/build-ooniprobe.sh index 3e616e7..c4e412c 100755 --- a/scripts/build-ooniprobe.sh +++ b/scripts/build-ooniprobe.sh @@ -1,6 +1,9 @@ #!/bin/bash set -e cd ../ +cat data/ooniprobe.conf.sample | sed s/'start_tor: true'/'start_tor: false'/ | sed s/'#socks_port: 8801'/'socks_port: 9050'/ > data/ooniprobe.conf.sample.new +mv data/ooniprobe.conf.sample data/ooniprobe.conf.sample.bak +mv data/ooniprobe.conf.sample.new data/ooniprobe.conf.sample python setup.py sdist cd dist/ py2dsc ooniprobe-*.tar.gz @@ -8,3 +11,5 @@ cd deb_dist/ooniprobe-* rm -rf debian/ cp -rf ../../../debian ./debian debuild +cd ../../../ +mv data/ooniprobe.conf.sample.bak data/ooniprobe.conf.sample

1 0

[ooni-probe/master] Automatically determine the test resolvers to use by reading /etc/resolv.conf
by art＠torproject.org 13 Jan '14

13 Jan '14

commit 05953ba5d3a1f69bd93875ef1e8482349dfaac0b Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Jan 13 13:11:51 2014 +0100 Automatically determine the test resolvers to use by reading /etc/resolv.conf --- ooni/nettests/blocking/dnsconsistency.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/ooni/nettests/blocking/dnsconsistency.py b/ooni/nettests/blocking/dnsconsistency.py index f10a48c..49e5360 100644 --- a/ooni/nettests/blocking/dnsconsistency.py +++ b/ooni/nettests/blocking/dnsconsistency.py @@ -53,7 +53,12 @@ class DNSConsistencyTest(dnst.DNSTest): def setUp(self): if (not self.localOptions['testresolvers'] and \ not self.localOptions['testresolver']): - raise usage.UsageError("You did not specify a testresolver") + self.test_resolvers = [] + with open('/etc/resolv.conf') as f: + for line in f: + if line.startswith('nameserver'): + self.test_resolvers.append(line.split(' ')[1].strip()) + self.report['test_resolvers'] = self.test_resolvers elif self.localOptions['testresolvers']: test_resolvers_file = self.localOptions['testresolvers']

1 0

[ooni-probe/master] Only output fetching required test inputs once.
by art＠torproject.org 13 Jan '14

13 Jan '14

commit 79ad3987a48f628dc626145272d28699185fa861 Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Jan 13 13:50:43 2014 +0100 Only output fetching required test inputs once. --- ooni/deck.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ooni/deck.py b/ooni/deck.py index faddfa4..260f8b4 100644 --- a/ooni/deck.py +++ b/ooni/deck.py @@ -137,8 +137,8 @@ class Deck(InputFile): @defer.inlineCallbacks def setup(self): """ fetch and verify inputs for all NetTests in the deck """ + log.msg("Fetching required net test inputs...") for net_test_loader in self.netTestLoaders: - log.msg("Fetching required net test inputs...") yield self.fetchAndVerifyNetTestInput(net_test_loader) if self.bouncer:

1 0

[ooni-probe/master] Add the mlab default test deck
by art＠torproject.org 13 Jan '14

13 Jan '14

commit cc2f1ce49ca3dd69d51d02cc5773b368f6f8415c Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Jan 13 13:12:41 2014 +0100 Add the mlab default test deck --- data/decks/mlab.deck | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/data/decks/mlab.deck b/data/decks/mlab.deck new file mode 100644 index 0000000..5d5a299 --- /dev/null +++ b/data/decks/mlab.deck @@ -0,0 +1,60 @@ +- options: + collector: null + help: 0 + logfile: null + no-default-reporter: 0 + parallelism: '10' + pcapfile: null + reportfile: null + resume: 0 + subargs: [] + test_file: manipulation/http_invalid_request_line + testdeck: null +- options: + collector: null + help: 0 + logfile: null + no-default-reporter: 0 + parallelism: '10' + pcapfile: null + reportfile: null + resume: 0 + subargs: [] + test_file: manipulation/http_header_field_manipulation + testdeck: null +- options: + collector: null + help: 0 + logfile: null + no-default-reporter: 0 + parallelism: '10' + pcapfile: null + reportfile: null + resume: 0 + subargs: [] + test_file: manipulation/traceroute + testdeck: null +- options: + collector: null + help: 0 + logfile: null + no-default-reporter: 0 + parallelism: '10' + pcapfile: null + reportfile: null + resume: 0 + subargs: [-f, 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1'] + test_file: manipulation/http_host + testdeck: null +- options: + collector: null + help: 0 + logfile: null + no-default-reporter: 0 + parallelism: '10' + pcapfile: null + reportfile: null + resume: 0 + subargs: [-f, 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1'] + test_file: blocking/dnsconsistency + testdeck: null

1 0

[ooni-probe/master] Merge branch 'master' of github.com:TheTorProject/ooni-probe
by art＠torproject.org 13 Jan '14

13 Jan '14

commit 80ea75a906c2d6cefa88ca3b1b4e7e2cd5a9cc52 Merge: 79ad398 51c92f2 Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Jan 13 13:51:11 2014 +0100 Merge branch 'master' of github.com:TheTorProject/ooni-probe * 'master' of github.com:TheTorProject/ooni-probe: Update README.md README.md | 9 +++++++++ 1 file changed, 9 insertions(+)

1 0

[ooni-probe/master] Fix bug that emerges when using CGeoIP.
by art＠torproject.org 13 Jan '14

13 Jan '14

commit e82178bbf56c570878403023cdaa1d3a74f83067 Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Jan 13 14:14:02 2014 +0100 Fix bug that emerges when using CGeoIP. --- ooni/geoip.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ooni/geoip.py b/ooni/geoip.py index 6c539e9..a97a6f5 100644 --- a/ooni/geoip.py +++ b/ooni/geoip.py @@ -18,7 +18,7 @@ except ImportError: try: import GeoIP as CGeoIP def GeoIP(database_path, *args, **kwargs): - return CGeoIP.open(database_path) + return CGeoIP.open(database_path, CGeoIP.GEOIP_STANDARD) except ImportError: log.err("Unable to import pygeoip. We will not be able to run geo IP related measurements")

1 0