June 2013 - tor-commits - lists.torproject.org

[ooni-probe/develop] Fix tls_handshake input processing
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 50cf01773f820354a01cdc3681cc6e03b2f0729b Author: aagbsn <aagbsn(a)extc.org> Date: Tue Apr 30 19:32:33 2013 +0800 Fix tls_handshake input processing --- nettests/experimental/tls_handshake.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index 666e961..83b2868 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -133,6 +133,7 @@ class HandshakeTest(nettest.NetTestCase): requiresRoot = False usageOptions = HandshakeOptions + host = None inputFile = ['file', 'f', None, 'List of <IP>:<PORT>s to test'] #: Default SSL/TLS context method. @@ -141,7 +142,6 @@ class HandshakeTest(nettest.NetTestCase): def setUp(self, *args, **kwargs): """Set defaults for a :class:`HandshakeTest <HandshakeTest>`.""" - self.host = None self.ciphers = list() if self.localOptions: @@ -772,11 +772,9 @@ class HandshakeTest(nettest.NetTestCase): def deferMakeConnection(host): return threads.deferToThread(makeConnection, self.input) - - log.msg("Beginning handshake test for %s" % (self.input or self.host)) - if self.host and not self.input: self.input = self.splitInput(self.host) + log.msg("Beginning handshake test for %s:%s" % self.input) connection = deferMakeConnection(self.input) connection.addCallbacks(connectionSucceeded, connectionFailed,

1 0

[ooni-probe/develop] Merge pull request #76 from aagbsn/fix_import_rename
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 33ac20de5a4bb9e97f2fe8fbd081756a2022dd31 Merge: 875dc22 a96ceb2 Author: aagbsn <aagbsn(a)extc.org> Date: Tue Apr 30 03:51:41 2013 -0700 Merge pull request #76 from aagbsn/fix_import_rename Fix import conflict nettests/experimental/tls_handshake.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)

1 0

[ooni-probe/develop] Merge pull request #77 from aagbsn/fix_tls_handshake_input
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 4fdbd6214a8f09d56de0598146d3f6ce760e8d77 Merge: 33ac20d 50cf017 Author: aagbsn <aagbsn(a)extc.org> Date: Tue Apr 30 04:35:00 2013 -0700 Merge pull request #77 from aagbsn/fix_tls_handshake_input Fix tls_handshake input processing nettests/experimental/tls_handshake.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)

1 0

[ooni-probe/develop] Swallow the report creation failure so that it does not propagate
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit df06507a89a5878ede116228f3016f8ecdc58155 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Apr 30 15:00:32 2013 +0200 Swallow the report creation failure so that it does not propagate This fixes: https://trac.torproject.org/projects/tor/ticket/8803 --- ooni/reporter.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/ooni/reporter.py b/ooni/reporter.py index 732d516..05d59d6 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -394,7 +394,6 @@ class Report(object): for reporter in self.reporters[:]: def report_created(result): - print "CREATED REPORT %s" % reporter log.debug("Created report with %s" % reporter) self._reporters_openned += 1 are_all_openned() @@ -406,7 +405,7 @@ class Report(object): all_openned.errback(defer.fail(e)) else: are_all_openned() - return failure + return d = defer.maybeDeferred(reporter.createReport) d.addCallback(report_created)

1 0

[ooni-probe/develop] Merge branch 'master' of github.com:TheTorProject/ooni-probe
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 865ec32d230e18321df3ab006b8eb5fff747f9e0 Merge: df06507 4fdbd62 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Apr 30 15:01:06 2013 +0200 Merge branch 'master' of github.com:TheTorProject/ooni-probe * 'master' of github.com:TheTorProject/ooni-probe: (61 commits) Fix tls_handshake input processing Fix import conflict Set self.input to self.host if no inputFile, and update several log messages. Reformat docstrings for several HandshakeTest functions to Sphinx format. Rename HandshakeTest.test_tlsv1_handshake() to HandshakeTest.test_handshake(). Cleanup docstrings for getX509Name() and getPublicKey() functions. Add Sphinx docstring for getPeerCert() function. Move to call IPAddress explicitly from ipaddr.IPAdress. Add self.host to the inputProcessor(). If no ciphersuite list is provided, explain that we're using firefox's. Fix bug where conflicting SSL context choices was not explained to the user. Fix #8440, set the self.host attribute when given the --host option. Create a host attribute for storing the --host option, and set self.ciphers. Make 'context' be a class-level attribute and give it a Sphinx :attr: docstring. Bump the version number on HandshakeTest, and rename the usageOptions attribute. Rename TLSHandshakeTest to HandshakeTest, since it's not only TLS. Remove UsageOptions to HandshakeOptions, and make arg help strings more informative. Fix HostUnreachable docstring to have proper Sphinx formatting. Add dictionary of pre-defined error messages to SSLContextError. Update docstring for SSLContextError. ... .gitignore | 2 + nettests/experimental/tls_handshake.py | 794 ++++++++++++++++++++++++++++++++ ooni/utils/txscapy.py | 1 + 3 files changed, 797 insertions(+)

1 0

[ooni-probe/develop] Set pcap name for each NetTest, and use consistent timestamps.
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 9fc621ad23abf7ec9c8344741d2f2ba401d78b5c Author: aagbsn <aagbsn(a)extc.org> Date: Fri May 3 13:38:02 2013 +0800 Set pcap name for each NetTest, and use consistent timestamps. --- ooni/config.py | 23 ++++------------------- ooni/director.py | 20 ++++++++++++-------- ooni/nettest.py | 5 ++--- ooni/otime.py | 2 ++ 4 files changed, 20 insertions(+), 30 deletions(-) diff --git a/ooni/config.py b/ooni/config.py index 671f80f..103ace7 100644 --- a/ooni/config.py +++ b/ooni/config.py @@ -122,22 +122,7 @@ if not resume_filename: with open(resume_filename, 'w+') as f: pass -def generatePcapFilename(cmd_line_options=None): - if not cmd_line_options: - cmd_line_options = {} - - if 'pcapfile' in cmd_line_options: - pcap_filename = cmd_line_options['pcapfile'] - else: - if 'test' in cmd_line_options: - test_filename = os.path.basename(cmd_line_options['test']) - elif 'testdeck' in cmd_line_options: - test_filename = os.path.basename(cmd_line_options['testdeck']) - else: - test_filename = '' - - test_name = '.'.join(test_filename.split(".")[:-1]) - frm_str = "report_%s_" + otime.timestamp() + ".%s" - pcap_filename = frm_str % (test_name, "pcap") - - return pcap_filename +def generatePcapFilename(testDetails): + test_name, start_time = testDetails['test_name'], testDetails['start_time'] + start_time = otime.epochToTimestamp(start_time) + return "report-%s-%s.%s" % (test_name, start_time, "pcap") diff --git a/ooni/director.py b/ooni/director.py index 7ad99a9..93134af 100644 --- a/ooni/director.py +++ b/ooni/director.py @@ -86,14 +86,10 @@ class Director(object): # This deferred is fired once all the measurements and their reporting # tasks are completed. self.allTestsDone = defer.Deferred() + self.sniffer = None @defer.inlineCallbacks def start(self): - if config.privacy.includepcap: - log.msg("Starting") - if not config.reports.pcap: - config.reports.pcap = config.generatePcapFilename() - self.startSniffing() if config.advanced.start_tor: log.msg("Starting Tor...") @@ -192,6 +188,13 @@ class Director(object): _: #XXX very dirty hack """ + + if config.privacy.includepcap: + log.msg("Starting") + if not config.reports.pcap: + config.reports.pcap = config.generatePcapFilename(net_test_loader.testDetails) + self.startSniffing() + report = Report(reporters, self.reportEntryManager) net_test = NetTest(net_test_loader, report) @@ -228,9 +231,10 @@ class Director(object): print "Renaming files with such name..." pushFilenameStack(config.reports.pcap) - sniffer = ScapySniffer(config.reports.pcap) - config.scapyFactory.registerProtocol(sniffer) - + if self.sniffer: + config.scapyFactory.unRegisterProtocol(self.sniffer) + self.sniffer = ScapySniffer(config.reports.pcap) + config.scapyFactory.registerProtocol(self.sniffer) def startTor(self): """ Starts Tor diff --git a/ooni/nettest.py b/ooni/nettest.py index 50c0cf0..cd8ab05 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -9,8 +9,7 @@ from ooni import geoip from ooni.tasks import Measurement from ooni.utils import log, checkForRoot, geodata from ooni import config -from ooni import otime - +import time from ooni import errors as e from inspect import getmembers @@ -69,7 +68,7 @@ class NetTestLoader(object): or ('countrycode' not in client_geodata): client_geodata['countrycode'] = None - test_details = {'start_time': otime.utcTimeNow(), + test_details = {'start_time': time.time(), 'probe_asn': client_geodata['asn'], 'probe_cc': client_geodata['countrycode'], 'probe_ip': client_geodata['ip'], diff --git a/ooni/otime.py b/ooni/otime.py index 84758eb..0446d4d 100644 --- a/ooni/otime.py +++ b/ooni/otime.py @@ -87,3 +87,5 @@ def timestamp(t=None): ISO8601 = "%Y-%m-%dT%H%M%SZ" return t.strftime(ISO8601) +def epochToTimestamp(seconds): + return timestamp(datetime.utcfromtimestamp(seconds))

1 0

[ooni-probe/develop] Add inputProcessor to http_host
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 57a300305cd569f444d641cf1b25237e1e4b9be9 Author: aagbsn <aagbsn(a)extc.org> Date: Fri May 3 14:28:25 2013 +0800 Add inputProcessor to http_host --- nettests/manipulation/http_host.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/nettests/manipulation/http_host.py b/nettests/manipulation/http_host.py index d95d836..3f1e0c6 100644 --- a/nettests/manipulation/http_host.py +++ b/nettests/manipulation/http_host.py @@ -139,3 +139,13 @@ class HTTPHost(httpt.HTTPTest): self.report['transparent_http_proxy'] = True self.check_for_censorship(body) + def inputProcessor(self, filename=None): + """ + This inputProcessor extracts domain names from urls + """ + if filename: + fp = open(filename) + for x in fp.readlines(): + yield x.strip().split('//')[-1].split('/')[0] + fp.close() + else: pass

1 0

[ooni-probe/develop] Fix markup of code
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 79a538eff2acd272c02a04c252c4dafab4799c03 Author: Arturo Filastò <hellais(a)gmail.com> Date: Sun May 5 03:09:20 2013 +0300 Fix markup of code --- README.md | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 576dd85..7dda15a 100644 --- a/README.md +++ b/README.md @@ -19,8 +19,11 @@ network data that will assist you with your endeavors! ## Getting started with ooniprobe is easy 0) Open your favorite terminal + 1) Grab the source and install the dependencies + 2) Run ooniprobe! + 3) ... There is no step three ... ## The easy way to prep your system for running ooniprobe @@ -29,17 +32,21 @@ We believe that ooniprobe runs reasonably well on Debian GNU/Linux wheezy as well as versions of Ubuntu such as natty and later releases. Running ooniprobe without installing it is supported with the following commands: - git clone https://git.torproject.org/ooni-probe.git - cd ooni-probe - ./setup-dependencies.sh - ./bin/ooniprobe --asciilulz +``` +git clone https://git.torproject.org/ooni-probe.git +cd ooni-probe +./setup-dependencies.sh +./bin/ooniprobe --asciilulz +``` ## Your first network test We run ooniprobe with a test deck - this is a collection of tests in a single file that tells ooniprobe how to run and what data to check or process: - ./bin/ooniprobe -i decks/before_i_commit.testdeck +``` +./bin/ooniprobe -i decks/before_i_commit.testdeck +``` The report output files from the above command will be located in the reports/ directory of the source code checkout. The report output ends with the .yamloo @@ -64,7 +71,9 @@ Basic system requirements: On Debian or Ubuntu GNU/Linux based systems these can be installed with: - sudo apt-get install git-core python python-pip python-dev build-essential tor tor-geoipdb tcpdump +``` +sudo apt-get install git-core python python-pip python-dev build-essential tor tor-geoipdb tcpdump +``` Other packages that may be of interest:

1 0

[ooni-probe/develop] Add URL inputProcessor to tcpconnect
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 16f41c0f74f9d61b7e8439c719d73641da3e24c8 Author: aagbsn <aagbsn(a)extc.org> Date: Sat May 4 16:49:51 2013 +0800 Add URL inputProcessor to tcpconnect --- nettests/blocking/tcpconnect.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/nettests/blocking/tcpconnect.py b/nettests/blocking/tcpconnect.py index 3b22427..11c5558 100644 --- a/nettests/blocking/tcpconnect.py +++ b/nettests/blocking/tcpconnect.py @@ -43,3 +43,21 @@ class TCPConnectTest(nettest.NetTestCase): d.addErrback(connectionFailed) return d + def inputProcessor(self, filename=None): + """ + This inputProcessor extracts name:port pairs from urls + XXX: Does not support unusual port numbers + """ + if filename: + fp = open(filename) + for x in fp.readlines(): + proto, path = x.strip().split('://') + proto = proto.lower() + host = path.split('/')[0] + if proto == 'http': + yield "%s:80" % host + if proto == 'https': + yield "%s:443" % host + fp.close() + else: + pass

1 0

[ooni-probe/develop] Fix markup of README file
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 4bcac69969d48028f59e95d675781b963e15e5d2 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun May 5 16:30:14 2013 +0200 Fix markup of README file --- README.md | 115 ++++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 76 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index 7dda15a..75095ad 100644 --- a/README.md +++ b/README.md @@ -77,8 +77,10 @@ sudo apt-get install git-core python python-pip python-dev build-essential tor t Other packages that may be of interest: - libdumbnet1 python-dumbnet python-libpcap python-pypcap python-pcapy python-dnspython - python-virtualenv virtualenvwrapper tor tor-geoipdb +``` +libdumbnet1 python-dumbnet python-libpcap python-pypcap python-pcapy python-dnspython +python-virtualenv virtualenvwrapper tor tor-geoipdb +``` The Python dependencies required for running ooniprobe are: @@ -95,7 +97,7 @@ The Python dependencies required for running ooniprobe are: Install the latest version of Tor for your platform: - https://www.torproject.org/download/download.html +[Download Tor](https://www.torproject.org/download/download.html) ## Configurating a virtual environment @@ -105,46 +107,62 @@ to install it system wide. This will require you to have installed virtualenv. - sudo apt-get install python-virtualenv virtualenvwrapper +``` +sudo apt-get install python-virtualenv virtualenvwrapper +``` To create a new virtual environment do - mkdir $HOME/.virtualenvs - mkvirtualenv ooni-probe +``` +mkdir $HOME/.virtualenvs +mkvirtualenv ooni-probe +``` You will automatically enter the environment. To re-enter this environment in the future, type: - workon ooni-probe +``` +workon ooni-probe +``` For convenience, you may want to add the following to your .bashrc: - if [ -e ~/ooni-probe/bin ]; then - export PATH=~/ooni-probe/bin:$PATH - fi - if [ -e ~/ooni-probe ]; then - export PYTHONPATH=$PYTHONPATH:~/ooni-probe - fi +``` +if [ -e ~/ooni-probe/bin ]; then + export PATH=~/ooni-probe/bin:$PATH +fi +if [ -e ~/ooni-probe ]; then + export PYTHONPATH=$PYTHONPATH:~/ooni-probe +fi +``` Add the following to $HOME/.virtualenvs/ooni-probe/bin/postactivate to automatically cd into the working directory upon activation. - if [ -e ~/ooni-probe ] ; then - cd ~/ooni-probe - fi +``` +if [ -e ~/ooni-probe ] ; then + cd ~/ooni-probe +fi +``` ## Installing ooni-probe Clone the ooniprobe repository: - git clone https://git.torproject.org/ooni-probe.git - cd ooni-probe +``` +git clone https://git.torproject.org/ooni-probe.git +cd ooni-probe +``` Then install OONI with: - pip install -r requirements.txt +``` +pip install -r requirements.txt +``` If you are not in a virtualenv you will have to run the above command as root: - sudo pip install -r requirements.txt +``` +sudo pip install -r requirements.txt +``` ## Install libdnet and pypcap python bindings @@ -156,25 +174,31 @@ libdnet-1.12 and pypcap 1.1, any other version should be considered untested. If you don't already have Subversion installed: - sudo apt-get install subversion +``` +sudo apt-get install subversion +``` For libdnet: - wget https://libdnet.googlecode.com/files/libdnet-1.12.tgz - tar xzf libdnet-1.12.tgz - cd libdnet-1.12 - ./configure && make - cd python/ - python setup.py install - cd ../../ && rm -rf libdnet-1.12* +``` +wget https://libdnet.googlecode.com/files/libdnet-1.12.tgz +tar xzf libdnet-1.12.tgz +cd libdnet-1.12 +./configure && make +cd python/ +python setup.py install +cd ../../ && rm -rf libdnet-1.12* +``` For pypcap: - git clone https://github.com/hellais/pypcap - cd pypcap/ - pip install pyrex - make && make install - cd ../ && rm -rf pypcap-read-only +``` +git clone https://github.com/hellais/pypcap +cd pypcap/ +pip install pyrex +make && make install +cd ../ && rm -rf pypcap-read-only +``` ## Including your geo data in the test report @@ -185,28 +209,38 @@ inside of ooniprobe.conf If you wish to include geografical data in the test report, you will have to go to the data/ directory and run: - make geoip +``` +make geoip +``` Then edit your ooniprobe.conf to point to the absolute path of where the data/ directory is located for example: - geoip_data_dir: /home/your_user/ooni-probe/data/ +``` +geoip_data_dir: /home/your_user/ooni-probe/data/ +``` ## Running some tests To see the possible command line options run: - ./bin/ooniprobe --help +``` +./bin/ooniprobe --help +``` For interesting tests to run look in the nettests/core/ directory. To run a test you can do so with: - ./bin/ooniprobe -o report_file_name path/to/test.py +``` +./bin/ooniprobe -o report_file_name path/to/test.py +``` Normally tests take options, you can see them with: - ./bin/ooniprobe -o report_file_name path/to/test.py --help +``` +./bin/ooniprobe -o report_file_name path/to/test.py --help +``` ## Configuration @@ -218,4 +252,7 @@ editing your ooniprobe.conf configuration file. If your distributation supports capabilities you can avoid needing to run OONI as root: - setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python +``` +setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python +``` +

1 0