commit 3ed62967a0119e2bf3393be4af98111f534b88ed Author: Arturo Filastò <art@fuffa.org> Date: Sat Dec 22 00:56:26 2012 +0100 Revert back to 9212cba (0.0.9) that was the last known working state commits between 20cd1df9de76b54a77aac22e44f457272354c0cb and 8c0e47c66ca78d6a94beacd90b0bb07a00cde013 broke master in a way that is complicated to debug. Such commits should be resubmitted as a pull request on github so that I can review them. In future when making changes to core ooniprobe components, please submit a pull request on github so that I can review the changes and test them before merging. --- .gitignore | 9 +- .../_static/images/ooniprobe-architecture.png | Bin 96574 -> 0 bytes docs/source/architecture.rst | 144 +--------- nettests/experimental/bridge_reachability/echo.py | 150 +++------- .../experimental/bridge_reachability/tcpsyn.py | 191 ------------ ooni/inputunit.py | 20 +- ooni/lib/001-scapy_missing-exc.sh.patch | 78 ----- ooni/nettest.py | 83 ++---- ooni/oonicli.py | 105 ++----- ooni/reporter.py | 41 ++-- ooni/runner.py | 321 ++++++-------------- ooni/utils/__init__.py | 22 +- ooni/utils/geodata.py | 18 +- ooni/utils/hacks.py | 8 - ooni/utils/log.py | 44 +--- ooni/utils/net.py | 124 +------- ooni/utils/txscapy.py | 79 ++--- oonib/README.md | 94 ------ ooniprobe.conf.sample | 16 +- scripts/before_i_commit.sh | 7 +- 20 files changed, 287 insertions(+), 1267 deletions(-) diff --git a/.gitignore b/.gitignore index 40df4cf..4cb0775 100644 --- a/.gitignore +++ b/.gitignore @@ -1,12 +1,8 @@ *.pyc assets/* -build/* -docs/build/* -data/*.dat ENV/* -fluff/ +build/* package/* -pcaps/* proxy-lists/ip-cc.txt proxy-lists/ips.txt proxy-lists/italy-dns-ips.txt @@ -18,12 +14,9 @@ ooni/plugins/dropin.cache ooni/report*.yaml oonib/oonibackend.conf oonib/data/* -*.resume fluff/ pcaps/ report*.yaml docs/build/* data/*.dat ooniprobe.conf -report*.yaml -report*.yamloo diff --git a/docs/source/_static/images/ooniprobe-architecture.png b/docs/source/_static/images/ooniprobe-architecture.png deleted file mode 100644 index 06065fc..0000000 Binary files a/docs/source/_static/images/ooniprobe-architecture.png and /dev/null differ diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst index cb50402..7091c29 100644 --- a/docs/source/architecture.rst +++ b/docs/source/architecture.rst @@ -4,6 +4,8 @@ Architecture :Contact: art@torproject.org :Copyright: This document has been placed in the public domain. +.. [[Image(https://github.com/hellais/ooni-docsnspecs/raw/master/graphics/CnCView.png, width=550px, align=right)]] + The goal of this document is provide an overview of how ooni works, what are it's pieces and how they interact with one another. @@ -14,9 +16,6 @@ To get an idea of what is implemented and with what sort of quality see the The two main components of ooni are `oonib`_ and `ooniprobe`_. -.. image:: _static/images/ooniprobe-architecture.png - :width: 700px - ooniprobe --------- @@ -182,153 +181,24 @@ Draft API specification Through the ooniprobe API it will be possible to `start tests`_, `stop tests`_ and `monitor test progress`_. -List tests -.......... - -`GET /test` - -Shall return the list of available tests as an array. - -This is how a response looks like -:: - - [{'id': 'http_requests', - 'name': 'HTTP Requests Test', - 'description': 'This test perform a HTTP GET request for the / resource over the test network and over Tor', - 'type': [ 'blocking' ], - 'version': '0.1', - 'arguments': { - 'urllist': 'Specify the list of URLs to be used for the test' - } - }] - -*type* may be either **blocking** or **manipulation**. - Start tests ........... - -`POST /test/<test_id>/start` - -Is used to start a test with the specified test_id. - -Inside of the request you will specify the arguments supported by the test - -This is how a request could look like -:: - { - 'urllist': - ['http://google.com/', 'http://torproject.org/'] - } - -The server will then respond with the test object -:: - { - 'status': 'running', - 'percentage': 0, - 'current_input': 'http://google.com/', - 'urllist': - ['http://google.com/', 'http://torproject.org/'] - } - +.. TODO Stop tests ........... -`POST /test/<test_id>/stop` - -This will terminate the execution of the test with the specified test_id. - -The request may optionally contain a reason for stopping the test such as -:: - { - 'reason': 'some reason' - } +.. TODO Monitor test progress -..................... - -`GET /test/<test_id>` - -Will return the status of a test - -Like so for example -:: - { - 'status': 'running', - 'percentage': 0, - 'current_input': 'http://google.com/', - 'urllist': - ['http://google.com/', 'http://torproject.org/'] - } +...................... +.. TODO Implementation status ===================== -ooniprobe -......... - -**Reporting** - - * To flat YAML file: *alpha* - - * To remote httpo backend: *alpha* - -**Test templates** - - * HTTP test template: *alpha* - - * Scapy test template: *alpha* - - * DNS test template: *alpha* - - * TCP test template: *prototype* - -**Tests** - -To see the list of implemented tests see: -https://ooni.torproject.org/docs/#core-ooniprobe-tests - -**ooniprobe API** - - * Specification: *draft* - - * HTTP API: *not implemented* - -**ooniprobe HTML5/JS user interface** - - Not implemented. - -**ooniprobe build system** - - Not implemented. - -**ooniprobe command line interface** - - Implemented in alpha quality, though needs to be ported to use the HTTP based - API. - -oonib -..... - -**Collector** - - * collection of YAML reports to flat file: *alpha* - - * collection of pcap reports: *not implemented* - - * association of reports with test helpers: *not implemented* - -**Test helpers** - - * HTTP Return JSON Helper: *alpha* - - * DNS Test helper: *prototype* - - * Test Helper - collector mapping: *Not implemented* - - * TCP Test helper: *prototype* +.. TODO - * Daphn3 Test helper: *prototype* diff --git a/nettests/experimental/bridge_reachability/echo.py b/nettests/experimental/bridge_reachability/echo.py index 0f422ca..d4033dd 100644 --- a/nettests/experimental/bridge_reachability/echo.py +++ b/nettests/experimental/bridge_reachability/echo.py @@ -16,39 +16,34 @@ import os import sys from twisted.python import usage -from twisted.internet import reactor, defer, address +from twisted.internet import reactor, defer from ooni import nettest from ooni.utils import log, net, Storage, txscapy try: - from scapy.all import IP, ICMP - from scapy.all import sr1 - from ooni.utils import txscapy -except Exception, e: + from scapy.all import IP, ICMP + from scapy.all import sr1 + from ooni.lib import txscapy + from ooni.lib.txscapy import txsr, txsend + from ooni.templates.scapyt import BaseScapyTest +except: log.msg("This test requires scapy, see www.secdev.org/projects/scapy") - log.exception(e) class UsageOptions(usage.Options): - """ - Options for EchoTest. - - Note: 'count', 'size', and 'ttl' have yet to be implemented. - """ optParameters = [ ['dst', 'd', None, 'Host IP to ping'], ['file', 'f', None, 'File of list of IPs to ping'], - ['pcap', 'p', None, 'Save pcap to this file'], ['interface', 'i', None, 'Network interface to use'], - ['receive', 'r', True, 'Receive response packets'], - ['timeout', 't', 2, 'Seconds to wait if no response', int], ['count', 'c', 1, 'Number of packets to send', int], - ['size', 's', 56, 'Bytes to send in ICMP data field', int], - ['ttl', 'l', 25, 'Set the IP Time to Live', int]] + ['size', 's', 56, 'Number of bytes to send in ICMP data field', int], + ['ttl', 'l', 25, 'Set the IP Time to Live', int], + ['timeout', 't', 2, 'Seconds until timeout if no response', int], + ['pcap', 'p', None, 'Save pcap to this file'], + ['receive', 'r', True, 'Receive response packets']] class EchoTest(nettest.NetTestCase): """ - Basic ping test. This takes an input file containing one IP or hostname - per line. + xxx fill me in """ name = 'echo' author = 'Isis Lovecruft <isis@torproject.org>' @@ -60,22 +55,6 @@ class EchoTest(nettest.NetTestCase): #requiredOptions = ['dst'] def setUp(self, *a, **kw): - """ - Send an ICMP-8 packet to a host IP, and process the response. - - @param timeout: - Seconds after sending the last packet to timeout. - @param interface: - The interface to restrict listening to. - @param dst: - A single host to ping. - @param file: - A file of hosts to ping, one per line. - @param receive: - Whether or not to receive replies. Defaults to True. - @param pcap: - The file to save packet captures to. - """ self.destinations = {} if self.localOptions: @@ -90,7 +69,7 @@ class EchoTest(nettest.NetTestCase): iface = txscapy.getDefaultIface() except Exception, e: log.msg("No network interface specified!") - log.exception(e) + log.err(e) else: log.msg("Using system default interface: %s" % iface) self.interface = iface @@ -106,12 +85,13 @@ class EchoTest(nettest.NetTestCase): if not self.dst: if self.file: self.dstProcessor(self.file) - for address, details in self.destinations.items(): - for labels, data in details.items(): - if not 'response' in labels: - self.dst = details['dst_ip'] + for key, value in self.destinations.items(): + for label, data in value.items(): + if not 'ans' in data: + self.dst = label else: self.addDest(self.dst) + log.debug("self.dst is now: %s" % self.dst) log.debug("Initialization of %s test completed." % self.name) @@ -120,6 +100,8 @@ class EchoTest(nettest.NetTestCase): self.destinations[d] = {'dst_ip': d} def dstProcessor(self, inputfile): + from ipaddr import IPAddress + if os.path.isfile(inputfile): with open(inputfile) as f: for line in f.readlines(): @@ -127,74 +109,24 @@ class EchoTest(nettest.NetTestCase): continue self.addDest(line) - def build_packets(self): - """ - Construct a list of packets to send out. - """ - packets = [] - for dest, data in self.destinations.items(): - pkt = IP(dst=dest)/ICMP() - packets.append(pkt) - ## XXX if a domain was specified, we need a way to check that - ## its IP matches the one we're seeing in pkt.src - #try: - # address.IPAddress(dest) - #except: - # data['dst_ip'] = pkt.dst - return packets - def test_icmp(self): - """ - Send the list of ICMP packets. - - TODO: add end summary progress report for % answered, etc. - """ - try: - def nicely(packets): - """Print scapy summary nicely.""" - return list([x.summary() for x in packets]) - - def process_answered((answered, sent)): - """Callback function for txscapy.sr().""" - self.report['sent'] = nicely(sent) - self.report['answered'] = [nicely(ans) for ans in answered] - - for req, resp in answered: - log.msg("Received echo-reply:\n%s" % resp.summary()) - for dest, data in self.destinations.items(): - if data['dst_ip'] == resp.src: - data['response'] = resp.summary() - data['censored'] = False - for snd in sent: - if snd.dst == resp.src: - answered.remove((req, resp)) - return (answered, sent) - - def process_unanswered((unanswered, sent)): - """ - Callback function for remaining packets and destinations which - do not have an associated response. - """ - if len(unanswered) > 0: - nicer = [nicely(unans) for unans in unanswered] - log.msg("Unanswered/remaining packets:\n%s" - % nicer) - self.report['unanswered'] = nicer - for dest, data in self.destinations.items(): - if not 'response' in data: - log.msg("No reply from %s. Possible censorship event." - % dest) - data['response'] = None - data['censored'] = True - return (unanswered, sent) - - packets = self.build_packets() - d = txscapy.sr(packets, iface=self.interface, multi=True) - d.addCallback(process_answered) - d.addErrback(log.exception) - d.addCallback(process_unanswered) - d.addErrback(log.exception) - self.report['destinations'] = self.destinations - return d - except Exception, e: - log.exception(e) + def process_response(echo_reply, dest): + ans, unans = echo_reply + if ans: + log.msg("Recieved echo reply from %s: %s" % (dest, ans)) + else: + log.msg("No reply was received from %s. Possible censorship event." % dest) + log.debug("Unanswered packets: %s" % unans) + self.report[dest] = echo_reply + + for label, data in self.destinations.items(): + reply = sr1(IP(dst=lebal)/ICMP()) + process = process_reponse(reply, label) + + #(ans, unans) = ping + #self.destinations[self.dst].update({'ans': ans, + # 'unans': unans, + # 'response_packet': ping}) + #return ping + + #return reply diff --git a/nettests/experimental/bridge_reachability/tcpsyn.py b/nettests/experimental/bridge_reachability/tcpsyn.py deleted file mode 100644 index b3226c8..0000000 --- a/nettests/experimental/bridge_reachability/tcpsyn.py +++ /dev/null @@ -1,191 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- -# -# +-----------+ -# | tcpflags.py | -# +-----------+ -# Send packets with various TCP flags set to a test server -# to check that it is reachable. -# -# @authors: Isis Lovecruft, <isis@torproject.org> -# @version: 0.0.1-pre-alpha -# @license: copyright (c) 2012 Isis Lovecruft -# see attached LICENCE file -# - -import os -import sys - -from twisted.python import usage -from twisted.python.failure import Failure -from twisted.internet import reactor, defer -from ooni import nettest, config -from ooni.utils import net, log -from ooni.utils.otime import timestamp - -try: - from scapy.all import TCP, IP, sr - from ooni.utils import txscapy -except: - log.msg("This test requires scapy, see www.secdev.org/projects/scapy") - - -class TCPFlagsOptions(usage.Options): - """Options for TCPTest.""" - optParameters = [ - ['dst', 'd', None, 'Host IP to ping'], - ['port', 'p', None, 'Host port'], - ['flags', 's', 'S', 'Comma separated flags to set, eg. "SA"'], - ['count', 'c', 3, 'Number of SYN packets to send', int], - ['interface', 'i', None, 'Network interface to use'], - ['hexdump', 'x', False, 'Show hexdump of responses']] - -class TCPFlagsTest(nettest.NetTestCase): - """ - Sends only a TCP SYN packet to a host IP:PORT, and waits for either a - SYN/ACK, a RST, or an ICMP error. - - TCPSynTest can take an input file containing one IP:Port pair per line, or - the commandline switches --dst <IP> and --port <PORT> can be used. - """ - name = 'TCP Flags' - author = 'Isis Lovecruft <isis@torproject.org>' - description = 'A TCP SYN/ACK/FIN test to see if a host is reachable.' - version = '0.1.1' - requiresRoot = True - - usageOptions = TCPFlagsOptions - inputFile = ['file', 'f', None, 'File of list of IP:PORTs to ping'] - - destinations = {} - - def setUp(self, *a, **kw): - """Configure commandline parameters for TCPSynTest.""" - if self.localOptions: - for key, value in self.localOptions.items(): - setattr(self, key, value) - if not self.interface: - try: - iface = net.getDefaultIface() - self.interface = iface - except net.IfaceError: - self.abortClass("Could not find a working network interface!") - if self.flags: - self.flags = self.flags.split(',') - if config.advanced.debug: - defer.setDebugging('on') - - def addToDestinations(self, addr=None, port='443'): - """ - Validate and add an IP address and port to the dictionary of - destinations to send to. If the host's IP is already in the - destinations, then only add the port. - - @param addr: A string representing an IPv4 or IPv6 address. - @param port: A string representing a port number. - @returns: A 2-tuple containing the address and port. - """ - if addr is None: - return (None, None) # do we want to return SkipTest? - - dst, dport = net.checkIPandPort(addr, port) - if not dst in self.destinations.keys(): - self.destinations[dst] = {'dst': dst, - 'dport': [dport]} - else: - log.debug("Got additional port for destination.") - self.destinations[dst]['dport'].append(dport) - return (dst, dport) - - def inputProcessor(self, input_file=None): - """ - Pull the IPs and PORTs from the commandline options first, and then - from the input file, and place them in a dict for storing test results - as they arrive. - """ - if self.localOptions['dst'] is not None \ - and self.localOptions['port'] is not None: - log.debug("Processing commandline destination") - yield self.addToDestinations(self.localOptions['dst'], - self.localOptions['port']) - if input_file and os.path.isfile(input_file): - log.debug("Processing input file %s" % input_file) - with open(input_file) as f: - for line in f.readlines(): - if line.startswith('#'): - continue - one = line.strip() - raw_ip, raw_port = one.rsplit(':', 1) - yield self.addToDestinations(raw_ip, raw_port) - - def test_tcp_flags(self): - """ - Generate, send, and listen for responses to, a list of TCP/IP packets - to an address and port pair taken from the current input, and a string - specifying the TCP flags to set. - - @param flags: - A string representing the TCP flags to be set, i.e. "SA" or "F". - Defaults to "S". - """ - def build_packets(addr, port): - """Construct a list of packets to send out.""" - packets = [] - for flag in self.flags: - log.debug("Generating packets with %s flags for %s:%d..." - % (flag, addr, port)) - for x in xrange(self.count): - packets.append( IP(dst=addr)/TCP(dport=port, flags=flag) ) - return packets - - def process_packets(packet_list): - """ - If the source address of packet in :param:packet_list matches one of - our input destinations, then extract some of the information from it - to the test report. - - @param packet_list: - A :class:scapy.plist.PacketList - """ - log.msg("Processing received packets...") - results, unanswered = packet_list - - for (q, r) in results: - request = {'dst': q.dst, - 'dport': q.dport, - 'summary': q.summary(), - 'hexdump': None, - 'sent_time': q.time} - response = {'src': r['IP'].src, - 'flags': r['IP'].flags, - 'summary': r.summary(), - 'hexdump': None, - 'recv_time': r.time, - 'delay': r.time - q.time} - if self.hexdump: - request['hexdump'] = q.hexdump() - response['hexdump'] = r.hexdump() - - for dest, data in self.destinations.items(): - if response['src'] == data['dst']: - log.msg(" Received response from %s:\n%s ==> %s" % ( - response['src'], q.mysummary(), r.mysummary())) - if self.hexdump: - log.msg("%s\n%s" % (q.hexdump(), r.hexdump())) - - self.report['request'] = request - self.report['response'] = response - - if unanswered is not None: - unans = [un.summary() for un in unanswered] - log.msg(" Waiting on responses from:\n%s" % '\n'.join(unans)) - self.report['unanswered'] = unans - - try: - self.report = {} - (addr, port) = self.input - pkts = build_packets(addr, port) - d = process_packets(sr(pkts, iface=self.interface, timeout=5)) - return d - except Exception, ex: - log.exception(ex) diff --git a/ooni/inputunit.py b/ooni/inputunit.py index 0b4377f..2ef89d8 100644 --- a/ooni/inputunit.py +++ b/ooni/inputunit.py @@ -6,10 +6,9 @@ # units. Input units are how the inputs to be fed to tests are # split up into. # -# :authors: Arturo Filastò, Isis Lovecruft +# :authors: Arturo Filastò # :license: see included LICENSE file - class InputUnitFactory(object): """ This is a factory that takes the size of input units to be generated a set @@ -62,37 +61,24 @@ class InputUnitFactory(object): class InputUnit(object): """ This is a python iterable object that contains the input elements to be - passed onto a :class:`ooni.nettest.NetTestCase`. + passed onto a TestCase. """ def __init__(self, inputs=[]): - """ - Create an iterable from a list of inputs, which can be given to a NetTestCase. - - @param inputs: A list of inputs for a NetTestCase. - """ self._inputs = iter(inputs) - # _inputs_copy is to avoid stealing things from - # the iterator when __repr__ is called: - self._inputs_copy = inputs def __str__(self): - """Prints the original input list.""" - return "<%s inputs=%s>" % (self.__class__, self._inputs_copy) + return "<%s inputs=%s>" % (self.__class__, self._inputs) def __add__(self, inputs): - """Add a list of inputs to the iterator.""" for i in inputs: self._inputs.append(i) def __iter__(self): - """Self explanatory.""" return self def next(self): - """Return the next item from the InputUnit iterator.""" return self._inputs.next() def append(self, input): - """Add an item to the end of the InputUnit iterator.""" self._inputs.append(input) diff --git a/ooni/lib/001-scapy_missing-exc.sh.patch b/ooni/lib/001-scapy_missing-exc.sh.patch deleted file mode 100644 index 3f5095c..0000000 --- a/ooni/lib/001-scapy_missing-exc.sh.patch +++ /dev/null @@ -1,78 +0,0 @@ -# -# Add a missing Exception class to /scapy/arch/linux.py -# -# To apply this patch: -# STEP 1: Chdir to the source directory. If you have scapy installed, this is -# likely located at /usr/share/pyshared. -# STEP 2: Run the 'patch' program with this file as input, i.e.: -# -# /usr/share/pyshared$ patch -p1 ./scapy/arch/linux.py </path/to/this/file> -# -# To apply this patch with the use of 'applypatch': -# STEP 1: Chdir to the source directory. -# STEP 2: Run the 'applypatch' program with this patch file as input. -# -# @authors: Isis Lovecruft -# @license: This file is part of ooniprobe, see LICENSE file for details. -# @copyright: 2012 Isis Lovecruft -# -#### End of Preamble #### - -#### Patch data follows #### -diff -c 'scapy/arch/linux.py' 'scapy/arch/linux.py-patched' -Index: ./scapy/arch/linux.py -*** ./scapy/arch/linux.py Thu Dec 13 21:26:59 2012 ---- ./scapy/arch/linux.py-patched Thu Dec 13 21:27:53 2012 -*************** -*** 14,21 **** - from scapy.data import * - from scapy.supersocket import SuperSocket - import scapy.arch -! from scapy.error import warning -! - - - # From bits/ioctls.h ---- 14,20 ---- - from scapy.data import * - from scapy.supersocket import SuperSocket - import scapy.arch -! from scapy.error import warning,Scapy_Exception - - - # From bits/ioctls.h -#### End of Patch data #### - -#### ApplyPatch data follows #### -# Data version : 1.0 -# Date generated : Thu Dec 13 21:33:12 2012 -# Generated by : makepatch 2.03 -# Recurse directories : Yes -# Excluded files : (\A|/).*\~\Z -# (\A|/).*\.a\Z -# (\A|/).*\.bak\Z -# (\A|/).*\.BAK\Z -# (\A|/).*\.elc\Z -# (\A|/).*\.exe\Z -# (\A|/).*\.gz\Z -# (\A|/).*\.ln\Z -# (\A|/).*\.o\Z -# (\A|/).*\.obj\Z -# (\A|/).*\.olb\Z -# (\A|/).*\.old\Z -# (\A|/).*\.orig\Z -# (\A|/).*\.rej\Z -# (\A|/).*\.so\Z -# (\A|/).*\.Z\Z -# (\A|/)\.del\-.*\Z -# (\A|/)\.make\.state\Z -# (\A|/)\.nse_depinfo\Z -# (\A|/)core\Z -# (\A|/)tags\Z -# (\A|/)TAGS\Z -# p 'scapy/arch/linux.py' 16563 1355434073 0100644 -#### End of ApplyPatch data #### - -#### End of Patch kit [created: Thu Dec 13 21:33:12 2012] #### -#### Patch checksum: 56 1830 50583 #### -#### Checksum: 74 2507 41533 #### diff --git a/ooni/nettest.py b/ooni/nettest.py index 13012c7..fcc945a 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -1,22 +1,9 @@ -# -*- encoding: utf-8 -*- -# -# nettest.py -# ---------- -# In here is the NetTest API definition. This is how people -# interested in writing ooniprobe tests will be specifying them -# -# :authors: Arturo Filastò, Isis Lovecruft -# :license: see included LICENSE file - -import sys -import os import itertools import traceback -import inspect +import sys +import os -from twisted.trial import unittest, itrial -from twisted.trial import util as txtrutil -from twisted.trial.test import skipping +from twisted.trial import unittest, itrial, util from twisted.internet import defer, utils from twisted.python import usage @@ -148,13 +135,16 @@ class NetTestCase(object): requiresRoot = False localOptions = {} - def _setUp(self): - """This is the internal setup method to be overwritten by templates.""" + """ + This is the internal setup method to be overwritten by templates. + """ pass def setUp(self): - """Place your logic to be executed when the test is being setup here.""" + """ + Place here your logic to be executed when the test is being setup. + """ pass def postProcessor(self, report): @@ -197,6 +187,12 @@ class NetTestCase(object): else: pass + def _checkRequiredOptions(self): + for required_option in self.requiredOptions: + log.debug("Checking if %s is present" % required_option) + if not self.localOptions[required_option]: + raise usage.UsageError("%s not specified!" % required_option) + def _processOptions(self): if self.inputFilename: inputProcessor = self.inputProcessor @@ -210,53 +206,10 @@ class NetTestCase(object): return inputProcessor(inputFilename) self.inputs = inputProcessorIterator() - return {'inputs': self.inputs, - 'name': self.name, - 'version': self.version} + return {'inputs': self.inputs, + 'name': self.name, 'version': self.version + } def __repr__(self): return "<%s inputs=%s>" % (self.__class__, self.inputs) - def _abortMethod(self, reason, method=None): - if method is None: - test_method = self._testMethod - else: - test_method = getattr(self.__class__, method, False) - - if inspect.ismethod(test_method): - method_name = test_method.im_func.func_name - setattr(test_method, 'skip', reason) - raise skipping.SkipTest("Aborting %s for reason: %s" - % (method_name, reason) ) - else: - log.debug("_abortMethod(): could not find method %s" % test_method) - - def abortInput(self, reason): - """ - Abort the current input. - - @param reason: A string explaining why this test is being skipped. - @raises: A :class:`twisted.trial.test.skipping.SkipTest <SkipTest>` - """ - raise skipping.SkipTest(" Reason: %s\nCurrent input: %s" - % (reason, self.input)) - - def abortMethod(self, reason, test_method=None): - """ - Abort all remaining inputs for the current test method. - - @param reason: A string explaining why the current test_method is - being skipped. - @param test_method: (optional) The test_method to skip, defaults to - the currently running test_method. - """ - return self._abortMethod(reason, test_method) - - def abortClass(self, reason='unspecified'): - """ - Abort the entire NetTestCase class. - - @param reason: A string explaining why the class is being skipped. - """ - log.msg("Aborting %s: %s" % (self.__class__.name, reason)) - setattr(self.__class__, 'skip', reason) diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 1c3edf6..b06bde9 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -1,13 +1,4 @@ -# -*- coding: UTF-8 -# -# oonicli -# ------- -# In here we take care of running ooniprobe from the command -# line interface -# -# :authors: Arturo Filastò, Isis Lovecruft -# :license: see included LICENSE file - +#-*- coding: utf-8 -*- import sys import os @@ -20,17 +11,14 @@ from twisted.application import app from twisted.python import usage, failure from twisted.python.util import spewer -# Supress scapy's "No route found for IPv6 destination" warnings: -import logging as pylogging -pylogging.getLogger("scapy.runtime").setLevel(pylogging.ERROR) - from ooni import nettest, runner, reporter, config + from ooni.inputunit import InputUnitFactory + from ooni.utils import net -from ooni.utils import checkForRoot, PermissionsError +from ooni.utils import checkForRoot, NotRootError from ooni.utils import log - class Options(usage.Options): synopsis = """%s [options] [path to test].py """ % (os.path.basename(sys.argv[0]),) @@ -42,14 +30,13 @@ class Options(usage.Options): optFlags = [["help", "h"], ["resume", "r"]] - optParameters = [ - ["reportfile", "o", None, "report file name"], - ["testdeck", "i", None, - "Specify a test deck: a yaml file containing tests and their arguments"], - ["collector", "c", None, - "Address of the collector of test results. (e.g.: http://127.0.0.1:8888)"], - ["logfile", "l", None, "log file name"], - ["pcapfile", "p", None, "pcap file name"]] + optParameters = [["reportfile", "o", None, "report file name"], + ["testdeck", "i", None, + "Specify as input a test deck: a yaml file containig the tests to run an their arguments"], + ["collector", "c", None, + "Address of the collector of test results. (example: http://127.0.0.1:8888)"], + ["logfile", "l", None, "log file name"], + ["pcapfile", "p", None, "pcap file name"]] compData = usage.Completions( extraActions=[usage.CompleteFiles( @@ -82,42 +69,23 @@ class Options(usage.Options): except: raise usage.UsageError("No test filename specified!") -class CooperativeTimer(object): - """ - A simple timer for the callback to functions on - :class:`twisted.internet.task.Cooperator <t.i.t.Cooperator>`. see - :meth:`oonicli.runTestList <runTestList>`. - - @param seconds: - An integer specifying the second to wait in between updating the - status and ETA bars. - """ - def __init__(self, seconds=5): - self.max_timer_interval = float(seconds) - self.end = time.time() + self.max_timer_interval - - def __call__(self): - return time.time() >= self.end - -def updateStatusBar(stop_func): +def updateStatusBar(): for test_filename in config.state.keys(): # The ETA is not updated so we we will not print it out for the # moment. eta = config.state[test_filename].eta() progress = config.state[test_filename].progress() - while progress is not None: - print "[%s] %s%%" % (test_filename, progress) - else: - print "[%s] All tests in file completed." % test_filename - stop_func() + progress_bar_frmt = "[%s] %s%%" % (test_filename, progress) + print progress_bar_frmt def testsEnded(*arg, **kw): - """You can place here all the post shutdown tasks.""" - log.debug("Finished running all tests") + """ + You can place here all the post shutdown tasks. + """ + log.debug("testsEnded: Finished running all tests") config.start_reactor = False - if not reactor.running: - try: reactor.stop() - except: reactor.runUntilCurrent() + try: reactor.stop() + except: pass def testFailed(failure): log.err("Failed in running a test inside a test list") @@ -141,14 +109,9 @@ def runTestList(none, test_list): d2.addCallback(testsEnded) d2.addErrback(testFailed) - try: - # Print every 5 second the list of current tests running - coop = task.Cooperator(started=False) - coop.cooperate(updateStatusBar) #this will need a .next() method - coop.start() - except StopIteration: - return d2 - + # Print every 5 second the list of current tests running + l = task.LoopingCall(updateStatusBar) + l.start(5.0) return d2 def errorRunningTests(failure): @@ -156,8 +119,9 @@ def errorRunningTests(failure): failure.printTraceback() def run(): - """Call me to begin testing from a file.""" - + """ + Parses command line arguments of test. + """ cmd_line_options = Options() if len(sys.argv) == 1: cmd_line_options.getUsage() @@ -171,15 +135,10 @@ def run(): config.cmd_line_options = cmd_line_options if config.privacy.includepcap: - try: - checkForRoot() - except PermissionsError, pe: - log.warn("Capturing packets requires administrator/root privileges. ") - log.warn("Run ooniprobe as root or set 'includepcap = false' in ooniprobe.conf .") - sys.exit(1) - else: - log.msg("Starting packet capture") - runner.startSniffing() + log.msg("Starting") + if not config.reports.pcap: + config.generatePcapFilename() + runner.startSniffing() resume = cmd_line_options['resume'] @@ -212,6 +171,4 @@ def run(): d = runTestList(None, test_list) d.addErrback(errorRunningTests) - # XXX I believe we don't actually need this: - #reactor.run() - + reactor.run() diff --git a/ooni/reporter.py b/ooni/reporter.py index aac163d..728c3f5 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -1,20 +1,11 @@ -#-*- coding: utf-8 -*- -# -# reporter.py -# ----------- -# In here goes the logic for the creation of ooniprobe reports. -# -# :authors: Arturo Filastò, Isis Lovecruft -# :license: see included LICENSE file - import traceback import itertools import logging -import sys -import os import time import yaml import json +import sys +import os import re from yaml.representer import * @@ -26,9 +17,7 @@ from twisted.trial import reporter from twisted.internet import defer, reactor from twisted.internet.error import ConnectionRefusedError -from ooni import config, otime -from ooni.utils import log, geodata, pushFilenameStack -from ooni.utils.net import BodyReceiver, StringProducer, userAgents +from ooni.utils import log try: from scapy.packet import Packet @@ -36,6 +25,12 @@ except ImportError: log.err("Scapy is not installed.") +from ooni import otime +from ooni.utils import geodata, pushFilenameStack +from ooni.utils.net import BodyReceiver, StringProducer, userAgents + +from ooni import config + def createPacketReport(packet_list): """ Takes as input a packet a list. @@ -157,7 +152,7 @@ def getTestDetails(options): 'test_version': options['version'], 'software_name': 'ooniprobe', 'software_version': software_version - } + } return test_details class OReporter(object): @@ -180,7 +175,7 @@ class OReporter(object): pass def testDone(self, test, test_name): - log.debug("Calling reporter to record results") + log.msg("Finished running %s" % test_name) test_report = dict(test.report) if isinstance(test.input, Packet): @@ -278,8 +273,8 @@ class OONIBReporter(OReporter): try: self.agent = Agent(reactor, sockshost="127.0.0.1", socksport=int(config.tor.socks_port)) - except Exception, ex: - log.exception(ex) + except Exception, e: + log.exception(e) OReporter.__init__(self, cmd_line_options) @@ -319,8 +314,8 @@ class OONIBReporter(OReporter): try: test_details = getTestDetails(options) - except Exception, ex: - log.exception(ex) + except Exception, e: + log.exception(e) test_details['options'] = self.cmd_line_options @@ -339,7 +334,7 @@ class OONIBReporter(OReporter): 'test_name': test_name, 'test_version': test_version, 'content': content - } + } log.msg("Reporting %s" % url) request_json = json.dumps(request) @@ -357,8 +352,8 @@ class OONIBReporter(OReporter): log.err("Connection to reporting backend failed (ConnectionRefusedError)") raise OONIBReportCreationError - except Exception, ex: - log.exception(ex) + except Exception, e: + log.exception(e) raise OONIBReportCreationError # This is a little trix to allow us to unspool the response. We create diff --git a/ooni/runner.py b/ooni/runner.py index d8b6c4e..4ebfa0b 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -1,79 +1,31 @@ -#-*- coding: utf-8 -*- -# -# runner.py -# --------- -# Handles running ooni.nettests as well as -# ooni.plugoo.tests.OONITests. -# -# :authors: Arturo Filastò, Isis Lovecruft -# :license: see included LICENSE file - import os import sys import time import inspect import traceback import itertools + import yaml -from twisted.python import reflect, usage, failure -from twisted.internet import defer, reactor, threads -from twisted.trial import reporter as txreporter -from twisted.trial import util as txutil +from twisted.python import reflect, usage +from twisted.internet import defer from twisted.trial.runner import filenameToModule -from twisted.trial.unittest import utils as txutils -from twisted.trial.unittest import SkipTest +from twisted.internet import reactor, threads from txtorcon import TorProtocolFactory, TorConfig from txtorcon import TorState, launch_tor -from ooni import config, nettest, reporter -from ooni.inputunit import InputUnitFactory +from ooni import config + from ooni.reporter import OONIBReporter, YAMLReporter, OONIBReportError from ooni.inputunit import InputUnitFactory from ooni.nettest import NetTestCase, NoPostProcessor -from ooni.utils import log, checkForRoot -from ooni.utils import PermissionsError, Storage +from ooni.utils import log, checkForRoot, pushFilenameStack +from ooni.utils import NotRootError, Storage from ooni.utils.net import randomFreePort - -class NoTestCasesFound(Exception): - pass - -class InvalidResumeFile(Exception): - pass - -class noResumeSession(Exception): - pass - -class InvalidConfigFile(Exception): - message = "Invalid setting in ooniprobe.conf: " - -class UnableToStartTor(Exception): - pass - - -def isTestCase(obj): - """Return True if obj is a subclass of NetTestCase, False otherwise.""" - try: - return issubclass(obj, nettest.NetTestCase) - except TypeError: - return False - -def checkRequiredOptions(test_instance): - """ - If test_instance has an attribute 'requiredOptions', then check that - those options were utilised on the commandline. - """ - required = getattr(test_instance, 'requiredOptions', None) - if required: - for required_option in required: - log.debug("Checking if %s is present" % required_option) - if not test_instance.localOptions[required_option]: - raise usage.UsageError("%s not specified!" % required_option) - def processTest(obj, cmd_line_options): """ Process the parameters and :class:`twisted.python.usage.Options` of a @@ -86,46 +38,58 @@ def processTest(obj, cmd_line_options): :param cmd_line_options: A configured and instantiated :class:`twisted.python.usage.Options` class. - """ - if obj.requiresRoot: - try: - checkForRoot() - except PermissionsError: - log.err("%s requires root to run" % obj.name) - sys.exit(1) + """ if not hasattr(obj.usageOptions, 'optParameters'): obj.usageOptions.optParameters = [] + if obj.inputFile: + obj.usageOptions.optParameters.append(obj.inputFile) + if obj.baseParameters: for parameter in obj.baseParameters: obj.usageOptions.optParameters.append(parameter) + if obj.baseFlags: if not hasattr(obj.usageOptions, 'optFlags'): obj.usageOptions.optFlags = [] for flag in obj.baseFlags: obj.usageOptions.optFlags.append(flag) - if obj.inputFile: # inputFile is the optParameters list - obj.usageOptions.optParameters.append(obj.inputFile) options = obj.usageOptions() + options.parseOptions(cmd_line_options['subargs']) obj.localOptions = options - if obj.inputFile: # inputFilename is the actual filename + if obj.inputFile: obj.inputFilename = options[obj.inputFile[0]] try: - log.debug("Parsing commandline options") - tmp_test_instance = obj() - checkRequiredOptions(tmp_test_instance) - except usage.UsageError, ue: - log.err("%s" % ue) + log.debug("processing options") + tmp_test_case_object = obj() + tmp_test_case_object._checkRequiredOptions() + + except usage.UsageError, e: + test_name = tmp_test_case_object.name + log.err("There was an error in running %s!" % test_name) + log.err("%s" % e) options.opt_help() - raise usage.UsageError("Error parsing command line args for %s" - % tmp_test_case_object.name) - else: - return obj + raise usage.UsageError("Error in parsing command line args for %s" % test_name) + + if obj.requiresRoot: + try: + checkForRoot() + except NotRootError: + log.err("%s requires root to run" % obj.name) + sys.exit(1) + + return obj + +def isTestCase(obj): + try: + return issubclass(obj, NetTestCase) + except TypeError: + return False def findTestClassesFromFile(cmd_line_options): """ @@ -139,19 +103,13 @@ def findTestClassesFromFile(cmd_line_options): A list of class objects found in a file or module given on the commandline. """ - classes = [] filename = cmd_line_options['test'] - relative = filename.rsplit('/', 1)[1] - try: - module = filenameToModule(filename) - except ValueError, ve: - log.fail("%r doesn't exist." % relative) - else: - for name, val in inspect.getmembers(module): - if isTestCase(val): - classes.append(processTest(val, cmd_line_options)) - finally: - return classes + classes = [] + module = filenameToModule(filename) + for name, val in inspect.getmembers(module): + if isTestCase(val): + classes.append(processTest(val, cmd_line_options)) + return classes def makeTestCases(klass, tests, method_prefix): """ @@ -163,6 +121,9 @@ def makeTestCases(klass, tests, method_prefix): cases.append((klass, method_prefix+test)) return cases +class NoTestCasesFound(Exception): + pass + def loadTestsAndOptions(classes, cmd_line_options): """ Takes a list of test classes and returns their testcases and options. @@ -184,45 +145,8 @@ def loadTestsAndOptions(classes, cmd_line_options): return test_cases, options -def getTestTimeout(test_instance, test_method): - """ - Returns the timeout value set on this test. Check on the instance first, - the the class, then the module, then package. As soon as it finds - something with a timeout attribute, returns that. Returns the value set in - ooniprobe.conf, :attr:`ooni.config.advanced.default_timeout - <default_timeout>` if it cannot find anything. - - See twisted.trial.unittest.TestCase docstring for more details. - - @param test_instance: - The instance of a :class:`ooni.nettest.NetTestCase` currently running. - @param test_method: - The test_instance.test_method currently being processed. - """ - default = config.advanced.default_timeout - - try: - tm = getattr(test_instance, test_method) - except: - log.debug("runner.getTestTimeout() couldn't find %s.%s!" - % (test_instance, test_method)) - try: - return float(default) - except (ValueError, TypeError): - raise InvalidConfigFile("'default_timeout' must be a number!") - else: - test_instance._parents = [tm, test_instance] - test_instance._parents.extend(txutil.getPythonContainers(tm)) - timeout = txutil.acquireAttribute( - test_instance._parents, 'timeout', default) - try: - return float(timeout) - except (ValueError, TypeError): - log.warn("'timeout' attribute must be a number!") - return float(default) - def runTestCasesWithInput(test_cases, test_input, yaml_reporter, - oonib_reporter=None): + oonib_reporter=None): """ Runs in parallel all the test methods that are inside of the specified test case. Reporting happens every time a Test Method has concluded running. @@ -248,30 +172,8 @@ def runTestCasesWithInput(test_cases, test_input, yaml_reporter, # This is used to store a copy of all the test reports tests_report = {} - def test_timeout(d, test_instance): - timeout_error = defer.TimeoutError( - "%s test for %s timed out after %s seconds" - % (test_instance.name, test_instance.input, test_instance.timeout)) - timeout_fail = failure.Failure(err) - try: - d.errback(timeout_fail) - except defer.AlreadyCalledError: - # if the deferred has already been called but the *back chain is - # still unfinished, safely crash the reactor and report the timeout - reactor.crash() - test_instance._timedOut = True # see test_instance._wait - test_instance._test_result.addExpectedFailure(test_instance, fail) - test_timeout = txutils.suppressWarnings( - test_timeout, txutil.suppress(category=DeprecationWarning)) - - def test_skip_class(reason): - try: - d.errback(failure.Failure(SkipTest("%s" % reason))) - except defer.AlreadyCalledError: - pass # XXX not sure what to do here... - def test_done(result, test_instance, test_name): - log.msg("Successfully finished running %s" % test_name) + log.msg("Finished running %s" % test_name) log.debug("Deferred callback result: %s" % result) tests_report[test_name] = dict(test_instance.report) if not oonib_reporter: @@ -280,12 +182,9 @@ def runTestCasesWithInput(test_cases, test_input, yaml_reporter, d2 = yaml_reporter.testDone(test_instance, test_name) return defer.DeferredList([d1, d2]) - def test_error(error, test_instance, test_name): - if isinstance(error, SkipTest): - log.warn("%s" % error.message) - else: - log.err("Error in running %s" % test_name) - log.exception(error) + def test_error(failure, test_instance, test_name): + log.err("Error in running %s" % test_name) + log.exception(failure) return def tests_done(result, test_class): @@ -301,54 +200,29 @@ def runTestCasesWithInput(test_cases, test_input, yaml_reporter, d1 = oonib_reporter.testDone(test_instance, 'summary') d2 = yaml_reporter.testDone(test_instance, 'summary') return defer.DeferredList([d1, d2]) - except nettest.NoPostProcessor: + except NoPostProcessor: log.debug("No post processor configured") return dl = [] for test_case in test_cases: + log.debug("Processing %s" % test_case[1]) test_class = test_case[0] test_method = test_case[1] - log.debug("%s: Setting up: %s" % (test_class.name, test_method)) + + log.msg("Running %s with %s..." % (test_method, test_input)) test_instance = test_class() test_instance.input = test_input test_instance.report = {} - - # XXX txreporter.TestResult is expected by test_timeout(), but we - # should eventually replace it with a stub class - test_instance._test_result = txreporter.TestResult() # use this to keep track of the test runtime test_instance._start_time = time.time() # call setups on the test test_instance._setUp() test_instance.setUp() - - # get the timeout and _parents, in case it was set in setUp() - test_instance.timeout = getTestTimeout(test_instance, test_method) - test_instance.timedOut = False - test = getattr(test_instance, test_method) - test_instance._testMethod = test d = defer.maybeDeferred(test) - - # register the timer with the reactor - call_timeout = reactor.callLater(test_instance.timeout, test_timeout, d, - test_instance) - d.addBoth(lambda x: call_timeout.active() and call_timeout.cancel() or x) - - # check if anything has been aborted or marked as 'skip' - if hasattr(test_instance.__class__, 'skip'): - reason = getattr(test_instance.__class__, 'skip') - else: - reason = txutil.acquireAttribute(test_instance._parents, 'skip', None) - if reason is not None: - log.warn("%s marked some tests to be skipped. Reason: %s" - % (test_instance.name, reason)) - call_skip = reactor.callLater(0, test_skip_class, reason) - d.addBoth(lambda x: call_skip.active() and call_skip.cancel() or x) - d.addCallback(test_done, test_instance, test_method) d.addErrback(test_error, test_instance, test_method) dl.append(d) @@ -357,8 +231,8 @@ def runTestCasesWithInput(test_cases, test_input, yaml_reporter, test_methods_d.addCallback(tests_done, test_cases[0][0]) return test_methods_d -def runTestCasesWithInputUnit(test_cases, input_unit, yaml_reporter, - oonib_reporter): +def runTestCasesWithInputUnit(test_cases, input_unit, yaml_reporter, + oonib_reporter): """ Runs the Test Cases that are given as input parallely. A Test Case is a subclass of ooni.nettest.NetTestCase and a list of @@ -367,21 +241,26 @@ def runTestCasesWithInputUnit(test_cases, input_unit, yaml_reporter, The deferred list will fire once all the test methods have been run once per item in the input unit. - @param test_cases: - A tuple containing the test_class and test_method as strings. - @param input_unit: - A generator that contains the inputs to be run on the test. - @return: - A DeferredList containing all the tests to be run at this time. + test_cases: A list of tuples containing the test class and the test method as a string. + + input_unit: A generator that yields an input per iteration + """ + log.debug("Running test cases with input unit") dl = [] for test_input in input_unit: - log.debug("Running test with this input %s" % str(test_input)) + log.debug("Running test with this input %s" % test_input) d = runTestCasesWithInput(test_cases, test_input, yaml_reporter, oonib_reporter) dl.append(d) return defer.DeferredList(dl) +class InvalidResumeFile(Exception): + pass + +class noResumeSession(Exception): + pass + def loadResumeFile(): """ Sets the singleton stateDict object to the content of the resume file. @@ -390,6 +269,7 @@ def loadResumeFile(): Raises: :class:ooni.runner.InvalidResumeFile if the resume file is not valid + """ if not config.stateDict: try: @@ -425,6 +305,7 @@ def resumeTest(test_filename, input_unit_factory): :class:ooni.inputunit.InputUnitFactory that is at the index of the previous test run. + """ try: idx = config.stateDict[test_filename] @@ -445,7 +326,7 @@ def resumeTest(test_filename, input_unit_factory): @defer.inlineCallbacks def updateResumeFile(test_filename): """ - Update the resume file with the current stateDict state. + update the resume file with the current stateDict state. """ log.debug("Acquiring lock for %s" % test_filename) yield config.resume_lock.acquire() @@ -467,17 +348,20 @@ def increaseInputUnitIdx(test_filename): including the .py extension. input_unit_idx (int): the current input unit index for the test. + """ config.stateDict[test_filename] += 1 yield updateResumeFile(test_filename) -def updateProgressMeters(test_filename, input_unit_factory, test_case_number): - """Update the progress meters for keeping track of test state.""" +def updateProgressMeters(test_filename, input_unit_factory, + test_case_number): + """ + Update the progress meters for keeping track of test state. + """ if not config.state.test_filename: config.state[test_filename] = Storage() - per_item_avg = float(2) - config.state[test_filename].per_item_average = per_item_avg + config.state[test_filename].per_item_average = 2.0 input_unit_idx = float(config.stateDict[test_filename]) input_unit_items = float(len(input_unit_factory) + 1) @@ -485,36 +369,29 @@ def updateProgressMeters(test_filename, input_unit_factory, test_case_number): total_iterations = input_unit_items * test_case_number current_iteration = input_unit_idx * test_case_number - log.debug("Total InputUnits: %s" % input_unit_items) + log.debug("input_unit_items: %s" % input_unit_items) + log.debug("test_case_number: %s" % test_case_number) + log.debug("Test case number: %s" % test_case_number) log.debug("Total iterations: %s" % total_iterations) log.debug("Current iteration: %s" % current_iteration) def progress(): - current_progress = (current_iteration / total_iterations) * 100.0 - while float(current_progress) < float(100): - return current_progress + return (current_iteration / total_iterations) * 100.0 + config.state[test_filename].progress = progress def eta(): - return (total_iterations - current_iteration) * per_item_avg + return (total_iterations - current_iteration) \ + * config.state[test_filename].per_item_average config.state[test_filename].eta = eta config.state[test_filename].input_unit_idx = input_unit_idx config.state[test_filename].input_unit_items = input_unit_items + @defer.inlineCallbacks def runTestCases(test_cases, options, cmd_line_options): - """ - Run all test cases found in specified files and modules. - - @param test_cases: - A list of tuples, each tuple in containing the test_class - and test_method to run. - @param cmd_line_options: - The parsed :attr:`twisted.python.usage.Options.optParameters` - obtained from the main ooni commandline. - """ log.debug("Running %s" % test_cases) log.debug("Options %s" % options) log.debug("cmd_line_options %s" % dict(cmd_line_options)) @@ -576,6 +453,9 @@ def runTestCases(test_cases, options, cmd_line_options): log.exception("Problem in running test") yaml_reporter.finish() +class UnableToStartTor(Exception): + pass + def startTor(): """ Starts Tor Launches a Tor with :param: socks_port :param: control_port @@ -651,7 +531,7 @@ def startSniffing(): from ooni.utils.txscapy import ScapyFactory, ScapySniffer try: checkForRoot() - except PermissionsError: + except NotRootError: print "[!] Includepcap options requires root priviledges to run" print " you should run ooniprobe as root or disable the options in ooniprobe.conf" sys.exit(1) @@ -659,8 +539,7 @@ def startSniffing(): print "Starting sniffer" config.scapyFactory = ScapyFactory(config.advanced.interface) - pcapfile = config.reports.pcap - if pcapfile and os.path.exists(pcapfile): + if os.path.exists(config.reports.pcap): print "Report PCAP already exists with filename %s" % config.reports.pcap print "Renaming files with such name..." pushFilenameStack(config.reports.pcap) @@ -677,12 +556,6 @@ def loadTest(cmd_line_options): # Ideally this would get all wrapped in a nice little class that get's # instanced with it's cmd_line_options as an instance attribute classes = findTestClassesFromFile(cmd_line_options) - try: - test_cases, options = loadTestsAndOptions(classes, cmd_line_options) - return test_cases, options, cmd_line_options - except NoTestCasesFound, ntcf: - log.err(ntcf) - if not 'testdeck' in cmd_line_options: # exit if this was this only test - sys.exit(1) # file and there aren't any tests - else: - pass # there are more tests, so continue + test_cases, options = loadTestsAndOptions(classes, cmd_line_options) + + return test_cases, options, cmd_line_options diff --git a/ooni/utils/__init__.py b/ooni/utils/__init__.py index efa609a..8510a3b 100644 --- a/ooni/utils/__init__.py +++ b/ooni/utils/__init__.py @@ -1,10 +1,10 @@ -import imp -import os import logging import string import random import glob import yaml +import imp +import os class Storage(dict): """ @@ -48,18 +48,16 @@ class Storage(dict): for (k, v) in value.items(): self[k] = v -class PermissionsError(Exception): - """This test requires administrator or root permissions.""" +class NotRootError(Exception): + pass def checkForRoot(): - """Check permissions.""" if os.getuid() != 0: - raise PermissionsError + raise NotRootError("This test requires root") def randomSTR(length, num=True): """ - Returns a random, all-uppercase, alpha-numeric (if num=True), string of - specified character length. + Returns a random all uppercase alfa-numerical (if num True) string long length """ chars = string.ascii_uppercase if num: @@ -68,8 +66,7 @@ def randomSTR(length, num=True): def randomstr(length, num=True): """ - Returns a random, all-lowercase, alpha-numeric (if num=True), string - specified character length. + Returns a random all lowercase alfa-numerical (if num True) string long length """ chars = string.ascii_lowercase if num: @@ -78,14 +75,15 @@ def randomstr(length, num=True): def randomStr(length, num=True): """ - Returns a random a mixed lowercase, uppercase, alpha-numeric (if num=True) - string of specified character length. + Returns a random a mixed lowercase, uppercase, alfanumerical (if num True) + string long length """ chars = string.ascii_lowercase + string.ascii_uppercase if num: chars += string.digits return ''.join(random.choice(chars) for x in range(length)) + def pushFilenameStack(filename): """ Takes as input a target filename and checks to see if a file by such name diff --git a/ooni/utils/geodata.py b/ooni/utils/geodata.py index d0e730e..d9883ba 100644 --- a/ooni/utils/geodata.py +++ b/ooni/utils/geodata.py @@ -1,12 +1,3 @@ -# -*- encoding: utf-8 -*- -# -# geodata.py -# ********** -# In here go functions related to the understanding of -# geographical information of the probe -# -# :licence: see LICENSE - import re import os @@ -39,12 +30,11 @@ def IPToLocation(ipaddr): asn_dat = pygeoip.GeoIP(asn_file) location['asn'] = asn_dat.org_by_addr(ipaddr) + except IOError: - try: - raise GeoIPDataFilesNotFound( - "Couldn't find GeoIP files. Go to ./data and run \"make geoip\".") - except GeoIPDataFilesNotFound, gnf: - log.err(gnf) + log.err("Could not find GeoIP data files. Go into data/ " + "and run make geoip") + raise GeoIPDataFilesNotFound return location diff --git a/ooni/utils/hacks.py b/ooni/utils/hacks.py index 4bbdf48..64b5a53 100644 --- a/ooni/utils/hacks.py +++ b/ooni/utils/hacks.py @@ -1,13 +1,5 @@ -# -*- encoding: utf-8 -*- -# -# hacks.py -# ******** # When some software has issues and we need to fix it in a # hackish way, we put it in here. This one day will be empty. -# -# :authors: Arturo Filastò, Isis Lovecruft -# :licence: see LICENSE - import copy_reg diff --git a/ooni/utils/log.py b/ooni/utils/log.py index 95163f3..0740c10 100644 --- a/ooni/utils/log.py +++ b/ooni/utils/log.py @@ -1,13 +1,7 @@ -# -*- encoding: utf-8 -*- -# -# :authors: Arturo Filastò -# :licence: see LICENSE - -from functools import wraps import sys import os -import traceback import logging +import traceback from twisted.python import log as txlog from twisted.python import util @@ -17,6 +11,9 @@ from twisted.python.logfile import DailyLogFile from ooni import otime from ooni import config +## Get rid of the annoying "No route found for +## IPv6 destination warnings": +logging.getLogger("scapy.runtime").setLevel(logging.ERROR) class LogWithNoPrefix(txlog.FileLogObserver): def emit(self, eventDict): @@ -50,16 +47,12 @@ def stop(): def msg(msg, *arg, **kw): print "%s" % msg -def debug(message, *arg, **kw): +def debug(msg, *arg, **kw): if config.advanced.debug: - print "[D] %s" % message - -def warn(message, *arg, **kw): - if config.advanced.show_warnings: - print "[W] %s" % message + print "[D] %s" % msg -def err(message, *arg, **kw): - print "[!] %s" % message +def err(msg, *arg, **kw): + print "[!] %s" % msg def exception(error): """ @@ -72,27 +65,6 @@ def exception(error): exc_type, exc_value, exc_traceback = sys.exc_info() traceback.print_exception(exc_type, exc_value, exc_traceback) -def fail(*failure): - logging.critical(failure) - -def catch(func): - """ - Quick wrapper to add around test methods for debugging purposes, - catches the given Exception. Use like so: - - @log.catcher - def foo(bar): - if bar == 'baz': - raise Exception("catch me no matter what I am") - foo("baz") - """ - def _catch(*args, **kwargs): - try: - func(*args, **kwargs) - except Exception, exc: - exception(exc) - return _catch - class LoggerFactory(object): """ This is a logger factory to be used by oonib diff --git a/ooni/utils/net.py b/ooni/utils/net.py index 2495197..824d720 100644 --- a/ooni/utils/net.py +++ b/ooni/utils/net.py @@ -1,39 +1,17 @@ -# -*- encoding: utf-8 -*- -# -# net.py -# -------- -# OONI utilities for network infrastructure and hardware. -# -# :authors: Isis Lovecruft, Arturo Filasto -# :version: 0.0.1-pre-alpha -# :license: (c) 2012 Isis Lovecruft, Arturo Filasto -# see attached LICENCE file - import sys import socket from random import randint -from ipaddr import IPAddress from zope.interface import implements from twisted.internet import protocol, defer from twisted.internet import threads, reactor from twisted.web.iweb import IBodyProducer -from scapy.all import utils from ooni.utils import log, txscapy -from ooni.utils import PermissionsError #if sys.platform.system() == 'Windows': # import _winreg as winreg -PLATFORMS = {'LINUX': sys.platform.startswith("linux"), - 'OPENBSD': sys.platform.startswith("openbsd"), - 'FREEBSD': sys.platform.startswith("freebsd"), - 'NETBSD': sys.platform.startswith("netbsd"), - 'DARWIN': sys.platform.startswith("darwin"), - 'SOLARIS': sys.platform.startswith("sunos"), - 'WINDOWS': sys.platform.startswith("win32")} - userAgents = [ ("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6", "Firefox 2.0, Windows XP"), ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)", "Internet Explorer 7, Windows Vista"), @@ -48,13 +26,23 @@ userAgents = [ ("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20060127 Netscape/8.1", "Netscape 8.1, Windows XP") ] - class UnsupportedPlatform(Exception): """Support for this platform is not currently available.""" class IfaceError(Exception): """Could not find default network interface.""" +class PermissionsError(SystemExit): + """This test requires admin or root privileges to run. Exiting...""" + +PLATFORMS = {'LINUX': sys.platform.startswith("linux"), + 'OPENBSD': sys.platform.startswith("openbsd"), + 'FREEBSD': sys.platform.startswith("freebsd"), + 'NETBSD': sys.platform.startswith("netbsd"), + 'DARWIN': sys.platform.startswith("darwin"), + 'SOLARIS': sys.platform.startswith("sunos"), + 'WINDOWS': sys.platform.startswith("win32")} + class StringProducer(object): implements(IBodyProducer) @@ -112,10 +100,6 @@ def getWindowsIfaces(): return ifup def getIfaces(platform_name=None): - """ - Determines if the client's OS is Windows or Posix based, and then calls - the appropriate function for retrieving interfaces. - """ client, test = getClientPlatform(platform_name) if client: if client == ('LINUX' or 'DARWIN') or client[-3:] == 'BSD': @@ -154,16 +138,8 @@ def randomFreePort(addr="127.0.0.1"): def checkInterfaces(ifaces=None, timeout=1): """ - Check given network interfaces to see that they can send and receive - packets. This is similar to :func:`getDefaultIface`, except that function - only retrieves the name of the interface which is associated with the LAN, - whereas this function validates tx/rx capabilities. - @param ifaces: - (optional) A dictionary in the form of ifaces['if_name'] = 'if_addr'. - @param timeout: - An integer specifying the number of seconds to timeout if - no reply is received for our pings. + A dictionary in the form of ifaces['if_name'] = 'if_addr'. """ try: from scapy.all import IP, ICMP @@ -200,15 +176,6 @@ def checkInterfaces(ifaces=None, timeout=1): raise IfaceError def getNonLoopbackIfaces(platform_name=None): - """ - Get the iface names of all client network interfaces which are not - the loopback interface, regardless of whether they route to internal - or external networks. - - @param platform_name: (optional) The client interface, if known. Should - be given precisely as listed in ooni.utils.net.PLATFORMS. - @return: A list of strings of non-loopback iface names. - """ try: ifaces = getIfaces(platform_name) except UnsupportedPlatform, up: @@ -229,75 +196,8 @@ def getNonLoopbackIfaces(platform_name=None): else: return interfaces -def getNetworksFromRoutes(): - """ - Get the networks this client is current on from the kernel routing table. - Each network is returned as a :class:`ipaddr.IPNetwork`, with the - network range as the name of the network, i.e.: - - network.compressed = '127.0.0.1/32' - network.netmask = IPv4Address('255.0.0.0') - network.ipaddr = IPv4Address('127.0.0.1') - network.gateway = IPv4Address('0.0.0.0') - network.iface = 'lo' - - This is mostly useful for retrieving the default network interface in a - portable manner, though it could be used to conduct local network checks - for things like rogue DHCP servers, or perhaps test that the clients NAT - router is not the mistakenly the source of a perceived censorship event. - - @return: A list of :class:`ipaddr.IPNetwork` objects with routing table - information. - """ - from scapy.all import conf, ltoa, read_routes - from ipaddr import IPNetwork, IPAddress - - conf.verb = 0 # Hide the warnings - networks = [] - for nw, nm, gw, iface, addr in read_routes(): - n = IPNetwork( ltoa(nw) ) - (n.netmask, n.gateway, n.ipaddr) = [ IPAddress(x) for x in - [nm, gw, addr] ] - n.iface = iface - if not n.compressed in networks: - networks.append(n) - return networks - -def getDefaultIface(): - """ - Get the client's default network interface. - - @return: A string containing the name of the default working interface. - @raise IfaceError: If no working interface is found. - """ - networks = getNetworksFromRoutes() - for net in networks: - if net.is_private: - return net.iface - raise IfaceError def getLocalAddress(): - """ - Get the rfc1918 IP address of the default working network interface. - - @return: The properly-formatted, validated, local IPv4/6 address of the - client's default working network interface. - """ default_iface = getDefaultIface() return default_iface.ipaddr -def checkIPandPort(raw_ip, raw_port): - """ - Check that IP and Port are a legitimate address and portnumber. - - @return: The validated ip and port, else None. - """ - try: - port = int(raw_port) - assert port in xrange(1, 65535), "Port out of range." - ip = IPAddress(raw_ip) ## either IPv4 or IPv6 - except Exception, e: - log.err(e) - return - else: - return ip.compressed, port diff --git a/ooni/utils/txscapy.py b/ooni/utils/txscapy.py index 7fa31fa..62bde94 100644 --- a/ooni/utils/txscapy.py +++ b/ooni/utils/txscapy.py @@ -1,12 +1,3 @@ -# -*- coding: utf-8 -*- -# -# txscapy -# ******* -# Here shall go functions related to using scapy with twisted. -# -# This software has been written to be part of OONI, the Open Observatory of -# Network Interference. More information on that here: http://ooni.nu/ - import struct import socket import os @@ -18,21 +9,25 @@ from twisted.internet import reactor, threads, error from twisted.internet import defer, abstract from zope.interface import implements -from scapy.all import BasePacketList, conf, PcapReader -from scapy.all import Gen, SetGen, MTU -from scapy.error import Scapy_Exception +from scapy.config import conf -from ooni import config from ooni.utils import log +from ooni import config try: + conf.use_pcap = True + conf.use_dnet = True + from scapy.all import PcapWriter from scapy.arch import pcapdnet + config.pcap_dnet = True - conf.use_pcap = True - conf.use_dnet = True -except ImportError: - log.err("pypcap or dnet not installed. Certain tests may not work.") + from scapy.all import Gen, SetGen, MTU + +except ImportError, e: + log.err("pypcap or dnet not installed. " + "Certain tests may not work.") + config.pcap_dnet = False conf.use_pcap = False conf.use_dnet = False @@ -40,16 +35,12 @@ except ImportError: class DummyPcapWriter: def __init__(self, pcap_filename, *arg, **kw): log.err("Initializing DummyPcapWriter. We will not actually write to a pcapfile") + def write(self): pass - PcapWriter = DummyPcapWriter + PcapWriter = DummyPcapWriter -class ProtocolNotRegistered(Exception): - pass - -class ProtocolAlreadyRegistered(Exception): - pass def getNetworksFromRoutes(): @@ -57,6 +48,9 @@ def getNetworksFromRoutes(): from scapy.all import conf, ltoa, read_routes from ipaddr import IPNetwork, IPAddress + ## Hide the 'no routes' warnings + conf.verb = 0 + networks = [] for nw, nm, gw, iface, addr in read_routes(): n = IPNetwork( ltoa(nw) ) @@ -81,6 +75,11 @@ def getDefaultIface(): return net.iface raise IfaceError +class ProtocolNotRegistered(Exception): + pass + +class ProtocolAlreadyRegistered(Exception): + pass class ScapyFactory(abstract.FileDescriptor): """ @@ -88,27 +87,14 @@ class ScapyFactory(abstract.FileDescriptor): https://github.com/enki/muXTCP/blob/master/scapyLink.py """ def __init__(self, interface, super_socket=None, timeout=5): + abstract.FileDescriptor.__init__(self, reactor) if interface == 'auto': interface = getDefaultIface() if not super_socket: - try: - # scapy is missing an import in /scapy/arch/linux.py - # see /ooni/lib/000-scapy-missing-exc.patch - super_socket = conf.L3socket(iface=interface, - promisc=True, filter='') - #super_socket = conf.L2socket(iface=interface) - except NameError, ne: - raise Scapy_Exception("Filter parse error") - except Scapy_Exception, se: - log.err("txscapy: %s" % se.message) - log.debug("txscapy: Trying socket setup again without filter") - try: - super_socket = conf.L3socket(iface=interface, - promisc=True) - except: - log.err("txscapy: Socket setup failed, giving up...") - raise sys.exit(1) + super_socket = conf.L3socket(iface=interface, + promisc=True, filter='') + #super_socket = conf.L2socket(iface=interface) self.protocols = [] fdesc._setCloseOnExec(super_socket.ins.fileno()) @@ -167,13 +153,9 @@ class ScapyProtocol(object): raise NotImplementedError class ScapySender(ScapyProtocol): - - # This deferred will fire when we have finished sending and receiving - # packets. timeout = 5 - if config.advanced.default_timeout: - timeout = int(config.advanced.default_timeout) + # This deferred will fire when we have finished sending a receiving packets. # Should we look for multiple answers for the same sent packet? multi = False @@ -197,7 +179,7 @@ class ScapySender(ScapyProtocol): break if len(self.answered_packets) == len(self.sent_packets): - # All of our questions have been answered. + log.debug("All of our questions have been answered.") self.stopSending() return @@ -254,10 +236,7 @@ class ScapySender(ScapyProtocol): class ScapySniffer(ScapyProtocol): def __init__(self, pcap_filename, *arg, **kw): - # The "str(pcap_filename)" explicit typing is due to an error where - # scapy.utils.PcapWriter expects strings, and it's getting unicode - # due to the "# -*- coding: utf-8 -*-"...this might be a problem... - self.pcapwriter = PcapWriter(str(pcap_filename), *arg, **kw) + self.pcapwriter = PcapWriter(pcap_filename, *arg, **kw) def packetReceived(self, packet): self.pcapwriter.write(packet) diff --git a/oonib/README.md b/oonib/README.md index d11f876..27c1163 100644 --- a/oonib/README.md +++ b/oonib/README.md @@ -5,12 +5,8 @@ The extra dependencies necessary to run OONIB are: * twisted-names * cyclone: https://github.com/fiorix/cyclone -We recommend that you use a python virtualenv. See OONI's README.md. - # Generate self signed certs for OONIB -If you want to use the HTTPS test helper, you will need to create a certificate: - openssl genrsa -des3 -out private.key 4096 openssl req -new -key private.key -out server.csr cp private.key private.key.org @@ -19,99 +15,9 @@ If you want to use the HTTPS test helper, you will need to create a certificate: openssl x509 -req -days 365 -in server.csr -signkey private.key -out certificate.crt rm private.key.org -Don't forget to update oonib/config.py options helpers.ssl.private_key and -helpers.ssl.certificate - # Redirect low ports with iptables -The following iptables commands will map connections on low ports to those bound by oonib - # Map port 80 to config.helpers.http_return_request.port (default: 57001) iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 57001 # Map port 443 to config.helpers.ssl.port (default: 57006) iptables -t nat -A PREROUTING -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 57006 - # Map port 53 udp to config.helpers.dns.udp_port (default: 57004) - iptables -t nat -A PREROUTING -p tcp -m udp --dport 53 -j REDIRECT --tor-ports - # Map port 53 tcp to config.helpers.dns.tcp_port (default: 57005) - iptables -t nat -A PREROUTING -p tcp -m tcp --dport 53 -j REDIRECT --tor-ports - -# Install Tor (Debian). - -You will need a Tor binary on your system. For complete instructions, see also: - - https://www.torproject.org/docs/tor-doc-unix.html.en - https://www.torproject.org/docs/rpms.html.en - -Add this line to your /etc/apt/sources.list, replacing <DISTRIBUTION> -where appropriate: - - deb http://deb.torproject.org/torproject.org <DISTRIBUTION> main - -Add the Tor Project gpg key to apt: - - gpg --keyserver keys.gnupg.net --recv 886DDD89 - gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add - - # Update apt and install the torproject keyring, tor, and geoipdb - apt-get update - apt-get install deb.torproject.org-keyring tor tor-geoipdb - -# Update ooni-probe/oonib/config.py - - Set config.main.tor_binary to your Tor path - Set config.main.tor2webmode = False - -# (For Experts Only) Tor2webmode: - -WARNING: provides no anonymity! Use only if you know what you are doing! -Tor2webmode will improve the performance of the collector Hidden Service -by discarding server-side anonymity. - -You will need to build Tor from source. At the time of writing, the latest stable Tor is tor-0.2.3.25. You should use the most recent stable Tor. - -Example: - - git clone https://git.torproject.org/tor.git - git checkout tor-0.2.3.25 - git verify-tag -v tor-0.2.3.25 - -You should see: - - object 17c24b3118224d6536c41fa4e1493a831fb29f0a - type commit - tag tor-0.2.3.25 - tagger Roger Dingledine <arma@torproject.org> 1353399116 -0500 - - tag 0.2.3.25 - gpg: Signature made Tue 20 Nov 2012 08:11:59 AM UTC using RSA key ID 19F78451 - gpg: Good signature from "Roger Dingledine <arma@mit.edu>" - gpg: aka "Roger Dingledine <arma@freehaven.net>" - gpg: aka "Roger Dingledine <arma@torproject.org>" - -It is always good idea to verify. - - gpg --fingerprint 19F78451 - pub 4096R/19F78451 2010-05-07 - Key fingerprint = F65C E37F 04BA 5B36 0AE6 EE17 C218 5258 19F7 8451 - uid Roger Dingledine <arma@mit.edu> - uid Roger Dingledine <arma@freehaven.net> - uid Roger Dingledine <arma@torproject.org> - sub 4096R/9B11185C 2012-05-02 [expires: 2013-05-02] - -Build Tor with enable-tor2web-mode - - ./autogen.sh ; ./configure --enable-tor2web-mode ; make - -Copy the tor binary from src/or/tor somewhere and set the corresponding -options in oonib/config.py - -# To launch oonib on system boot - -To launch oonib on startup, you may want to use supervisord (www.supervisord.org) -The following supervisord config will use the virtual environment in -/home/ooni/venv_oonib and start oonib on boot: - - [program:oonib] - command=/home/ooni/venv_oonib/bin/python /home/ooni/ooni-probe/bin/oonib - autostart=true - user=oonib - directory=/home/oonib/ diff --git a/ooniprobe.conf.sample b/ooniprobe.conf.sample index 99faa4b..ed77dfe 100644 --- a/ooniprobe.conf.sample +++ b/ooniprobe.conf.sample @@ -17,21 +17,15 @@ privacy: # Should we collect a full packet capture on the client? includepcap: false advanced: - # XXX change this to point to the directory where you have stored the - # GeoIP database file. This should be the directory in which OONI is - # installed /path/to/ooni-probe/data/ + # XXX change this to point to the directory where you have stored the GeoIP + # database file. This should be the directory in which OONI is installed + # /path/to/ooni-probe/data/ geoip_data_dir: /usr/share/GeoIP/ - # Should we display debug messages?: debug: true - # Should we display warning messages?: - show_warnings: true - # How many seconds should we wait for connections before timing out?: - default_timeout: 30 - # Location where Tor is installed: tor_binary: '/usr/sbin/tor' - # To automatically detect the system default networking interface, use: + # For auto detection interface: auto - # Or specify a specific interface: + # Of specify a specific interface #interface: wlan0 # If you do not specify start_tor, you will have to have Tor running and # explicitly set the control port and orport. diff --git a/scripts/before_i_commit.sh b/scripts/before_i_commit.sh index a504ad8..918b137 100755 --- a/scripts/before_i_commit.sh +++ b/scripts/before_i_commit.sh @@ -1,4 +1,4 @@ -#!/bin/sh +#!/bin/bash # This script should be run before you commit to verify that the basic tests # are working as they should # Once you have run it you can inspect the log file via @@ -32,12 +32,11 @@ fi echo "Below you should not see anything" echo "---------------------------------" -[ -f before_i_commit.log ] && grep "Error: " before_i_commit.log +grep "Error: " before_i_commit.log echo "---------------------------------" echo "If you do, it means something is wrong." echo "Read through the log file and fix it." echo "If you are having some problems fixing some things that have to do with" echo "the core of OONI, let's first discuss it on IRC, or open a ticket" read -cat *yamloo | less -rm -f *yamloo +#cat *.yamloo | less