November 2012 - tor-commits - lists.torproject.org

[ooni-probe/master] Update documentation on how to setup ooniprobe
by art＠torproject.org 11 Nov '12

11 Nov '12

commit fa476d07dccbfd58a186700401d2f337e8213e4b Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 18:50:09 2012 +0100 Update documentation on how to setup ooniprobe --- INSTALL | 1 - README.md | 44 ++++++++++++++++- docs/source/index.rst | 51 ++------------------ docs/source/install.rst | 119 ----------------------------------------------- 4 files changed, 45 insertions(+), 170 deletions(-) diff --git a/INSTALL b/INSTALL deleted file mode 120000 index 5c72744..0000000 --- a/INSTALL +++ /dev/null @@ -1 +0,0 @@ -docs/source/install.rst \ No newline at end of file diff --git a/README.md b/README.md index 137386a..bb17cc7 100644 --- a/README.md +++ b/README.md @@ -29,9 +29,13 @@ On debian based systems these can be installed with: The python dependencies required for running ooniprobe are: - * Twisted - * Scapy >= 2.2.0 - * txtorcon + * Twisted (>12.0.0): http://twistedmatrix.com/trac/ + * PyYAML: http://pyyaml.org/ + * Scapy: http://www.secdev.org/projects/scapy/ + * pypcap: http://code.google.com/p/pypcap/ + * libdnet: http://code.google.com/p/libdnet/ + * BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ + * txtorcon: https://github.com/meejah/txtorcon They can be installed from the requirements.txt with: @@ -53,6 +57,40 @@ Then install OONI with: pip install -r requirements.txt +## Installing scapy + +Unfortunately the version of scapy that is stored on pypi is out of date, this +means that you will have to manually download and install scapy. + +This can be done like so: + + wget http://www.secdev.org/projects/scapy/files/scapy-latest.tar.gz + tar xzf scapy-latest.tar.gz + cd scapy-latest.tar.gz + cd scapy-latest + python setup.py install + +If you are not inside of a virtual env the last command will have to be: + + sudo python setup.py install + +## Including your geo data in the test report + +Including geografical information on where your probe is located helps us +better assess the value of the test. You can personalize these setting from +inside of ooniprobe.conf + +If you wish to include geografical data in the test report, you will have to go +to the data/ directory and run: + + make geoip + +Then edit your ooniprobe.conf to point to the absolute path of where the data/ +directory is located for example: + + geoip_data_dir: /home/your_user/ooni-probe/data/ + + ## Running some tests To see the possible command line options run: diff --git a/docs/source/index.rst b/docs/source/index.rst index 132381f..ccfaad4 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,56 +1,13 @@ .. OONI documentation master file. -Welcome to the OONI documentation! -================================== +Welcome to the OONI developer documentation +=========================================== - The Net interprets censorship as damage and routes around it. - John Gilmore; TIME magazine (6 December 1993) +If you are looking for how to start running the ooniprobe tool see: -OONI, the Open Observatory of Network Interference, is a global observation -network which aims is to collect high quality data using open methodologies, -using Free and Open Source Software (FL/OSS) to share observations and data -about the various types, methods, and amounts of network tampering in the world. + https://github.com/hellais/ooni-probe#getting-started -Getting started -*************** - -Requirements: - - * Git: http://git-scm.com/book/en/Getting-Started-Installing-Git - * Python >= 2.6: http://www.python.org/download/releases/ - * pip: http://www.pip-installer.org/en/latest/ - -On debian based systems these can be installed with: - - apt-get install git-core python python-pip python-dev - -The python dependencies required for running ooniprobe are: - - * Twisted - * Scapy >= 2.2.0 - * txtorcon - -They can be installed from the requirements.txt with: - - pip install -r requirements.txt - -You are highly recommended to do so from inside of a virtual environment, since -pip does not download the packages via SSL and you will need to install it -system wide. - -This will require you to have installed virtualenv. - - apt-get install python-virtualenv - -To create a new virtual environment do - - virtualenv env - -Then install OONI with: - - pip install -r requirements.txt - Contents ******** diff --git a/docs/source/install.rst b/docs/source/install.rst deleted file mode 100644 index 4a24d3a..0000000 --- a/docs/source/install.rst +++ /dev/null @@ -1,119 +0,0 @@ - -Installing OONI -=============== - -Dependencies -************ - -OONI depends on the following pieces of software. - -* Twisted (>12.0.0): http://twistedmatrix.com/trac/ -* PyYAML: http://pyyaml.org/ -* Scapy: http://www.secdev.org/projects/scapy/ - * pypcap: http://code.google.com/p/pypcap/ - * libdnet: http://code.google.com/p/libdnet/ -* BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ -* Txtorcon: https://github.com/meejah/txtorcon - -*Optional* - -* Dnspython [1]: http://www.dnspython.org/ -* Paramiko [2]: http://www.lag.net/paramiko/ - -If you would like to use Scapy to plot graphs of network activity, -or be capable of dumping obtained information to a PostScript or PDF -file, then you'll need: - - * gnuplot-py - * numpy - * Pyx - -[1][2] These dependencies will soon be removed completely. - -Debian packages ---------------- - -On debian you can install most of the dependecies with apt-get with this command:: - - apt-get install python-yaml python-scapy python-beautifulsoup python-pypcap python-dumbnet python-dnspython - -Note that debian (squeeze) distributes version 10.1.0 of python-twisted, -and ubuntu (precise) distributes version 11.1.0. -You need the following packages for twisted to compile:: - - apt-get install python-dev build-essential - -Txtorcon has the following addtitional dependencies:: - - apt-get install python-geoip python-ipaddr python-psutil - -Python virtual environmentsa (virtualenv) ------------------------------------------ - -You may prefer to install these dependencies within a python virtualenv[3]:: - - sudo apt-get install python-virtualenv virtualenvwrapper - -From the docs (/usr/share/doc/virtualenvwrapper/README.Debian): - - Virtualenvwrapper is enabled if you install the package bash-completion and - enable bash completion support in /etc/bash.bashrc or your ~/.bashrc. - - If you only want to use virtualenvwrapper you may just add - source /etc/bash_completion.d/virtualenvwrapper to your ~/.bashrc. - -Source the script or login again and set up a virtualenv:: - - mkdir $HOME/.virtualenvs - mkvirtualenv ooni-probe - -You will automatically enter the environment. easy_install and pip will install -packages inside this environment, and will not require root privileges. -[3] http://www.virtualenv.org - -Manual installation of python dependencies -------------------------------------------- - -This involves installing the dependencies installable via easy_install/pip and -the ones that are not by building them from source. - -"simple" dependencies via easy_install:: - - sudo easy_install pyyaml twisted beautifulsoup pygeoip six ipaddr psutil txtorcon - -"simple" dependencies via pip:: - - sudo pip install pyyaml twisted beautifulsoup pygeoip six ipaddr psutil txtorcon - -Manual installation of scapy ----------------------------- - -It is optimal to install scapy, libdnet and pypcap from source. This can be -done with the following code snippets. - -libdnet:: - - wget http://libdnet.googlecode.com/files/libdnet-1.12.tgz - tar xzf libdnet-1.12.tgz - cd libdnet-1.12 - ./configure && make - cd python/ - sudo python setup.py install - cd ../../ && rm -rf libdnet-1.12* - -pypcap:: - - svn checkout http://pypcap.googlecode.com/svn/trunk/ pypcap-read-only - cd pypcap-read-only/ - sudo pip install pyrex - make - sudo python setup.py install - cd ../ && rm -rf pypcap-read-only - -scapy:: - - wget http://www.secdev.org/projects/scapy/files/scapy-latest.zip - unzip scapy-latest.zip - cd scapy-2.2.0/ - sudo python setup.py install - cd ../ && rm -rf scapy-*

1 0

[ooni-probe/master] Start working on remote reporting system
by art＠torproject.org 11 Nov '12

11 Nov '12

commit 187b8e98d061da05168ac1f9406154ef6d587038 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 00:43:44 2012 +0100 Start working on remote reporting system * Still broken --- ooni/reporter.py | 86 +++++++++++++++++++++++++++++++++++++++++------ ooni/runner.py | 10 ++++-- ooni/templates/httpt.py | 3 +- 3 files changed, 84 insertions(+), 15 deletions(-) diff --git a/ooni/reporter.py b/ooni/reporter.py index 5d29d54..84137b6 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -15,11 +15,6 @@ import yaml import json import traceback -from yaml.representer import * -from yaml.emitter import * -from yaml.serializer import * -from yaml.resolver import * - from twisted.python.util import untilConcludes from twisted.trial import reporter from twisted.internet import defer, reactor @@ -114,12 +109,20 @@ class OONIBReporter(object): 'test_name': test_name, 'test_version': test_version, 'progress': 0} - log.debug("Creating report via url %s" % url) - bodyProducer = StringProducer(json.dumps(request)) - d = self.agent.request("POST", url, bodyProducer=bodyProducer) - d.addCallback(self._processResponseBody, self._newReportCreated) - return d - + report = {'input': test_input, + 'test_name': test_name, + 'test_started': test_started, + 'report': test_report} + self.writeReportEntry(report) + + def allDone(self): + log.debug("allDone: Finished running all tests") + self.finish() + try: + reactor.stop() + except: + pass + return None class YamlReporter(object): """ @@ -167,6 +170,7 @@ class OReporter(YamlReporter): client_geodata = {} +<<<<<<< Updated upstream if config.privacy.includeip or \ config.privacy.includeasn or \ config.privacy.includecountry or \ @@ -176,12 +180,21 @@ class OReporter(YamlReporter): client_location = geodata.IPToLocation(client_ip) else: client_ip = "127.0.0.1" +======= +class OONIBReporter(OReporter): + def __init__(self, backend_url): + from twisted.web.client import Agent + from twisted.internet import reactor + self.agent = Agent(reactor) + self.backend_url = backend_url +>>>>>>> Stashed changes if config.privacy.includeip: client_geodata['ip'] = client_ip else: client_geodata['ip'] = "127.0.0.1" +<<<<<<< Updated upstream client_geodata['asn'] = None client_geodata['city'] = None client_geodata['countrycode'] = None @@ -194,6 +207,54 @@ class OReporter(YamlReporter): if config.privacy.includecountry: client_geodata['countrycode'] = client_location['countrycode'] +======= + def _processResponseBody(self, *arg, **kw): + log.msg("processResponseBody ------------------") + print arg, kw + #done = defer.Deferred() + #response.deliverBody(BodyReceiver(done)) + #done.addCallback(self._newReportCreated) + #return done + + def createReport(self, options): + test_name = options['name'] + test_version = options['version'] + + log.debug("Creating report with OONIB Reporter") + url = self.backend_url + '/report/new' + software_version = '0.0.1' + + def gotDetails(test_details): + content = '---\n' + content += safe_dump(test_details) + content += '...\n' + + request = {'software_name': 'ooniprobe', + 'software_version': software_version, + 'test_name': test_name, + 'test_version': test_version, + 'progress': 0, + 'content': content + } + log.debug("Creating report via url %s" % url) + request_json = json.dumps(request) + log.debug("Sending %s" % request_json) + + def bothCalls(*arg, **kw): + print "In HERE y0" + print arg, kw + + body_producer = StringProducer(request_json) + d = self.agent.request("POST", url, None, + body_producer) + d.addBoth(self._processResponseBody) + return d + + d = getTestDetails(options) + d.addCallback(gotDetails) + # XXX handle errors + return d +>>>>>>> Stashed changes test_details = {'start_time': otime.utcTimeNow(), @@ -224,6 +285,7 @@ class OReporter(YamlReporter): 'report': test_report} self.writeReportEntry(report) +<<<<<<< Updated upstream def allDone(self): log.debug("allDone: Finished running all tests") self.finish() @@ -233,3 +295,5 @@ class OReporter(YamlReporter): pass return None +======= +>>>>>>> Stashed changes diff --git a/ooni/runner.py b/ooni/runner.py index 1affb15..c2105ea 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -223,10 +223,14 @@ def runTestCases(test_cases, options, test_inputs = [None] reportFile = open(yamloo_filename, 'w+') - oreporter = reporter.OReporter(reportFile) + #oreporter = reporter.YAMLReporter(reportFile) + oreporter = reporter.OONIBReporter('http://127.0.0.1:8888') + input_unit_factory = InputUnitFactory(test_inputs) - yield oreporter.writeReportHeader(options) + log.debug("Creating report") + yield oreporter.createReport(options) + # This deferred list is a deferred list of deferred lists # it is used to store all the deferreds of the tests that # are run @@ -238,7 +242,7 @@ def runTestCases(test_cases, options, test_class = test_case[0] test_method = test_case[1] yield runTestWithInputUnit(test_class, - test_method, input_list, + test_method, input_list, oreporter) oreporter.allDone() diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index 4f3be3d..8360d7d 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -143,7 +143,8 @@ class HTTPTest(NetTestCase): d.addCallback(finished) return d - def build_request(self, url, method="GET", headers=None, body=None): + def build_request(self, url, method="GET", + headers=None, body=None): self.request['method'] = method self.request['url'] = url self.request['headers'] = headers if headers else {}

1 0

[ooni-probe/master] Respect the includeip = false option in ooniprobe.conf for scapyt derivate tests
by art＠torproject.org 11 Nov '12

11 Nov '12

commit dc01cb60c73d4a57df3e01e6acf0d136d2e79e2c Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 15:38:05 2012 +0100 Respect the includeip = false option in ooniprobe.conf for scapyt derivate tests * If the option to not include the IP address of the probe is set, change the source and destination ip address of the sent and received packets to 127.0.0.1 --- ooni/reporter.py | 10 +--------- ooni/templates/scapyt.py | 15 +++++++++++++-- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/ooni/reporter.py b/ooni/reporter.py index eea44c7..b5e2039 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -15,11 +15,6 @@ import yaml import json import traceback -from yaml.representer import * -from yaml.emitter import * -from yaml.serializer import * -from yaml.resolver import * - from twisted.python.util import untilConcludes from twisted.trial import reporter from twisted.internet import defer, reactor @@ -94,7 +89,6 @@ def getTestDetails(options): } defer.returnValue(test_details) - class OReporter(object): def createReport(options): """ @@ -114,10 +108,8 @@ class OReporter(object): def testDone(self, test, test_name): test_report = dict(test.report) - # XXX the scapy test has an example of how - # to do this properly. if isinstance(test.input, packet.Packet): - test_input = repr(test.input) + test_input = createPacketReport(test.input) else: test_input = test.input diff --git a/ooni/templates/scapyt.py b/ooni/templates/scapyt.py index feaf720..d137276 100644 --- a/ooni/templates/scapyt.py +++ b/ooni/templates/scapyt.py @@ -14,6 +14,8 @@ from scapy.all import send, sr, IP, TCP from ooni.nettest import NetTestCase from ooni.utils import log +from ooni import config + from ooni.utils.txscapy import ScapyProtocol def createPacketReport(packet_list): @@ -53,8 +55,17 @@ class BaseScapyTest(NetTestCase): self.report['sent_packets'] = [] for snd, rcv in answered: log.debug("Writing report for scapy test") - pkt_report_r = createPacketReport(rcv) - pkt_report_s = createPacketReport(snd) + sent_packet = snd + received_packet = rcv + + if not config.privacy.includeip: + log.msg("Detected you would not like to include your ip in the report") + log.msg("Stripping source and destination IPs from the reports") + sent_packet.src = '127.0.0.1' + received_packet.dst = '127.0.0.1' + + pkt_report_r = createPacketReport(received_packet) + pkt_report_s = createPacketReport(sent_packet) self.report['answered_packets'].append(pkt_report_r) self.report['sent_packets'].append(pkt_report_s) log.debug("Done")

1 0

[ooni-probe/master] Add ooniprobe prefix to logs
by art＠torproject.org 11 Nov '12

11 Nov '12

commit bee2164106ad2a6e8be6b6334a2efe90d7521f84 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 13:50:21 2012 +0100 Add ooniprobe prefix to logs --- ooni/utils/log.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/ooni/utils/log.py b/ooni/utils/log.py index c05c776..743161f 100644 --- a/ooni/utils/log.py +++ b/ooni/utils/log.py @@ -13,7 +13,7 @@ from twisted.python.logfile import DailyLogFile from ooni.utils import otime from ooni import config -def start(logfile=None, application_name="OONI"): +def start(logfile=None, application_name="ooniprobe"): daily_logfile = None if not logfile: @@ -27,7 +27,7 @@ def start(logfile=None, application_name="OONI"): txlog.msg("Starting %s on %s (%s UTC)" % (application_name, otime.prettyDateNow(), otime.utcPrettyDateNow())) logging.basicConfig() - python_logging = txlog.PythonLoggingObserver() + python_logging = txlog.PythonLoggingObserver(application_name) if config.advanced.debug: python_logging.logger.setLevel(logging.DEBUG)

1 0

[ooni-probe/master] Do some major refactoring of the ooniprobe backend
by art＠torproject.org 11 Nov '12

11 Nov '12

commit dce255af61ae3ee89a0d37776103919dd86e4374 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 15:41:28 2012 +0100 Do some major refactoring of the ooniprobe backend * Implement TCP Echo test helper * Refactor testhelpers into proper namespace * Remove unused imports * Remove files that were not being imported anywhere --- oonib/config.py | 39 ++++++++++++------ oonib/lib/__init__.py | 1 - oonib/lib/ssl.py | 8 ---- oonib/oonibackend.conf.sample | 12 ------ oonib/oonibackend.py | 79 ++++++++++++++++++++---------------- oonib/testhelpers/__init__.py | 5 ++ oonib/testhelpers/dns.py | 16 ------- oonib/testhelpers/dns_helpers.py | 16 +++++++ oonib/testhelpers/http_helpers.py | 80 +++++++++++++++++++++++++++++++++++++ oonib/testhelpers/httph.py | 80 ------------------------------------- oonib/testhelpers/ssl_helpers.py | 9 ++++ oonib/testhelpers/tcp_helpers.py | 14 ++++++ ooniprobe.conf | 3 +- 13 files changed, 195 insertions(+), 167 deletions(-) diff --git a/oonib/config.py b/oonib/config.py index 0156b69..dc2be2f 100644 --- a/oonib/config.py +++ b/oonib/config.py @@ -9,21 +9,34 @@ def get_root_path(): # XXX convert this to something that is a proper config file main = Storage() -main.reporting_port = 8888 -main.http_port = 8080 -main.dns_udp_port = 5354 -main.dns_tcp_port = 8002 -main.daphn3_port = 9666 -main.server_version = "Apache" +main.collector_port = 8888 + +# XXX make this be the server name that is of main.database_uri = "sqlite:"+get_root_path()+"oonib_test_db.db" main.db_threadpool_size = 10 -#main.ssl_private_key = /path/to/data/private.key -#main.ssl_certificate = /path/to/data/certificate.crt -#main.ssl_port = 8433 helpers = Storage() -helpers.http_return_request_port = 1234 -daphn3 = Storage() -daphn3.yaml_file = "/path/to/data/oonib/daphn3.yaml" -daphn3.pcap_file = "/path/to/data/server.pcap" +helpers.http_return_request = Storage() +helpers.http_return_request.port = 57001 +helpers.http_return_request.server_version = "Apache" + +helpers.tcp_echo = Storage() +helpers.tcp_echo.port = 57002 + +helpers.daphn3 = Storage() +helpers.daphn3.yaml_file = "/path/to/data/oonib/daphn3.yaml" +helpers.daphn3.pcap_file = "/path/to/data/server.pcap" + +helpers.daphn3.port = 57003 + +helpers.dns = Storage() +helpers.dns.udp_port = 57004 +helpers.dns.tcp_port = 57005 + +helpers.ssl = Storage() +#helpers.ssl.private_key = /path/to/data/private.key +#helpers.ssl.certificate = /path/to/data/certificate.crt +#helpers.ssl.port = 57007 + + diff --git a/oonib/lib/__init__.py b/oonib/lib/__init__.py deleted file mode 100644 index 8b13789..0000000 --- a/oonib/lib/__init__.py +++ /dev/null @@ -1 +0,0 @@ - diff --git a/oonib/lib/ssl.py b/oonib/lib/ssl.py deleted file mode 100644 index 20e99e8..0000000 --- a/oonib/lib/ssl.py +++ /dev/null @@ -1,8 +0,0 @@ -from twisted.internet import ssl -from oonib import config - -class SSLContext(ssl.DefaultOpenSSLContextFactory): - def __init__(self, *args, **kw): - ssl.DefaultOpenSSLContextFactory.__init__(self, config.main.ssl_private_key, - config.main.ssl_certificate) - diff --git a/oonib/oonibackend.conf.sample b/oonib/oonibackend.conf.sample deleted file mode 100644 index dc0a662..0000000 --- a/oonib/oonibackend.conf.sample +++ /dev/null @@ -1,12 +0,0 @@ -[main] -reporting_port = 8888 -http_port = 8080 -dns_udp_port = 5354 -dns_tcp_port = 8002 -daphn3_port = 9666 -server_version = Apache -ssl_private_key = /path/to/private.key -ssl_certificate = /path/to/certificate.crt -[daphn3] -pcap_file = /path/to/server.pcap -yaml_file = /path/to/server.yaml diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py index 799dcf0..8f47e73 100644 --- a/oonib/oonibackend.py +++ b/oonib/oonibackend.py @@ -3,57 +3,53 @@ # # This is the backend system responsible for running certain services that are # useful for censorship detection. - -import json -import random -import string +# +# In here we start all the test helpers that are required by ooniprobe and +# start the report collector from twisted.application import internet - -from twisted.internet import protocol, reactor, defer +from twisted.internet import reactor from twisted.application import internet, service from twisted.application.service import Application - -from twisted.web import resource, server, static -from twisted.web.microdom import escape -from twisted.protocols import basic from twisted.names import dns from ooni.utils import log from oonib.report.api import reportingBackend -from oonib.lib.ssl import SSLContext + from oonib import config -from oonib.testhelpers.httph import HTTPReturnJSONHeadersHelper -from oonib.testhelpers.dns import ProxyDNSServer +from oonib.testhelpers import dns_helpers, ssl_helpers +from oonib.testhelpers import http_helpers, tcp_helpers + #from oonib.testhelpers.daphn3 import Daphn3Server from oonib import db_threadpool from cyclone import web -# This tells twisted to set the -server.version = config.main.server_version - application = service.Application('oonibackend') serviceCollection = service.IServiceCollection(application) -if config.main.ssl_port: - internet.SSLServer(int(config.main.ssl_port), +if config.helpers.ssl.port: + log.msg("Starting SSL helper") + ssl_helper = internet.SSLServer(int(config.helpers.ssl.port), server.Site(HTTPBackend()), - SSLContext(config), - ).setServiceParent(serviceCollection) + SSLContext(config)) + ssl_helper.setServiceParent(serviceCollection) # Start the DNS Server related services -if config.main.dns_tcp_port: - TCPDNSServer = ProxyDNSServer() - internet.TCPServer(int(config.main.dns_tcp_port), - TCPDNSServer).setServiceParent(serviceCollection) - -if config.main.dns_udp_port: - UDPFactory = dns.DNSDatagramProtocol(TCPDNSServer) - internet.UDPServer(int(config.main.dns_udp_port), - UDPFactory).setServiceParent(serviceCollection) +if config.helpers.dns.tcp_port: + log.msg("Starting TCP DNS Helper on %s" % config.helpers.dns.tcp_port) + tcp_dns_helper = internet.TCPServer(int(config.helpers.dns.tcp_port), + dns_helpers.DNSTestHelper()) + tcp_dns_helper.setServiceParent(serviceCollection) + +if config.helpers.dns.udp_port: + log.msg("Starting UDP DNS Helper on %s" % config.helpers.dns.udp_port) + udp_dns_factory = dns.DNSDatagramProtocol(dns_helpers.DNSTestHelper()) + udp_dns_helper = internet.UDPServer(int(config.helpers.dns.udp_port), + udp_dns_factory) + udp_dns_helper.setServiceParent(serviceCollection) # XXX this needs to be ported # Start the OONI daphn3 backend @@ -62,12 +58,23 @@ if config.main.dns_udp_port: # internet.TCPServer(int(config.main.daphn3_port), # daphn3).setServiceParent(serviceCollection) -if config.main.reporting_port: - internet.TCPServer(int(config.main.reporting_port), - reportingBackend).setServiceParent(serviceCollection) - -if config.helpers.http_return_request_port: - internet.TCPServer(int(config.helpers.http_return_request_port), - HTTPReturnJSONHeadersHelper).setServiceParent(serviceCollection) +if config.main.collector_port: + log.msg("Starting Collector on %s" % config.main.collector_port) + collector = internet.TCPServer(int(config.main.collector_port), + reportingBackend) + collector.setServiceParent(serviceCollection) + +if config.helpers.tcp_echo.port: + log.msg("Starting TCP echo helper on %s" % config.helpers.tcp_echo.port) + tcp_echo_helper = internet.TCPServer(int(config.helpers.tcp_echo.port), + tcp_helpers.TCPEchoHelper()) + tcp_echo_helper.setServiceParent(serviceCollection) + +if config.helpers.http_return_request.port: + log.msg("Starting HTTP return request helper on %s" % config.helpers.http_return_request.port) + http_return_request_helper = internet.TCPServer( + int(config.helpers.http_return_request.port), + http_helpers.HTTPReturnJSONHeadersHelper) + http_return_request_helper.setServiceParent(serviceCollection) reactor.addSystemEventTrigger('after', 'shutdown', db_threadpool.stop) diff --git a/oonib/testhelpers/__init__.py b/oonib/testhelpers/__init__.py index e69de29..4dbb547 100644 --- a/oonib/testhelpers/__init__.py +++ b/oonib/testhelpers/__init__.py @@ -0,0 +1,5 @@ +from . import dns_helpers +from . import http_helpers +from . import tcp_helpers + +__all__ = ['dns_helpers', 'http_helpers', 'tcp_helpers'] diff --git a/oonib/testhelpers/dns.py b/oonib/testhelpers/dns.py deleted file mode 100644 index 689d1a4..0000000 --- a/oonib/testhelpers/dns.py +++ /dev/null @@ -1,16 +0,0 @@ -from twisted.internet.protocol import Factory, Protocol -from twisted.internet import reactor -from twisted.names import dns -from twisted.names import client, server - -class ProxyDNSServer(server.DNSServerFactory): - def __init__(self, authorities = None, - caches = None, clients = None, - verbose = 0): - resolver = client.Resolver(servers=[('8.8.8.8', 53)]) - server.DNSServerFactory.__init__(self, authorities = authorities, - caches = caches, clients = [resolver], - verbose = verbose) - def handleQuery(self, message, protocol, address): - print message, protocol, address - server.DNSServerFactory.handleQuery(self, message, protocol, address) diff --git a/oonib/testhelpers/dns_helpers.py b/oonib/testhelpers/dns_helpers.py new file mode 100644 index 0000000..cb4ff9f --- /dev/null +++ b/oonib/testhelpers/dns_helpers.py @@ -0,0 +1,16 @@ +from twisted.internet.protocol import Factory, Protocol +from twisted.internet import reactor +from twisted.names import dns +from twisted.names import client, server + +class DNSTestHelper(server.DNSServerFactory): + def __init__(self, authorities = None, + caches = None, clients = None, + verbose = 0): + resolver = client.Resolver(servers=[('8.8.8.8', 53)]) + server.DNSServerFactory.__init__(self, authorities = authorities, + caches = caches, clients = [resolver], + verbose = verbose) + def handleQuery(self, message, protocol, address): + print message, protocol, address + server.DNSServerFactory.handleQuery(self, message, protocol, address) diff --git a/oonib/testhelpers/http_helpers.py b/oonib/testhelpers/http_helpers.py new file mode 100644 index 0000000..7001fa6 --- /dev/null +++ b/oonib/testhelpers/http_helpers.py @@ -0,0 +1,80 @@ +import json +import random +import string + +from twisted.application import internet, service +from twisted.internet import protocol, reactor, defer +from twisted.protocols import basic +from twisted.web import resource, server, static, http +from twisted.web.microdom import escape + +from cyclone.web import RequestHandler, Application + +class HTTPTrapAll(RequestHandler): + """ + Master class to be used to trap all the HTTP methods and make capitalized + requests pass. + """ + def _execute(self, transforms, *args, **kwargs): + self._transforms = transforms + defer.maybeDeferred(self.prepare).addCallbacks( + self._execute_handler, + lambda f: self._handle_request_exception(f.value), + callbackArgs=(args, kwargs)) + + def _execute_handler(self, r, args, kwargs): + if not self._finished: + args = [self.decode_argument(arg) for arg in args] + kwargs = dict((k, self.decode_argument(v, name=k)) + for (k, v) in kwargs.iteritems()) + # This is where we do the patching + # XXX this is somewhat hackish + d = defer.maybeDeferred(self.all, *args, **kwargs) + d.addCallbacks(self._execute_success, self._execute_failure) + self.notifyFinish().addCallback(self.on_connection_close) + +class HTTPReturnJSONHeaders(HTTPTrapAll): + def all(self): + # XXX make sure that the request headers are in the correct order + submitted_data = {'request_body': self.request.body, + 'request_headers': self.request.headers, + 'request_uri': self.request.uri, + 'request_method': self.request.method} + response = json.dumps(submitted_data) + self.write(response) + +HTTPReturnJSONHeadersHelper = Application([ + (r"/*", HTTPReturnJSONHeaders) +]) + + +class HTTPRandomPage(HTTPTrapAll): + """ + This generates a random page of arbitrary length and containing the string + selected by the user. + /<length>/<keyword> + XXX this is currently disabled as it is not of use to any test. + """ + isLeaf = True + def _gen_random_string(self, length): + return ''.join(random.choice(string.letters) for x in range(length)) + + def genRandomPage(self, length=100, keyword=None): + data = self._gen_random_string(length/2) + if keyword: + data += keyword + data += self._gen_random_string(length - length/2) + data += '\n' + return data + + def all(self, length, keyword): + length = 100 + if length > 100000: + length = 100000 + return self.genRandomPage(length, keyword) + +HTTPRandomPageHelper = Application([ + # XXX add regexps here + (r"/(.*)/(.*)", HTTPRandomPage) +]) + diff --git a/oonib/testhelpers/httph.py b/oonib/testhelpers/httph.py deleted file mode 100644 index 7001fa6..0000000 --- a/oonib/testhelpers/httph.py +++ /dev/null @@ -1,80 +0,0 @@ -import json -import random -import string - -from twisted.application import internet, service -from twisted.internet import protocol, reactor, defer -from twisted.protocols import basic -from twisted.web import resource, server, static, http -from twisted.web.microdom import escape - -from cyclone.web import RequestHandler, Application - -class HTTPTrapAll(RequestHandler): - """ - Master class to be used to trap all the HTTP methods and make capitalized - requests pass. - """ - def _execute(self, transforms, *args, **kwargs): - self._transforms = transforms - defer.maybeDeferred(self.prepare).addCallbacks( - self._execute_handler, - lambda f: self._handle_request_exception(f.value), - callbackArgs=(args, kwargs)) - - def _execute_handler(self, r, args, kwargs): - if not self._finished: - args = [self.decode_argument(arg) for arg in args] - kwargs = dict((k, self.decode_argument(v, name=k)) - for (k, v) in kwargs.iteritems()) - # This is where we do the patching - # XXX this is somewhat hackish - d = defer.maybeDeferred(self.all, *args, **kwargs) - d.addCallbacks(self._execute_success, self._execute_failure) - self.notifyFinish().addCallback(self.on_connection_close) - -class HTTPReturnJSONHeaders(HTTPTrapAll): - def all(self): - # XXX make sure that the request headers are in the correct order - submitted_data = {'request_body': self.request.body, - 'request_headers': self.request.headers, - 'request_uri': self.request.uri, - 'request_method': self.request.method} - response = json.dumps(submitted_data) - self.write(response) - -HTTPReturnJSONHeadersHelper = Application([ - (r"/*", HTTPReturnJSONHeaders) -]) - - -class HTTPRandomPage(HTTPTrapAll): - """ - This generates a random page of arbitrary length and containing the string - selected by the user. - /<length>/<keyword> - XXX this is currently disabled as it is not of use to any test. - """ - isLeaf = True - def _gen_random_string(self, length): - return ''.join(random.choice(string.letters) for x in range(length)) - - def genRandomPage(self, length=100, keyword=None): - data = self._gen_random_string(length/2) - if keyword: - data += keyword - data += self._gen_random_string(length - length/2) - data += '\n' - return data - - def all(self, length, keyword): - length = 100 - if length > 100000: - length = 100000 - return self.genRandomPage(length, keyword) - -HTTPRandomPageHelper = Application([ - # XXX add regexps here - (r"/(.*)/(.*)", HTTPRandomPage) -]) - diff --git a/oonib/testhelpers/ssl_helpers.py b/oonib/testhelpers/ssl_helpers.py new file mode 100644 index 0000000..2c07275 --- /dev/null +++ b/oonib/testhelpers/ssl_helpers.py @@ -0,0 +1,9 @@ +from twisted.internet import ssl +from oonib import config + +class SSLContext(ssl.DefaultOpenSSLContextFactory): + def __init__(self, *args, **kw): + ssl.DefaultOpenSSLContextFactory.__init__(self, + config.helpers.ssl.private_key, + config.main.ssl.certificate) + diff --git a/oonib/testhelpers/tcp_helpers.py b/oonib/testhelpers/tcp_helpers.py new file mode 100644 index 0000000..57b93d2 --- /dev/null +++ b/oonib/testhelpers/tcp_helpers.py @@ -0,0 +1,14 @@ +from twisted.internet.protocol import Protocol, Factory + +class TCPEchoProtocol(Protocol): + def dataReceived(self, data): + self.transport.write(data) + +class TCPEchoHelper(Factory): + """ + A very simple echo protocol implementation + """ + protocol = TCPEchoProtocol + + + diff --git a/ooniprobe.conf b/ooniprobe.conf index 47f480a..191d809 100644 --- a/ooniprobe.conf +++ b/ooniprobe.conf @@ -20,7 +20,8 @@ advanced: # XXX change this to point to the directory where you have stored the GeoIP # database file. This should be the directory in which OONI is installed # /path/to/ooni-probe/data/ - geoip_data_dir: /usr/share/GeoIP/ + #geoip_data_dir: /usr/share/GeoIP/ + geoip_data_dir: /home/x/code/networking/ooni-probe/data/ debug: true threadpool_size: 10

1 0

[ooni-probe/master] Implement basic keyword filtering detection test.
by art＠torproject.org 11 Nov '12

11 Nov '12

commit 9e7cf8c35eb29ed56e8bca2da1bc3055a50df1aa Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 15:48:22 2012 +0100 Implement basic keyword filtering detection test. * XXX Does not use the bisection method * XXX Does not establish a TCP session, but just sends --- nettests/core/keyword_filtering.py | 52 ++++++++++++++++++++++++++++++++++++ 1 files changed, 52 insertions(+), 0 deletions(-) diff --git a/nettests/core/keyword_filtering.py b/nettests/core/keyword_filtering.py new file mode 100644 index 0000000..9eec4ff --- /dev/null +++ b/nettests/core/keyword_filtering.py @@ -0,0 +1,52 @@ +# -*- encoding: utf-8 -*- +# +# :authors: Arturo Filastò +# :licence: see LICENSE + +from twisted.python import usage +from twisted.internet import defer + +from ooni.utils import log +from ooni.templates import scapyt + +from scapy.all import * + +class UsageOptions(usage.Options): + optParameters = [ + ['backend', 'b', '127.0.0.1:57002', 'Test backend running TCP echo'], + ['timeout', 't', 5, 'Timeout after which to give up waiting for RST packets'] + ] + +class KeywordFiltering(scapyt.BaseScapyTest): + name = "Keyword Filtering detection based on RST packets" + author = "Arturo Filastò" + version = "0.1" + + usageOptions = UsageOptions + + inputFile = ['file', 'f', None, + 'List of keywords to use for censorship testing'] + + def test_tcp_keyword_filtering(self): + """ + Places the keyword to be tested in the payload of a TCP packet. + XXX need to implement bisection method for enumerating keywords. + though this should not be an issue since we are testing all + the keywords in parallel. + """ + def finished(packets): + log.debug("Finished running TCP traceroute test on port %s" % port) + answered, unanswered = packets + self.report['rst_packets'] = [] + for snd, rcv in answered: + # The received packet has the RST flag + if rcv[TCP].flags == 4: + self.report['rst_packets'].append(rcv) + + backend_ip, backend_port = self.localOptions['backend'] + keyword_to_test = str(self.input) + packets = IP(dst=backend_ip,id=RandShort())/TCP(dport=backend_port)/keyword_to_test + d = self.sr(packets, timeout=timeout) + d.addCallback(finished) + return d +

1 0

[ooni-probe/master] Merge branch 'master' into remotereport
by art＠torproject.org 11 Nov '12

11 Nov '12

commit fb7c4962063161e621691b128f23d70402c4da36 Merge: 10f0d90 9e7cf8c Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 16:48:30 2012 +0100 Merge branch 'master' into remotereport * master: Implement basic keyword filtering detection test. * XXX Does not use the bisection method * XXX Does not establish a TCP session, but just sends Do some major refactoring of the ooniprobe backend * Implement TCP Echo test helper * Refactor testhelpers into proper namespace * Remove unused imports * Remove files that were not being imported anywhere Respect the includeip = false option in ooniprobe.conf for scapyt derivate tests * If the option to not include the IP address of the probe is set, change the source and destination ip address of the sent and received packets to 127.0.0.1 Add ooniprobe prefix to logs Clean up filenames of generated reports * They now follow the format $testName_report_$timestamp.yamloo Implement ICMP traceroute test * parametrize max_ttl and timeout * make max_ttl and timeout be included in the report Add not about the fact that tests should return a deferred. Add software version to the report * Cleanup some debug messages * Remove unused ooni-probe.conf config file Implement basic reporting to flat files containing the report ID. * XXX the location of such files is not configurable, they will leave tons of crap in your cwd. Refactor reporter Object. * We now have a parent OReporter object that is subclassed by OONIBReporter for remote reporting and YAMLReporter for reporting to YAML format on file system. * Move secure YAML serialization hacks to the hacks module * Do more progress on the implementation of reporting to remote systems Conflicts: ooni/reporter.py ooni/runner.py nettests/core/keyword_filtering.py | 52 +++++++ nettests/core/squid.py | 2 +- nettests/core/traceroute.py | 70 ++++++++-- ooni/__init__.py | 4 + ooni/config.py | 9 +- ooni/inputunit.py | 2 +- ooni/nettest.py | 3 + ooni/ooni-probe.conf | 100 -------------- ooni/oonicli.py | 8 +- ooni/reporter.py | 265 ++++++++++++++++++------------------ ooni/runner.py | 14 ++- ooni/templates/scapyt.py | 18 ++- ooni/utils/hacks.py | 47 +++++++ ooni/utils/log.py | 4 +- oonib/config.py | 39 ++++-- oonib/lib/__init__.py | 1 - oonib/lib/ssl.py | 8 - oonib/oonibackend.conf.sample | 12 -- oonib/oonibackend.py | 79 ++++++----- oonib/report/api.py | 123 +++++++++++++++-- oonib/testhelpers/__init__.py | 5 + oonib/testhelpers/dns.py | 16 -- oonib/testhelpers/dns_helpers.py | 16 ++ oonib/testhelpers/http_helpers.py | 80 +++++++++++ oonib/testhelpers/httph.py | 80 ----------- oonib/testhelpers/ssl_helpers.py | 9 ++ oonib/testhelpers/tcp_helpers.py | 14 ++ ooniprobe.conf | 3 +- 28 files changed, 646 insertions(+), 437 deletions(-) diff --cc nettests/core/squid.py index 675119c,675119c..777bc3e --- a/nettests/core/squid.py +++ b/nettests/core/squid.py @@@ -17,7 -17,7 +17,7 @@@ class SquidTest(httpt.HTTPTest) """ name = "Squid test" author = "Arturo Filastò" -- version = 0.1 ++ version = "0.1" optParameters = [['backend', 'b', 'http://ooni.nu/test/', 'Test backend to use']] diff --cc ooni/reporter.py index 31c53a9,b5e2039..7147b35 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@@ -168,122 -162,78 +162,129 @@@ class YAMLReporter(OReporter) self._writeln("# %s" % otime.prettyDateNow()) self._writeln("###########################################") - client_geodata = {} + test_details = yield getTestDetails(options) - if config.privacy.includeip or \ - config.privacy.includeasn or \ - config.privacy.includecountry or \ - config.privacy.includecity: - log.msg("Running geo IP lookup via check.torproject.org") - client_ip = yield geodata.myIP() - client_location = geodata.IPToLocation(client_ip) - else: - client_ip = "127.0.0.1" + self.writeReportEntry(test_details) - if config.privacy.includeip: - client_geodata['ip'] = client_ip - else: - client_geodata['ip'] = "127.0.0.1" + def finish(self): + self._stream.close() - client_geodata['asn'] = None - client_geodata['city'] = None - client_geodata['countrycode'] = None + class OONIBReporter(object): + def __init__(self, backend_url): + from twisted.web.client import Agent + from twisted.internet import reactor + self.agent = Agent(reactor) + self.backend_url = backend_url - if config.privacy.includeasn: - client_geodata['asn'] = client_location['asn'] + def _newReportCreated(self, data): + log.debug("newReportCreated %s" % data) + return data - if config.privacy.includecity: - client_geodata['city'] = client_location['city'] + def _processResponseBody(self, response, body_cb): + log.debug("processResponseBody %s" % response) + done = defer.Deferred() + response.deliverBody(BodyReceiver(done)) + done.addCallback(body_cb) + return done - if config.privacy.includecountry: - client_geodata['countrycode'] = client_location['countrycode'] + def createReport(self, test_name, + test_version, report_header): - + url = self.backend_url + '/new' + software_version = '0.0.1' - test_details = {'start_time': otime.utcTimeNow(), - 'probe_asn': client_geodata['asn'], - 'probe_cc': client_geodata['countrycode'], - 'probe_ip': client_geodata['ip'], - 'test_name': options['name'], - 'test_version': options['version'], - } - self.writeReportEntry(test_details) + request = {'software_name': 'ooni-probe', + 'software_version': software_version, + 'test_name': test_name, + 'test_version': test_version, + 'progress': 0, + 'content': report_header + } + def gotDetails(test_details): + log.debug("Creating report via url %s" % url) + + bodyProducer = StringProducer(json.dumps(request)) + d = self.agent.request("POST", url, + bodyProducer=bodyProducer) + d.addCallback(self._processResponseBody, + self._newReportCreated) + return d + + d = getTestDetails(options) + d.addCallback(gotDetails) + return d + + def writeReportEntry(self, entry, test_id=None): + if not test_id: + log.err("Write report entry on OONIB requires test id") + raise NoTestIDSpecified + + report = '---\n' + report += safe_dump(entry) + report += '...\n' - def testDone(self, test, test_name): - test_report = dict(test.report) + url = self.backend_url + '/new' - # XXX the scapy test has an example of how - # to do this properly. - if isinstance(test.input, packet.Packet): - test_input = repr(test.input) - else: - test_input = test.input + request = {'test_id': test_id, + 'content': report} - test_started = test._start_time - test_runtime = test_started - time.time() + bodyProducer = StringProducer(json.dumps(request)) + d = self.agent.request("PUT", url, + bodyProducer=bodyProducer) - report = {'input': test_input, - 'test_name': test_name, - 'test_started': test_started, - 'report': test_report} - self.writeReportEntry(report) + d.addCallback(self._processResponseBody, + self._newReportCreated) + return d - def allDone(self): - log.debug("allDone: Finished running all tests") - self.finish() - try: - reactor.stop() - except: - pass - return None + +class OONIBReporter(OReporter): + def __init__(self, backend_url): + from twisted.web.client import Agent + from twisted.internet import reactor + self.agent = Agent(reactor) + self.backend_url = backend_url + + def _processResponseBody(self, *arg, **kw): + #done = defer.Deferred() + #response.deliverBody(BodyReceiver(done)) + #done.addCallback(self._newReportCreated) + #return done + + def createReport(self, options): + test_name = options['name'] + test_version = options['version'] + + log.debug("Creating report with OONIB Reporter") + url = self.backend_url + '/report/new' + software_version = '0.0.1' + + def gotDetails(test_details): + content = '---\n' + content += safe_dump(test_details) + content += '...\n' + + request = {'software_name': 'ooniprobe', + 'software_version': software_version, + 'test_name': test_name, + 'test_version': test_version, + 'progress': 0, + 'content': content + } + log.debug("Creating report via url %s" % url) + request_json = json.dumps(request) + log.debug("Sending %s" % request_json) + + def bothCalls(*arg, **kw): + print arg, kw + + body_producer = StringProducer(request_json) + d = self.agent.request("POST", url, None, + body_producer) + d.addBoth(self._processResponseBody) + return d + + d = getTestDetails(options) + d.addCallback(gotDetails) + # XXX handle errors + return d + diff --cc ooni/runner.py index c2105ea,23ff898..b92ad7e --- a/ooni/runner.py +++ b/ooni/runner.py @@@ -221,16 -221,13 +221,22 @@@ def runTestCases(test_cases, options log.msg("Could not find inputs!") log.msg("options[0] = %s" % first) test_inputs = [None] - + reportFile = open(yamloo_filename, 'w+') ++ + #oreporter = reporter.YAMLReporter(reportFile) + oreporter = reporter.OONIBReporter('http://127.0.0.1:8888') + + input_unit_factory = InputUnitFactory(test_inputs) + + log.debug("Creating report") + yield oreporter.createReport(options) + + oreporter = reporter.YAMLReporter(reportFile) + + input_unit_factory = InputUnitFactory(test_inputs) + + yield oreporter.createReport(options) # This deferred list is a deferred list of deferred lists # it is used to store all the deferreds of the tests that # are run

1 0

[ooni-probe/master] Do some progress on implementing reporter
by art＠torproject.org 11 Nov '12

11 Nov '12

commit 10f0d902a12d52b24009bb7def0c59c10280f7dc Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 16:47:10 2012 +0100 Do some progress on implementing reporter --- ooni/reporter.py | 104 ++++++++++++++++++++++++----------------------------- 1 files changed, 47 insertions(+), 57 deletions(-) diff --git a/ooni/reporter.py b/ooni/reporter.py index 84137b6..31c53a9 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -114,7 +114,7 @@ class OONIBReporter(object): 'test_started': test_started, 'report': test_report} self.writeReportEntry(report) - + def allDone(self): log.debug("allDone: Finished running all tests") self.finish() @@ -170,7 +170,6 @@ class OReporter(YamlReporter): client_geodata = {} -<<<<<<< Updated upstream if config.privacy.includeip or \ config.privacy.includeasn or \ config.privacy.includecountry or \ @@ -180,21 +179,12 @@ class OReporter(YamlReporter): client_location = geodata.IPToLocation(client_ip) else: client_ip = "127.0.0.1" -======= -class OONIBReporter(OReporter): - def __init__(self, backend_url): - from twisted.web.client import Agent - from twisted.internet import reactor - self.agent = Agent(reactor) - self.backend_url = backend_url ->>>>>>> Stashed changes if config.privacy.includeip: client_geodata['ip'] = client_ip else: client_geodata['ip'] = "127.0.0.1" -<<<<<<< Updated upstream client_geodata['asn'] = None client_geodata['city'] = None client_geodata['countrycode'] = None @@ -207,10 +197,53 @@ class OONIBReporter(OReporter): if config.privacy.includecountry: client_geodata['countrycode'] = client_location['countrycode'] -======= + + test_details = {'start_time': otime.utcTimeNow(), + 'probe_asn': client_geodata['asn'], + 'probe_cc': client_geodata['countrycode'], + 'probe_ip': client_geodata['ip'], + 'test_name': options['name'], + 'test_version': options['version'], + } + self.writeReportEntry(test_details) + + def testDone(self, test, test_name): + test_report = dict(test.report) + + # XXX the scapy test has an example of how + # to do this properly. + if isinstance(test.input, packet.Packet): + test_input = repr(test.input) + else: + test_input = test.input + + test_started = test._start_time + test_runtime = test_started - time.time() + + report = {'input': test_input, + 'test_name': test_name, + 'test_started': test_started, + 'report': test_report} + self.writeReportEntry(report) + + def allDone(self): + log.debug("allDone: Finished running all tests") + self.finish() + try: + reactor.stop() + except: + pass + return None + + +class OONIBReporter(OReporter): + def __init__(self, backend_url): + from twisted.web.client import Agent + from twisted.internet import reactor + self.agent = Agent(reactor) + self.backend_url = backend_url + def _processResponseBody(self, *arg, **kw): - log.msg("processResponseBody ------------------") - print arg, kw #done = defer.Deferred() #response.deliverBody(BodyReceiver(done)) #done.addCallback(self._newReportCreated) @@ -241,7 +274,6 @@ class OONIBReporter(OReporter): log.debug("Sending %s" % request_json) def bothCalls(*arg, **kw): - print "In HERE y0" print arg, kw body_producer = StringProducer(request_json) @@ -254,46 +286,4 @@ class OONIBReporter(OReporter): d.addCallback(gotDetails) # XXX handle errors return d ->>>>>>> Stashed changes - - - test_details = {'start_time': otime.utcTimeNow(), - 'probe_asn': client_geodata['asn'], - 'probe_cc': client_geodata['countrycode'], - 'probe_ip': client_geodata['ip'], - 'test_name': options['name'], - 'test_version': options['version'], - } - self.writeReportEntry(test_details) - - def testDone(self, test, test_name): - test_report = dict(test.report) - - # XXX the scapy test has an example of how - # to do this properly. - if isinstance(test.input, packet.Packet): - test_input = repr(test.input) - else: - test_input = test.input - - test_started = test._start_time - test_runtime = test_started - time.time() - - report = {'input': test_input, - 'test_name': test_name, - 'test_started': test_started, - 'report': test_report} - self.writeReportEntry(report) - -<<<<<<< Updated upstream - def allDone(self): - log.debug("allDone: Finished running all tests") - self.finish() - try: - reactor.stop() - except: - pass - return None -======= ->>>>>>> Stashed changes

1 0

[ooni-probe/master] Implement basic collector for ooniprobe reports
by art＠torproject.org 11 Nov '12

11 Nov '12

commit 5544ee71c001d64aecc5a729724af53c8ec2f246 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 18:15:54 2012 +0100 Implement basic collector for ooniprobe reports * Reports can be submitted over the network via http to a remote collector * Implement the backend component of the collector that writes submitted reports to flat files, following the report_id naming convention. * XXX add support for connecting to the collector via Tor Hidden Services --- nettests/core/http_host.py | 2 +- ooni/oonicli.py | 2 + ooni/reporter.py | 176 ++++++++++++++++++++------------------------ ooni/runner.py | 19 +++-- ooni/utils/net.py | 2 +- oonib/config.py | 6 +- oonib/report/api.py | 44 ++++++++++- 7 files changed, 138 insertions(+), 113 deletions(-) diff --git a/nettests/core/http_host.py b/nettests/core/http_host.py index 0e73f82..662cc40 100644 --- a/nettests/core/http_host.py +++ b/nettests/core/http_host.py @@ -16,7 +16,7 @@ from ooni.utils import log from ooni.templates import httpt class UsageOptions(usage.Options): - optParameters = [['backend', 'b', 'http://127.0.0.1:1234', + optParameters = [['backend', 'b', 'http://127.0.0.1:57001', 'URL of the test backend to use'], ['content', 'c', None, 'The file to read from containing the content of a block page']] diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 9287174..9473ad8 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -40,6 +40,8 @@ class Options(usage.Options, app.ReactorSelectionMixin): 'Report deferred creation and callback stack traces'],] optParameters = [["reportfile", "o", None, "report file name"], + ["collector", "c", None, + "Address of the collector of test results. (example: http://127.0.0.1:8888)"], ["logfile", "l", None, "log file name"], ["pcapfile", "p", None, "pcap file name"]] diff --git a/ooni/reporter.py b/ooni/reporter.py index 7147b35..05ea94a 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -106,6 +106,8 @@ class OReporter(object): pass def testDone(self, test, test_name): + log.debug("Finished running %s" % test_name) + log.debug("Writing report") test_report = dict(test.report) if isinstance(test.input, packet.Packet): @@ -120,14 +122,15 @@ class OReporter(object): 'test_name': test_name, 'test_started': test_started, 'report': test_report} - self.writeReportEntry(report) + return self.writeReportEntry(report) def allDone(self): log.debug("allDone: Finished running all tests") - self.finish() try: + log.debug("Stopping the reactor") reactor.stop() except: + log.debug("Unable to stop the reactor") pass return None @@ -151,6 +154,7 @@ class YAMLReporter(OReporter): untilConcludes(self._stream.flush) def writeReportEntry(self, entry): + log.debug("Writing report with YAML reporter") self._write('---\n') self._write(safe_dump(entry)) self._write('...\n') @@ -169,88 +173,60 @@ class YAMLReporter(OReporter): def finish(self): self._stream.close() -class OONIBReporter(object): + +class OONIBReportUpdateFailed(Exception): + pass + +class OONIBReportCreationFailed(Exception): + pass + +class OONIBTestDetailsLookupFailed(Exception): + pass + +class OONIBReporter(OReporter): def __init__(self, backend_url): from twisted.web.client import Agent from twisted.internet import reactor self.agent = Agent(reactor) self.backend_url = backend_url - def _newReportCreated(self, data): - log.debug("newReportCreated %s" % data) - return data - - def _processResponseBody(self, response, body_cb): - log.debug("processResponseBody %s" % response) - done = defer.Deferred() - response.deliverBody(BodyReceiver(done)) - done.addCallback(body_cb) - return done - - def createReport(self, test_name, - test_version, report_header): - url = self.backend_url + '/new' - software_version = '0.0.1' - - request = {'software_name': 'ooni-probe', - 'software_version': software_version, - 'test_name': test_name, - 'test_version': test_version, - 'progress': 0, - 'content': report_header - } - def gotDetails(test_details): - log.debug("Creating report via url %s" % url) - - bodyProducer = StringProducer(json.dumps(request)) - d = self.agent.request("POST", url, - bodyProducer=bodyProducer) - d.addCallback(self._processResponseBody, - self._newReportCreated) - return d - - d = getTestDetails(options) - d.addCallback(gotDetails) - return d - - def writeReportEntry(self, entry, test_id=None): - if not test_id: - log.err("Write report entry on OONIB requires test id") - raise NoTestIDSpecified + @defer.inlineCallbacks + def writeReportEntry(self, entry): + log.debug("Writing report with OONIB reporter") + content = '---\n' + content += safe_dump(entry) + content += '...\n' - report = '---\n' - report += safe_dump(entry) - report += '...\n' + url = self.backend_url + '/report/new' - url = self.backend_url + '/new' + request = {'report_id': self.report_id, + 'content': content} - request = {'test_id': test_id, - 'content': report} + log.debug("Updating report with id %s" % self.report_id) + request_json = json.dumps(request) + log.debug("Sending %s" % request_json) bodyProducer = StringProducer(json.dumps(request)) - d = self.agent.request("PUT", url, - bodyProducer=bodyProducer) - - d.addCallback(self._processResponseBody, - self._newReportCreated) - return d - + log.debug("Creating report via url %s" % url) + try: + response = yield self.agent.request("PUT", url, + bodyProducer=bodyProducer) + except: + # XXX we must trap this in the runner and make sure to report the data later. + raise OONIBReportUpdateFailed -class OONIBReporter(OReporter): - def __init__(self, backend_url): - from twisted.web.client import Agent - from twisted.internet import reactor - self.agent = Agent(reactor) - self.backend_url = backend_url + #parsed_response = json.loads(backend_response) + #self.report_id = parsed_response['report_id'] + #self.backend_version = parsed_response['backend_version'] + #log.debug("Created report with id %s" % parsed_response['report_id']) - def _processResponseBody(self, *arg, **kw): - #done = defer.Deferred() - #response.deliverBody(BodyReceiver(done)) - #done.addCallback(self._newReportCreated) - #return done + @defer.inlineCallbacks def createReport(self, options): + """ + Creates a report on the oonib collector. + """ test_name = options['name'] test_version = options['version'] @@ -258,33 +234,41 @@ class OONIBReporter(OReporter): url = self.backend_url + '/report/new' software_version = '0.0.1' - def gotDetails(test_details): - content = '---\n' - content += safe_dump(test_details) - content += '...\n' + test_details = yield getTestDetails(options) + + content = '---\n' + content += safe_dump(test_details) + content += '...\n' - request = {'software_name': 'ooniprobe', - 'software_version': software_version, - 'test_name': test_name, - 'test_version': test_version, - 'progress': 0, - 'content': content - } - log.debug("Creating report via url %s" % url) - request_json = json.dumps(request) - log.debug("Sending %s" % request_json) - - def bothCalls(*arg, **kw): - print arg, kw - - body_producer = StringProducer(request_json) - d = self.agent.request("POST", url, None, - body_producer) - d.addBoth(self._processResponseBody) - return d - - d = getTestDetails(options) - d.addCallback(gotDetails) - # XXX handle errors - return d + request = {'software_name': 'ooniprobe', + 'software_version': software_version, + 'test_name': test_name, + 'test_version': test_version, + 'progress': 0, + 'content': content + } + log.debug("Creating report via url %s" % url) + request_json = json.dumps(request) + log.debug("Sending %s" % request_json) + + bodyProducer = StringProducer(json.dumps(request)) + log.debug("Creating report via url %s" % url) + + try: + response = yield self.agent.request("POST", url, + bodyProducer=bodyProducer) + except: + raise OONIBReportCreationFailed + + # This is a little trix to allow us to unspool the response. We create + # a deferred and call yield on it. + response_body = defer.Deferred() + response.deliverBody(BodyReceiver(response_body)) + + backend_response = yield response_body + + parsed_response = json.loads(backend_response) + self.report_id = parsed_response['report_id'] + self.backend_version = parsed_response['backend_version'] + log.debug("Created report with id %s" % parsed_response['report_id']) diff --git a/ooni/runner.py b/ooni/runner.py index b92ad7e..1aac145 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -153,8 +153,10 @@ def loadTestsAndOptions(classes, cmd_line_options): def runTestWithInput(test_class, test_method, test_input, oreporter): log.debug("Running %s with %s" % (test_method, test_input)) + def test_done(result, test_instance, test_name): - oreporter.testDone(test_instance, test_name) + log.debug("runTestWithInput: concluded %s" % test_name) + return oreporter.testDone(test_instance, test_name) def test_error(error, test_instance, test_name): log.err("%s\n" % error) @@ -221,22 +223,21 @@ def runTestCases(test_cases, options, log.msg("Could not find inputs!") log.msg("options[0] = %s" % first) test_inputs = [None] - + reportFile = open(yamloo_filename, 'w+') - #oreporter = reporter.YAMLReporter(reportFile) - oreporter = reporter.OONIBReporter('http://127.0.0.1:8888') + + if cmd_line_options['collector']: + oreporter = reporter.OONIBReporter(cmd_line_options['collector']) + else: + oreporter = reporter.YAMLReporter(reportFile) input_unit_factory = InputUnitFactory(test_inputs) log.debug("Creating report") - yield oreporter.createReport(options) - - oreporter = reporter.YAMLReporter(reportFile) - - input_unit_factory = InputUnitFactory(test_inputs) yield oreporter.createReport(options) + # This deferred list is a deferred list of deferred lists # it is used to store all the deferreds of the tests that # are run diff --git a/ooni/utils/net.py b/ooni/utils/net.py index d43261a..3ddba61 100644 --- a/ooni/utils/net.py +++ b/ooni/utils/net.py @@ -7,7 +7,7 @@ import sys from zope.interface import implements -from twisted.internet import protocol +from twisted.internet import protocol, defer from twisted.internet import threads, reactor from twisted.web.iweb import IBodyProducer diff --git a/oonib/config.py b/oonib/config.py index dc2be2f..b34a8ae 100644 --- a/oonib/config.py +++ b/oonib/config.py @@ -18,11 +18,13 @@ main.db_threadpool_size = 10 helpers = Storage() helpers.http_return_request = Storage() -helpers.http_return_request.port = 57001 +helpers.http_return_request.port = 57001 +# XXX this actually needs to be the advertised Server HTTP header of our web +# server helpers.http_return_request.server_version = "Apache" helpers.tcp_echo = Storage() -helpers.tcp_echo.port = 57002 +helpers.tcp_echo.port = 57002 helpers.daphn3 = Storage() helpers.daphn3.yaml_file = "/path/to/data/oonib/daphn3.yaml" diff --git a/oonib/report/api.py b/oonib/report/api.py index 489cc25..bd409b0 100644 --- a/oonib/report/api.py +++ b/oonib/report/api.py @@ -59,6 +59,28 @@ def parseNewReportRequest(request): raise InvalidRequestField(k) return parsed_request +def parseUpdateReportRequest(request): + # XXX this and the function above can probably be refactored into something + # more compact. There is quite a bit of code duplication going on here. + + report_id_regexp = re.compile("[a-zA-Z0-9]+$") + + # XXX here we are actually parsing a json object that could be quite big. + # If we want this to scale properly we only want to look at the test_id + # field. + # We are also keeping in memory multiple copies of the same object. A lot + # of optimization can be done. + parsed_request = json.loads(request) + try: + report_id = parsed_request['report_id'] + except KeyError: + raise MissingField('report_id') + + if not re.match(report_id_regexp, report_id): + raise InvalidRequestField('report_id') + + return parsed_request + class NewReportHandlerFile(web.RequestHandler): """ Responsible for creating and updating reports by writing to flat file. @@ -129,7 +151,20 @@ class NewReportHandlerFile(web.RequestHandler): 'content': 'XXX' } """ - pass + parsed_request = parseUpdateReportRequest(self.request.body) + report_id = parsed_request['report_id'] + print "Got this request %s" % parsed_request + + report_filename = report_id + report_filename += '.yamloo' + try: + with open(report_filename, 'a+') as f: + # XXX this could be quite big. We should probably use the + # twisted.internet.fdesc module + print parsed_request['content'] + f.write(parsed_request['content']) + except IOError as e: + web.HTTPError(404, "Report not found") class NewReportHandlerDB(web.RequestHandler): """ @@ -188,7 +223,8 @@ class PCAPReportHandler(web.RequestHandler): def post(self): pass -spec = [(r"/report/new", NewReportHandlerFile), - (r"/report/pcap", PCAPReportHandler)] +reportingBackendAPI = [(r"/report/new", NewReportHandlerFile), + (r"/report/pcap", PCAPReportHandler) +] -reportingBackend = web.Application(spec, debug=True) +reportingBackend = web.Application(reportingBackendAPI, debug=True)

1 0

[translation/vidalia_alpha_completed] Update translations for vidalia_alpha_completed
by translation＠torproject.org 11 Nov '12

11 Nov '12

commit 00ce276c9feeede1b6bf675ecbf1bababb8870ee Author: Translation commit bot <translation(a)torproject.org> Date: Sun Nov 11 12:15:20 2012 +0000 Update translations for vidalia_alpha_completed --- es/vidalia_es.po | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/es/vidalia_es.po b/es/vidalia_es.po index ba51e72..34bd7aa 100644 --- a/es/vidalia_es.po +++ b/es/vidalia_es.po @@ -9,7 +9,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2012-03-21 17:46+0000\n" -"PO-Revision-Date: 2012-11-11 03:25+0000\n" +"PO-Revision-Date: 2012-11-11 12:14+0000\n" "Last-Translator: strel <strelnic(a)gmail.com>\n" "Language-Team: Spanish (http://www.transifex.com/projects/p/torproject/language/es/)\n" "MIME-Version: 1.0\n" @@ -2069,7 +2069,7 @@ msgid "" "You are currently running a relay. Terminating your relay will interrupt any open connections from clients.\n" "\n" "Would you like to shutdown gracefully and give clients time to find a new relay?" -msgstr "Actualmente está ejecutando un repetidor (relay). Detenerlo interrumpiría cualquier conexión abierta desde los clientes.\n\n¿Desea cerrar con un periodo de gracia para dar tiempo de encontrar un nuevo repetidor a los clientes?" +msgstr "Actualmente está ejecutando un repetidor (relay). Detenerlo interrumpiría cualquier conexión abierta desde los clientes.\n\n¿Desea cerrar con un periodo de gracia para dar tiempo a los clientes de encontrar un nuevo repetidor (relay)?" msgctxt "MainWindow" msgid ""

1 0