November 2012 - tor-commits - lists.torproject.org

[translation/vidalia_alpha] Update translations for vidalia_alpha
by translation＠torproject.org 10 Nov '12

10 Nov '12

commit 40f2409f8c858fdb2bb7af911297ca429813053a Author: Translation commit bot <translation(a)torproject.org> Date: Sat Nov 10 01:15:18 2012 +0000 Update translations for vidalia_alpha --- es/vidalia_es.po | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/es/vidalia_es.po b/es/vidalia_es.po index 4d166fe..5598040 100644 --- a/es/vidalia_es.po +++ b/es/vidalia_es.po @@ -9,7 +9,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2012-03-21 17:46+0000\n" -"PO-Revision-Date: 2012-11-09 05:33+0000\n" +"PO-Revision-Date: 2012-11-10 01:10+0000\n" "Last-Translator: strel <strelnic(a)gmail.com>\n" "Language-Team: Spanish (http://www.transifex.com/projects/p/torproject/language/es/)\n" "MIME-Version: 1.0\n" @@ -46,7 +46,7 @@ msgstr "'%1' no es una dirección IP válida. " msgctxt "AdvancedPage" msgid "" "You selected 'Password' authentication, but did not specify a password." -msgstr "Ha seleccionado autenticación por 'Contraseña', pero no ha especificado ninguna. " +msgstr "Ha seleccionado autentificación por 'Contraseña', pero no ha especificado ninguna. " msgctxt "AdvancedPage" msgid "Select Tor Configuration File" @@ -134,11 +134,11 @@ msgstr "Carpeta de Datos " msgctxt "AdvancedPage" msgid "Store data for the Tor software in the following directory" -msgstr "Almacenar datos de Tor en la siguiente carpeta " +msgstr "Almacenar datos para Tor en la siguiente carpeta " msgctxt "AdvancedPage" msgid "Select the directory used to store data for the Tor software" -msgstr "Seleccione la carpeta que se utilizará para almacenar los datos de Tor " +msgstr "Seleccione la carpeta que se utilizará para almacenar datos de Tor " msgctxt "AdvancedPage" msgid "Tor Control"

1 0

[ooni-probe/master] Update example for writing scapy tests.
by art＠torproject.org 10 Nov '12

10 Nov '12

commit 50df3ff29e42c733fc6dfbe64da7f4ae1cd89b9e Author: Arturo Filastò <art(a)fuffa.org> Date: Sat Nov 10 00:22:55 2012 +0100 Update example for writing scapy tests. * Make changes to test template of scapy tests --- nettests/examples/example_scapyt.py | 22 ++++++++++++++++------ ooni/templates/scapyt.py | 19 ++++++++++++++----- 2 files changed, 30 insertions(+), 11 deletions(-) diff --git a/nettests/examples/example_scapyt.py b/nettests/examples/example_scapyt.py index 51ffd02..a6199e2 100644 --- a/nettests/examples/example_scapyt.py +++ b/nettests/examples/example_scapyt.py @@ -5,13 +5,23 @@ from ooni.utils import log from ooni.templates import scapyt -from scapy.all import IP, TCP +from scapy.all import IP, ICMP -class ExampleBasicScapy(scapyt.BaseScapyTest): - name = "Example Scapy Test" + +class ExampleICMPPingScapy(scapyt.BaseScapyTest): + name = "Example ICMP Ping Test" author = "Arturo Filastò" version = 0.1 - def test_send_raw_ip_frame(self): - log.msg("Running send receive") - ans, unans = self.sr(IP(dst='8.8.8.8')/TCP(), timeout=1) + def test_icmp_ping(self): + log.msg("Pinging 8.8.8.8") + def finished(packets): + print packets + answered, unanswered = packets + for snd, rcv in answered: + rcv.show() + + packets = IP(dst='8.8.8.8')/ICMP() + d = self.sr(packets) + d.addCallback(finished) + return d diff --git a/ooni/templates/scapyt.py b/ooni/templates/scapyt.py index 4c18f0a..3087ac7 100644 --- a/ooni/templates/scapyt.py +++ b/ooni/templates/scapyt.py @@ -47,14 +47,23 @@ class BaseScapyTest(NetTestCase): Wrapper around scapy.sendrecv.sr for sending and receiving of packets at layer 3. """ - def finished(result): - answered, unanswered = result - sent_packets, received_packets = answered - self.report['answered_packets'] = createPacketReport(received_packets) - self.report['sent_packets'] = createPacketReport(sent_packets) + def finished(packets): + log.debug("Got this bullshit") + answered, unanswered = packets + self.report['answered_packets'] = [] + self.report['sent_packets'] = [] + for snd, rcv in answered: + log.debug("Writing report %s") + pkt_report_r = createPacketReport(rcv) + pkt_report_s = createPacketReport(snd) + self.report['answered_packets'].append(pkt_report_r) + self.report['sent_packets'].append(pkt_report_s) + log.debug("Done") + return packets scapyProtocol = ScapyProtocol(*arg, **kw) d = scapyProtocol.startSending(packets) + d.addCallback(finished) return d def send(self, pkts, *arg, **kw):

1 0

[ooni-probe/master] Do a very big cleanup and refactor of all the code in the repo.
by art＠torproject.org 09 Nov '12

09 Nov '12

commit 10c63e0fceb0479c5893bfd2cb7e77af2af703cc Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Nov 9 17:37:32 2012 +0100 Do a very big cleanup and refactor of all the code in the repo. --- CHANGES.yaml | 5 - HACKING | 141 ++++++++++---------- TODO | 2 +- before_i_commit.sh | 2 +- nettests/core/http_host.py | 3 +- ooni/config.py | 4 +- ooni/inputunit.py | 12 ++ ooni/lib/rfc3339.py | 283 ---------------------------------------- ooni/nettest.py | 20 +-- ooni/nodes.py | 24 ++-- ooni/oonicli.py | 23 ++-- ooni/protocols/daphn3.py | 311 -------------------------------------------- ooni/reporter.py | 18 ++- ooni/runner.py | 16 ++- ooni/templates/httpt.py | 44 +------ ooni/utils/__init__.py | 104 +-------------- ooni/utils/date.py | 30 ----- ooni/utils/geodata.py | 29 ++-- ooni/utils/hacks.py | 38 +----- ooni/utils/net.py | 47 ++++++- ooni/utils/otime.py | 8 + ooniprobe.conf | 2 +- 22 files changed, 220 insertions(+), 946 deletions(-) diff --git a/CHANGES.yaml b/CHANGES.yaml deleted file mode 100644 index 86206f1..0000000 --- a/CHANGES.yaml +++ /dev/null @@ -1,5 +0,0 @@ -# Add each new entry to the top of the file. - -- version: 0.0.1 - date: Thu Jul 7 16:03:36 EDT 2011 - changes: Initial version. diff --git a/HACKING b/HACKING index a0bf89c..2fdc392 100644 --- a/HACKING +++ b/HACKING @@ -34,80 +34,85 @@ Code Structure --------- - HACKING - The document you are currently reading. + The document you are currently reading. -- oonib/ - Contains the OONI probe backend to be run on the ooni-net +- inputs/ + Contains input files for tests. -- oonid/ - Contains the OONI daemon that can be used to interrogated from the cli to - run tests. +- oonib/ + Contains the OONI probe backend to be run on the ooni-net - ooni/ - Contains the main ooni probe comand line client - -- ooni/assets/ - Where we store all the asset files that are - used when running OONI tests. + Contains the main ooni probe comand line client - ooni/config.py - Parts of the code related to parsing OONI - configuration files and making them accessible - to other components of the software. - -- ooni/logo.py - File containing some funny ASCII art. Yes, we - do enjoy ASCII art and are not afraid to admit it! - -- ooni/nodes.conf - The configuration file for nodes. This contains the - list of network and code execution nodes that can be - used to run tests off of. - -- ooni/ooniprobe.py - The main OONI-probe command line interface. This is - responsible for parsing the command line arguments and - passing the arguments to the underlying components. - -- ooni/ooni-probe.conf - The main OONI-probe configuration file. This can be used - to configure your OONI CLI, tell it where it should report - to, where the asset files are located, what should be used - for control, etc. - -- ooni/plugoo/__init__.py - All the necessary "goo" for making OONI probe work. This - means loading Assets, creating Reports, running Tests, - interacting with Nodes. - -- ooni/plugoo/assets.py - This is a python object representation of the data that is - located inside the asset directory. - -- ooni/plugoo/nodes.py - The part of code responsible for interacting with OONI Nodes. - Nodes can be Network or Code Execution. Network nodes are - capable of receiving packets and fowarding them onto their - network. This means that to run a test on X network nodes you - consume X*test_cost bandwith. Code Execution nodes accept units - of work, they are therefore capable of receiving a set of tests - that should be completed by a set of Network nodes or run locally. - -- ooni/plugoo/reports.py - Takes care of transforming the output of a test into a report. This - may mean pushing the result data to a remote backend or simply writing - a local file. - -- ooni/plugoo/tests.py - The main OONI probe test class. This provides all the necessary scaffold - to write your own test based on the OONI paradigm. - -- ooni/oonitests/ - Contains all the "offical" OONI tests that are shipped. - -- ooni/utils.py - Helper functions that don't fit into any place, but are not big enough to - be a dependency by themselves. + Parts of the code related to parsing OONI + configuration files and making them accessible + to other components of the software. + +- ooni/inputunit.py + In here we have functions related to the creation of input + units. Input units are how the inputs to be fed to tests are + split up into. + +- ooni/nettest.py + In here is the NetTest API definition. This is how people + interested in writing ooniprobe tests will be specifying + them. + +- ooni/nodes.py + Mostly broken code for the remote dispatching of tests. + +- ooni/oonicli.py + In here we take care of running ooniprobe from the command + line interface + +- ooni/reporter.py + In here goes the logic for the creation of ooniprobe + reports. + +- ooni/runner.py + Handles running ooni.nettests as well as + ooni.plugoo.tests.OONITests. + +- ooni/kit/ + In here go utilities that can be used by tests. + +- ooni/lib/ + XXX this directory is to be removed. + +- ooni/utils/ + In here go internal utilities that are useful to ooniprobe + +- ooni/utils/geodata.py + In here go functions related to the understanding of + geographical information of the probe + +- ooni/utils/hacks.py + When some software has issues and we need to fix it in a + hackish way, we put it in here. This one day will be empty. + +- ooni/utils/log.py + log realted functions. + +- ooni/utils/net.py + utilities for networking related operations + +- ooni/utils/onion.py + Utilities for working with Tor. + XXX this code should be removed and merged into txtorcon. + +- ooni/utils/otime.py + Generation of timestamps, time conversions and all the rest + +- ooni/utils/txscapy.py + Tools for making scapy work well with twisted. + +- ooniprobe.conf + The main OONI-probe configuration file. This can be used + to configure your OONI CLI, tell it where it should report + to, where the asset files are located, what should be used + for control, etc. Style guide ----------- diff --git a/TODO b/TODO index 63d950c..b8524f3 100644 --- a/TODO +++ b/TODO @@ -105,4 +105,4 @@ It's important to make the new code asych and based on Twisted. It should respect the design goals of the new ooni-probe model. Also, importing new, non-standard libraries should be discussed first, if the new test is to be used in the core of OONI (packaging scapy and twisted already makes our -codebase quite large). \ No newline at end of file +codebase quite large). diff --git a/before_i_commit.sh b/before_i_commit.sh index f08a5f5..4a37686 100755 --- a/before_i_commit.sh +++ b/before_i_commit.sh @@ -34,5 +34,5 @@ echo "If you do, it means something is wrong." echo "Read through the log file and fix it." echo "If you are having some problems fixing some things that have to do with" echo "the core of OONI, let's first discuss it on IRC, or open a ticket" - +less *yamloo rm -f *yamloo diff --git a/nettests/core/http_host.py b/nettests/core/http_host.py index e09fc84..b87594d 100644 --- a/nettests/core/http_host.py +++ b/nettests/core/http_host.py @@ -14,8 +14,7 @@ from ooni.templates import httpt class UsageOptions(usage.Options): optParameters = [ - ['url', 'u', 'http://torproject.org/', 'Test single site'], - ['backend', 'b', 'http://ooni.nu/test/', 'Test backend to use'], + ['backend', 'b', 'http://ooni.nu/test/', 'Test backend to use'] ] diff --git a/ooni/config.py b/ooni/config.py index f3d1a80..cec9146 100644 --- a/ooni/config.py +++ b/ooni/config.py @@ -8,7 +8,7 @@ import yaml from twisted.internet import reactor, threads -from ooni.utils import date +from ooni.utils import otime from ooni.utils import Storage def get_root_path(): @@ -24,7 +24,7 @@ def oreport_filenames(): returns yamloo_filename, pcap_filename """ - base_filename = "%s_"+date.timestamp()+".%s" + base_filename = "%s_"+otime.timestamp()+".%s" yamloo_filename = base_filename % ("report", "yamloo") pcap_filename = base_filename % ("packets", "pcap") return yamloo_filename, pcap_filename diff --git a/ooni/inputunit.py b/ooni/inputunit.py index 3b0c491..484631b 100644 --- a/ooni/inputunit.py +++ b/ooni/inputunit.py @@ -1,3 +1,15 @@ +#-*- coding: utf-8 -*- +# +# inputunit.py +# ------------- +# IN here we have functions related to the creation of input +# units. Input units are how the inputs to be fed to tests are +# split up into. +# +# :authors: Arturo Filastò, Isis Lovecruft +# :license: see included LICENSE file + + class InputUnitFactory(object): """ This is a factory that takes the size of input units to be generated a set diff --git a/ooni/lib/rfc3339.py b/ooni/lib/rfc3339.py deleted file mode 100644 index e664ce7..0000000 --- a/ooni/lib/rfc3339.py +++ /dev/null @@ -1,283 +0,0 @@ -#!/usr/bin/env python -# -# Copyright (c) 2009, 2010, Henry Precheur <henry(a)precheur.org> -# -# Permission to use, copy, modify, and/or distribute this software for any -# purpose with or without fee is hereby granted, provided that the above -# copyright notice and this permission notice appear in all copies. -# -# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH -# REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND -# FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, -# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM -# LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR -# OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR -# PERFORMANCE OF THIS SOFTWARE. -# -'''Formats dates according to the :RFC:`3339`. - -Report bugs & problems on BitBucket_ - -.. _BitBucket: https://bitbucket.org/henry/clan.cx/issues -''' - -__author__ = 'Henry Precheur <henry(a)precheur.org>' -__license__ = 'ISCL' -__version__ = '5.1' -__all__ = ('rfc3339', ) - -import datetime -import time -import unittest - -def _timezone(utc_offset): - ''' - Return a string representing the timezone offset. - - >>> _timezone(0) - '+00:00' - >>> _timezone(3600) - '+01:00' - >>> _timezone(-28800) - '-08:00' - >>> _timezone(-1800) - '-00:30' - ''' - # Python's division uses floor(), not round() like in other languages: - # -1 / 2 == -1 and not -1 / 2 == 0 - # That's why we use abs(utc_offset). - hours = abs(utc_offset) // 3600 - minutes = abs(utc_offset) % 3600 // 60 - sign = (utc_offset < 0 and '-') or '+' - return '%c%02d:%02d' % (sign, hours, minutes) - -def _timedelta_to_seconds(timedelta): - ''' - >>> _timedelta_to_seconds(datetime.timedelta(hours=3)) - 10800 - >>> _timedelta_to_seconds(datetime.timedelta(hours=3, minutes=15)) - 11700 - ''' - return (timedelta.days * 86400 + timedelta.seconds + - timedelta.microseconds // 1000) - -def _utc_offset(date, use_system_timezone): - ''' - Return the UTC offset of `date`. If `date` does not have any `tzinfo`, use - the timezone informations stored locally on the system. - - >>> if time.localtime().tm_isdst: - ... system_timezone = -time.altzone - ... else: - ... system_timezone = -time.timezone - >>> _utc_offset(datetime.datetime.now(), True) == system_timezone - True - >>> _utc_offset(datetime.datetime.now(), False) - 0 - ''' - if isinstance(date, datetime.datetime) and date.tzinfo is not None: - return _timedelta_to_seconds(date.dst() or date.utcoffset()) - elif use_system_timezone: - if date.year < 1970: - # We use 1972 because 1970 doesn't have a leap day (feb 29) - t = time.mktime(date.replace(year=1972).timetuple()) - else: - t = time.mktime(date.timetuple()) - if time.localtime(t).tm_isdst: # pragma: no cover - return -time.altzone - else: - return -time.timezone - else: - return 0 - -def _string(d, timezone): - return ('%04d-%02d-%02dT%02d:%02d:%02d%s' % - (d.year, d.month, d.day, d.hour, d.minute, d.second, timezone)) - -def rfc3339(date, utc=False, use_system_timezone=True): - ''' - Return a string formatted according to the :RFC:`3339`. If called with - `utc=True`, it normalizes `date` to the UTC date. If `date` does not have - any timezone information, uses the local timezone:: - - >>> d = datetime.datetime(2008, 4, 2, 20) - >>> rfc3339(d, utc=True, use_system_timezone=False) - '2008-04-02T20:00:00Z' - >>> rfc3339(d) # doctest: +ELLIPSIS - '2008-04-02T20:00:00...' - - If called with `user_system_timezone=False` don't use the local timezone if - `date` does not have timezone informations and consider the offset to UTC - to be zero:: - - >>> rfc3339(d, use_system_timezone=False) - '2008-04-02T20:00:00+00:00' - - `date` must be a `datetime.datetime`, `datetime.date` or a timestamp as - returned by `time.time()`:: - - >>> rfc3339(0, utc=True, use_system_timezone=False) - '1970-01-01T00:00:00Z' - >>> rfc3339(datetime.date(2008, 9, 6), utc=True, - ... use_system_timezone=False) - '2008-09-06T00:00:00Z' - >>> rfc3339(datetime.date(2008, 9, 6), - ... use_system_timezone=False) - '2008-09-06T00:00:00+00:00' - >>> rfc3339('foo bar') - Traceback (most recent call last): - ... - TypeError: Expected timestamp or date object. Got <type 'str'>. - - For dates before January 1st 1970, the timezones will be the ones used in - 1970. It might not be accurate, but on most sytem there is no timezone - information before 1970. - ''' - # Try to convert timestamp to datetime - try: - if use_system_timezone: - date = datetime.datetime.fromtimestamp(date) - else: - date = datetime.datetime.utcfromtimestamp(date) - except TypeError: - pass - - if not isinstance(date, datetime.date): - raise TypeError('Expected timestamp or date object. Got %r.' % - type(date)) - - if not isinstance(date, datetime.datetime): - date = datetime.datetime(*date.timetuple()[:3]) - utc_offset = _utc_offset(date, use_system_timezone) - if utc: - return _string(date + datetime.timedelta(seconds=utc_offset), 'Z') - else: - return _string(date, _timezone(utc_offset)) - - -class LocalTimeTestCase(unittest.TestCase): - ''' - Test the use of the timezone saved locally. Since it is hard to test using - doctest. - ''' - - def setUp(self): - local_utcoffset = _utc_offset(datetime.datetime.now(), True) - self.local_utcoffset = datetime.timedelta(seconds=local_utcoffset) - self.local_timezone = _timezone(local_utcoffset) - - def test_datetime(self): - d = datetime.datetime.now() - self.assertEqual(rfc3339(d), - d.strftime('%Y-%m-%dT%H:%M:%S') + self.local_timezone) - - def test_datetime_timezone(self): - - class FixedNoDst(datetime.tzinfo): - 'A timezone info with fixed offset, not DST' - - def utcoffset(self, dt): - return datetime.timedelta(hours=2, minutes=30) - - def dst(self, dt): - return None - - fixed_no_dst = FixedNoDst() - - class Fixed(FixedNoDst): - 'A timezone info with DST' - - def dst(self, dt): - return datetime.timedelta(hours=3, minutes=15) - - fixed = Fixed() - - d = datetime.datetime.now().replace(tzinfo=fixed_no_dst) - timezone = _timezone(_timedelta_to_seconds(fixed_no_dst.\ - utcoffset(None))) - self.assertEqual(rfc3339(d), - d.strftime('%Y-%m-%dT%H:%M:%S') + timezone) - - d = datetime.datetime.now().replace(tzinfo=fixed) - timezone = _timezone(_timedelta_to_seconds(fixed.dst(None))) - self.assertEqual(rfc3339(d), - d.strftime('%Y-%m-%dT%H:%M:%S') + timezone) - - def test_datetime_utc(self): - d = datetime.datetime.now() - d_utc = d + self.local_utcoffset - self.assertEqual(rfc3339(d, utc=True), - d_utc.strftime('%Y-%m-%dT%H:%M:%SZ')) - - def test_date(self): - d = datetime.date.today() - self.assertEqual(rfc3339(d), - d.strftime('%Y-%m-%dT%H:%M:%S') + self.local_timezone) - - def test_date_utc(self): - d = datetime.date.today() - # Convert `date` to `datetime`, since `date` ignores seconds and hours - # in timedeltas: - # >>> datetime.date(2008, 9, 7) + datetime.timedelta(hours=23) - # datetime.date(2008, 9, 7) - d_utc = datetime.datetime(*d.timetuple()[:3]) + self.local_utcoffset - self.assertEqual(rfc3339(d, utc=True), - d_utc.strftime('%Y-%m-%dT%H:%M:%SZ')) - - def test_timestamp(self): - d = time.time() - self.assertEqual(rfc3339(d), - datetime.datetime.fromtimestamp(d).\ - strftime('%Y-%m-%dT%H:%M:%S') + self.local_timezone) - - def test_timestamp_utc(self): - d = time.time() - d_utc = datetime.datetime.utcfromtimestamp(d) + self.local_utcoffset - self.assertEqual(rfc3339(d), - (d_utc.strftime('%Y-%m-%dT%H:%M:%S') + - self.local_timezone)) - - def test_before_1970(self): - d = datetime.date(1885, 01, 04) - self.failUnless(rfc3339(d).startswith('1885-01-04T00:00:00')) - self.assertEqual(rfc3339(d, utc=True, use_system_timezone=False), - '1885-01-04T00:00:00Z') - - def test_1920(self): - d = datetime.date(1920, 02, 29) - x = rfc3339(d, utc=False, use_system_timezone=True) - self.failUnless(x.startswith('1920-02-29T00:00:00')) - - # If these tests start failing it probably means there was a policy change - # for the Pacific time zone. - # See http://en.wikipedia.org/wiki/Pacific_Time_Zone. - if 'PST' in time.tzname: - def testPDTChange(self): - '''Test Daylight saving change''' - # PDT switch happens at 2AM on March 14, 2010 - - # 1:59AM PST - self.assertEqual(rfc3339(datetime.datetime(2010, 3, 14, 1, 59)), - '2010-03-14T01:59:00-08:00') - # 3AM PDT - self.assertEqual(rfc3339(datetime.datetime(2010, 3, 14, 3, 0)), - '2010-03-14T03:00:00-07:00') - - def testPSTChange(self): - '''Test Standard time change''' - # PST switch happens at 2AM on November 6, 2010 - - # 0:59AM PDT - self.assertEqual(rfc3339(datetime.datetime(2010, 11, 7, 0, 59)), - '2010-11-07T00:59:00-07:00') - - # 1:00AM PST - # There's no way to have 1:00AM PST without a proper tzinfo - self.assertEqual(rfc3339(datetime.datetime(2010, 11, 7, 1, 0)), - '2010-11-07T01:00:00-07:00') - - -if __name__ == '__main__': # pragma: no cover - import doctest - doctest.testmod() - unittest.main() diff --git a/ooni/nettest.py b/ooni/nettest.py index 6221a3f..896ae2a 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -1,15 +1,12 @@ # -*- encoding: utf-8 -*- # -# nettest.py -# -------------------> +# nettest.py +# ---------- +# In here is the NetTest API definition. This is how people +# interested in writing ooniprobe tests will be specifying them # -# :authors: Arturo "hellais" Filastò <art(a)fuffa.org>, -# Isis Lovecruft <isis(a)torproject.org> -# :licence: see LICENSE -# :copyright: 2012 Arturo Filasto, Isis Lovecruft -# :version: 0.1.0-alpha -# -# <------------------- +# :authors: Arturo Filastò, Isis Lovecruft +# :license: see included LICENSE file import sys import os @@ -22,8 +19,6 @@ from twisted.python import usage from ooni.utils import log -pyunit = __import__('unittest') - class NetTestCase(object): """ This is the base of the OONI nettest universe. When you write a nettest @@ -102,7 +97,6 @@ class NetTestCase(object): usageOptions = None requiredOptions = [] - requiresRoot = False localOptions = {} @@ -159,7 +153,7 @@ class NetTestCase(object): raise usage.UsageError("No input file specified!") self._checkRequiredOptions() - + # XXX perhaps we may want to name and version to be inside of a # different method that is not called options. return {'inputs': self.inputs, diff --git a/ooni/nodes.py b/ooni/nodes.py index 155f183..070ffe7 100644 --- a/ooni/nodes.py +++ b/ooni/nodes.py @@ -1,16 +1,14 @@ -#!/usr/bin/env python -# -*- coding: UTF-8 -""" - nodes - ***** - - This contains all the code related to Nodes - both network and code execution. - - :copyright: (c) 2012 by Arturo Filastò, Isis Lovecruft - :license: see LICENSE for more details. - -""" +#-*- coding: utf-8 -*- +# +# nodes.py +# -------- +# here is code for handling the interaction with remote +# services that will run ooniprobe tests. +# XXX most of the code in here is broken or not tested and +# probably should be trashed +# +# :authors: Arturo Filastò, Isis Lovecruft +# :license: see included LICENSE file import os from binascii import hexlify diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 4b63a61..8e4fa14 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -1,16 +1,12 @@ -#!/usr/bin/env python # -*- coding: UTF-8 # -# oonicli -# ********* +# oonicli +# ------- +# In here we take care of running ooniprobe from the command +# line interface # -# oonicli is the next generation ooniprober. It based off of twisted's trial -# unit testing framework. -# -# :copyright: (c) 2012 by Arturo Filastò, Isis Lovecruft -# :license: see LICENSE for more details. -# -# original copyright (c) by Twisted Matrix Laboratories. +# :authors: Arturo Filastò, Isis Lovecruft +# :license: see included LICENSE file import sys @@ -27,7 +23,8 @@ from ooni import nettest, runner, reporter, config from ooni.inputunit import InputUnitFactory -from ooni.utils import net, checkForRoot +from ooni.utils import net +from ooni.utils import checkForRoot, NotRootError from ooni.utils import log @@ -118,9 +115,9 @@ def run(): if config.privacy.includepcap: try: checkForRoot() - except: + except NotRootError: log.err("includepcap options requires root priviledges to run") - log.err("disable it in your ooniprobe.conf file") + log.err("you should run ooniprobe as root or disable the options in ooniprobe.conf") sys.exit(1) log.debug("Starting sniffer") sniffer_d = net.capturePackets(pcap_filename) diff --git a/ooni/protocols/__init__.py b/ooni/protocols/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/ooni/protocols/daphn3.py b/ooni/protocols/daphn3.py deleted file mode 100644 index 37c94c7..0000000 --- a/ooni/protocols/daphn3.py +++ /dev/null @@ -1,311 +0,0 @@ -import sys -import yaml - -from twisted.internet import protocol, defer -from twisted.internet.error import ConnectionDone - -from scapy.all import IP, Raw, rdpcap - -from ooni.utils import log -from ooni.plugoo import reports - -def read_pcap(filename): - """ - @param filename: Filesystem path to the pcap. - - Returns: - [{"sender": "client", "data": "\x17\x52\x15"}, {"sender": "server", "data": "\x17\x15\x13"}] - """ - packets = rdpcap(filename) - - checking_first_packet = True - client_ip_addr = None - server_ip_addr = None - - ssl_packets = [] - messages = [] - - """ - pcap assumptions: - - pcap only contains packets exchanged between a Tor client and a Tor server. - (This assumption makes sure that there are only two IP addresses in the - pcap file) - - The first packet of the pcap is sent from the client to the server. (This - assumption is used to get the IP address of the client.) - - All captured packets are TLS packets: that is TCP session - establishment/teardown packets should be filtered out (no SYN/SYN+ACK) - """ - - """Minimally validate the pcap and also find out what's the client - and server IP addresses.""" - for packet in packets: - if checking_first_packet: - client_ip_addr = packet[IP].src - checking_first_packet = False - else: - if packet[IP].src != client_ip_addr: - server_ip_addr = packet[IP].src - - try: - if (packet[Raw]): - ssl_packets.append(packet) - except IndexError: - pass - - """Form our list.""" - for packet in ssl_packets: - if packet[IP].src == client_ip_addr: - messages.append({"sender": "client", "data": str(packet[Raw])}) - elif packet[IP].src == server_ip_addr: - messages.append({"sender": "server", "data": str(packet[Raw])}) - else: - raise("Detected third IP address! pcap is corrupted.") - - return messages - -def read_yaml(filename): - f = open(filename) - obj = yaml.load(f) - f.close() - return obj - -class Mutator: - idx = 0 - step = 0 - - waiting = False - waiting_step = 0 - - def __init__(self, steps): - """ - @param steps: array of dicts for the steps that must be gone over by - the mutator. Looks like this: - [{"sender": "client", "data": "\xde\xad\xbe\xef"}, - {"sender": "server", "data": "\xde\xad\xbe\xef"}] - """ - self.steps = steps - - def _mutate(self, data, idx): - """ - Mutate the idx bytes by increasing it's value by one - - @param data: the data to be mutated. - - @param idx: what byte should be mutated. - """ - print "idx: %s, data: %s" % (idx, data) - ret = data[:idx] - ret += chr(ord(data[idx]) + 1) - ret += data[idx+1:] - return ret - - def state(self): - """ - Return the current mutation state. As in what bytes are being mutated. - - Returns a dict containg the packet index and the step number. - """ - print "[Mutator.state()] Giving out my internal state." - current_state = {'idx': self.idx, 'step': self.step} - return current_state - - def next(self): - """ - Increases by one the mutation state. - - ex. (* is the mutation state, i.e. the byte to be mutated) - before [___*] [____] - step1 step2 - after [____] [*___] - - Should be called every time you need to proceed onto the next mutation. - It changes the internal state of the mutator to that of the next - mutatation. - - returns True if another mutation is available. - returns False if all the possible mutations have been done. - """ - if (self.step) == len(self.steps): - # Hack to stop once we have gone through all the steps - print "[Mutator.next()] I believe I have gone over all steps" - print " Stopping!" - self.waiting = True - return False - - self.idx += 1 - current_idx = self.idx - current_step = self.step - current_data = self.steps[current_step]['data'] - - if 0: - print "current_step: %s" % current_step - print "current_idx: %s" % current_idx - print "current_data: %s" % current_data - print "steps: %s" % len(self.steps) - print "waiting_step: %s" % self.waiting_step - - data_to_receive = len(self.steps[current_step]['data']) - - if self.waiting and self.waiting_step == data_to_receive: - print "[Mutator.next()] I am no longer waiting" - log.debug("I am no longer waiting.") - self.waiting = False - self.waiting_step = 0 - self.idx = 0 - - elif self.waiting: - print "[Mutator.next()] Waiting some more." - log.debug("Waiting some more.") - self.waiting_step += 1 - - elif current_idx >= len(current_data): - print "[Mutator.next()] Entering waiting mode." - log.debug("Entering waiting mode.") - self.step += 1 - self.idx = 0 - self.waiting = True - - log.debug("current index %s" % current_idx) - log.debug("current data %s" % len(current_data)) - return True - - def get(self, step): - """ - Returns the current packet to be sent to the wire. - If no mutation is necessary it will return the plain data. - Should be called when you are interested in obtaining the data to be - sent for the selected state. - - @param step: the current step you want the mutation for - - returns the mutated packet for the specified step. - """ - if step != self.step or self.waiting: - log.debug("[Mutator.get()] I am not going to do anything :)") - return self.steps[step]['data'] - - data = self.steps[step]['data'] - #print "Mutating %s with idx %s" % (data, self.idx) - return self._mutate(data, self.idx) - -class Daphn3Protocol(protocol.Protocol): - """ - This implements the Daphn3 protocol for the server side. - It gets instanced once for every client that connects to the oonib. - For every instance of protocol there is only 1 mutation. - Once the last step is reached the connection is closed on the serverside. - """ - steps = [] - mutator = None - - current_state = None - - role = 'client' - state = 0 - total_states = len(steps) - 1 - received_data = 0 - to_receive_data = 0 - report = reports.Report('daphn3', 'daphn3.yamlooni') - - test = None - - def next_state(self): - """ - This is called once I have completed one step of the protocol and need - to proceed to the next step. - """ - if not self.mutator: - print "[Daphn3Protocol.next_state] No mutator. There is no point to stay on this earth." - self.transport.loseConnection() - return - - if self.role is self.steps[self.state]['sender']: - print "[Daphn3Protocol.next_state] I am a sender" - data = self.mutator.get(self.state) - self.transport.write(data) - self.to_receive_data = 0 - - else: - print "[Daphn3Protocol.next_state] I am a receiver" - self.to_receive_data = len(self.steps[self.state]['data']) - - self.state += 1 - self.received_data = 0 - - def dataReceived(self, data): - """ - This is called every time some data is received. I transition to the - next step once the amount of data that I expect to receive is received. - - @param data: the data that has been sent by the client. - """ - if not self.mutator: - print "I don't have a mutator. My life means nothing." - self.transport.loseConnection() - return - - if len(self.steps) == self.state: - self.transport.loseConnection() - return - - self.received_data += len(data) - if self.received_data >= self.to_receive_data: - print "Moving to next state %s" % self.state - self.next_state() - - def censorship_detected(self, report): - """ - I have detected the possible presence of censorship we need to write a - report on it. - - @param report: a dict containing the report to be written. Must contain - the keys 'reason', 'proto_state' and 'mutator_state'. - The reason is the reason for which the connection was - closed. The proto_state is the current state of the - protocol instance and mutator_state is what was being - mutated. - """ - print "The connection was closed because of %s" % report['reason'] - print "State %s, Mutator %s" % (report['proto_state'], - report['mutator_state']) - if self.test: - self.test.result['censored'] = True - self.test.result['state'] = report - self.mutator.next() - - def connectionLost(self, reason): - """ - The connection was closed. This may be because of a legittimate reason - or it may be because of a censorship event. - """ - if not self.mutator: - print "Terminated because of little interest in life." - return - report = {'reason': reason, 'proto_state': self.state, - 'trigger': None, 'mutator_state': self.current_state} - - if self.state < self.total_states: - report['trigger'] = 'did not finish state walk' - self.censorship_detected(report) - - else: - print "I have reached the end of the state machine" - print "Censorship fingerprint bruteforced!" - if self.test: - print "In the test thing" - self.test.result['censored'] = False - self.test.result['state'] = report - self.test.result['state_walk_finished'] = True - self.test.report(self.test.result) - return - - if reason.check(ConnectionDone): - print "Connection closed cleanly" - else: - report['trigger'] = 'unclean connection closure' - self.censorship_detected(report) - - diff --git a/ooni/reporter.py b/ooni/reporter.py index cdbf355..6d37838 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -1,3 +1,12 @@ +#-*- coding: utf-8 -*- +# +# reporter.py +# ----------- +# In here goes the logic for the creation of ooniprobe reports. +# +# :authors: Arturo Filastò, Isis Lovecruft +# :license: see included LICENSE file + import itertools import logging import sys @@ -11,13 +20,12 @@ from yaml.emitter import * from yaml.serializer import * from yaml.resolver import * -from datetime import datetime from twisted.python.util import untilConcludes from twisted.trial import reporter from twisted.internet import defer, reactor from ooni.templates.httpt import BodyReceiver, StringProducer -from ooni.utils import date, log, geodata +from ooni.utils import otime, log, geodata from ooni import config try: @@ -160,7 +168,7 @@ class OReporter(YamlReporter): self.firstrun = False self._writeln("###########################################") self._writeln("# OONI Probe Report for %s test" % options['name']) - self._writeln("# %s" % date.pretty_date()) + self._writeln("# %s" % otime.prettyDateNow()) self._writeln("###########################################") client_geodata = {} @@ -194,7 +202,7 @@ class OReporter(YamlReporter): client_geodata['countrycode'] = client_location['countrycode'] - test_details = {'start_time': repr(date.now()), + test_details = {'start_time': otime.utcTimeNow(), 'probe_asn': client_geodata['asn'], 'probe_cc': client_geodata['countrycode'], 'probe_ip': client_geodata['ip'], @@ -222,7 +230,7 @@ class OReporter(YamlReporter): self.writeReportEntry(report) def allDone(self): - log.debug("Finished running everything") + log.debug("Finished running all tests") self.finish() try: reactor.stop() diff --git a/ooni/runner.py b/ooni/runner.py index d8b7df8..238d7d5 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -2,12 +2,11 @@ # # runner.py # --------- -# Handles running ooni.nettests as well as ooni.plugoo.tests.OONITests. +# Handles running ooni.nettests as well as +# ooni.plugoo.tests.OONITests. # -# :authors: Isis Lovecruft, Arturo Filasto +# :authors: Arturo Filastò, Isis Lovecruft # :license: see included LICENSE file -# :copyright: (c) 2012 Isis Lovecruft, Arturo Filasto, The Tor Project, Inc. -# :version: 0.1.0-pre-alpha import os import sys @@ -25,7 +24,7 @@ from ooni.nettest import NetTestCase from ooni import reporter -from ooni.utils import log, date, checkForRoot +from ooni.utils import log, checkForRoot, NotRootError def processTest(obj, cmd_line_options): """ @@ -42,7 +41,12 @@ def processTest(obj, cmd_line_options): input_file = obj.inputFile if obj.requiresRoot: - checkForRoot("test") + try: + checkForRoot() + except NotRootError: + log.err("%s requires root to run" % obj.name) + sys.exit(1) + if obj.optParameters or input_file \ or obj.usageOptions or obj.optFlags: diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index 1491cbc..4c42a3a 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -13,50 +13,10 @@ from twisted.internet import protocol, defer from twisted.internet.ssl import ClientContextFactory from twisted.web.http_headers import Headers -from twisted.web.iweb import IBodyProducer - from ooni.nettest import NetTestCase from ooni.utils import log -useragents = [("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6", "Firefox 2.0, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)", "Internet Explorer 7, Windows Vista"), - ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)", "Internet Explorer 7, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)", "Internet Explorer 6, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 5.0; Windows NT 5.1; .NET CLR 1.1.4322)", "Internet Explorer 5, Windows XP"), - ("Opera/9.20 (Windows NT 6.0; U; en)", "Opera 9.2, Windows Vista"), - ("Opera/9.00 (Windows NT 5.1; U; en)", "Opera 9.0, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.50", "Opera 8.5, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.0", "Opera 8.0, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Opera 7.02 [en]", "Opera 7.02, Windows XP"), - ("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20060127 Netscape/8.1", "Netscape 8.1, Windows XP")] - -class StringProducer(object): - implements(IBodyProducer) - - def __init__(self, body): - self.body = body - self.length = len(body) - - def startProducing(self, consumer): - consumer.write(self.body) - return defer.succeed(None) - - def pauseProducing(self): - pass - - def stopProducing(self): - pass - -class BodyReceiver(protocol.Protocol): - def __init__(self, finished): - self.finished = finished - self.data = "" - - def dataReceived(self, bytes): - self.data += bytes - - def connectionLost(self, reason): - self.finished.callback(self.data) +from ooni.utils.net import BodyReceiver, StringProducer, userAgents class HTTPTest(NetTestCase): """ @@ -204,7 +164,7 @@ class HTTPTest(NetTestCase): return finished def randomize_useragent(self): - user_agent = random.choice(useragents) + user_agent = random.choice(userAgents) self.request['headers']['User-Agent'] = [user_agent] def build_request(self, url, method="GET", headers=None, body=None): diff --git a/ooni/utils/__init__.py b/ooni/utils/__init__.py index 9961e03..5947519 100644 --- a/ooni/utils/__init__.py +++ b/ooni/utils/__init__.py @@ -7,11 +7,7 @@ import os import logging import string import random - -try: - import yaml -except: - print "Error in importing YAML" +import yaml class Storage(dict): """ @@ -30,7 +26,6 @@ class Storage(dict): >>> o.a None """ - def __getattr__(self, key): try: return self[key] @@ -56,99 +51,12 @@ class Storage(dict): for (k, v) in value.items(): self[k] = v -def checkForRoot(what): - if os.getuid() != 0: - raise Exception("This %s requires root to run" % what) - -def get_logger(config): - loglevel = getattr(logging, config.loglevel.upper()) - logging.basicConfig(level=loglevel, - format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', - filename=config.logfile, - filemode='w') - - console = logging.StreamHandler() - console.setLevel(getattr(logging, config.consoleloglevel.upper())) - # Set the console logger to a different format - formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s') - console.setFormatter(formatter) - logging.getLogger('').addHandler(console) - - return logging.getLogger('ooniprobe') - -def parse_asset(asset): - parsed = Storage() - try: - with open(asset, 'r') as f: - for line in f.readlines(): - # XXX This should be rewritten, if the values contain - # #: they will be rewritten with blank. - # should not be an issue but this is not a very good parser - if line.startswith("#:"): - n = line.split(' ')[0].replace('#:','') - v = line.replace('#:'+n+' ', '').strip() - if n in ('tests', 'files'): - parsed[n] = v.split(",") - else: - parsed[n] = v - - elif line.startswith("#"): - continue - else: - break - finally: - if not parsed.name: - parsed.name = asset - if not parsed.files: - parsed.files = asset - return parsed - -def import_test(name, config): - if name.endswith(".py"): - test = Storage() - test_name = name.split(".")[0] - fp, pathname, description = imp.find_module(test_name, - [config.main.testdir]) - module = imp.load_module(name, fp, pathname, description) - - try: - test.name = module.__name__ - test.desc = module.__desc__ - test.module = module - except: - test.name = test_name - test.desc = "" - test.module = module - - return test_name, test - - return None, None +class NotRootError(Exception): + pass -class Log(): - """ - This is a class necessary for parsing YAML log files. - It is required because pyYaml has a bug in parsing - log format YAML files. - """ - def __init__(self, file=None): - if file: - self.fh = open(file) - - def __iter__(self): - return self - - def next(self): - lines = [] - try: - line = self.fh.readline() - if not line: - raise StopIteration - while not line.startswith("---"): - lines.append(line) - line = self.fh.readline() - return lines - except: - raise StopIteration +def checkForRoot(): + if os.getuid() != 0: + raise NotRootError("This test requires root") def randomSTR(length, num=True): """ diff --git a/ooni/utils/date.py b/ooni/utils/date.py deleted file mode 100644 index 25250a6..0000000 --- a/ooni/utils/date.py +++ /dev/null @@ -1,30 +0,0 @@ -from ooni.lib.rfc3339 import rfc3339 -from datetime import datetime - -class odate(datetime): - def __str__(self): - return "%s" % rfc3339(self) - - def __repr__(self): - return "%s" % rfc3339(self) - - def from_rfc(self, datestr): - pass - -def now(): - return odate.utcnow() - -def pretty_date(): - cur_time = datetime.utcnow() - d_format = "%d %B %Y %H:%M:%S" - pretty = cur_time.strftime(d_format) - return pretty - -def timestamp(): - cur_time = datetime.utcnow() - d_format = "%d_%B_%Y_%H-%M-%S" - pretty = cur_time.strftime(d_format) - return pretty - - - diff --git a/ooni/utils/geodata.py b/ooni/utils/geodata.py index bd61dfd..5c3c481 100644 --- a/ooni/utils/geodata.py +++ b/ooni/utils/geodata.py @@ -1,23 +1,22 @@ +# -*- encoding: utf-8 -*- +# +# geodata.py +# ********** +# In here go functions related to the understanding of +# geographical information of the probe +# +# :authors: Arturo Filastò +# :licence: see LICENSE + import re -import pygeoip import os - -from ooni import config -from ooni.utils import log +import pygeoip from twisted.web.client import Agent from twisted.internet import reactor, defer, protocol -class BodyReceiver(protocol.Protocol): - def __init__(self, finished): - self.finished = finished - self.data = "" - - def dataReceived(self, bytes): - self.data += bytes - - def connectionLost(self, reason): - self.finished.callback(self.data) +from ooni.utils import log, net +from ooni import config @defer.inlineCallbacks def myIP(): @@ -28,7 +27,7 @@ def myIP(): result = yield myAgent.request('GET', target_site) finished = defer.Deferred() - result.deliverBody(BodyReceiver(finished)) + result.deliverBody(net.BodyReceiver(finished)) body = yield finished diff --git a/ooni/utils/hacks.py b/ooni/utils/hacks.py index e778540..4eef366 100644 --- a/ooni/utils/hacks.py +++ b/ooni/utils/hacks.py @@ -1,5 +1,10 @@ # -*- encoding: utf-8 -*- # +# hacks.py +# ******** +# When some software has issues and we need to fix it in a +# hackish way, we put it in here. This one day will be empty. +# # :authors: Arturo Filastò # :licence: see LICENSE @@ -56,36 +61,3 @@ def patched_reduce_ex(self, proto): return copy_reg._reconstructor, args, dict else: return copy_reg._reconstructor, args - -class MetaSuper(type): - """ - Metaclass for creating subclasses which have builtin name munging, so that - they are able to call self.__super.method() from an instance function - without knowing the instance class' base class name. - - For example: - - from hacks import MetaSuper - class A: - __metaclass__ = MetaSuper - def method(self): - return "A" - class B(A): - def method(self): - return "B" + self.__super.method() - class C(A): - def method(self): - return "C" + self.__super.method() - class D(C, B): - def method(self): - return "D" + self.__super.method() - - assert D().method() == "DCBA" - - Subclasses should not override "__init__", nor should subclasses have - the same name as any of their bases, or else much pain and suffering - will occur. - """ - def __init__(cls, name, bases, dict): - super(autosuper, cls).__init__(name, bases, dict) - setattr(cls, "_%s__super" % name, super(cls)) diff --git a/ooni/utils/net.py b/ooni/utils/net.py index 3fd4b41..d43261a 100644 --- a/ooni/utils/net.py +++ b/ooni/utils/net.py @@ -5,16 +5,55 @@ # OONI utilities for networking related operations import sys +from zope.interface import implements + +from twisted.internet import protocol from twisted.internet import threads, reactor +from twisted.web.iweb import IBodyProducer from scapy.all import utils from ooni.utils import log, txscapy -def getClientAddress(): - address = {'asn': 'REPLACE_ME', - 'ip': 'REPLACE_ME'} - return address +userAgents = [("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6", "Firefox 2.0, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)", "Internet Explorer 7, Windows Vista"), + ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)", "Internet Explorer 7, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)", "Internet Explorer 6, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 5.0; Windows NT 5.1; .NET CLR 1.1.4322)", "Internet Explorer 5, Windows XP"), + ("Opera/9.20 (Windows NT 6.0; U; en)", "Opera 9.2, Windows Vista"), + ("Opera/9.00 (Windows NT 5.1; U; en)", "Opera 9.0, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.50", "Opera 8.5, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.0", "Opera 8.0, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Opera 7.02 [en]", "Opera 7.02, Windows XP"), + ("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20060127 Netscape/8.1", "Netscape 8.1, Windows XP")] + +class StringProducer(object): + implements(IBodyProducer) + + def __init__(self, body): + self.body = body + self.length = len(body) + + def startProducing(self, consumer): + consumer.write(self.body) + return defer.succeed(None) + + def pauseProducing(self): + pass + + def stopProducing(self): + pass + +class BodyReceiver(protocol.Protocol): + def __init__(self, finished): + self.finished = finished + self.data = "" + + def dataReceived(self, bytes): + self.data += bytes + + def connectionLost(self, reason): + self.finished.callback(self.data) def capturePackets(pcap_filename): from scapy.all import sniff diff --git a/ooni/utils/otime.py b/ooni/utils/otime.py index 11f7be1..719230e 100644 --- a/ooni/utils/otime.py +++ b/ooni/utils/otime.py @@ -41,3 +41,11 @@ def utcPrettyDateNow(): def timeToPrettyDate(time_val): return time.ctime(time_val) + +def timestamp(): + cur_time = datetime.utcnow() + d_format = "%d_%B_%Y_%H-%M-%S" + pretty = cur_time.strftime(d_format) + return pretty + + diff --git a/ooniprobe.conf b/ooniprobe.conf index 1e76ad7..47f480a 100644 --- a/ooniprobe.conf +++ b/ooniprobe.conf @@ -20,7 +20,7 @@ advanced: # XXX change this to point to the directory where you have stored the GeoIP # database file. This should be the directory in which OONI is installed # /path/to/ooni-probe/data/ - geoip_data_dir: /home/x/code/networking/ooni-probe/data/ + geoip_data_dir: /usr/share/GeoIP/ debug: true threadpool_size: 10

1 0

[ooni-probe/master] Do a very big cleanup and refactor of all the code in the repo.
by art＠torproject.org 09 Nov '12

09 Nov '12

commit 7d6901f1552067bce9595db6a84f8f5245d8f28c Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Nov 9 17:37:32 2012 +0100 Do a very big cleanup and refactor of all the code in the repo. * Move daphn3 protocol to to-be-ported * Remove rfc3339 support, we will use seconds since epoch * Refactor code that should have been in nettests * Eliminate code duplication * Remove python metaclass virtuosism instanity --- CHANGES.yaml | 5 - HACKING | 141 +++++++++-------- TODO | 2 +- before_i_commit.sh | 2 +- nettests/core/http_host.py | 3 +- ooni/config.py | 4 +- ooni/inputunit.py | 12 ++ ooni/lib/rfc3339.py | 283 ---------------------------------- ooni/nettest.py | 20 +-- ooni/nodes.py | 24 ++-- ooni/oonicli.py | 23 ++-- ooni/protocols/daphn3.py | 311 -------------------------------------- ooni/reporter.py | 18 ++- ooni/runner.py | 16 ++- ooni/templates/httpt.py | 44 +----- ooni/utils/__init__.py | 104 +------------ ooni/utils/date.py | 30 ---- ooni/utils/geodata.py | 29 ++-- ooni/utils/hacks.py | 38 +---- ooni/utils/net.py | 47 ++++++- ooni/utils/otime.py | 8 + ooniprobe.conf | 2 +- to-be-ported/protocols/daphn3.py | 311 ++++++++++++++++++++++++++++++++++++++ 23 files changed, 531 insertions(+), 946 deletions(-) diff --git a/CHANGES.yaml b/CHANGES.yaml deleted file mode 100644 index 86206f1..0000000 --- a/CHANGES.yaml +++ /dev/null @@ -1,5 +0,0 @@ -# Add each new entry to the top of the file. - -- version: 0.0.1 - date: Thu Jul 7 16:03:36 EDT 2011 - changes: Initial version. diff --git a/HACKING b/HACKING index a0bf89c..2fdc392 100644 --- a/HACKING +++ b/HACKING @@ -34,80 +34,85 @@ Code Structure --------- - HACKING - The document you are currently reading. + The document you are currently reading. -- oonib/ - Contains the OONI probe backend to be run on the ooni-net +- inputs/ + Contains input files for tests. -- oonid/ - Contains the OONI daemon that can be used to interrogated from the cli to - run tests. +- oonib/ + Contains the OONI probe backend to be run on the ooni-net - ooni/ - Contains the main ooni probe comand line client - -- ooni/assets/ - Where we store all the asset files that are - used when running OONI tests. + Contains the main ooni probe comand line client - ooni/config.py - Parts of the code related to parsing OONI - configuration files and making them accessible - to other components of the software. - -- ooni/logo.py - File containing some funny ASCII art. Yes, we - do enjoy ASCII art and are not afraid to admit it! - -- ooni/nodes.conf - The configuration file for nodes. This contains the - list of network and code execution nodes that can be - used to run tests off of. - -- ooni/ooniprobe.py - The main OONI-probe command line interface. This is - responsible for parsing the command line arguments and - passing the arguments to the underlying components. - -- ooni/ooni-probe.conf - The main OONI-probe configuration file. This can be used - to configure your OONI CLI, tell it where it should report - to, where the asset files are located, what should be used - for control, etc. - -- ooni/plugoo/__init__.py - All the necessary "goo" for making OONI probe work. This - means loading Assets, creating Reports, running Tests, - interacting with Nodes. - -- ooni/plugoo/assets.py - This is a python object representation of the data that is - located inside the asset directory. - -- ooni/plugoo/nodes.py - The part of code responsible for interacting with OONI Nodes. - Nodes can be Network or Code Execution. Network nodes are - capable of receiving packets and fowarding them onto their - network. This means that to run a test on X network nodes you - consume X*test_cost bandwith. Code Execution nodes accept units - of work, they are therefore capable of receiving a set of tests - that should be completed by a set of Network nodes or run locally. - -- ooni/plugoo/reports.py - Takes care of transforming the output of a test into a report. This - may mean pushing the result data to a remote backend or simply writing - a local file. - -- ooni/plugoo/tests.py - The main OONI probe test class. This provides all the necessary scaffold - to write your own test based on the OONI paradigm. - -- ooni/oonitests/ - Contains all the "offical" OONI tests that are shipped. - -- ooni/utils.py - Helper functions that don't fit into any place, but are not big enough to - be a dependency by themselves. + Parts of the code related to parsing OONI + configuration files and making them accessible + to other components of the software. + +- ooni/inputunit.py + In here we have functions related to the creation of input + units. Input units are how the inputs to be fed to tests are + split up into. + +- ooni/nettest.py + In here is the NetTest API definition. This is how people + interested in writing ooniprobe tests will be specifying + them. + +- ooni/nodes.py + Mostly broken code for the remote dispatching of tests. + +- ooni/oonicli.py + In here we take care of running ooniprobe from the command + line interface + +- ooni/reporter.py + In here goes the logic for the creation of ooniprobe + reports. + +- ooni/runner.py + Handles running ooni.nettests as well as + ooni.plugoo.tests.OONITests. + +- ooni/kit/ + In here go utilities that can be used by tests. + +- ooni/lib/ + XXX this directory is to be removed. + +- ooni/utils/ + In here go internal utilities that are useful to ooniprobe + +- ooni/utils/geodata.py + In here go functions related to the understanding of + geographical information of the probe + +- ooni/utils/hacks.py + When some software has issues and we need to fix it in a + hackish way, we put it in here. This one day will be empty. + +- ooni/utils/log.py + log realted functions. + +- ooni/utils/net.py + utilities for networking related operations + +- ooni/utils/onion.py + Utilities for working with Tor. + XXX this code should be removed and merged into txtorcon. + +- ooni/utils/otime.py + Generation of timestamps, time conversions and all the rest + +- ooni/utils/txscapy.py + Tools for making scapy work well with twisted. + +- ooniprobe.conf + The main OONI-probe configuration file. This can be used + to configure your OONI CLI, tell it where it should report + to, where the asset files are located, what should be used + for control, etc. Style guide ----------- diff --git a/TODO b/TODO index 63d950c..b8524f3 100644 --- a/TODO +++ b/TODO @@ -105,4 +105,4 @@ It's important to make the new code asych and based on Twisted. It should respect the design goals of the new ooni-probe model. Also, importing new, non-standard libraries should be discussed first, if the new test is to be used in the core of OONI (packaging scapy and twisted already makes our -codebase quite large). \ No newline at end of file +codebase quite large). diff --git a/before_i_commit.sh b/before_i_commit.sh index f08a5f5..4a37686 100755 --- a/before_i_commit.sh +++ b/before_i_commit.sh @@ -34,5 +34,5 @@ echo "If you do, it means something is wrong." echo "Read through the log file and fix it." echo "If you are having some problems fixing some things that have to do with" echo "the core of OONI, let's first discuss it on IRC, or open a ticket" - +less *yamloo rm -f *yamloo diff --git a/nettests/core/http_host.py b/nettests/core/http_host.py index e09fc84..b87594d 100644 --- a/nettests/core/http_host.py +++ b/nettests/core/http_host.py @@ -14,8 +14,7 @@ from ooni.templates import httpt class UsageOptions(usage.Options): optParameters = [ - ['url', 'u', 'http://torproject.org/', 'Test single site'], - ['backend', 'b', 'http://ooni.nu/test/', 'Test backend to use'], + ['backend', 'b', 'http://ooni.nu/test/', 'Test backend to use'] ] diff --git a/ooni/config.py b/ooni/config.py index f3d1a80..cec9146 100644 --- a/ooni/config.py +++ b/ooni/config.py @@ -8,7 +8,7 @@ import yaml from twisted.internet import reactor, threads -from ooni.utils import date +from ooni.utils import otime from ooni.utils import Storage def get_root_path(): @@ -24,7 +24,7 @@ def oreport_filenames(): returns yamloo_filename, pcap_filename """ - base_filename = "%s_"+date.timestamp()+".%s" + base_filename = "%s_"+otime.timestamp()+".%s" yamloo_filename = base_filename % ("report", "yamloo") pcap_filename = base_filename % ("packets", "pcap") return yamloo_filename, pcap_filename diff --git a/ooni/inputunit.py b/ooni/inputunit.py index 3b0c491..484631b 100644 --- a/ooni/inputunit.py +++ b/ooni/inputunit.py @@ -1,3 +1,15 @@ +#-*- coding: utf-8 -*- +# +# inputunit.py +# ------------- +# IN here we have functions related to the creation of input +# units. Input units are how the inputs to be fed to tests are +# split up into. +# +# :authors: Arturo Filastò, Isis Lovecruft +# :license: see included LICENSE file + + class InputUnitFactory(object): """ This is a factory that takes the size of input units to be generated a set diff --git a/ooni/lib/rfc3339.py b/ooni/lib/rfc3339.py deleted file mode 100644 index e664ce7..0000000 --- a/ooni/lib/rfc3339.py +++ /dev/null @@ -1,283 +0,0 @@ -#!/usr/bin/env python -# -# Copyright (c) 2009, 2010, Henry Precheur <henry(a)precheur.org> -# -# Permission to use, copy, modify, and/or distribute this software for any -# purpose with or without fee is hereby granted, provided that the above -# copyright notice and this permission notice appear in all copies. -# -# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH -# REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND -# FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, -# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM -# LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR -# OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR -# PERFORMANCE OF THIS SOFTWARE. -# -'''Formats dates according to the :RFC:`3339`. - -Report bugs & problems on BitBucket_ - -.. _BitBucket: https://bitbucket.org/henry/clan.cx/issues -''' - -__author__ = 'Henry Precheur <henry(a)precheur.org>' -__license__ = 'ISCL' -__version__ = '5.1' -__all__ = ('rfc3339', ) - -import datetime -import time -import unittest - -def _timezone(utc_offset): - ''' - Return a string representing the timezone offset. - - >>> _timezone(0) - '+00:00' - >>> _timezone(3600) - '+01:00' - >>> _timezone(-28800) - '-08:00' - >>> _timezone(-1800) - '-00:30' - ''' - # Python's division uses floor(), not round() like in other languages: - # -1 / 2 == -1 and not -1 / 2 == 0 - # That's why we use abs(utc_offset). - hours = abs(utc_offset) // 3600 - minutes = abs(utc_offset) % 3600 // 60 - sign = (utc_offset < 0 and '-') or '+' - return '%c%02d:%02d' % (sign, hours, minutes) - -def _timedelta_to_seconds(timedelta): - ''' - >>> _timedelta_to_seconds(datetime.timedelta(hours=3)) - 10800 - >>> _timedelta_to_seconds(datetime.timedelta(hours=3, minutes=15)) - 11700 - ''' - return (timedelta.days * 86400 + timedelta.seconds + - timedelta.microseconds // 1000) - -def _utc_offset(date, use_system_timezone): - ''' - Return the UTC offset of `date`. If `date` does not have any `tzinfo`, use - the timezone informations stored locally on the system. - - >>> if time.localtime().tm_isdst: - ... system_timezone = -time.altzone - ... else: - ... system_timezone = -time.timezone - >>> _utc_offset(datetime.datetime.now(), True) == system_timezone - True - >>> _utc_offset(datetime.datetime.now(), False) - 0 - ''' - if isinstance(date, datetime.datetime) and date.tzinfo is not None: - return _timedelta_to_seconds(date.dst() or date.utcoffset()) - elif use_system_timezone: - if date.year < 1970: - # We use 1972 because 1970 doesn't have a leap day (feb 29) - t = time.mktime(date.replace(year=1972).timetuple()) - else: - t = time.mktime(date.timetuple()) - if time.localtime(t).tm_isdst: # pragma: no cover - return -time.altzone - else: - return -time.timezone - else: - return 0 - -def _string(d, timezone): - return ('%04d-%02d-%02dT%02d:%02d:%02d%s' % - (d.year, d.month, d.day, d.hour, d.minute, d.second, timezone)) - -def rfc3339(date, utc=False, use_system_timezone=True): - ''' - Return a string formatted according to the :RFC:`3339`. If called with - `utc=True`, it normalizes `date` to the UTC date. If `date` does not have - any timezone information, uses the local timezone:: - - >>> d = datetime.datetime(2008, 4, 2, 20) - >>> rfc3339(d, utc=True, use_system_timezone=False) - '2008-04-02T20:00:00Z' - >>> rfc3339(d) # doctest: +ELLIPSIS - '2008-04-02T20:00:00...' - - If called with `user_system_timezone=False` don't use the local timezone if - `date` does not have timezone informations and consider the offset to UTC - to be zero:: - - >>> rfc3339(d, use_system_timezone=False) - '2008-04-02T20:00:00+00:00' - - `date` must be a `datetime.datetime`, `datetime.date` or a timestamp as - returned by `time.time()`:: - - >>> rfc3339(0, utc=True, use_system_timezone=False) - '1970-01-01T00:00:00Z' - >>> rfc3339(datetime.date(2008, 9, 6), utc=True, - ... use_system_timezone=False) - '2008-09-06T00:00:00Z' - >>> rfc3339(datetime.date(2008, 9, 6), - ... use_system_timezone=False) - '2008-09-06T00:00:00+00:00' - >>> rfc3339('foo bar') - Traceback (most recent call last): - ... - TypeError: Expected timestamp or date object. Got <type 'str'>. - - For dates before January 1st 1970, the timezones will be the ones used in - 1970. It might not be accurate, but on most sytem there is no timezone - information before 1970. - ''' - # Try to convert timestamp to datetime - try: - if use_system_timezone: - date = datetime.datetime.fromtimestamp(date) - else: - date = datetime.datetime.utcfromtimestamp(date) - except TypeError: - pass - - if not isinstance(date, datetime.date): - raise TypeError('Expected timestamp or date object. Got %r.' % - type(date)) - - if not isinstance(date, datetime.datetime): - date = datetime.datetime(*date.timetuple()[:3]) - utc_offset = _utc_offset(date, use_system_timezone) - if utc: - return _string(date + datetime.timedelta(seconds=utc_offset), 'Z') - else: - return _string(date, _timezone(utc_offset)) - - -class LocalTimeTestCase(unittest.TestCase): - ''' - Test the use of the timezone saved locally. Since it is hard to test using - doctest. - ''' - - def setUp(self): - local_utcoffset = _utc_offset(datetime.datetime.now(), True) - self.local_utcoffset = datetime.timedelta(seconds=local_utcoffset) - self.local_timezone = _timezone(local_utcoffset) - - def test_datetime(self): - d = datetime.datetime.now() - self.assertEqual(rfc3339(d), - d.strftime('%Y-%m-%dT%H:%M:%S') + self.local_timezone) - - def test_datetime_timezone(self): - - class FixedNoDst(datetime.tzinfo): - 'A timezone info with fixed offset, not DST' - - def utcoffset(self, dt): - return datetime.timedelta(hours=2, minutes=30) - - def dst(self, dt): - return None - - fixed_no_dst = FixedNoDst() - - class Fixed(FixedNoDst): - 'A timezone info with DST' - - def dst(self, dt): - return datetime.timedelta(hours=3, minutes=15) - - fixed = Fixed() - - d = datetime.datetime.now().replace(tzinfo=fixed_no_dst) - timezone = _timezone(_timedelta_to_seconds(fixed_no_dst.\ - utcoffset(None))) - self.assertEqual(rfc3339(d), - d.strftime('%Y-%m-%dT%H:%M:%S') + timezone) - - d = datetime.datetime.now().replace(tzinfo=fixed) - timezone = _timezone(_timedelta_to_seconds(fixed.dst(None))) - self.assertEqual(rfc3339(d), - d.strftime('%Y-%m-%dT%H:%M:%S') + timezone) - - def test_datetime_utc(self): - d = datetime.datetime.now() - d_utc = d + self.local_utcoffset - self.assertEqual(rfc3339(d, utc=True), - d_utc.strftime('%Y-%m-%dT%H:%M:%SZ')) - - def test_date(self): - d = datetime.date.today() - self.assertEqual(rfc3339(d), - d.strftime('%Y-%m-%dT%H:%M:%S') + self.local_timezone) - - def test_date_utc(self): - d = datetime.date.today() - # Convert `date` to `datetime`, since `date` ignores seconds and hours - # in timedeltas: - # >>> datetime.date(2008, 9, 7) + datetime.timedelta(hours=23) - # datetime.date(2008, 9, 7) - d_utc = datetime.datetime(*d.timetuple()[:3]) + self.local_utcoffset - self.assertEqual(rfc3339(d, utc=True), - d_utc.strftime('%Y-%m-%dT%H:%M:%SZ')) - - def test_timestamp(self): - d = time.time() - self.assertEqual(rfc3339(d), - datetime.datetime.fromtimestamp(d).\ - strftime('%Y-%m-%dT%H:%M:%S') + self.local_timezone) - - def test_timestamp_utc(self): - d = time.time() - d_utc = datetime.datetime.utcfromtimestamp(d) + self.local_utcoffset - self.assertEqual(rfc3339(d), - (d_utc.strftime('%Y-%m-%dT%H:%M:%S') + - self.local_timezone)) - - def test_before_1970(self): - d = datetime.date(1885, 01, 04) - self.failUnless(rfc3339(d).startswith('1885-01-04T00:00:00')) - self.assertEqual(rfc3339(d, utc=True, use_system_timezone=False), - '1885-01-04T00:00:00Z') - - def test_1920(self): - d = datetime.date(1920, 02, 29) - x = rfc3339(d, utc=False, use_system_timezone=True) - self.failUnless(x.startswith('1920-02-29T00:00:00')) - - # If these tests start failing it probably means there was a policy change - # for the Pacific time zone. - # See http://en.wikipedia.org/wiki/Pacific_Time_Zone. - if 'PST' in time.tzname: - def testPDTChange(self): - '''Test Daylight saving change''' - # PDT switch happens at 2AM on March 14, 2010 - - # 1:59AM PST - self.assertEqual(rfc3339(datetime.datetime(2010, 3, 14, 1, 59)), - '2010-03-14T01:59:00-08:00') - # 3AM PDT - self.assertEqual(rfc3339(datetime.datetime(2010, 3, 14, 3, 0)), - '2010-03-14T03:00:00-07:00') - - def testPSTChange(self): - '''Test Standard time change''' - # PST switch happens at 2AM on November 6, 2010 - - # 0:59AM PDT - self.assertEqual(rfc3339(datetime.datetime(2010, 11, 7, 0, 59)), - '2010-11-07T00:59:00-07:00') - - # 1:00AM PST - # There's no way to have 1:00AM PST without a proper tzinfo - self.assertEqual(rfc3339(datetime.datetime(2010, 11, 7, 1, 0)), - '2010-11-07T01:00:00-07:00') - - -if __name__ == '__main__': # pragma: no cover - import doctest - doctest.testmod() - unittest.main() diff --git a/ooni/nettest.py b/ooni/nettest.py index 6221a3f..896ae2a 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -1,15 +1,12 @@ # -*- encoding: utf-8 -*- # -# nettest.py -# -------------------> +# nettest.py +# ---------- +# In here is the NetTest API definition. This is how people +# interested in writing ooniprobe tests will be specifying them # -# :authors: Arturo "hellais" Filastò <art(a)fuffa.org>, -# Isis Lovecruft <isis(a)torproject.org> -# :licence: see LICENSE -# :copyright: 2012 Arturo Filasto, Isis Lovecruft -# :version: 0.1.0-alpha -# -# <------------------- +# :authors: Arturo Filastò, Isis Lovecruft +# :license: see included LICENSE file import sys import os @@ -22,8 +19,6 @@ from twisted.python import usage from ooni.utils import log -pyunit = __import__('unittest') - class NetTestCase(object): """ This is the base of the OONI nettest universe. When you write a nettest @@ -102,7 +97,6 @@ class NetTestCase(object): usageOptions = None requiredOptions = [] - requiresRoot = False localOptions = {} @@ -159,7 +153,7 @@ class NetTestCase(object): raise usage.UsageError("No input file specified!") self._checkRequiredOptions() - + # XXX perhaps we may want to name and version to be inside of a # different method that is not called options. return {'inputs': self.inputs, diff --git a/ooni/nodes.py b/ooni/nodes.py index 155f183..070ffe7 100644 --- a/ooni/nodes.py +++ b/ooni/nodes.py @@ -1,16 +1,14 @@ -#!/usr/bin/env python -# -*- coding: UTF-8 -""" - nodes - ***** - - This contains all the code related to Nodes - both network and code execution. - - :copyright: (c) 2012 by Arturo Filastò, Isis Lovecruft - :license: see LICENSE for more details. - -""" +#-*- coding: utf-8 -*- +# +# nodes.py +# -------- +# here is code for handling the interaction with remote +# services that will run ooniprobe tests. +# XXX most of the code in here is broken or not tested and +# probably should be trashed +# +# :authors: Arturo Filastò, Isis Lovecruft +# :license: see included LICENSE file import os from binascii import hexlify diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 4b63a61..8e4fa14 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -1,16 +1,12 @@ -#!/usr/bin/env python # -*- coding: UTF-8 # -# oonicli -# ********* +# oonicli +# ------- +# In here we take care of running ooniprobe from the command +# line interface # -# oonicli is the next generation ooniprober. It based off of twisted's trial -# unit testing framework. -# -# :copyright: (c) 2012 by Arturo Filastò, Isis Lovecruft -# :license: see LICENSE for more details. -# -# original copyright (c) by Twisted Matrix Laboratories. +# :authors: Arturo Filastò, Isis Lovecruft +# :license: see included LICENSE file import sys @@ -27,7 +23,8 @@ from ooni import nettest, runner, reporter, config from ooni.inputunit import InputUnitFactory -from ooni.utils import net, checkForRoot +from ooni.utils import net +from ooni.utils import checkForRoot, NotRootError from ooni.utils import log @@ -118,9 +115,9 @@ def run(): if config.privacy.includepcap: try: checkForRoot() - except: + except NotRootError: log.err("includepcap options requires root priviledges to run") - log.err("disable it in your ooniprobe.conf file") + log.err("you should run ooniprobe as root or disable the options in ooniprobe.conf") sys.exit(1) log.debug("Starting sniffer") sniffer_d = net.capturePackets(pcap_filename) diff --git a/ooni/protocols/__init__.py b/ooni/protocols/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/ooni/protocols/daphn3.py b/ooni/protocols/daphn3.py deleted file mode 100644 index 37c94c7..0000000 --- a/ooni/protocols/daphn3.py +++ /dev/null @@ -1,311 +0,0 @@ -import sys -import yaml - -from twisted.internet import protocol, defer -from twisted.internet.error import ConnectionDone - -from scapy.all import IP, Raw, rdpcap - -from ooni.utils import log -from ooni.plugoo import reports - -def read_pcap(filename): - """ - @param filename: Filesystem path to the pcap. - - Returns: - [{"sender": "client", "data": "\x17\x52\x15"}, {"sender": "server", "data": "\x17\x15\x13"}] - """ - packets = rdpcap(filename) - - checking_first_packet = True - client_ip_addr = None - server_ip_addr = None - - ssl_packets = [] - messages = [] - - """ - pcap assumptions: - - pcap only contains packets exchanged between a Tor client and a Tor server. - (This assumption makes sure that there are only two IP addresses in the - pcap file) - - The first packet of the pcap is sent from the client to the server. (This - assumption is used to get the IP address of the client.) - - All captured packets are TLS packets: that is TCP session - establishment/teardown packets should be filtered out (no SYN/SYN+ACK) - """ - - """Minimally validate the pcap and also find out what's the client - and server IP addresses.""" - for packet in packets: - if checking_first_packet: - client_ip_addr = packet[IP].src - checking_first_packet = False - else: - if packet[IP].src != client_ip_addr: - server_ip_addr = packet[IP].src - - try: - if (packet[Raw]): - ssl_packets.append(packet) - except IndexError: - pass - - """Form our list.""" - for packet in ssl_packets: - if packet[IP].src == client_ip_addr: - messages.append({"sender": "client", "data": str(packet[Raw])}) - elif packet[IP].src == server_ip_addr: - messages.append({"sender": "server", "data": str(packet[Raw])}) - else: - raise("Detected third IP address! pcap is corrupted.") - - return messages - -def read_yaml(filename): - f = open(filename) - obj = yaml.load(f) - f.close() - return obj - -class Mutator: - idx = 0 - step = 0 - - waiting = False - waiting_step = 0 - - def __init__(self, steps): - """ - @param steps: array of dicts for the steps that must be gone over by - the mutator. Looks like this: - [{"sender": "client", "data": "\xde\xad\xbe\xef"}, - {"sender": "server", "data": "\xde\xad\xbe\xef"}] - """ - self.steps = steps - - def _mutate(self, data, idx): - """ - Mutate the idx bytes by increasing it's value by one - - @param data: the data to be mutated. - - @param idx: what byte should be mutated. - """ - print "idx: %s, data: %s" % (idx, data) - ret = data[:idx] - ret += chr(ord(data[idx]) + 1) - ret += data[idx+1:] - return ret - - def state(self): - """ - Return the current mutation state. As in what bytes are being mutated. - - Returns a dict containg the packet index and the step number. - """ - print "[Mutator.state()] Giving out my internal state." - current_state = {'idx': self.idx, 'step': self.step} - return current_state - - def next(self): - """ - Increases by one the mutation state. - - ex. (* is the mutation state, i.e. the byte to be mutated) - before [___*] [____] - step1 step2 - after [____] [*___] - - Should be called every time you need to proceed onto the next mutation. - It changes the internal state of the mutator to that of the next - mutatation. - - returns True if another mutation is available. - returns False if all the possible mutations have been done. - """ - if (self.step) == len(self.steps): - # Hack to stop once we have gone through all the steps - print "[Mutator.next()] I believe I have gone over all steps" - print " Stopping!" - self.waiting = True - return False - - self.idx += 1 - current_idx = self.idx - current_step = self.step - current_data = self.steps[current_step]['data'] - - if 0: - print "current_step: %s" % current_step - print "current_idx: %s" % current_idx - print "current_data: %s" % current_data - print "steps: %s" % len(self.steps) - print "waiting_step: %s" % self.waiting_step - - data_to_receive = len(self.steps[current_step]['data']) - - if self.waiting and self.waiting_step == data_to_receive: - print "[Mutator.next()] I am no longer waiting" - log.debug("I am no longer waiting.") - self.waiting = False - self.waiting_step = 0 - self.idx = 0 - - elif self.waiting: - print "[Mutator.next()] Waiting some more." - log.debug("Waiting some more.") - self.waiting_step += 1 - - elif current_idx >= len(current_data): - print "[Mutator.next()] Entering waiting mode." - log.debug("Entering waiting mode.") - self.step += 1 - self.idx = 0 - self.waiting = True - - log.debug("current index %s" % current_idx) - log.debug("current data %s" % len(current_data)) - return True - - def get(self, step): - """ - Returns the current packet to be sent to the wire. - If no mutation is necessary it will return the plain data. - Should be called when you are interested in obtaining the data to be - sent for the selected state. - - @param step: the current step you want the mutation for - - returns the mutated packet for the specified step. - """ - if step != self.step or self.waiting: - log.debug("[Mutator.get()] I am not going to do anything :)") - return self.steps[step]['data'] - - data = self.steps[step]['data'] - #print "Mutating %s with idx %s" % (data, self.idx) - return self._mutate(data, self.idx) - -class Daphn3Protocol(protocol.Protocol): - """ - This implements the Daphn3 protocol for the server side. - It gets instanced once for every client that connects to the oonib. - For every instance of protocol there is only 1 mutation. - Once the last step is reached the connection is closed on the serverside. - """ - steps = [] - mutator = None - - current_state = None - - role = 'client' - state = 0 - total_states = len(steps) - 1 - received_data = 0 - to_receive_data = 0 - report = reports.Report('daphn3', 'daphn3.yamlooni') - - test = None - - def next_state(self): - """ - This is called once I have completed one step of the protocol and need - to proceed to the next step. - """ - if not self.mutator: - print "[Daphn3Protocol.next_state] No mutator. There is no point to stay on this earth." - self.transport.loseConnection() - return - - if self.role is self.steps[self.state]['sender']: - print "[Daphn3Protocol.next_state] I am a sender" - data = self.mutator.get(self.state) - self.transport.write(data) - self.to_receive_data = 0 - - else: - print "[Daphn3Protocol.next_state] I am a receiver" - self.to_receive_data = len(self.steps[self.state]['data']) - - self.state += 1 - self.received_data = 0 - - def dataReceived(self, data): - """ - This is called every time some data is received. I transition to the - next step once the amount of data that I expect to receive is received. - - @param data: the data that has been sent by the client. - """ - if not self.mutator: - print "I don't have a mutator. My life means nothing." - self.transport.loseConnection() - return - - if len(self.steps) == self.state: - self.transport.loseConnection() - return - - self.received_data += len(data) - if self.received_data >= self.to_receive_data: - print "Moving to next state %s" % self.state - self.next_state() - - def censorship_detected(self, report): - """ - I have detected the possible presence of censorship we need to write a - report on it. - - @param report: a dict containing the report to be written. Must contain - the keys 'reason', 'proto_state' and 'mutator_state'. - The reason is the reason for which the connection was - closed. The proto_state is the current state of the - protocol instance and mutator_state is what was being - mutated. - """ - print "The connection was closed because of %s" % report['reason'] - print "State %s, Mutator %s" % (report['proto_state'], - report['mutator_state']) - if self.test: - self.test.result['censored'] = True - self.test.result['state'] = report - self.mutator.next() - - def connectionLost(self, reason): - """ - The connection was closed. This may be because of a legittimate reason - or it may be because of a censorship event. - """ - if not self.mutator: - print "Terminated because of little interest in life." - return - report = {'reason': reason, 'proto_state': self.state, - 'trigger': None, 'mutator_state': self.current_state} - - if self.state < self.total_states: - report['trigger'] = 'did not finish state walk' - self.censorship_detected(report) - - else: - print "I have reached the end of the state machine" - print "Censorship fingerprint bruteforced!" - if self.test: - print "In the test thing" - self.test.result['censored'] = False - self.test.result['state'] = report - self.test.result['state_walk_finished'] = True - self.test.report(self.test.result) - return - - if reason.check(ConnectionDone): - print "Connection closed cleanly" - else: - report['trigger'] = 'unclean connection closure' - self.censorship_detected(report) - - diff --git a/ooni/reporter.py b/ooni/reporter.py index cdbf355..6d37838 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -1,3 +1,12 @@ +#-*- coding: utf-8 -*- +# +# reporter.py +# ----------- +# In here goes the logic for the creation of ooniprobe reports. +# +# :authors: Arturo Filastò, Isis Lovecruft +# :license: see included LICENSE file + import itertools import logging import sys @@ -11,13 +20,12 @@ from yaml.emitter import * from yaml.serializer import * from yaml.resolver import * -from datetime import datetime from twisted.python.util import untilConcludes from twisted.trial import reporter from twisted.internet import defer, reactor from ooni.templates.httpt import BodyReceiver, StringProducer -from ooni.utils import date, log, geodata +from ooni.utils import otime, log, geodata from ooni import config try: @@ -160,7 +168,7 @@ class OReporter(YamlReporter): self.firstrun = False self._writeln("###########################################") self._writeln("# OONI Probe Report for %s test" % options['name']) - self._writeln("# %s" % date.pretty_date()) + self._writeln("# %s" % otime.prettyDateNow()) self._writeln("###########################################") client_geodata = {} @@ -194,7 +202,7 @@ class OReporter(YamlReporter): client_geodata['countrycode'] = client_location['countrycode'] - test_details = {'start_time': repr(date.now()), + test_details = {'start_time': otime.utcTimeNow(), 'probe_asn': client_geodata['asn'], 'probe_cc': client_geodata['countrycode'], 'probe_ip': client_geodata['ip'], @@ -222,7 +230,7 @@ class OReporter(YamlReporter): self.writeReportEntry(report) def allDone(self): - log.debug("Finished running everything") + log.debug("Finished running all tests") self.finish() try: reactor.stop() diff --git a/ooni/runner.py b/ooni/runner.py index d8b7df8..238d7d5 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -2,12 +2,11 @@ # # runner.py # --------- -# Handles running ooni.nettests as well as ooni.plugoo.tests.OONITests. +# Handles running ooni.nettests as well as +# ooni.plugoo.tests.OONITests. # -# :authors: Isis Lovecruft, Arturo Filasto +# :authors: Arturo Filastò, Isis Lovecruft # :license: see included LICENSE file -# :copyright: (c) 2012 Isis Lovecruft, Arturo Filasto, The Tor Project, Inc. -# :version: 0.1.0-pre-alpha import os import sys @@ -25,7 +24,7 @@ from ooni.nettest import NetTestCase from ooni import reporter -from ooni.utils import log, date, checkForRoot +from ooni.utils import log, checkForRoot, NotRootError def processTest(obj, cmd_line_options): """ @@ -42,7 +41,12 @@ def processTest(obj, cmd_line_options): input_file = obj.inputFile if obj.requiresRoot: - checkForRoot("test") + try: + checkForRoot() + except NotRootError: + log.err("%s requires root to run" % obj.name) + sys.exit(1) + if obj.optParameters or input_file \ or obj.usageOptions or obj.optFlags: diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index 1491cbc..4c42a3a 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -13,50 +13,10 @@ from twisted.internet import protocol, defer from twisted.internet.ssl import ClientContextFactory from twisted.web.http_headers import Headers -from twisted.web.iweb import IBodyProducer - from ooni.nettest import NetTestCase from ooni.utils import log -useragents = [("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6", "Firefox 2.0, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)", "Internet Explorer 7, Windows Vista"), - ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)", "Internet Explorer 7, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)", "Internet Explorer 6, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 5.0; Windows NT 5.1; .NET CLR 1.1.4322)", "Internet Explorer 5, Windows XP"), - ("Opera/9.20 (Windows NT 6.0; U; en)", "Opera 9.2, Windows Vista"), - ("Opera/9.00 (Windows NT 5.1; U; en)", "Opera 9.0, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.50", "Opera 8.5, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.0", "Opera 8.0, Windows XP"), - ("Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Opera 7.02 [en]", "Opera 7.02, Windows XP"), - ("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20060127 Netscape/8.1", "Netscape 8.1, Windows XP")] - -class StringProducer(object): - implements(IBodyProducer) - - def __init__(self, body): - self.body = body - self.length = len(body) - - def startProducing(self, consumer): - consumer.write(self.body) - return defer.succeed(None) - - def pauseProducing(self): - pass - - def stopProducing(self): - pass - -class BodyReceiver(protocol.Protocol): - def __init__(self, finished): - self.finished = finished - self.data = "" - - def dataReceived(self, bytes): - self.data += bytes - - def connectionLost(self, reason): - self.finished.callback(self.data) +from ooni.utils.net import BodyReceiver, StringProducer, userAgents class HTTPTest(NetTestCase): """ @@ -204,7 +164,7 @@ class HTTPTest(NetTestCase): return finished def randomize_useragent(self): - user_agent = random.choice(useragents) + user_agent = random.choice(userAgents) self.request['headers']['User-Agent'] = [user_agent] def build_request(self, url, method="GET", headers=None, body=None): diff --git a/ooni/utils/__init__.py b/ooni/utils/__init__.py index 9961e03..5947519 100644 --- a/ooni/utils/__init__.py +++ b/ooni/utils/__init__.py @@ -7,11 +7,7 @@ import os import logging import string import random - -try: - import yaml -except: - print "Error in importing YAML" +import yaml class Storage(dict): """ @@ -30,7 +26,6 @@ class Storage(dict): >>> o.a None """ - def __getattr__(self, key): try: return self[key] @@ -56,99 +51,12 @@ class Storage(dict): for (k, v) in value.items(): self[k] = v -def checkForRoot(what): - if os.getuid() != 0: - raise Exception("This %s requires root to run" % what) - -def get_logger(config): - loglevel = getattr(logging, config.loglevel.upper()) - logging.basicConfig(level=loglevel, - format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', - filename=config.logfile, - filemode='w') - - console = logging.StreamHandler() - console.setLevel(getattr(logging, config.consoleloglevel.upper())) - # Set the console logger to a different format - formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s') - console.setFormatter(formatter) - logging.getLogger('').addHandler(console) - - return logging.getLogger('ooniprobe') - -def parse_asset(asset): - parsed = Storage() - try: - with open(asset, 'r') as f: - for line in f.readlines(): - # XXX This should be rewritten, if the values contain - # #: they will be rewritten with blank. - # should not be an issue but this is not a very good parser - if line.startswith("#:"): - n = line.split(' ')[0].replace('#:','') - v = line.replace('#:'+n+' ', '').strip() - if n in ('tests', 'files'): - parsed[n] = v.split(",") - else: - parsed[n] = v - - elif line.startswith("#"): - continue - else: - break - finally: - if not parsed.name: - parsed.name = asset - if not parsed.files: - parsed.files = asset - return parsed - -def import_test(name, config): - if name.endswith(".py"): - test = Storage() - test_name = name.split(".")[0] - fp, pathname, description = imp.find_module(test_name, - [config.main.testdir]) - module = imp.load_module(name, fp, pathname, description) - - try: - test.name = module.__name__ - test.desc = module.__desc__ - test.module = module - except: - test.name = test_name - test.desc = "" - test.module = module - - return test_name, test - - return None, None +class NotRootError(Exception): + pass -class Log(): - """ - This is a class necessary for parsing YAML log files. - It is required because pyYaml has a bug in parsing - log format YAML files. - """ - def __init__(self, file=None): - if file: - self.fh = open(file) - - def __iter__(self): - return self - - def next(self): - lines = [] - try: - line = self.fh.readline() - if not line: - raise StopIteration - while not line.startswith("---"): - lines.append(line) - line = self.fh.readline() - return lines - except: - raise StopIteration +def checkForRoot(): + if os.getuid() != 0: + raise NotRootError("This test requires root") def randomSTR(length, num=True): """ diff --git a/ooni/utils/date.py b/ooni/utils/date.py deleted file mode 100644 index 25250a6..0000000 --- a/ooni/utils/date.py +++ /dev/null @@ -1,30 +0,0 @@ -from ooni.lib.rfc3339 import rfc3339 -from datetime import datetime - -class odate(datetime): - def __str__(self): - return "%s" % rfc3339(self) - - def __repr__(self): - return "%s" % rfc3339(self) - - def from_rfc(self, datestr): - pass - -def now(): - return odate.utcnow() - -def pretty_date(): - cur_time = datetime.utcnow() - d_format = "%d %B %Y %H:%M:%S" - pretty = cur_time.strftime(d_format) - return pretty - -def timestamp(): - cur_time = datetime.utcnow() - d_format = "%d_%B_%Y_%H-%M-%S" - pretty = cur_time.strftime(d_format) - return pretty - - - diff --git a/ooni/utils/geodata.py b/ooni/utils/geodata.py index bd61dfd..5c3c481 100644 --- a/ooni/utils/geodata.py +++ b/ooni/utils/geodata.py @@ -1,23 +1,22 @@ +# -*- encoding: utf-8 -*- +# +# geodata.py +# ********** +# In here go functions related to the understanding of +# geographical information of the probe +# +# :authors: Arturo Filastò +# :licence: see LICENSE + import re -import pygeoip import os - -from ooni import config -from ooni.utils import log +import pygeoip from twisted.web.client import Agent from twisted.internet import reactor, defer, protocol -class BodyReceiver(protocol.Protocol): - def __init__(self, finished): - self.finished = finished - self.data = "" - - def dataReceived(self, bytes): - self.data += bytes - - def connectionLost(self, reason): - self.finished.callback(self.data) +from ooni.utils import log, net +from ooni import config @defer.inlineCallbacks def myIP(): @@ -28,7 +27,7 @@ def myIP(): result = yield myAgent.request('GET', target_site) finished = defer.Deferred() - result.deliverBody(BodyReceiver(finished)) + result.deliverBody(net.BodyReceiver(finished)) body = yield finished diff --git a/ooni/utils/hacks.py b/ooni/utils/hacks.py index e778540..4eef366 100644 --- a/ooni/utils/hacks.py +++ b/ooni/utils/hacks.py @@ -1,5 +1,10 @@ # -*- encoding: utf-8 -*- # +# hacks.py +# ******** +# When some software has issues and we need to fix it in a +# hackish way, we put it in here. This one day will be empty. +# # :authors: Arturo Filastò # :licence: see LICENSE @@ -56,36 +61,3 @@ def patched_reduce_ex(self, proto): return copy_reg._reconstructor, args, dict else: return copy_reg._reconstructor, args - -class MetaSuper(type): - """ - Metaclass for creating subclasses which have builtin name munging, so that - they are able to call self.__super.method() from an instance function - without knowing the instance class' base class name. - - For example: - - from hacks import MetaSuper - class A: - __metaclass__ = MetaSuper - def method(self): - return "A" - class B(A): - def method(self): - return "B" + self.__super.method() - class C(A): - def method(self): - return "C" + self.__super.method() - class D(C, B): - def method(self): - return "D" + self.__super.method() - - assert D().method() == "DCBA" - - Subclasses should not override "__init__", nor should subclasses have - the same name as any of their bases, or else much pain and suffering - will occur. - """ - def __init__(cls, name, bases, dict): - super(autosuper, cls).__init__(name, bases, dict) - setattr(cls, "_%s__super" % name, super(cls)) diff --git a/ooni/utils/net.py b/ooni/utils/net.py index 3fd4b41..d43261a 100644 --- a/ooni/utils/net.py +++ b/ooni/utils/net.py @@ -5,16 +5,55 @@ # OONI utilities for networking related operations import sys +from zope.interface import implements + +from twisted.internet import protocol from twisted.internet import threads, reactor +from twisted.web.iweb import IBodyProducer from scapy.all import utils from ooni.utils import log, txscapy -def getClientAddress(): - address = {'asn': 'REPLACE_ME', - 'ip': 'REPLACE_ME'} - return address +userAgents = [("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6", "Firefox 2.0, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)", "Internet Explorer 7, Windows Vista"), + ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)", "Internet Explorer 7, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)", "Internet Explorer 6, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 5.0; Windows NT 5.1; .NET CLR 1.1.4322)", "Internet Explorer 5, Windows XP"), + ("Opera/9.20 (Windows NT 6.0; U; en)", "Opera 9.2, Windows Vista"), + ("Opera/9.00 (Windows NT 5.1; U; en)", "Opera 9.0, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.50", "Opera 8.5, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.0", "Opera 8.0, Windows XP"), + ("Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Opera 7.02 [en]", "Opera 7.02, Windows XP"), + ("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20060127 Netscape/8.1", "Netscape 8.1, Windows XP")] + +class StringProducer(object): + implements(IBodyProducer) + + def __init__(self, body): + self.body = body + self.length = len(body) + + def startProducing(self, consumer): + consumer.write(self.body) + return defer.succeed(None) + + def pauseProducing(self): + pass + + def stopProducing(self): + pass + +class BodyReceiver(protocol.Protocol): + def __init__(self, finished): + self.finished = finished + self.data = "" + + def dataReceived(self, bytes): + self.data += bytes + + def connectionLost(self, reason): + self.finished.callback(self.data) def capturePackets(pcap_filename): from scapy.all import sniff diff --git a/ooni/utils/otime.py b/ooni/utils/otime.py index 11f7be1..719230e 100644 --- a/ooni/utils/otime.py +++ b/ooni/utils/otime.py @@ -41,3 +41,11 @@ def utcPrettyDateNow(): def timeToPrettyDate(time_val): return time.ctime(time_val) + +def timestamp(): + cur_time = datetime.utcnow() + d_format = "%d_%B_%Y_%H-%M-%S" + pretty = cur_time.strftime(d_format) + return pretty + + diff --git a/ooniprobe.conf b/ooniprobe.conf index 1e76ad7..47f480a 100644 --- a/ooniprobe.conf +++ b/ooniprobe.conf @@ -20,7 +20,7 @@ advanced: # XXX change this to point to the directory where you have stored the GeoIP # database file. This should be the directory in which OONI is installed # /path/to/ooni-probe/data/ - geoip_data_dir: /home/x/code/networking/ooni-probe/data/ + geoip_data_dir: /usr/share/GeoIP/ debug: true threadpool_size: 10 diff --git a/to-be-ported/protocols/daphn3.py b/to-be-ported/protocols/daphn3.py new file mode 100644 index 0000000..37c94c7 --- /dev/null +++ b/to-be-ported/protocols/daphn3.py @@ -0,0 +1,311 @@ +import sys +import yaml + +from twisted.internet import protocol, defer +from twisted.internet.error import ConnectionDone + +from scapy.all import IP, Raw, rdpcap + +from ooni.utils import log +from ooni.plugoo import reports + +def read_pcap(filename): + """ + @param filename: Filesystem path to the pcap. + + Returns: + [{"sender": "client", "data": "\x17\x52\x15"}, {"sender": "server", "data": "\x17\x15\x13"}] + """ + packets = rdpcap(filename) + + checking_first_packet = True + client_ip_addr = None + server_ip_addr = None + + ssl_packets = [] + messages = [] + + """ + pcap assumptions: + + pcap only contains packets exchanged between a Tor client and a Tor server. + (This assumption makes sure that there are only two IP addresses in the + pcap file) + + The first packet of the pcap is sent from the client to the server. (This + assumption is used to get the IP address of the client.) + + All captured packets are TLS packets: that is TCP session + establishment/teardown packets should be filtered out (no SYN/SYN+ACK) + """ + + """Minimally validate the pcap and also find out what's the client + and server IP addresses.""" + for packet in packets: + if checking_first_packet: + client_ip_addr = packet[IP].src + checking_first_packet = False + else: + if packet[IP].src != client_ip_addr: + server_ip_addr = packet[IP].src + + try: + if (packet[Raw]): + ssl_packets.append(packet) + except IndexError: + pass + + """Form our list.""" + for packet in ssl_packets: + if packet[IP].src == client_ip_addr: + messages.append({"sender": "client", "data": str(packet[Raw])}) + elif packet[IP].src == server_ip_addr: + messages.append({"sender": "server", "data": str(packet[Raw])}) + else: + raise("Detected third IP address! pcap is corrupted.") + + return messages + +def read_yaml(filename): + f = open(filename) + obj = yaml.load(f) + f.close() + return obj + +class Mutator: + idx = 0 + step = 0 + + waiting = False + waiting_step = 0 + + def __init__(self, steps): + """ + @param steps: array of dicts for the steps that must be gone over by + the mutator. Looks like this: + [{"sender": "client", "data": "\xde\xad\xbe\xef"}, + {"sender": "server", "data": "\xde\xad\xbe\xef"}] + """ + self.steps = steps + + def _mutate(self, data, idx): + """ + Mutate the idx bytes by increasing it's value by one + + @param data: the data to be mutated. + + @param idx: what byte should be mutated. + """ + print "idx: %s, data: %s" % (idx, data) + ret = data[:idx] + ret += chr(ord(data[idx]) + 1) + ret += data[idx+1:] + return ret + + def state(self): + """ + Return the current mutation state. As in what bytes are being mutated. + + Returns a dict containg the packet index and the step number. + """ + print "[Mutator.state()] Giving out my internal state." + current_state = {'idx': self.idx, 'step': self.step} + return current_state + + def next(self): + """ + Increases by one the mutation state. + + ex. (* is the mutation state, i.e. the byte to be mutated) + before [___*] [____] + step1 step2 + after [____] [*___] + + Should be called every time you need to proceed onto the next mutation. + It changes the internal state of the mutator to that of the next + mutatation. + + returns True if another mutation is available. + returns False if all the possible mutations have been done. + """ + if (self.step) == len(self.steps): + # Hack to stop once we have gone through all the steps + print "[Mutator.next()] I believe I have gone over all steps" + print " Stopping!" + self.waiting = True + return False + + self.idx += 1 + current_idx = self.idx + current_step = self.step + current_data = self.steps[current_step]['data'] + + if 0: + print "current_step: %s" % current_step + print "current_idx: %s" % current_idx + print "current_data: %s" % current_data + print "steps: %s" % len(self.steps) + print "waiting_step: %s" % self.waiting_step + + data_to_receive = len(self.steps[current_step]['data']) + + if self.waiting and self.waiting_step == data_to_receive: + print "[Mutator.next()] I am no longer waiting" + log.debug("I am no longer waiting.") + self.waiting = False + self.waiting_step = 0 + self.idx = 0 + + elif self.waiting: + print "[Mutator.next()] Waiting some more." + log.debug("Waiting some more.") + self.waiting_step += 1 + + elif current_idx >= len(current_data): + print "[Mutator.next()] Entering waiting mode." + log.debug("Entering waiting mode.") + self.step += 1 + self.idx = 0 + self.waiting = True + + log.debug("current index %s" % current_idx) + log.debug("current data %s" % len(current_data)) + return True + + def get(self, step): + """ + Returns the current packet to be sent to the wire. + If no mutation is necessary it will return the plain data. + Should be called when you are interested in obtaining the data to be + sent for the selected state. + + @param step: the current step you want the mutation for + + returns the mutated packet for the specified step. + """ + if step != self.step or self.waiting: + log.debug("[Mutator.get()] I am not going to do anything :)") + return self.steps[step]['data'] + + data = self.steps[step]['data'] + #print "Mutating %s with idx %s" % (data, self.idx) + return self._mutate(data, self.idx) + +class Daphn3Protocol(protocol.Protocol): + """ + This implements the Daphn3 protocol for the server side. + It gets instanced once for every client that connects to the oonib. + For every instance of protocol there is only 1 mutation. + Once the last step is reached the connection is closed on the serverside. + """ + steps = [] + mutator = None + + current_state = None + + role = 'client' + state = 0 + total_states = len(steps) - 1 + received_data = 0 + to_receive_data = 0 + report = reports.Report('daphn3', 'daphn3.yamlooni') + + test = None + + def next_state(self): + """ + This is called once I have completed one step of the protocol and need + to proceed to the next step. + """ + if not self.mutator: + print "[Daphn3Protocol.next_state] No mutator. There is no point to stay on this earth." + self.transport.loseConnection() + return + + if self.role is self.steps[self.state]['sender']: + print "[Daphn3Protocol.next_state] I am a sender" + data = self.mutator.get(self.state) + self.transport.write(data) + self.to_receive_data = 0 + + else: + print "[Daphn3Protocol.next_state] I am a receiver" + self.to_receive_data = len(self.steps[self.state]['data']) + + self.state += 1 + self.received_data = 0 + + def dataReceived(self, data): + """ + This is called every time some data is received. I transition to the + next step once the amount of data that I expect to receive is received. + + @param data: the data that has been sent by the client. + """ + if not self.mutator: + print "I don't have a mutator. My life means nothing." + self.transport.loseConnection() + return + + if len(self.steps) == self.state: + self.transport.loseConnection() + return + + self.received_data += len(data) + if self.received_data >= self.to_receive_data: + print "Moving to next state %s" % self.state + self.next_state() + + def censorship_detected(self, report): + """ + I have detected the possible presence of censorship we need to write a + report on it. + + @param report: a dict containing the report to be written. Must contain + the keys 'reason', 'proto_state' and 'mutator_state'. + The reason is the reason for which the connection was + closed. The proto_state is the current state of the + protocol instance and mutator_state is what was being + mutated. + """ + print "The connection was closed because of %s" % report['reason'] + print "State %s, Mutator %s" % (report['proto_state'], + report['mutator_state']) + if self.test: + self.test.result['censored'] = True + self.test.result['state'] = report + self.mutator.next() + + def connectionLost(self, reason): + """ + The connection was closed. This may be because of a legittimate reason + or it may be because of a censorship event. + """ + if not self.mutator: + print "Terminated because of little interest in life." + return + report = {'reason': reason, 'proto_state': self.state, + 'trigger': None, 'mutator_state': self.current_state} + + if self.state < self.total_states: + report['trigger'] = 'did not finish state walk' + self.censorship_detected(report) + + else: + print "I have reached the end of the state machine" + print "Censorship fingerprint bruteforced!" + if self.test: + print "In the test thing" + self.test.result['censored'] = False + self.test.result['state'] = report + self.test.result['state_walk_finished'] = True + self.test.report(self.test.result) + return + + if reason.check(ConnectionDone): + print "Connection closed cleanly" + else: + report['trigger'] = 'unclean connection closure' + self.censorship_detected(report) + +

1 0

[ooni-probe/master] Completely rewrite the txscapy.
by art＠torproject.org 09 Nov '12

09 Nov '12

commit 12726ca463d9e68e93d49fcb418421648d054744 Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Nov 9 22:19:44 2012 +0100 Completely rewrite the txscapy. * It is now much cleaner and does not start the packet capture in a separate thread. It subclasses the twisted filedescriptor protocol and returns a deferred that will fire the callback with the packets it received and the ones it considers answers to the request. --- nettests/bridge_reachability/echo.py | 2 +- nettests/core/chinatrigger.py | 9 ++- ooni/templates/scapyt.py | 105 +++------------------------ ooni/utils/txscapy.py | 133 ++++++++++++++++++++++++++++++---- 4 files changed, 137 insertions(+), 112 deletions(-) diff --git a/nettests/bridge_reachability/echo.py b/nettests/bridge_reachability/echo.py index 542e017..5060ffd 100644 --- a/nettests/bridge_reachability/echo.py +++ b/nettests/bridge_reachability/echo.py @@ -164,5 +164,5 @@ class EchoTest(BaseScapyTest): raise IfaceError("Could not find a working network interface.") def test_icmp(self): - self.sr(IP(dst=self.input)/ICMP()) + return self.sr(IP(dst=self.input)/ICMP()) diff --git a/nettests/core/chinatrigger.py b/nettests/core/chinatrigger.py index 53fadb9..de1f64d 100644 --- a/nettests/core/chinatrigger.py +++ b/nettests/core/chinatrigger.py @@ -23,6 +23,7 @@ class ChinaTriggerTest(BaseScapyTest): name = "chinatrigger" usageOptions = UsageOptions requiredOptions = ['dst', 'port'] + timeout = 2 def setUp(self): self.dst = self.localOptions['dst'] @@ -47,7 +48,7 @@ class ChinaTriggerTest(BaseScapyTest): def set_random_field(pkt): ret = pkt[:15] for i in range(28): - ret += chr(random.randint(0, 256)) + ret += chr(random.randint(0, 255)) ret += pkt[15+28:] return ret @@ -57,9 +58,9 @@ class ChinaTriggerTest(BaseScapyTest): Slightly changed mutate function. """ ret = pkt[:idx-1] - mutation = chr(random.randint(0, 256)) + mutation = chr(random.randint(0, 255)) while mutation == pkt[idx]: - mutation = chr(random.randint(0, 256)) + mutation = chr(random.randint(0, 255)) ret += mutation ret += pkt[idx:] return ret @@ -103,5 +104,5 @@ class ChinaTriggerTest(BaseScapyTest): for x in range(len(pkt)): mutation = IP(dst=self.dst)/TCP(dport=self.port)/ChinaTriggerTest.mutate(pkt, x) pkts.append(mutation) - self.send(pkts) + return self.sr(pkts, timeout=2) diff --git a/ooni/templates/scapyt.py b/ooni/templates/scapyt.py index a1d2969..4c18f0a 100644 --- a/ooni/templates/scapyt.py +++ b/ooni/templates/scapyt.py @@ -14,7 +14,7 @@ from scapy.all import send, sr, IP, TCP from ooni.nettest import NetTestCase from ooni.utils import log -from ooni.lib.txscapy import TXScapy +from ooni.utils.txscapy import ScapyProtocol def createPacketReport(packet_list): """ @@ -42,106 +42,25 @@ class BaseScapyTest(NetTestCase): requiresRoot = True - sentPackets = [] - answeredPackets = [] - - def sr(self, pkts, *arg, **kw): + def sr(self, packets, *arg, **kw): """ Wrapper around scapy.sendrecv.sr for sending and receiving of packets at layer 3. """ - answered_packets, unanswered = sr(pkts, *arg, **kw) - self.report['answered_packets'] = createPacketReport(answered_packets) - self.report['sent_packets'] = createPacketReport(pkts) - return (answered_packets, sent_packets) + def finished(result): + answered, unanswered = result + sent_packets, received_packets = answered + self.report['answered_packets'] = createPacketReport(received_packets) + self.report['sent_packets'] = createPacketReport(sent_packets) + + scapyProtocol = ScapyProtocol(*arg, **kw) + d = scapyProtocol.startSending(packets) + return d def send(self, pkts, *arg, **kw): """ Wrapper around scapy.sendrecv.send for sending of packets at layer 3 """ - sent_packets = send(pkts, *arg, **kw) - self.report['sent_packets'] = createPacketReport(pkts) - return sent_packets - -class TXScapyTest(BaseScapyTest): - """ - A utility class for writing scapy driven OONI tests. - - * pcapfile: specify where to store the logged pcapfile - - * timeout: timeout in ms of when we should stop waiting to receive packets - - * receive: if we should also receive packets and not just send - - XXX This is currently not working - """ - name = "TX Scapy Test" - version = 0.1 - - receive = True - timeout = 1 - pcapfile = 'packet_capture.pcap' - packet = IP()/TCP() - reactor = None - - answered = None - unanswered = None - - def processInputs(self): - """ - Place here the logic for validating and processing of inputs and - command line arguments. - """ - pass - - def tearDown(self): - log.debug("Tearing down reactor") - - def finished(self, *arg): - log.debug("Calling final close") - - self.questions = self.txscapy.questions - self.answers = self.txscapy.answers + raise Exception("Not implemented") - log.debug("These are the questions: %s" % self.questions) - log.debug("These are the answers: %s" % self.answers) - - self.txscapy.finalClose() - - def sendReceivePackets(self): - packets = self.buildPackets() - - log.debug("Sending and receiving %s" % packets) - - self.txscapy = TXScapy(packets, pcapfile=self.pcapfile, - timeout=self.timeout, reactor=self.reactor) - - self.txscapy.sr(packets, pcapfile=self.pcapfile, - timeout=self.timeout, reactor=self.reactor) - - d = self.txscapy.deferred - d.addCallback(self.finished) - - return d - - def sendPackets(self): - log.debug("Sending and receiving of packets %s" % packets) - - packets = self.buildPackets() - - self.txscapy = TXScapy(packets, pcapfile=self.pcapfile, - timeout=self.timeout, reactor=self.reactor) - - self.txscapy.send(packets, reactor=self.reactor).deferred - - d = self.txscapy.deferred - d.addCallback(self.finished) - - return d - - def buildPackets(self): - """ - Override this method to build scapy packets. - """ - pass diff --git a/ooni/utils/txscapy.py b/ooni/utils/txscapy.py index a3a5610..2559d19 100644 --- a/ooni/utils/txscapy.py +++ b/ooni/utils/txscapy.py @@ -12,29 +12,134 @@ import os import sys import time -from twisted.internet import protocol, base, fdesc, error, defer -from twisted.internet import reactor, threads +from twisted.internet import protocol, base, fdesc +from twisted.internet import reactor, threads, error +from twisted.internet import defer, abstract from zope.interface import implements -from scapy.all import Gen -from scapy.all import SetGen - -from ooni.utils import log from scapy.all import PcapWriter, MTU from scapy.all import BasePacketList, conf, PcapReader +from scapy.all import conf, Gen, SetGen + +from ooni.utils import log + class TXPcapWriter(PcapWriter): def __init__(self, *arg, **kw): PcapWriter.__init__(self, *arg, **kw) fdesc.setNonBlocking(self.f) -def txSniff(count=0, store=1, offline=None, - prn = None, lfilter=None, - L2socket=None, timeout=None, - opened_socket=None, stop_filter=None, - *arg, **karg): - """ - XXX we probably want to rewrite the scapy sniff function to better suite our needs. - """ +class ScapyProtocol(abstract.FileDescriptor): + def __init__(self, super_socket=None, + reactor=None, timeout=None, receive=True): + abstract.FileDescriptor.__init__(self, reactor) + # By default we use the conf.L3socket + if not super_socket: + super_socket = conf.L3socket() + self.super_socket = super_socket + + self.timeout = timeout + + # This dict is used to store the unique hashes that allow scapy to + # match up request with answer + self.hr_sent_packets = {} + + # These are the packets we have received as answer to the ones we sent + self.answered_packets = [] + + # These are the packets we send + self.sent_packets = [] + + # This deferred will fire when we have finished sending a receiving packets. + self.d = defer.Deferred() + self.debug = False + self.multi = False + # XXX this needs to be implemented. It would involve keeping track of + # the state of the sending via the super socket file descriptor and + # firing the callback when we have concluded sending. Check out + # twisted.internet.udp to see how this is done. + self.receive = receive + + def fileno(self): + return self.super_socket.ins.fileno() + + def processPacket(self, packet): + """ + Hook useful for processing packets as they come in. + """ + + def processAnswer(self, packet, answer_hr): + log.debug("Got an answer processing it") + for i in range(len(answer_hr)): + if packet.answers(answer_hr[i]): + self.answered_packets.append((answer_hr[i], packet)) + if self.debug: + print packet.src, packet.ttl + #answer.show() + + if not self.multi: + del(answer_hr[i]) + break + if len(self.answered_packets) == len(self.sent_packets): + # All of our questions have been answered. + self.stopSending() + + def doRead(self): + timeout = time.time() - self._start_time + log.debug("Checking for timeout %s > %s" % (timeout, self.timeout)) + if self.timeout and time.time() - self._start_time > self.timeout: + self.stopSending() + packet = self.super_socket.recv() + if packet: + self.processPacket(packet) + # A string that has the same value for the request than for the + # response. + hr = packet.hashret() + if hr in self.hr_sent_packets: + answer_hr = self.hr_sent_packets[hr] + self.processAnswer(packet, answer_hr) + + def stopSending(self): + self.stopReading() + self.super_socket.close() + if hasattr(self, "d"): + result = (self.answered_packets, self.sent_packets) + self.d.callback(result) + del self.d + + def write(self, packet): + """ + Write a scapy packet to the wire + """ + hashret = packet.hashret() + if hashret in self.hr_sent_packets: + self.hr_sent_packets[hashret].append(packet) + else: + self.hr_sent_packets[hashret] = [packet] + self.sent_packets.append(packet) + return self.super_socket.send(packet) + + def sendPackets(self, packets): + if not isinstance(packets, Gen): + packets = SetGen(packets) + for packet in packets: + self.write(packet) + + def startSending(self, packets): + self._start_time = time.time() + self.startReading() + self.sendPackets(packets) + return self.d + +def sr(x, filter=None, iface=None, nofilter=0, timeout=None): + super_socket = conf.L3socket(filter=filter, iface=iface, nofilter=nofilter) + sp = ScapyProtocol(super_socket=super_socket, timeout=timeout) + return sp.startSending(x) + +def send(x, filter=None, iface=None, nofilter=0, timeout=None): + super_socket = conf.L3socket(filter=filter, iface=iface, nofilter=nofilter) + sp = ScapyProtocol(super_socket=super_socket, timeout=timeout) + return sp.startSending(x) +

1 0

[ooni-probe/master] Merge branch 'cleanup'
by art＠torproject.org 09 Nov '12

09 Nov '12

commit 6e30857efa9223f3852d9e0d42bd02f14ee1b76e Merge: 10c63e0 7d6901f Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Nov 9 17:40:25 2012 +0100 Merge branch 'cleanup' * cleanup: Do a very big cleanup and refactor of all the code in the repo. Conflicts: to-be-ported/protocols/daphn3.py to-be-ported/protocols/daphn3.py | 311 ++++++++++++++++++++++++++++++++++++++ 1 files changed, 311 insertions(+), 0 deletions(-)

1 0

r25867: {website} you don't need firefox installed before trying tbb (website/trunk/docs/en)
by Roger Dingledine 09 Nov '12

09 Nov '12

Author: arma Date: 2012-11-09 20:49:30 +0000 (Fri, 09 Nov 2012) New Revision: 25867 Modified: website/trunk/docs/en/documentation.wml Log: you don't need firefox installed before trying tbb Modified: website/trunk/docs/en/documentation.wml =================================================================== --- website/trunk/docs/en/documentation.wml 2012-11-09 20:28:18 UTC (rev 25866) +++ website/trunk/docs/en/documentation.wml 2012-11-09 20:49:30 UTC (rev 25867) @@ -38,8 +38,8 @@ </li> <li> - <a href="<page download/download>">Install the Tor bundle</a> and try it out. - Make sure you've got Firefox installed first, and be sure to read the + <a href="<page download/download>">Install the Tor Browser Bundle</a> and try it out. + Be sure to read the <a href="<page download/download>#Warning">list of warnings</a> about ways you can screw up your anonymity. Look through the <a href="https://www.torproject.org/projects/torbrowser/design/">Tor

1 0

r25866: {website} make the obfsproxy bridge debian instructions more likely to (website/trunk/projects/en)
by Roger Dingledine 09 Nov '12

09 Nov '12

Author: arma Date: 2012-11-09 20:28:18 +0000 (Fri, 09 Nov 2012) New Revision: 25866 Modified: website/trunk/projects/en/obfsproxy-debian-instructions.wml Log: make the obfsproxy bridge debian instructions more likely to work Modified: website/trunk/projects/en/obfsproxy-debian-instructions.wml =================================================================== --- website/trunk/projects/en/obfsproxy-debian-instructions.wml 2012-11-06 22:40:33 UTC (rev 25865) +++ website/trunk/projects/en/obfsproxy-debian-instructions.wml 2012-11-09 20:28:18 UTC (rev 25866) @@ -2,7 +2,7 @@ # Revision: $Revision$ # Translation-Priority: 4-optional -#include "head.wmi" TITLE="obfsproxy: Installation instructions" CHARSET="UTF-8" +#include "head.wmi" TITLE="obfsproxy: Setting up an Obfsproxy Bridge on Debian/Ubuntu" CHARSET="UTF-8" <div id="content" class="clearfix"> <div id="breadcrumbs"> @@ -14,65 +14,55 @@  - <h1 id="instructions">Obfsproxy Bridge Instructions on Debian/Ubuntu</h1> + <h1 id="instructions">Setting up an Obfsproxy Bridge on Debian/Ubuntu</h1> <img src="$(IMGROOT)/obfsproxy_diagram.png" alt="obfsproxy diagram"></a> <p> - This guide will help you setup an obfuscated bridge on a Debian/Ubuntu system. + This guide will help you set up an obfuscated bridge on a Debian/Ubuntu system. </p> - <h3>Step 0: Add Tor repositories to APT</h3> + <h3>Step 0: Move to the development version of Tor</h3> <br> <p> - You need - to <a href="https://www.torproject.org/docs/debian#development">install - the experimental official Tor Project APT repositories</a>, - because a fresh version of Tor (0.2.4.x) is required (Older - versions of Tor don't report their bridge addresses to BridgeDB). + Add the <a href="<page docs/debian>#development">development Tor + APT repository</a> and run the specified commands to install tor + and deb.torproject.org-keyring. You need Tor 0.2.4.x Tor because + it knows how to automatically report your obfsproxy address to <a + href="https://bridges.torproject.org/?transport=obfs2">BridgeDB</a>. </p> - <h3>Step 1: Install Tor and obfsproxy</h3> + <h3>Step 1: Install obfsproxy</h3> <br> - <p> - Now install tor and obfsproxy: - </p> - <pre style="margin: 1.5em 0 1.5em 2em"> -\# apt-get update -\# apt-get install obfsproxy tor +\# apt-get install obfsproxy </pre> <p> - Note that obfsproxy requires - libevent2 and your distribution (e.g. Debian stable) might not - have it in its repos. You can - <a href="https://trac.torproject.org/projects/tor/ticket/5009#comment:9">try - our experimental backport libevent2 debs</a>, - or <a href="https://trac.torproject.org/projects/tor/ticket/5009#comment:17">build - libevent2 from source</a>. + Obfsproxy requires libevent2. If your distribution (e.g. Debian + squeeze) doesn't include it, you can get it from the <a + href="http://packages.debian.org/search?keywords=libevent-2.0-5">backports</a> + repository. </p> - <h3>Step 2: Set up Tor</h3> + <h3>Step 2: Configure Tor</h3> <br> <p> - You will need an appropriate - Tor <a href="<page docs/faq>#torrc">configuration file</a> - (usually at <i>/etc/tor/torrc</i>): + Edit your <i>/etc/tor/torrc</i> to add: </p> <pre style="margin: 1.5em 0 1.5em 2em"> SocksPort 0 -ORPort auto +ORPort 443 # or some other port if you already run a webserver/skype BridgeRelay 1 Exitpolicy reject *:* -\## CHANGEME_1 -> provide a nickname for your bridge, can be anything you like. +\## CHANGEME_1 -> provide a nickname for your bridge, can be anything you like Nickname CHANGEME_1 -\## CHANGEME_2 -> If you want others to be able to contact you uncomment this line and put your GPG fingerprint for example. +\## CHANGEME_2 -> provide some email address so we can contact you if there's a problem \#ContactInfo CHANGEME_2 ServerTransportPlugin obfs2 exec /usr/bin/obfsproxy --managed @@ -82,11 +72,12 @@ Don't forget to edit the <i>CHANGEME</i> fields! </p> - <h3>Step 3: Launch Tor and verify that it works</h3> + <h3>Step 3: Launch Tor and verify that it bootstraps</h3> <br> <p> - Restart Tor for the the new configuration file to be in effect: + Restart Tor to use the new configuration file. + (Preface with sudo if needed.) </p> <pre style="margin: 1.5em 0 1.5em 2em"> @@ -112,10 +103,16 @@ 100%. </p> + <h3>Step 4: Set up port forwarding if needed</h3> + <br> + <p> - Now you need to find the address on which obfsproxy is - listening. To do this, check your Tor logs for a line similar to - this one: + If you're behind a NAT/firewall, you'll need to make your bridge + reachable from the outside world — both on the ORPort and + the obfsproxy port. The ORPort is whatever you defined in step two + above. To find your obfsproxy port, check your Tor logs for a line + similar to this one: + </p> <pre style="margin: 1.5em 0 1.5em 2em"> Oct 05 20:00:41.000 [notice] Registered server transport 'obfs2' at '0.0.0.0:26821 @@ -123,19 +120,11 @@ <p> The last number, in this case <i>26821</i>, is the TCP port number - that your clients should point their obfsproxy to. So for example, - if your public IP is 1.2.3.4, your clients should put <i>Bridge - obfs2 1.2.3.4:26821</i> in their configuration file. - </pre> + that you need to forward through your firewall. (This port is randomly + chosen the first time Tor starts, but Tor will cache and reuse the + same number in future runs.) </p> - <p> - <img width="7%" height="7%" style="float: left;" src="$(IMGROOT)/icon-Obfsproxy.jpg"> - <b>Don't forget!</b> If you are behind a NAT, you should <b>port - forward</b> the port that obfsproxy is listening on. In the - example above you would have to forward port <i>26821</i>. - </p> - </div>  <div id = "sidecol">

1 0

[tor-design-2012/master] Add local copy of the "top changes in tor" blogposts so we can work
by nickm＠torproject.org 09 Nov '12

09 Nov '12

commit 0aa14b70a3a94ff06c1a0de53e55f2618c00d57b Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Nov 9 11:03:52 2012 -0500 Add local copy of the "top changes in tor" blogposts so we can work offline when integrating their content. --- blog/blogpost-1.txt | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ blog/blogpost-2.txt | 34 ++++++++++++++++++++++++++++++++++ blog/blogpost-3.txt | 23 +++++++++++++++++++++++ 3 files changed, 106 insertions(+), 0 deletions(-) diff --git a/blog/blogpost-1.txt b/blog/blogpost-1.txt new file mode 100644 index 0000000..222c4a8 --- /dev/null +++ b/blog/blogpost-1.txt @@ -0,0 +1,49 @@ +The main academic reference for Tor is "Tor: The Second-Generation Onion Router" by Dingledine, Mathewson, and Syverson. But that paper was published back in 2004, and Tor has evolved since then. So Steven Murdoch and Nick Mathewson are currently preparing an updated version of the Tor design paper, to include new design changes and research results concerning Tor over the last 8 years. +In this series of posts, we (Steven and Nick) will try to summarize the most interesting or significant changes to Tor's design since the publication of the original paper. We're only going to cover the stuff we think is most interesting, and we'll aim to do so in an interesting way. +We think this will be a three part series. In this first part, we'll cover the evolution of Tor's directory system, and some performance improvements in Tor's circuit creation and cell scheduling logic. +1. Node discovery and the directory protocol +Since the earliest versions of Tor, we knew that node discovery would require a better implementation than we had. There are a few key issues that any anonymity network's node discovery system needs to solve. +Every client needs to be selecting nodes from the same probability distribution, or an adversary could be able to exploit differences in client knowledge. In the worst case, if an adversary can cause one client to know about an exit node that no other client uses, the adversary can know that all traffic leaving that exit is coming from the targeted client. But even in more mild cases, where (say) every client knows every node with P=90%, an adversary can use this information to attack users. +While there has been some work on quantifying these so-called "epistemic attacks," we're proceeding with the conservative assumption that clients using separate sets of nodes are likely to be partitionable from one other, and so the set of nodes used by all clients needs to be uniform. +The earliest versions of Tor solved this problem with a "Directory" object – each server generated a signed "router descriptor", and uploaded it to one of a small set (three) of "directory authorities". Each of these authorities generated a signed concatenated list of router descriptors, and served that list to clients over HTTP. +This system had several notable problems: +Clients needed to download the same descriptors over and over, whether they needed them or not. +Each client would believe whichever directory authority it had spoken to most recently: rather than providing distributed trust, each authority was fully trusted in its own right, and any one misbehaving authority could completely compromise all the clients that talked to it. +To the extent that directory authorities disagreed, they created partitions in client knowledge, which could in the worst case allow an adversary to partition clients based on which authority's directory each client had downloaded most recently. +The load on the authorities promised to grow excessive, as every client contacted every authority. +The contents of the directory were sent unencrypted, which made them trivially easy to fingerprint on the wire. +Early changes in the Version 1 Directory System +Our earliest changes focused on improving scalability rather than improving the trust model. We began by having each authority publish two documents, rather than one: a directory that contained all the router descriptors, and a "network status" document that was a list of which descriptors were up and which were down (Tor 0.0.8pre1, in Jul 2004). Clients could download the latter more frequently to avoid using down servers, and refresh the former only periodically. +We also added a caching feature in Tor 0.0.8pre1, where nodes could act as directory caches. With this feature, once a client had a directory, it no longer needed to contact the directory authorities every time it wanted to update it, but rather could contact a cache instead. +The Version 2 Directory System (deprecated) +In Tor 0.1.1.8-alpha (Oct 2005), we took our first shot at solving the trust model. Now, we had each authority sign a more complete network status statement, including a list of all the nodes that it believed should be in the network, a digest of each node's public key, and a digest of each node's router descriptor. Clients would download one of these documents signed by each authority, and then compute, based on all the documents they had, which collection of nodes represented the consensus view of all the authorities. +To get router descriptors, clients would then contact one or more caches, and ask for the descriptors they believed represented the consensus view of all the authorities. +This approach meant that a single rogue directory authority could no longer completely control any client's view of the network. Clients became more fragmented, however, since instead of falling into one of N possible groups based on which authority they contacted most recently, they fell into one of MN groups where N was the number of authorities and M was the number of recently valid opinions from each authority. +Around this time we also had authorities begin assigning flags to nodes, so that in addition to recording "up" or "down" for each node, authorities could also declare whether nodes were fast, stable, likely to be good guard nodes, and so forth. +All of the improvements so far were oriented toward saving bandwidth at the server side: we figured that clients had plenty of bandwidth, and we wanted to avoid overloading the authorities and caches. But if we wanted to add more directory authorities (a majority of 5 is still an uncomfortably small number), bootstrapping clients would have to fetch one more network status for every new authority. By early 2008, each status document listed 2500 relay summaries and came in around 175KB compressed, meaning you needed 875KB of status docs when starting up, and then another megabyte of descriptors after that. And we couldn't add more authorities without making the problem even worse. +Version 3: Consensus and Directory Voting +To solve the problems with the v2 directory protocol, Tor 0.2.0.3-alpha (Jul 2007) introduced a directory voting system, where the authorities themselves would exchange vote documents periodically (currently once per hour), compute a consensus document based on everyone's votes, and all sign the consensus. +Now clients only need to download a single signed consensus document periodically, and check that it is signed by a sufficiently large fraction of the authorities that the client knows about. This gives clients a uniform view of the network, makes it harder still for a small group of corrupt authorities to attack a client, and limits the number of documents they need to download. +The voting algorithm is ad hoc, and is by no means the state of the art in byzantine fault tolerance. Our approach to failures to reach a consensus (which have been mercifully infrequent) is to find what's wrong and fix it manually. +Saving bytes with microdescriptors +Now that the consensus algorithm finally matched what we had in mind when we wrote the first Tor paper, it was time to address the protocol's verbosity. +Out of all the data in a typical 1500-byte server descriptor, a client really only needs to know what ports it supports exiting to, its current RSA1024 onion key, and other information that is fully redundant with its listing in the consensus network status document. +One crucial observation is that signatures on the router descriptors themselves don't actually help clients: if there were enough hostile authorities to successfully convince the clients to use a descriptor that a router didn't actually sign, they could as easily convince the clients to use a descriptor signed with a phony identity key. +This observation let us move (in Tor 0.2.3.1-alpha, May 2011) to a design where the authorities, as part of their voting process, create an abbreviated version of each descriptor they recommend. Currently, these contain only a short summary of the router's exit policy, and the router's current onion key. Clients now download these abbreviated "microdescriptors", which cuts the information downloaded each node by about 75%. Further, because the data here change relatively infrequently, it cuts down the frequency with which clients fetch new information about each router at all. +Tunneling directory connections over Tor +In 0.1.2.5-alpha (Jan 2007), we added support by default for clients to download all directory documents over HTTP over Tor, rather than by contacting directories and caches over unencrypted HTTP. This change helps clients resist fingerprinting. +Because clients aren't using Tor for anonymity on directory connections, they build single-hop circuits. We use HTTP over a one hop Tor circuit, rather than plain old HTTPS, so that clients can use the same TLS connection both for a directory fetch and for other Tor traffic. +2. Security improvements for hidden services +Decentralized hidden-service directory system +A partly-centralized directory infrastructure makes sense for Tor nodes, since every client is supposed to be able to know about every node, but it doesn't make a great deal of sense for hidden services. +To become more censorship-resistant, we started (in Tor 0.2.0.10-alpha, Nov 2007) to instead use the Tor network itself to cache and serve hidden service descriptors. Now, instead of publishing their hidden service descriptors anonymously to a small fixed set of hidden service authorities, hidden services publish to a set of nodes whose identity keys are closest to a hash of the service's identity, the current date, and a replica number. +Improved authorization model for hidden services +We also added improved support for authentication to hidden services. Optionally, to use a hidden service, a client must know a shared key, and use this key to decrypt the part of a hidden service descriptor containing the introduction points. It later must use information in that encrypted part to authenticate to any introduction point it uses, and later to the hidden service itself. One of the main uses of authentication here is to hide presence -- only authenticated users can learn whether the hidden service is online. +3. Faster first-hop circuit establishment with CREATE_FAST +At each stage of extending a circuit to the next hop, the client carries out a Diffie-Hellman (DH) key agreement protocol with that next hop. This step provides confidentiality (and forward secrecy) of the relay cell payload as it is passed on by intermediate hops. Originally Tor also carried out DH with the first hop, even though there was already a DH exchange as part of the TLS handshake. DH is quite computationally expensive for both ends, so Tor 0.1.1.10-alpha (Dec 2005) onwards skipped the DH exchange on the first hop by sending a CREATE_FAST (as opposed to a standard CREATE) cell, which generates key material simply by hashing random numbers sent by the client and server. +4. Cell queueing and scheduling +The original Tor design left the fine-grained handling of incoming cells unspecified: every circuit's cells were to be decrypted and delivered in order, but nodes were free to choose which circuits to handle in any order they pleased. +Early versions of Tor also punted on the question: they handled cells in the order they were received on incoming OR connections, encrypting/decrypting them and handing them off immediately to the next node on the circuit, or to the appropriate exit or entry connection. This approach, however, frequently created huge output buffers where quiet circuits couldn't get a cell in edgewise. +Instead, Tor currently places incoming cells on a per-circuit queue associated with each circuit. Rather than filling all output buffers to capacity, Tor instead fills them up with cells on a near just-in-time basis. +When we first implemented these cell queues in 0.2.0.1-alpha (Jun 2007), we chose which cells to deliver by rotating the circuits in a round-robin approach. In Tor 0.2.2.7-alpha (Jan 2010), we began to instead favor the circuits on each connection that had been quiet recently, so that a circuit with small, infrequent amounts of cells will get better latency than a circuit being used for a bulk transfer. (Specifically, when we are about to put a cell on an outgoing connection, we choose the circuit which has sent the lowest total exponentially-decaying number of cells so far. Currently, each cell has a 30-second half-life.) +In Part 2 we will look at changes to how Tor selects paths and the new anti-censorship measures. diff --git a/blog/blogpost-2.txt b/blog/blogpost-2.txt new file mode 100644 index 0000000..eb9c416 --- /dev/null +++ b/blog/blogpost-2.txt @@ -0,0 +1,34 @@ +This is part 2 of Nick Mathewson and Steven Murdoch's series on what has changed in Tor's design since the original design paper in 2004. Part one is back over here. +In this installment, we cover changes in how we pick and use nodes in our circuits, and general anticensorship features. +5. Guard nodes +We assume, based on a fairly large body of research, that if an attacker controls or monitors the first hop and last hop of a circuit, then the attacker can de-anonymize the user by correlating timing and volume information. Many of the security improvements to path selection discussed in this post concentrate on reducing the probability that an attacker can be in this position, but no reasonably efficient proposal can eliminate the possibility. +Therefore, each time a user creates a circuit, there is a small chance that the circuit will be compromised. However, most users create a large number of Tor circuits, so with the original path selection algorithm, these small chances would build up into a potentially large chance that at least one of their circuits will be compromised. +To help improve this situation, in Tor 0.1.1.2-alpha, the guard node feature was implemented (initially called "helper nodes", invented by Wright, Adler, Levine, and Shields and proposed for use in Tor by Øverlier and Syverson). In Tor 0.1.1.11-alpha it was enabled by default. Now, the Tor client picks a few Tor nodes as its "guards", and uses one of them as the first hop for all circuits (as long as those nodes remain operational). +This doesn't affect the probability that the first circuit is compromised, but it does mean that if the guard nodes chosen by a user are not attacker-controlled all their future circuits will be safe. On the other hand, users who choose attacker-controlled guards will have about M/N of their circuits compromised, where M is the amount of attacker-controlled network resource and N is the total network resource. Without guard nodes every circuit has a (M/N)2 probability of being compromised. +Essentially, the guard node approach recognises that some circuits are going to be compromised, but it's better to increase your probability of having no compromised circuits at the expense of also increasing the proportion of your circuits that will be compromised if any of them are. This is because compromising a fraction of a user's circuits—sometimes even just one—can be enough to compromise a user's anonymity. For users who have good guard nodes, the situation is much better, and for users with bad guard nodes the situation is not much worse than before. +6. Bridges, censorship resistance, and pluggable transports +While Tor was originally designed as an anonymous communication system, more and more users need to circumvent censorship rather than to just preserve their privacy. The two goals are closely linked – to prevent a censor from blocking access to certain websites, it is necessary to hide where a user is connecting to. Also, many censored Internet users live in repressive regimes which might punish people who access banned websites, so here anonymity is also of critical importance. +However, anonymity is not enough. Censors can't block access to certain websites browsed over Tor, but it was easy for censors to block access to the whole of the Tor network in the original design. This is because there were a handful of directory authorities which users needed to connect to before they could discover the addresses of Tor nodes, and indeed some censors blocked the directory authorities. Even if users could discover the current list of Tor nodes, censors also blocked the IP addresses of all Tor nodes too. +Therefore, the Tor censorship resistance design introduced bridges – special Tor nodes which were not published in the directory, and could be used as entry points to the network (both for downloading the directory and also for building circuits). Users need to find out about these somehow, so the bridge authority collects the IP addresses and gives them out via email, on the web, and via personal contacts, so as to make it difficult for the censor to enumerate them all. +That's not enough to provide censorship resistance though. Preventing the censor from knowing all the IP addresses they need to block to block access to the Tor network will be enough to defeat some censors. But others have the capability to block not only by IP address but also by content (deep packet inspection). Some censors have tried to do this already and Tor has, in response, gradually changed its TLS handshake to better imitate web browsers. +Impersonating web browsers is difficult, and even if Tor perfectly impersonated one, some censors could just block encrypted web browsing (like Iran did, for some time). So it would be better if Tor could impersonate multiple protocols. Even better would be if other people could contribute to this goal, rather than the Tor developers being a bottleneck. This is the motivation of the pluggable transports design which allows Tor to manage an external program which transforms Tor traffic into some hard-to-fingerprint obfuscation. +7. Changes and complexities in our path selection algorithms +The original Tor paper never specified how clients should pick which nodes to use when constructing a circuit through the network. This question has proven unexpectedly complex. +Weighting node selection by bandwidth +The simplest possible approach to path construction, which we used in the earliest versions of Tor, is simply to pick uniformly at random from all advertised nodes that could be used for a given position in the path. But this approach creates terrible bandwidth bottlenecks: a server that would allow 10x as many bytes per second as another would still get the same number of circuits constructed through it. +Therefore, Tor 0.0.8rc1 started to have clients weight their choice of nodes by servers' advertised bandwidths, so that a server with 10x as much bandwidth would get 10x as many circuits, and therefore (probabilistically) 10x as much of the traffic. +(In the original paper, we imagined that we might take Morphmix's approach, and divide nodes into "bandwidth classes", such that clients would choose only from among nodes having at least the same approximate bandwidth as the clients. This may be a good design for peer-to-peer anonymity networks, but it doesn't seem to work for the Tor network: the most useful high-capacity nodes have more capacity than nearly any typical client.) +Later, it proved that weighting by bandwidth was also suboptimal, because of nonuniformity in path selection rules. Consider that if node A is suitable for use at any point in a circuit, but node B is suitable only as the middle node, then node A will be considered for use three times as often as B. If the two nodes have equal bandwidth, node A will be chosen three times as often, leading to it being overloaded in comparison with B. So eventually, in Tor 0.2.2.10-alpha, we moved to a more sophisticated approach, where nodes are chosen proportionally to their bandwidth, as weighted by an algorithm to optimize load-balancing between nodes of different capabilities. +Bandwidth authorities +Of course, once you choose nodes with unequal probability, you open the possibility of an attacker trying to see a disproportionate number of circuits -- not by running an extra-high number of nodes -- but by claiming to have a very large bandwidth. +For a while, we tried to limit the impact of this attack by limiting the maximum bandwidth that a client would believe, so that a single rogue node couldn't just claim to have infinite bandwidth. +In 0.2.1.17-rc, clients switched from using bandwidth values advertised by nodes themselves to using values published in the network status consensus document. A subset of the authorities measure and vote on nodes' observed bandwidth, to prevent misbehaving nodes from claiming (intentionally or accidentally) to have too much capacity. +Avoiding duplicate families in a single circuit +As mentioned above, if the first and last node in a circuit are controlled by an adversary, they can use traffic correlation attacks to notice that the traffic entering the network at the first hop matches traffic leaving the circuit at the last hop, and thereby trace a client's activity with high probability. Research on preventing this attack has not yet come up with any affordable, effective defense suitable for use in a low-latency anonymity network. Therefore, the most promising mitigation strategies seem to involve lowering the attacker's chances of controlling both ends of a circuit. +To this end, clients do not use any two nodes in a circuit whose IP addresses are in the same /16 – when we designed the network, it was marginally more difficult to acquire a large number of disparate addresses than it was to get a large number of concentrated addresses. (Roger and Nick may have been influenced by their undergraduacy at MIT, where their dormitory occupied the entirety of 18.244.0.0/16.) This approach is imperfect, but possibly better than nothing. +To allow honest node operators to run more than one server without inadvertently giving themselves the chance to see more traffic than they should, we also allow nodes to declare themselves to be members of the same "family", such that a client won't use two nodes in the same family in the same circuit. (Clients only believe mutual family declarations, so that an adversary can't capture routes by having his nodes claim unilaterally to be in a family with every node the adversary doesn't control.) +8. Stream isolation +Building a circuit is fairly expensive (in terms of computation and bandwidth) for the network, and the setup takes time, so the Tor client tries to re-use existing circuits if possible, by sending multiple TCP streams down them. Streams which share a circuit are linkable, because the exit node can tell that they have the same circuit ID. If the user sends some information on one stream which gives their identity away, the other streams on the same circuit will be de-anonymized. +To reduce the risk of this occurring, Tor will not re-use a circuit which the client first used more than 10 minutes ago. Users can also use their Tor controller to send the "NEWNYM" signal, preventing any old circuits being used for new streams. As long as users don't mix anonymous and non-anoymous tasks at the same time, this form of circuit re-use is probably a good tradeoff. +However, Manils et al. discovered that some Tor users simultaneously ran BitTorrent over the same Tor client as they did web browsing. Running BitTorrent over Tor is a bad idea because the network can't handle the load, and because BitTorrent packets include the user's real IP address in the payload, so it isn't anonymous. But running BitTorrent while doing anonymous web browsing is an especially bad idea. An exit node can find the user's IP address in the BitTorrent payload then trivially de-anonymize all streams sharing the circuit. +Running BitTorrent over Tor is still strongly discouraged, but this paper did illustrate some potential problems with circuit reuse so proposal 171 was written, and implemented in Tor 0.2.3.3-alpha, to help isolate streams which shouldn't share the same circuit. By default streams which were initiated by different clients, which came from SOCKS connections with different authentication credentials, or which came to a different SOCKS port on the Tor client, are separated. In this way, a user can isolate applications by either setting up multiple SOCKS ports on Tor and using one per application, or by setting up a single SOCKS port but using different username/passwords for each application. Tor can also be configured to isolate streams based on destination address and/or port. diff --git a/blog/blogpost-3.txt b/blog/blogpost-3.txt new file mode 100644 index 0000000..b76a840 --- /dev/null +++ b/blog/blogpost-3.txt @@ -0,0 +1,23 @@ +In this third and final installment of Nick Mathewson and Steven Murdoch's blog series (previously part 1 and part 2) we discuss how Tor has made its traffic harder to fingerprint, as well as usability and security improvements to how users interact with Tor. +9. Link protocol TLS, renegotiation +Tor's original (version 1) TLS handshake was fairly straightforward. The client said that it supported a sensible set of cryptographic algorithms and parameters (ciphersuites, in TLS terminology) and the server selected one. If one side wanted to prove to the other that it was a Tor node, it would send a two-element certificate chain signed by the key published in the Tor directory. +This approach met all the security properties envisaged at the time the 2004 design paper was written, but Tor's increasing use in censorship resistance changed the requirements – Tor's protocol signature also had to look like that of HTTPS web traffic, to prevent censors using deep-packet-inspection to detect and block Tor. +It turned out that Tor's original design looked very different from HTTPS. Firstly, web browsers offer a wide range of ciphersuites which Tor cannot use, such as those using RC4 (due to the narrow security margins) and RSA key exchange (due to lack of forward secrecy). Secondly, in HTTPS web traffic, the client seldom offers a certificate, and the server usually offers a one-element certificate chain, whereas in Tor node-to-node communication both sides offer a two-element certificate chain. +Therefore proposal 124, later superseded by proposal 130, tried to resolve the situation and the resulting version 2 connection protocol was implemented in Tor 0.2.0.20-rc. Here, the client presents a large selection of ciphersuites (including some it doesn't actually support), selected to appear similar to that of a web browser. The server then chooses one which is suitable for use in Tor, but if the server chooses one which is not adequately secure, the client will pull down the connection. +To make the certificate part of the handshake look closer to HTTPS, the client sends no certificate, and the server sends a one-element dummy certificate chain. The certificate offered by the server is designed to not contain distinctive strings which could be used for blocking (version 1 certificates used "Tor" or "TOR" as the organization name). Once the handshake is complete, Tor then restarts the handshake (via TLS renegotiation), but now encrypted under the keys established in the first handshake, and sends the two-element certificate chains as before. +This improves the situation for anti-blocking considerably, although more could still be done. In particular, the fact that renegotiation is occurring is not hidden from an observer because the type of TLS messages (known as records) is not encrypted in TLS, and renegotiation records are of a different type from data records. Therefore version 3 of the connection protocol, described in proposal 176 and implemented in Tor 0.2.3.6-alpha, moves the second stage of the handshake into data records, binding the inner to the outer handshake through sharing some key material. +10. Rise and fall of .exit +In Tor 0.0.9rc5, Tor had the .exit feature added. Here, if the user requested domain.nickname.exit then Tor would make a connection to domain using the Tor node called nickname as the last hop (if possible). This was a convenient feature for exploring how the Internet looked from different locations, but it also raised some security concerns. +In particular, a malicious website could embed an image with a .exit hostname, forcing the Tor client to select an attacker-controlled exit node. Then, if the user also chooses an attacker-controlled entry node the circuit could be de-anonymized. This strategy increases the probability of a successful attack from about (M/N)2 to M/N (where M is the amount of attacker-controlled network resource and N is the total network resource). +Therefore, in Tor 0.2.2.1-alpha, .exit notation was disabled by default. In Tor 0.2.3.17-beta an exception was made, allowing .exit notation when it is specified in the configuration file or by a controller. These sources are assumed to be safe, and by combining the .exit notation with the MapAddress option it is possible for the client to always contact some domain names via a particular exit node. This is useful when a service is running on the same machine as a Tor node, as then the user can choose for circuits to never leave the Tor network. +11. Controller protocol +Tor has always had a minimalist user interface – it can be configured on the command line or a configuration file and sends output to a log file. This is fine for advanced users, but most users will prefer a GUI. Building a GUI into Tor would be difficult, and would force certain choices (e.g. GUI toolkit) to be made which might not suit all users and all platforms. Therefore the approach taken by Tor in 0.0.9pre5 is to build an interface for other programs – the control protocol – to communicate with the Tor daemon, extracting information to display on the GUI and changing the Tor configuration based on user actions. +The control protocol has also proven useful to researchers experimenting with Tor. Initially the functionality exposed in the control protocol was simply that exposed by the configuration file and log files. Providing status information in a specified and machine-readable format made the task of monitoring and controlling Tor easier. Later, functionality was added to the control protocol which should not be exposed to ordinary Tor users but is useful to researchers, such as allowing controllers to arbitrarily control the path selection process (added in 0.1.0.1-rc). +In 0.1.1.1-alpha the protocol was changed to version 1, which used ASCII rather than binary commands to make it easier to write and debug controllers as well as allow advanced users to telnet into the control port and manually type commands. +12. Torbutton +The 2004 design paper stated that Tor explicitly did not make any attempt to scrub application data which might contain identifying information. By adopting the near universal SOCKS protocol, almost any application could send its traffic over Tor, but there was no guarantee it would be safe to do so. This is in contrast to the the predecessors to Tor from the Onion Routing project which required an "application proxy" to be written for each protocol carried by Tor. These proxies greatly increased the cost for supporting each additional application. +Still, there was clear need for a place to perform the protocol scrubbing, and so Tor recommended that Privoxy take the place of an application proxy for HTTP. However, the disadvantages of this approach gradually became clear, in particular Privoxy could not inspect or modify HTTPS traffic and so malicious websites could send their tracking code over HTTPS and avoid scrubbing. +Therefore, more and more of the scrubbing was performed by a Firefox add-on, Torbutton, which also could turn Tor on and off – hence the name. Torbutton had full access to content regardless of whether it was HTTP or HTTPS and could also disable features of Firefox which were bad for privacy. A proxy was still needed though, because Firefox's SOCKS support handled high-latency connections badly, so the lighter-weight Polipo was adopted instead. +13. Tor Browser Bundle +Now to use Tor, most users would need to download and install Tor, Firefox, Torbutton and Polipo, probably along with a GUI controller such as Vidalia. This was inconvenient, especially for customers of Internet cafes who could not install software on the computer they were using. So the Tor Browser Bundle was created which included all this software, pre-configured to be run from a USB drive. +This was far easier to use than the previous way to install Tor, and eventually became the default. It had the added advantage that we could modify the browser to include patches which made Polipo unnecessary and to fix some privacy problems which could not be solved from within a Firefox add-on. It was also safer for users because now Torbutton could not be disabled, meaning that the user had different web browsers for anonymous and non-anonymous browsing and were less likely to muddle up the two.

1 0

[tor/master] Fix a memory leak in handling errors on CERTS cells. bug 7422
by nickm＠torproject.org 09 Nov '12

09 Nov '12

commit 713736a6a738aa371176241e11e2f7e06f63523b Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Nov 8 23:01:39 2012 -0500 Fix a memory leak in handling errors on CERTS cells. bug 7422 --- changes/bug7422 | 3 +++ src/or/channeltls.c | 2 +- 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/changes/bug7422 b/changes/bug7422 new file mode 100644 index 0000000..652f1a2 --- /dev/null +++ b/changes/bug7422 @@ -0,0 +1,3 @@ + o Minor bugfixes: + - Fix a memory leak on failing cases of channel_tls_process_certs_cell. + Fixes bug 7422; bugfix on 0.2.4.4-alpha. diff --git a/src/or/channeltls.c b/src/or/channeltls.c index d094d15..ede2458 100644 --- a/src/or/channeltls.c +++ b/src/or/channeltls.c @@ -1522,7 +1522,7 @@ channel_tls_process_certs_cell(var_cell_t *cell, channel_tls_t *chan) safe_str(chan->conn->base_.address), \ chan->conn->base_.port, (s)); \ connection_or_close_for_error(chan->conn, 0); \ - return; \ + goto err; \ } while (0) if (chan->conn->base_.state != OR_CONN_STATE_OR_HANDSHAKING_V3)

1 0