December 2012 - tor-commits - lists.torproject.org

[ooni-probe/master] Cleanup all the source tree from dead code and cruft
by art＠torproject.org 06 Dec '12

06 Dec '12

commit 5fc1204fd56275e4dde7116fb0db2564a00d55f4 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Dec 6 23:16:26 2012 +0100 Cleanup all the source tree from dead code and cruft * Create directory for storing test decks --- before_i_commit.sh | 29 -- before_i_commit.testdeck | 33 -- bin/INSTRUCTIONS | 15 - bin/Makefile | 54 --- bin/… [View More]

1 0

[ooni-probe/master] Use HTTPS where available
by art＠torproject.org 06 Dec '12

06 Dec '12

commit b55355f1ec24a50b3a9cbf3e670a89fc5e0a3528 Author: aagbsn <aagbsn(a)extc.org> Date: Wed Dec 5 14:08:10 2012 +0000 Use HTTPS where available --- README.md | 14 +++++++------- 1 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index be35943..77b5b91 100644 --- a/README.md +++ b/README.md @@ -29,13 +29,13 @@ On debian based systems these can be installed with: The python dependencies required for running ooniprobe are: - * Tor (>2.2.… [View More]

1 0

[ooni-probe/master] call launch_tor with the tor_binary from config
by art＠torproject.org 06 Dec '12

06 Dec '12

commit 3fd67d201061ca22c5fcdbc7bbaafb9897a888c5 Author: aagbsn <aagbsn(a)extc.org> Date: Wed Dec 5 16:56:49 2012 +0000 call launch_tor with the tor_binary from config Otherwise, the default is supplied by txtorcon as /usr/sbin/tor --- ooni/runner.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/ooni/runner.py b/ooni/runner.py index bc9b874..2123a7d 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -516,6 +516,7 @@ def startTor(): log.… [View More]

1 0

[ooni-probe/master] Add docstrings to network helper functions
by art＠torproject.org 06 Dec '12

06 Dec '12

commit 7f29689606b9661c53c4164c8bb660df87567223 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 12:20:14 2012 +0000 Add docstrings to network helper functions Adds documentation for getDefaultIface and getNetworkFromRoutes. Also adds #XXX warning for users of OpenVZ environments. --- ooni/utils/txscapy.py | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/ooni/utils/txscapy.py b/ooni/utils/txscapy.py index 7902133..210f21b 100644 --- a/… [View More]

1 0

[ooni-probe/master] Fix spelling in test documentation
by art＠torproject.org 06 Dec '12

06 Dec '12

commit 6dea02f4de4cd32741b3f9a7404750dff7136183 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 11:56:16 2012 +0000 Fix spelling in test documentation --- docs/source/tests/dnstamper.rst | 6 +++--- docs/source/tests/http_host.rst | 10 +++++----- docs/source/tests/http_invalid_request_line.rst | 4 ++-- docs/source/tests/http_requests.rst | 4 ++-- docs/source/tests/tcpconnect.rst | 2 +- docs/source/… [View More]tests/traceroute.rst | 2 +- docs/source/writing_tests.rst | 8 ++++---- 7 files changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/source/tests/dnstamper.rst b/docs/source/tests/dnstamper.rst index c4e6f17..8fdb572 100644 --- a/docs/source/tests/dnstamper.rst +++ b/docs/source/tests/dnstamper.rst @@ -22,8 +22,8 @@ lookup on the first A record address of both sets and check if they both resolve to the same name. NOTE: This test frequently results in false positives due to GeoIP-based -load balancing on major global sites such as google, facebook, and -youtube, etc. +load balancing on major global sites such as Google, Facebook, and +Youtube, etc. How to run the test =================== @@ -176,5 +176,5 @@ From running: test_started: 1354187839.512434 ... -Notes: Query is the string repsentation of :class:twisted.names.dns.Query +Notes: Query is the string representation of :class:twisted.names.dns.Query diff --git a/docs/source/tests/http_host.rst b/docs/source/tests/http_host.rst index febdc4a..ebe10e7 100644 --- a/docs/source/tests/http_host.rst +++ b/docs/source/tests/http_host.rst @@ -22,7 +22,7 @@ enumerating the sites that are being censored by it. It places inside of the Host header field the hostname of the site that is to be tested for censorship and then determines if the probe is behind a transparent HTTP proxy (because the response from the backend server does not -match) and if the site is censorsed, by checking if the page that it got back +match) and if the site is censored, by checking if the page that it got back matches the input block page. *Why do content blocking?* @@ -30,7 +30,7 @@ matches the input block page. Q: Why should be do content blocking measurements with this test when we have other tests that also do this? -A: Why not? Although you are correct that tecnically the two tests are +A: Why not? Although you are correct that technically the two tests are equivalent even though the IP layer differs in the two tests. Note: We may in the future remove the Content Blocking aspect of the HTTP Host @@ -46,9 +46,9 @@ How to run the test *backend url* is the url of the backend that will be used for checking if the site is blocked or not. -*content* is the content of a blockpage. When a transparent HTTP proxy is -present we will do comparisons against this to verify if the requested site is -blocked or not. +*content* is the content of a page. When a transparent HTTP proxy is present we +will do comparisons against this to verify if the requested site is blocked or +not. Sample report diff --git a/docs/source/tests/http_invalid_request_line.rst b/docs/source/tests/http_invalid_request_line.rst index 4ac223e..4a59df0 100644 --- a/docs/source/tests/http_invalid_request_line.rst +++ b/docs/source/tests/http_invalid_request_line.rst @@ -24,7 +24,7 @@ on the HTTP request line. We generate a series of requests that are not valid HTTP requests. The remote backend runs a TCP echo server. If the response from the backend -does not match with what we have sent then we say that tampering is occuring. +does not match with what we have sent then we say that tampering is occurring. The idea behind this is that certain transparent HTTP proxies may not be properly parsing the HTTP request line. @@ -75,7 +75,7 @@ This generates a request that looks like this: :: GET / HTTP/XxX -This attemps to trigger bugs in the parsing of the HTTP version number, that +This attempts to trigger bugs in the parsing of the HTTP version number, that is usually being split on the `.`. How to run the test diff --git a/docs/source/tests/http_requests.rst b/docs/source/tests/http_requests.rst index cc98882..6eb0d0a 100644 --- a/docs/source/tests/http_requests.rst +++ b/docs/source/tests/http_requests.rst @@ -21,8 +21,8 @@ and over Tor. It then compares the two responses to see if the response bodies o proportion between the expected body length (the one over Tor) and the one over the control network match. -If the proportion between the two body lengths is <= a certain tollerance -factor (by default set to 0.8), then we say that they do not match. +If the proportion between the two body lengths is <= a certain tolerance factor +(by default set to 0.8), then we say that they do not match. The reason for doing so is that a lot of sites serve geolocalized content based on the location from which the request originated from. diff --git a/docs/source/tests/tcpconnect.rst b/docs/source/tests/tcpconnect.rst index 7c7bd16..e92077d 100644 --- a/docs/source/tests/tcpconnect.rst +++ b/docs/source/tests/tcpconnect.rst @@ -16,7 +16,7 @@ Details Description =========== -This test performs TCP connections to a set of sepecified IP:PORT pairs and +This test performs TCP connections to a set of specified IP:PORT pairs and reports the reason for which it failed connecting to the target address. The reason for failure may be: "timeout", when the connection timed out, diff --git a/docs/source/tests/traceroute.rst b/docs/source/tests/traceroute.rst index 898fc30..5b18454 100644 --- a/docs/source/tests/traceroute.rst +++ b/docs/source/tests/traceroute.rst @@ -40,7 +40,7 @@ other means that are not the src and destination IP address. In particular the ICMP TTL expired citations will contain the IP headers. -We could theorically strip these though even if that were the case there would +We could theoretically strip these though even if that were the case there would still be at least a reduction of the anonymity set given by the fact that we received a TTL expired from a router in a certain network range. diff --git a/docs/source/writing_tests.rst b/docs/source/writing_tests.rst index f74f3f0..e4d03e4 100644 --- a/docs/source/writing_tests.rst +++ b/docs/source/writing_tests.rst @@ -41,7 +41,7 @@ specifies what command line option may be used to control this value. By default the ``inputProcessor`` is set to read the file line by line and strip newline characters. To change this behavior you must set the -``inputProcessor`` attribute to a function that takes as arugment a file +``inputProcessor`` attribute to a function that takes as argument a file descriptor and yield the next item. The default ``inputProcessor`` looks like this:: @@ -57,7 +57,7 @@ Setup and command line passing ------------------------------ Tests may define the `setUp` method that will be called every time the Test -Case object is intantiated, in here you may place some common logic to all your +Case object is instantiated, in here you may place some common logic to all your Test Methods that should be run before any testing occurs. Command line arguments can be parsed thanks to the twisted @@ -181,8 +181,8 @@ To implement a simple ICMP ping based on this function you can do like so return d The arguments taken by self.sr() are exactly the same as the scapy send and -receive function, the only difference is that instead of using the regualar -scapy super socket it uses our twisted drivven wrapper around it. +receive function, the only difference is that instead of using the regular +scapy super socket it uses our twisted driven wrapper around it. Alternatively this test can also be written using the `twisted.defer.inlineCallbacks` decorator, that makes it look more similar to [View Less]

1 0

[ooni-probe/master] Add dnst to ooni.templates.rst
by art＠torproject.org 06 Dec '12

06 Dec '12

commit e4ab3c57422824f68b906cbfaa2de8391b550184 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 12:14:03 2012 +0000 Add dnst to ooni.templates.rst --- docs/source/api/ooni.templates.rst | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/docs/source/api/ooni.templates.rst b/docs/source/api/ooni.templates.rst index 6951e31..c65a2da 100644 --- a/docs/source/api/ooni.templates.rst +++ b/docs/source/api/ooni.templates.rst @@ -17,4 +17,10 @@ templates … [View More]

1 0

[ooni-probe/master] Add docstrings to ooni/runner.py
by art＠torproject.org 06 Dec '12

06 Dec '12

commit a2468793713723c1811ab01d115b5bdd2cd8c465 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 12:28:23 2012 +0000 Add docstrings to ooni/runner.py Add docstring to startTor() Add docstring to startSniffing() --- ooni/runner.py | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/ooni/runner.py b/ooni/runner.py index bc9b874..723b209 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -464,6 +464,10 @@ class UnableToStartTor(Exception): … [View More]

1 0

[ooni-probe/master] Remove automodule for ooni/__init__.py
by art＠torproject.org 06 Dec '12

06 Dec '12

commit 95ff2cf20efd0a293e44ecfa80eb19da9e238043 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 12:24:07 2012 +0000 Remove automodule for ooni/__init__.py --- docs/source/api/ooni.rst | 8 -------- 1 files changed, 0 insertions(+), 8 deletions(-) diff --git a/docs/source/api/ooni.rst b/docs/source/api/ooni.rst index 83b4f83..cb79525 100644 --- a/docs/source/api/ooni.rst +++ b/docs/source/api/ooni.rst @@ -1,14 +1,6 @@ ooni Package ============ -:mod:`ooni` Package -------… [View More]

1 0

[ooni-probe/master] Clean up index.rst
by art＠torproject.org 06 Dec '12

06 Dec '12

commit 4cfb89c7f000a23cd0547a3ff0c513b22f4b7efa Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 12:34:09 2012 +0000 Clean up index.rst Fix link to http_header_field_manipulation Fix some spelling errors Remove daphne from list of tests Disable globbing in index TOC --- docs/source/index.rst | 5 +---- 1 files changed, 1 insertions(+), 4 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 770133d..1abb517 100644 --- a/docs/… [View More]

1 0

[ooni-probe/master] Add documentation and examples templates httpt and dnst
by art＠torproject.org 06 Dec '12

06 Dec '12

commit 256c340080cf8d721ac278c91bb97407b935b5dc Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 13:38:04 2012 +0000 Add documentation and examples templates httpt and dnst --- docs/source/writing_tests.rst | 180 +++++++++++++++++++++++++++- nettests/examples/example_http_checksum.py | 27 ++++ 2 files changed, 203 insertions(+), 4 deletions(-) diff --git a/docs/source/writing_tests.rst b/docs/source/writing_tests.rst index e4d03e4..a8e5409 100644 --- a/docs/… [View More]source/writing_tests.rst +++ b/docs/source/writing_tests.rst @@ -317,14 +317,186 @@ TODO finish this with more details HTTP based tests ................ -see nettests/examples/example_httpt.py +HTTP based tests will be a subclass of `ooni.templates.httpt.HTTPTest`. -TODO +It provides methods `ooni.templates.httpt.HTTPTest.processResponseBody` and +`ooni.templates.httpt.HTTPTest.processResponseHeaders` for interacting with the +response body and headers respectively. + +For example, to implement a HTTP test that returns the sha256 hash of the +response body (based on nettests/examples/example_httpt.py): + +:: + + from ooni.utils import log + from ooni.templates import httpt + from hashlib import sha256 + + class SHA256HTTPBodyTest(httpt.HTTPTest): + name = "ChecksumHTTPBodyTest" + author = "Aaron Gibson" + version = 0.1 + + inputFile = ['url file', 'f', None, + 'List of URLS to perform GET requests to'] + requiredOptions = ['url file'] + + def test_http(self): + if self.input: + url = self.input + return self.doRequest(url) + else: + raise Exception("No input specified") + + def processResponseBody(self, body): + body_sha256sum = sha256(body).hexdigest() + self.report['checksum'] = body_sha256sum + +The report for this test looks like this: + +:: + + ########################################### + # OONI Probe Report for ChecksumHTTPBodyTest test + # Thu Dec 6 17:31:57 2012 + ########################################### + --- + options: + collector: null + help: 0 + logfile: null + pcapfile: null + reportfile: null + resume: 0 + subargs: [-f, hosts] + test: nettests/examples/example_http_checksum.py + probe_asn: null + probe_cc: null + probe_ip: 127.0.0.1 + software_name: ooniprobe + software_version: 0.0.7.1-alpha + start_time: 1354786317.0 + test_name: ChecksumHTTPBodyTest + test_version: 0.1 + ... + --- + input: http://www.google.com + report: + agent: agent + checksum: d630fa2efd547d3656e349e96ff7af5496889dad959e8e29212af1ff843e7aa1 + requests: + - request: + body: null + headers: + - - User-Agent + - - [Opera/9.00 (Windows NT 5.1; U; en), 'Opera 9.0, Windows XP'] + method: GET + url: http://www.google.com + response: + body: '<!doctype html><html ... snip ... </html>' + code: 200 + headers: + - - X-XSS-Protection + - [1; mode=block] + - - Set-Cookie + - ['PREF=ID=fada4216eb3684f9:FF=0:TM=1354800717:LM=1354800717:S=IT-2GCkNAocyXlVa; + expires=Sat, 06-Dec-2014 13:31:57 GMT; path=/; domain=.google.com', 'NID=66=KWaLbNQumuGuYf0HrWlGm54u9l-DKJwhFCMQXfhQPZM-qniRhmF6QRGXUKXb_8CIUuCOHnyoC5oAX5jWNrsfk-LLJLW530UiMp6hemTtDMh_e6GSiEB4GR3yOP_E0TCN; + expires=Fri, 07-Jun-2013 13:31:57 GMT; path=/; domain=.google.com; HttpOnly'] + - - Expires + - ['-1'] + - - Server + - [gws] + - - Connection + - [close] + - - Cache-Control + - ['private, max-age=0'] + - - Date + - ['Thu, 06 Dec 2012 13:31:57 GMT'] + - - P3P + - ['CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 + for more info."'] + - - Content-Type + - [text/html; charset=UTF-8] + - - X-Frame-Options + - [SAMEORIGIN] + socksproxy: null + test_name: test_http + test_runtime: 0.08298492431640625 + test_started: 1354800717.478403 + ... + DNS based tests ............... -see nettests/core/dnstamper.py +DNS based tests will be a subclass of `ooni.templates.dnst.DNSTest`. + +It provides methods `ooni.templates.dnst.DNSTest.performPTRLookup` and +`ooni.templates.dnst.DNSTest.performALookup` -TODO +For example (taken from nettets/examples/example_dnst.py): + +:: + + from ooni.templates.dnst import DNSTest + + class ExampleDNSTest(DNSTest): + def test_a_lookup(self): + def gotResult(result): + # Result is an array containing all the A record lookup results + print result + + d = self.performALookup('torproject.org', ('8.8.8.8', 53)) + d.addCallback(gotResult) + return d + +The report looks like this: + +:: + + ########################################### + # OONI Probe Report for Base DNS Test test + # Thu Dec 6 17:42:51 2012 + ########################################### + --- + options: + collector: null + help: 0 + logfile: null + pcapfile: null + reportfile: null + resume: 0 + subargs: [] + test: nettests/examples/example_dnst.py + probe_asn: null + probe_cc: null + probe_ip: 127.0.0.1 + software_name: ooniprobe + software_version: 0.0.7.1-alpha + start_time: 1354786971.0 + test_name: Base DNS Test + test_version: 0.1 + ... + --- + input: null + report: + queries: + - addrs: [82.195.75.101, 86.59.30.40, 38.229.72.14, 38.229.72.16] + answers: + - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=82.195.75.101 + ttl=782>] + - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=86.59.30.40 + ttl=782>] + - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.14 + ttl=782>] + - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.16 + ttl=782>] + query: '[Query(''torproject.org'', 1, 1)]' + query_type: A + resolver: [8.8.8.8, 53] + test_name: test_a_lookup + test_runtime: 0.028924942016601562 + test_started: 1354801371.980114 + ... +For a more complex example, see: `DNS Tamper Test <https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/blocking/d…>`_ diff --git a/nettests/examples/example_http_checksum.py b/nettests/examples/example_http_checksum.py new file mode 100644 index 0000000..9226b52 --- /dev/null +++ b/nettests/examples/example_http_checksum.py @@ -0,0 +1,27 @@ +# -*- encoding: utf-8 -*- +# +# :authors: Aaron Gibson +# :licence: see LICENSE + +from ooni.utils import log +from ooni.templates import httpt +from hashlib import sha256 + +class SHA256HTTPBodyTest(httpt.HTTPTest): + name = "ChecksumHTTPBodyTest" + author = "Aaron Gibson" + version = 0.1 + + inputFile = ['file', 'f', None, + 'List of URLS to perform GET requests to'] + + def test_http(self): + if self.input: + url = self.input + return self.doRequest(url) + else: + raise Exception("No input specified") + + def processResponseBody(self, body): + body_sha256sum = sha256(body).digest() + self.report['checksum'] = body_sha256sum [View Less]

1 0