December 2012 - tor-commits - lists.torproject.org

[ooni-probe/master] Add documentation and examples templates httpt and dnst
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 256c340080cf8d721ac278c91bb97407b935b5dc Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 13:38:04 2012 +0000 Add documentation and examples templates httpt and dnst --- docs/source/writing_tests.rst | 180 +++++++++++++++++++++++++++- nettests/examples/example_http_checksum.py | 27 ++++ 2 files changed, 203 insertions(+), 4 deletions(-) diff --git a/docs/source/writing_tests.rst b/docs/source/writing_tests.rst index e4d03e4..a8e5409 100644 --- a/docs/source/writing_tests.rst +++ b/docs/source/writing_tests.rst @@ -317,14 +317,186 @@ TODO finish this with more details HTTP based tests ................ -see nettests/examples/example_httpt.py +HTTP based tests will be a subclass of `ooni.templates.httpt.HTTPTest`. -TODO +It provides methods `ooni.templates.httpt.HTTPTest.processResponseBody` and +`ooni.templates.httpt.HTTPTest.processResponseHeaders` for interacting with the +response body and headers respectively. + +For example, to implement a HTTP test that returns the sha256 hash of the +response body (based on nettests/examples/example_httpt.py): + +:: + + from ooni.utils import log + from ooni.templates import httpt + from hashlib import sha256 + + class SHA256HTTPBodyTest(httpt.HTTPTest): + name = "ChecksumHTTPBodyTest" + author = "Aaron Gibson" + version = 0.1 + + inputFile = ['url file', 'f', None, + 'List of URLS to perform GET requests to'] + requiredOptions = ['url file'] + + def test_http(self): + if self.input: + url = self.input + return self.doRequest(url) + else: + raise Exception("No input specified") + + def processResponseBody(self, body): + body_sha256sum = sha256(body).hexdigest() + self.report['checksum'] = body_sha256sum + +The report for this test looks like this: + +:: + + ########################################### + # OONI Probe Report for ChecksumHTTPBodyTest test + # Thu Dec 6 17:31:57 2012 + ########################################### + --- + options: + collector: null + help: 0 + logfile: null + pcapfile: null + reportfile: null + resume: 0 + subargs: [-f, hosts] + test: nettests/examples/example_http_checksum.py + probe_asn: null + probe_cc: null + probe_ip: 127.0.0.1 + software_name: ooniprobe + software_version: 0.0.7.1-alpha + start_time: 1354786317.0 + test_name: ChecksumHTTPBodyTest + test_version: 0.1 + ... + --- + input: http://www.google.com + report: + agent: agent + checksum: d630fa2efd547d3656e349e96ff7af5496889dad959e8e29212af1ff843e7aa1 + requests: + - request: + body: null + headers: + - - User-Agent + - - [Opera/9.00 (Windows NT 5.1; U; en), 'Opera 9.0, Windows XP'] + method: GET + url: http://www.google.com + response: + body: '<!doctype html><html ... snip ... </html>' + code: 200 + headers: + - - X-XSS-Protection + - [1; mode=block] + - - Set-Cookie + - ['PREF=ID=fada4216eb3684f9:FF=0:TM=1354800717:LM=1354800717:S=IT-2GCkNAocyXlVa; + expires=Sat, 06-Dec-2014 13:31:57 GMT; path=/; domain=.google.com', 'NID=66=KWaLbNQumuGuYf0HrWlGm54u9l-DKJwhFCMQXfhQPZM-qniRhmF6QRGXUKXb_8CIUuCOHnyoC5oAX5jWNrsfk-LLJLW530UiMp6hemTtDMh_e6GSiEB4GR3yOP_E0TCN; + expires=Fri, 07-Jun-2013 13:31:57 GMT; path=/; domain=.google.com; HttpOnly'] + - - Expires + - ['-1'] + - - Server + - [gws] + - - Connection + - [close] + - - Cache-Control + - ['private, max-age=0'] + - - Date + - ['Thu, 06 Dec 2012 13:31:57 GMT'] + - - P3P + - ['CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 + for more info."'] + - - Content-Type + - [text/html; charset=UTF-8] + - - X-Frame-Options + - [SAMEORIGIN] + socksproxy: null + test_name: test_http + test_runtime: 0.08298492431640625 + test_started: 1354800717.478403 + ... + DNS based tests ............... -see nettests/core/dnstamper.py +DNS based tests will be a subclass of `ooni.templates.dnst.DNSTest`. + +It provides methods `ooni.templates.dnst.DNSTest.performPTRLookup` and +`ooni.templates.dnst.DNSTest.performALookup` -TODO +For example (taken from nettets/examples/example_dnst.py): + +:: + + from ooni.templates.dnst import DNSTest + + class ExampleDNSTest(DNSTest): + def test_a_lookup(self): + def gotResult(result): + # Result is an array containing all the A record lookup results + print result + + d = self.performALookup('torproject.org', ('8.8.8.8', 53)) + d.addCallback(gotResult) + return d + +The report looks like this: + +:: + + ########################################### + # OONI Probe Report for Base DNS Test test + # Thu Dec 6 17:42:51 2012 + ########################################### + --- + options: + collector: null + help: 0 + logfile: null + pcapfile: null + reportfile: null + resume: 0 + subargs: [] + test: nettests/examples/example_dnst.py + probe_asn: null + probe_cc: null + probe_ip: 127.0.0.1 + software_name: ooniprobe + software_version: 0.0.7.1-alpha + start_time: 1354786971.0 + test_name: Base DNS Test + test_version: 0.1 + ... + --- + input: null + report: + queries: + - addrs: [82.195.75.101, 86.59.30.40, 38.229.72.14, 38.229.72.16] + answers: + - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=82.195.75.101 + ttl=782>] + - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=86.59.30.40 + ttl=782>] + - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.14 + ttl=782>] + - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.16 + ttl=782>] + query: '[Query(''torproject.org'', 1, 1)]' + query_type: A + resolver: [8.8.8.8, 53] + test_name: test_a_lookup + test_runtime: 0.028924942016601562 + test_started: 1354801371.980114 + ... +For a more complex example, see: `DNS Tamper Test <https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/blocking/d…>`_ diff --git a/nettests/examples/example_http_checksum.py b/nettests/examples/example_http_checksum.py new file mode 100644 index 0000000..9226b52 --- /dev/null +++ b/nettests/examples/example_http_checksum.py @@ -0,0 +1,27 @@ +# -*- encoding: utf-8 -*- +# +# :authors: Aaron Gibson +# :licence: see LICENSE + +from ooni.utils import log +from ooni.templates import httpt +from hashlib import sha256 + +class SHA256HTTPBodyTest(httpt.HTTPTest): + name = "ChecksumHTTPBodyTest" + author = "Aaron Gibson" + version = 0.1 + + inputFile = ['file', 'f', None, + 'List of URLS to perform GET requests to'] + + def test_http(self): + if self.input: + url = self.input + return self.doRequest(url) + else: + raise Exception("No input specified") + + def processResponseBody(self, body): + body_sha256sum = sha256(body).digest() + self.report['checksum'] = body_sha256sum

1 0

[ooni-probe/master] Add automodule for tcpt to ooni.templates.rst
by art＠torproject.org 07 Dec '12

07 Dec '12

commit bc6341301520903573e377d856c7ef05b27cf049 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 13:57:38 2012 +0000 Add automodule for tcpt to ooni.templates.rst --- docs/source/api/ooni.templates.rst | 20 ++++++++++++++------ 1 files changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/source/api/ooni.templates.rst b/docs/source/api/ooni.templates.rst index c65a2da..d9172dd 100644 --- a/docs/source/api/ooni.templates.rst +++ b/docs/source/api/ooni.templates.rst @@ -1,26 +1,34 @@ templates Package ================= -:mod:`httpt` Module -------------------- +:mod:`ooni.templates.httpt` Module +---------------------------------- .. automodule:: ooni.templates.httpt :members: :undoc-members: :show-inheritance: -:mod:`scapyt` Module --------------------- +:mod:`ooni.templates.scapyt` Module +----------------------------------- .. automodule:: ooni.templates.scapyt :members: :undoc-members: :show-inheritance: -:mod:`dnst` Module --------------------- +:mod:`ooni.templates.dnst` Module +--------------------------------- .. automodule:: ooni.templates.dnst :members: :undoc-members: :show-inheritance: + +:mod:`ooni.templates.tcpt` Module +--------------------------------- + +.. automodule:: ooni.templates.tcpt + :members: + :undoc-members: + :show-inheritance:

1 0

[ooni-probe/master] Clean up writing_tests.rst
by art＠torproject.org 07 Dec '12

07 Dec '12

commit d4fb235ce3d74c3d14f883b30b877c5c4ef81dd8 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 14:09:09 2012 +0000 Clean up writing_tests.rst Add URLs to source in writing_tests Fix spelling and grammar --- docs/source/writing_tests.rst | 18 ++++++++++++------ 1 files changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/source/writing_tests.rst b/docs/source/writing_tests.rst index 01998db..e31d37f 100644 --- a/docs/source/writing_tests.rst +++ b/docs/source/writing_tests.rst @@ -3,7 +3,7 @@ Writing OONI tests The OONI testing API is heavily influenced and partially based on the python -:class:`unittest` module and :class:`twsted.trial`. +:class:`unittest` module and :class:`twisted.trial`. Test Cases @@ -28,8 +28,8 @@ templates, then the setUp method that is overwritable by users. Inputs ------ -Inputs are what is given as input to every iteration of the Test Case. You have -100 inputs, then every test case will be run 100 times. +Inputs are what is given as input to every iteration of the Test Case. +Iflyou have 100 inputs, then every test case will be run 100 times. To configure a static set of inputs you should define the :class:`ooni.nettest.TestCase` attribute ``inputs``. The test will be run @@ -131,9 +131,15 @@ functionality that is useful to running a test of that type. They also take care of writing the data they collect that is relevant to the test run to the report file. -Currently implemented test templates are `ooni.templates.scapt` for tests based -on Scapy, `ooni.templates.tcpt` for tests based on TCP, `ooni.templates.httpt` -for tests based on HTTP, `ooni.templates.dnst` for tests based on DNS. +Currently implemented test templates are `ooni.templates.scapyt +<https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/templates/scap…>` +for tests based on Scapy, `ooni.templates.tcpt +<https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/templates/tcpt…>` +for tests based on TCP, `ooni.templates.httpt +<https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/templates/http…>` +for tests based on HTTP, `ooni.templates.dnst +<https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/templates/dnst…>` +for tests based on DNS. Scapy based tests

1 0

[ooni-probe/master] Use section headers
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 3a936e1a2e3e20e88bbf86f048b2c7998c6bc302 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 13:53:45 2012 +0000 Use section headers --- docs/source/writing_tests.rst | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/source/writing_tests.rst b/docs/source/writing_tests.rst index a8e5409..01998db 100644 --- a/docs/source/writing_tests.rst +++ b/docs/source/writing_tests.rst @@ -352,7 +352,8 @@ response body (based on nettests/examples/example_httpt.py): body_sha256sum = sha256(body).hexdigest() self.report['checksum'] = body_sha256sum -The report for this test looks like this: +Report format +************* :: @@ -450,7 +451,8 @@ For example (taken from nettets/examples/example_dnst.py): d.addCallback(gotResult) return d -The report looks like this: +Report format +************* ::

1 0

[ooni-probe/master] Remove redundant heading
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 84915cf5a06570654d47032a5792a97af7326c0b Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 13:58:43 2012 +0000 Remove redundant heading --- docs/source/api/ooni.utils.rst | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/docs/source/api/ooni.utils.rst b/docs/source/api/ooni.utils.rst index 057138b..1b0ca03 100644 --- a/docs/source/api/ooni.utils.rst +++ b/docs/source/api/ooni.utils.rst @@ -1,8 +1,6 @@ utils Package ============= -:mod:`utils` Package --------------------- .. automodule:: ooni.utils :members: :undoc-members:

1 0

[ooni-probe/master] Add documentation for HTTPHeaderFieldManipulation
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 4dd1045f24188195684f14727c4c1d56f19e24a0 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 15:21:59 2012 +0000 Add documentation for HTTPHeaderFieldManipulation --- .../tests/http_header_field_manipulation.rst | 346 ++++++++++++++++++++ 1 files changed, 346 insertions(+), 0 deletions(-) diff --git a/docs/source/tests/http_header_field_manipulation.rst b/docs/source/tests/http_header_field_manipulation.rst new file mode 100644 index 0000000..92bd0a4 --- /dev/null +++ b/docs/source/tests/http_header_field_manipulation.rst @@ -0,0 +1,346 @@ +Details +======= + +*Test Name*: HTTP Header Field Manipulation + +*Current version*: 0.1.3 + +*NetTest*: HTTPHeaderFieldManipulation (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/manipulati…) + +*Test Helper*: SimpleHTTPChannel (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/testhelpers/h…) + +*Test Type*: Traffic Manipulation + +*Requires Root*: No + +*WARNING*: This test is more dangerous to run than any other one and you +should do it only if you know what you are doing. + +Description +=========== +It performes HTTP requests with request headers that vary capitalization +towards a SimpleHTTPChannel test helper backend. If we detect that the +headers the backend received matches the ones we have sent then we have +detected tampering. + +How to run the test +=================== + +`./bin/ooniprobe nettests/manipulation/http_header_field_manipulation.py -b <address of backend> [-h <headers>]` +`address of backend` is the IP:PORT of the SimpleHTTPChannel backend. + +Sample report +============= + +From running: +`./bin/ooniprobe nettests/manipulation/http_header_field_manipulation.py` +If no backend is specified, the default backend is 127.0.0.1:57001, where you will need to have oonib listening. + +:: + + ########################################### + # OONI Probe Report for HTTP Header Field Manipulation test + # Thu Dec 6 19:22:00 2012 + ########################################### + --- + options: + collector: null + help: 0 + logfile: null + pcapfile: null + reportfile: null + resume: 0 + subargs: [] + test: nettests/manipulation/http_header_field_manipulation.py + probe_asn: null + probe_cc: null + probe_ip: 127.0.0.1 + software_name: ooniprobe + software_version: 0.0.7.1-alpha + start_time: 1354792920.0 + test_name: HTTP Header Field Manipulation + test_version: 0.1.3 + ... + --- + input: null + report: + agent: agent + requests: + - request: + body: null + headers: + - - Accept-Language + - ['en-US,en;q=0.8'] + - - Accept-Encoding + - ['gzip,deflate,sdch'] + - - Accept + - ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'] + - - User-Agent + - ['Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 + Firefox/2.0.0.6'] + - - Accept-Charset + - ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'] + - - Host + - [cDMxQx4pPcCnNC5.com] + method: PUT + url: http://127.0.0.1:57001 + response: + body: '{"headers_dict": {"Accept-Language": ["en-US,en;q=0.8"], "Accept-Encoding": + ["gzip,deflate,sdch"], "Host": ["cDMxQx4pPcCnNC5.com"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + "User-Agent": ["Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) + Gecko/20070725 Firefox/2.0.0.6"], "Accept-Charset": ["ISO-8859-1,utf-8;q=0.7,*;q=0.3"], + "Connection": ["close"]}, "request_line": "PUT / HTTP/1.1", "request_headers": + [["Connection", "close"], ["Accept-Language", "en-US,en;q=0.8"], ["Accept-Encoding", + "gzip,deflate,sdch"], ["Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + ["User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) + Gecko/20070725 Firefox/2.0.0.6"], ["Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3"], + ["Host", "cDMxQx4pPcCnNC5.com"]]}' + code: 200 + headers: [] + socksproxy: null + tampering: + header_field_name: false + header_field_number: false + header_field_value: false + header_name_capitalization: false + header_name_diff: [] + request_line_capitalization: false + total: false + test_name: test_put + test_runtime: 0.023853063583374023 + test_started: 1354807320.864641 + ... + --- + input: null + report: + agent: agent + requests: + - request: + body: null + headers: + - - aCcept-LANguage + - ['en-US,en;q=0.8'] + - - acCEPt-ENcODING + - ['gzip,deflate,sdch'] + - - AccEPT + - ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'] + - - usER-AGenT + - [Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)] + - - aCcEpt-ChARseT + - ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'] + - - hosT + - [Vw0mRN7DmC0IFU0.com] + method: Get + url: http://127.0.0.1:57001 + response: + body: '{"headers_dict": {"aCcept-LANguage": ["en-US,en;q=0.8"], "acCEPt-ENcODING": + ["gzip,deflate,sdch"], "hosT": ["Vw0mRN7DmC0IFU0.com"], "AccEPT": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + "usER-AGenT": ["Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR + 1.1.4322)"], "aCcEpt-ChARseT": ["ISO-8859-1,utf-8;q=0.7,*;q=0.3"], "Connection": + ["close"]}, "request_line": "Get / HTTP/1.1", "request_headers": [["Connection", + "close"], ["aCcept-LANguage", "en-US,en;q=0.8"], ["acCEPt-ENcODING", "gzip,deflate,sdch"], + ["AccEPT", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + ["usER-AGenT", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR + 1.1.4322)"], ["aCcEpt-ChARseT", "ISO-8859-1,utf-8;q=0.7,*;q=0.3"], ["hosT", + "Vw0mRN7DmC0IFU0.com"]]}' + code: 200 + headers: [] + socksproxy: null + tampering: + header_field_name: false + header_field_number: false + header_field_value: false + header_name_capitalization: false + header_name_diff: [] + request_line_capitalization: false + total: false + test_name: test_get_random_capitalization + test_runtime: 0.035381078720092773 + test_started: 1354807320.866462 + ... + --- + input: null + report: + agent: agent + requests: + - request: + body: null + headers: + - - ACcEPt-lANGuAgE + - ['en-US,en;q=0.8'] + - - AcCePT-EnCodiNg + - ['gzip,deflate,sdch'] + - - acCept + - ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'] + - - USEr-Agent + - [Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.0] + - - AccepT-cHARsEt + - ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'] + - - HOst + - [1numISAjBIEifu1.com] + method: pOst + url: http://127.0.0.1:57001 + response: + body: '{"headers_dict": {"ACcEPt-lANGuAgE": ["en-US,en;q=0.8"], "AcCePT-EnCodiNg": + ["gzip,deflate,sdch"], "HOst": ["1numISAjBIEifu1.com"], "acCept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + "USEr-Agent": ["Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera + 8.0"], "AccepT-cHARsEt": ["ISO-8859-1,utf-8;q=0.7,*;q=0.3"], "Connection": + ["close"]}, "request_line": "pOst / HTTP/1.1", "request_headers": [["Connection", + "close"], ["ACcEPt-lANGuAgE", "en-US,en;q=0.8"], ["AcCePT-EnCodiNg", "gzip,deflate,sdch"], + ["acCept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + ["USEr-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera + 8.0"], ["AccepT-cHARsEt", "ISO-8859-1,utf-8;q=0.7,*;q=0.3"], ["HOst", "1numISAjBIEifu1.com"]]}' + code: 200 + headers: [] + socksproxy: null + tampering: + header_field_name: false + header_field_number: false + header_field_value: false + header_name_capitalization: false + header_name_diff: [] + request_line_capitalization: false + total: false + test_name: test_post_random_capitalization + test_runtime: 0.046284914016723633 + test_started: 1354807320.868329 + ... + --- + input: null + report: + agent: agent + requests: + - request: + body: null + headers: + - - Accept-Language + - ['en-US,en;q=0.8'] + - - Accept-Encoding + - ['gzip,deflate,sdch'] + - - Accept + - ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'] + - - User-Agent + - [Opera/9.00 (Windows NT 5.1; U; en)] + - - Accept-Charset + - ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'] + - - Host + - [9ogjh0OCzT1arR8.com] + method: POST + url: http://127.0.0.1:57001 + response: + body: '{"headers_dict": {"Accept-Language": ["en-US,en;q=0.8"], "Accept-Encoding": + ["gzip,deflate,sdch"], "Host": ["9ogjh0OCzT1arR8.com"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + "User-Agent": ["Opera/9.00 (Windows NT 5.1; U; en)"], "Accept-Charset": ["ISO-8859-1,utf-8;q=0.7,*;q=0.3"], + "Connection": ["close"]}, "request_line": "POST / HTTP/1.1", "request_headers": + [["Connection", "close"], ["Accept-Language", "en-US,en;q=0.8"], ["Accept-Encoding", + "gzip,deflate,sdch"], ["Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + ["User-Agent", "Opera/9.00 (Windows NT 5.1; U; en)"], ["Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3"], + ["Host", "9ogjh0OCzT1arR8.com"]]}' + code: 200 + headers: [] + socksproxy: null + tampering: + header_field_name: false + header_field_number: false + header_field_value: false + header_name_capitalization: false + header_name_diff: [] + request_line_capitalization: false + total: false + test_name: test_post + test_runtime: 0.058208942413330078 + test_started: 1354807320.870338 + ... + --- + input: null + report: + agent: agent + requests: + - request: + body: null + headers: + - - Accept-laNguagE + - ['en-US,en;q=0.8'] + - - aCcEpt-EnCODIng + - ['gzip,deflate,sdch'] + - - acCePt + - ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'] + - - uSer-AGeNT + - [Opera/9.00 (Windows NT 5.1; U; en)] + - - aCcept-CHArSET + - ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'] + - - HosT + - [Upd9yWpA0TMhUua.com] + method: GET + url: http://127.0.0.1:57001 + response: + body: '{"headers_dict": {"Accept-laNguagE": ["en-US,en;q=0.8"], "aCcEpt-EnCODIng": + ["gzip,deflate,sdch"], "HosT": ["Upd9yWpA0TMhUua.com"], "acCePt": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + "uSer-AGeNT": ["Opera/9.00 (Windows NT 5.1; U; en)"], "aCcept-CHArSET": ["ISO-8859-1,utf-8;q=0.7,*;q=0.3"], + "Connection": ["close"]}, "request_line": "GET / HTTP/1.1", "request_headers": + [["Connection", "close"], ["Accept-laNguagE", "en-US,en;q=0.8"], ["aCcEpt-EnCODIng", + "gzip,deflate,sdch"], ["acCePt", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + ["uSer-AGeNT", "Opera/9.00 (Windows NT 5.1; U; en)"], ["aCcept-CHArSET", "ISO-8859-1,utf-8;q=0.7,*;q=0.3"], + ["HosT", "Upd9yWpA0TMhUua.com"]]}' + code: 200 + headers: [] + socksproxy: null + tampering: + header_field_name: false + header_field_number: false + header_field_value: false + header_name_capitalization: false + header_name_diff: [] + request_line_capitalization: false + total: false + test_name: test_get + test_runtime: 0.068952083587646484 + test_started: 1354807320.872004 + ... + --- + input: null + report: + agent: agent + requests: + - request: + body: null + headers: + - - accEpt-lANGuAGE + - ['en-US,en;q=0.8'] + - - acCePt-encodINg + - ['gzip,deflate,sdch'] + - - aCCepT + - ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'] + - - uSer-AGent + - [Opera/9.20 (Windows NT 6.0; U; en)] + - - ACcepT-cHarSEt + - ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'] + - - HOsT + - [UTqJhv92syxk0nj.com] + method: pUt + url: http://127.0.0.1:57001 + response: + body: '{"headers_dict": {"accEpt-lANGuAGE": ["en-US,en;q=0.8"], "acCePt-encodINg": + ["gzip,deflate,sdch"], "HOsT": ["UTqJhv92syxk0nj.com"], "aCCepT": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + "uSer-AGent": ["Opera/9.20 (Windows NT 6.0; U; en)"], "ACcepT-cHarSEt": ["ISO-8859-1,utf-8;q=0.7,*;q=0.3"], + "Connection": ["close"]}, "request_line": "pUt / HTTP/1.1", "request_headers": + [["Connection", "close"], ["accEpt-lANGuAGE", "en-US,en;q=0.8"], ["acCePt-encodINg", + "gzip,deflate,sdch"], ["aCCepT", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], + ["uSer-AGent", "Opera/9.20 (Windows NT 6.0; U; en)"], ["ACcepT-cHarSEt", "ISO-8859-1,utf-8;q=0.7,*;q=0.3"], + ["HOsT", "UTqJhv92syxk0nj.com"]]}' + code: 200 + headers: [] + socksproxy: null + tampering: + header_field_name: false + header_field_number: false + header_field_value: false + header_name_capitalization: false + header_name_diff: [] + request_line_capitalization: false + total: false + test_name: test_put_random_capitalization + test_runtime: 0.080827951431274414 + test_started: 1354807320.8738551 + ... +

1 0

[ooni-probe/master] Add documentation for the DNSSpoof test
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 6f7e91543e9b11a47efa6ca03a0f2df4fcd438d9 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 15:25:46 2012 +0000 Add documentation for the DNSSpoof test --- docs/source/tests/dnsspoof.rst | 111 ++++++++++++++++++++++++++++++++++++++++ 1 files changed, 111 insertions(+), 0 deletions(-) diff --git a/docs/source/tests/dnsspoof.rst b/docs/source/tests/dnsspoof.rst new file mode 100644 index 0000000..abb08e4 --- /dev/null +++ b/docs/source/tests/dnsspoof.rst @@ -0,0 +1,111 @@ +Details +======= + +*Test Name*: DNS Spoof + +*Current version*: 0.1 + +*NetTest*: DNSSpoof (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/manipulati…) + +*Test Helper*: DNS Test Helper (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/testhelpers/d…) + +*Test Type*: Traffic Manipulation + +*Requires Root*: Yes + +Description +=========== + +This test performs A queries to a test resolver and a known good control resolver. The query is considered tampered with if the two responses match. + +How to run the test +=================== + +`./bin/ooniprobe nettests/manipulation/dnsspoof.py [-s] [-k] [-i] -r <test resolver> -h <hostname> -b IP:PORT` + +*test resolver* is a single test resolver (IP address). +*hostname* is the hostname to query. +*IP:PORT* is the address of the known good "control" resolver. +*-s, --ipsrc* Do *not* check if IP src and ICMP IP citation match +*-k, --seqack* Check if TCP sequence number and ACK match in the ICMP citation +*-i, --ipid* Check if the IPID matches when processing answers + + +Sample report +============= + +From running: +`./bin/ooniprobe nettests/manipulation/dnsspoof.py -h torproject.org -r 4.2.2.2:53` + +:: + + ########################################### + # OONI Probe Report for DNS Spoof test + # Thu Dec 6 11:10:38 2012 + ########################################### + --- + options: + collector: null + help: 0 + logfile: null + pcapfile: null + reportfile: null + resume: 0 + subargs: [-h, torproject.org, -r, '4.2.2.2:53'] + test: nettests/manipulation/dnsspoof.py + probe_asn: null + probe_cc: null + probe_ip: 127.0.0.1 + software_name: ooniprobe + software_version: 0.0.7.1-alpha + start_time: 1354828238.0 + test_name: DNS Spoof + test_version: 0.10000000000000001 + ... + --- + input: null + report: + answer_flags: [ipsrc] + answered_packets: + - - raw_packet: !!binary | + RQAAfDj1AAA4EZJIBAICAn8AAAEANQA1AGjH/wAAgYAAAQAEAAAAAAp0b3Jwcm9qZWN0A29yZwAA + AQABCnRvcnByb2plY3QDb3JnAAABAAEAAADnAAQm5UgQCnRvcnByb2plY3QDb3JnAAABAAEAAADn + AARSw0tlCnRvcnByb2plY3QDb3JnAAABAAEAAADnAARWOx4oCnRvcnByb2plY3QDb3JnAAABAAEA + AADnAAQm5UgO + summary: 'IP / UDP / DNS Ans "38.229.72.16" ' + sent_packets: + - - raw_packet: !!binary | + RQAAPAABAABAEfWrfwAAAQQCAgIANQA1AChvjwAAAQAAAQAAAAAAAAp0b3Jwcm9qZWN0A29yZwAA + AQAB + summary: 'IP / UDP / DNS Qry "torproject.org" ' + test_name: test_a_lookup + test_runtime: 0.23476505279541016 + test_started: 1354810238.400979 + ... + --- + input: null + report: + answer_flags: [ipsrc] + answered_packets: + - - raw_packet: !!binary | + RQAAfGQmAAAvEWYLCAgICH8AAAEANQA1AGizfwAAgYAAAQAEAAAAAAp0b3Jwcm9qZWN0A29yZwAA + AQABCnRvcnByb2plY3QDb3JnAAABAAEAAAOEAAQm5UgQCnRvcnByb2plY3QDb3JnAAABAAEAAAOE + AARSw0tlCnRvcnByb2plY3QDb3JnAAABAAEAAAOEAARWOx4oCnRvcnByb2plY3QDb3JnAAABAAEA + AAOEAAQm5UgO + summary: 'IP / UDP / DNS Ans "38.229.72.16" ' + sent_packets: + - - raw_packet: !!binary | + RQAAPAABAABAEeuffwAAAQgICAgANQA1AChlgwAAAQAAAQAAAAAAAAp0b3Jwcm9qZWN0A29yZwAA + AQAB + summary: 'IP / UDP / DNS Qry "torproject.org" ' + test_name: test_control_a_lookup + test_runtime: 0.23965692520141602 + test_started: 1354810238.625988 + ... + --- + input: null + report: {spoofing: false} + test_name: summary + test_runtime: 0.00017499923706054688 + test_started: 1354810238.8703561 + ...

1 0

[ooni-probe/master] Merge pull request #17 from aagbsn/update_readme_with_https_urls
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 0b40aa8923dfa1b1469421050a66cb25bf248580 Merge: 87a3099 b55355f Author: Arturo Filastò <hellais(a)gmail.com> Date: Thu Dec 6 09:44:12 2012 -0800 Merge pull request #17 from aagbsn/update_readme_with_https_urls Use HTTPS where available README.md | 14 +++++++------- 1 files changed, 7 insertions(+), 7 deletions(-)

1 0

[ooni-probe/master] Add references to classes in writing_tests
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 23c0d737fe60cac3947eeeb20453ea4ccfaee6c8 Author: aagbsn <aagbsn(a)extc.org> Date: Thu Dec 6 18:03:01 2012 +0000 Add references to classes in writing_tests --- docs/source/writing_tests.rst | 103 ++++++++++++++++++++--------------------- 1 files changed, 50 insertions(+), 53 deletions(-) diff --git a/docs/source/writing_tests.rst b/docs/source/writing_tests.rst index e31d37f..4ce6268 100644 --- a/docs/source/writing_tests.rst +++ b/docs/source/writing_tests.rst @@ -3,7 +3,7 @@ Writing OONI tests The OONI testing API is heavily influenced and partially based on the python -:class:`unittest` module and :class:`twisted.trial`. +:mod:`unittest` module and :mod:`twisted.trial`. Test Cases @@ -32,8 +32,9 @@ Inputs are what is given as input to every iteration of the Test Case. Iflyou have 100 inputs, then every test case will be run 100 times. To configure a static set of inputs you should define the -:class:`ooni.nettest.TestCase` attribute ``inputs``. The test will be run -``len(inputs)`` times. Any iterable object is a valid ``inputs`` attribute. +:class:`ooni.nettest.NetTestCase` attribute ``inputs``. The test will be +run ``len(inputs)`` times. Any iterable object is a valid ``inputs`` +attribute. If you would like to have inputs be determined from a user specified input file, then you must set the ``inputFile`` attribute. This is an array that @@ -56,12 +57,12 @@ this:: Setup and command line passing ------------------------------ -Tests may define the `setUp` method that will be called every time the Test -Case object is instantiated, in here you may place some common logic to all your -Test Methods that should be run before any testing occurs. +Tests may define the `setUp` method that will be called every time the +Test Case object is instantiated, in here you may place some common logic +to all your Test Methods that should be run before any testing occurs. Command line arguments can be parsed thanks to the twisted -`twisted.python.usage.UsageOptions` class. +:class:`twisted.python.usage.UsageOptions` class. You will have to subclass this and define the NetTestCase attribute usageOptions to point to a subclass of this. @@ -73,7 +74,7 @@ usageOptions to point to a subclass of this. 'URL of the test backend to use'] ] - class MyTestCase(nettest.TestCase): + class MyTestCase(nettest.NetTestCase): usageOptions = UsageOptions inputFile = ['file', 'f', None, "Some foo file"] @@ -83,30 +84,31 @@ usageOptions to point to a subclass of this. self.localOptions['backend'] -You will then be able to access the parsed command line arguments via the class -attribute localOptions. +You will then be able to access the parsed command line arguments via the +class attribute localOptions. The `requiredOptions` attributes specifies an array of parameters that are required for the test to run properly. -`inputFile` is a special class attribute that will be used for processing of -the inputFile. The filename that is read here will be given to the -`ooni.nettest.NetTestCase.inputProcessor` method that will yield, by default, -one line of the file at a time. +`inputFile` is a special class attribute that will be used for processing +of the inputFile. The filename that is read here will be given to the +:class:`ooni.nettest.NetTestCase.inputProcessor` method that will yield, by +default, one line of the file at a time. Test Methods ------------ -These shall be defined inside of your :class:`ooni.nettest.TestCase` subclass. -These will be class methods. +These shall be defined inside of your :class:`ooni.nettest.NetTestCase` +subclass. These will be class methods. -All class methods that are prefixed with test\_ shall be run. Functions that -are relevant to your test should be all lowercase separated by underscore. +All class methods that are prefixed with test\_ shall be run. Functions +that are relevant to your test should be all lowercase separated by +underscore. -To add data to the test report you may write directly to the report object like -so:: +To add data to the test report you may write directly to the report object +like so:: def test_my_function(): result = do_something() @@ -126,36 +128,31 @@ input. Test Templates -------------- -Test templates assist you in writing tests. They already contain all the common -functionality that is useful to running a test of that type. They also take -care of writing the data they collect that is relevant to the test run to the -report file. +Test templates assist you in writing tests. They already contain all the +common functionality that is useful to running a test of that type. They +also take care of writing the data they collect that is relevant to the +test run to the report file. -Currently implemented test templates are `ooni.templates.scapyt -<https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/templates/scap…>` -for tests based on Scapy, `ooni.templates.tcpt -<https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/templates/tcpt…>` -for tests based on TCP, `ooni.templates.httpt -<https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/templates/http…>` -for tests based on HTTP, `ooni.templates.dnst -<https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/ooni/templates/dnst…>` -for tests based on DNS. +Currently implemented test templates are :mod:`ooni.templates.scapyt` for +tests based on Scapy, :mod:`ooni.templates.tcpt` for tests based on TCP, +:mod:`ooni.templates.httpt` for tests based on HTTP, and +:mod:`ooni.templates.dnst` for tests based on DNS. Scapy based tests ................. -Scapy based tests will be a subclass of `ooni.templates.scapyt.BaseScapyTest`. +Scapy based tests will be a subclass of :class:`ooni.templates.scapyt.BaseScapyTest`. It provides a wrapper around the scapy send and receive function that will -write the sent and received packets to the report with sanitization of the src -and destination IP addresses. +write the sent and received packets to the report with sanitization of the +src and destination IP addresses. -It has the same syntax as the Scapy sr function, except that it will return a -deferred. +It has the same syntax as the Scapy sr function, except that it will +return a deferred. To implement a simple ICMP ping based on this function you can do like so -(taken from nettest/examples/example_scapyt.py): +(Taken from :class:`nettest.examples.example_scapyt.ExampleICMPPingScapy`) :: @@ -191,7 +188,7 @@ receive function, the only difference is that instead of using the regular scapy super socket it uses our twisted driven wrapper around it. Alternatively this test can also be written using the -`twisted.defer.inlineCallbacks` decorator, that makes it look more similar to +:func:`twisted.defer.inlineCallbacks` decorator, that makes it look more similar to regular sequential code. :: @@ -253,7 +250,7 @@ Report Format TCP based tests ............... -TCP based tests will subclass `ooni.templates.tcpt.TCPTest`. +TCP based tests will subclass :class:`ooni.templates.tcpt.TCPTest`. This test template facilitates the sending of TCP payloads to the wire and recording the response. @@ -284,12 +281,12 @@ recording the response. The possible failures for a TCP connection are: -`twisted.internet.error.NoRouteError` that corresponds to errno.ENETUNREACH +:class:`twisted.internet.error.NoRouteError` that corresponds to errno.ENETUNREACH -`twisted.internet.error.ConnectionRefusedError` that corresponds to +:class:`twisted.internet.error.ConnectionRefusedError` that corresponds to errno.ECONNREFUSED -`twisted.internet.error.TCPTimedOutError` that corresponds to errno.ETIMEDOUT +:class:`twisted.internet.error.TCPTimedOutError` that corresponds to errno.ETIMEDOUT Report format ************* @@ -323,14 +320,14 @@ TODO finish this with more details HTTP based tests ................ -HTTP based tests will be a subclass of `ooni.templates.httpt.HTTPTest`. +HTTP based tests will be a subclass of :class:`ooni.templates.httpt.HTTPTest`. -It provides methods `ooni.templates.httpt.HTTPTest.processResponseBody` and -`ooni.templates.httpt.HTTPTest.processResponseHeaders` for interacting with the +It provides methods :func:`ooni.templates.httpt.HTTPTest.processResponseBody` and +:func:`ooni.templates.httpt.HTTPTest.processResponseHeaders` for interacting with the response body and headers respectively. For example, to implement a HTTP test that returns the sha256 hash of the -response body (based on nettests/examples/example_httpt.py): +response body (based on :mod:`nettests.examples.example_httpt`): :: @@ -436,12 +433,12 @@ Report format DNS based tests ............... -DNS based tests will be a subclass of `ooni.templates.dnst.DNSTest`. +DNS based tests will be a subclass of :class:`ooni.templates.dnst.DNSTest`. -It provides methods `ooni.templates.dnst.DNSTest.performPTRLookup` and -`ooni.templates.dnst.DNSTest.performALookup` +It provides methods :func:`ooni.templates.dnst.DNSTest.performPTRLookup` +and :func:`ooni.templates.dnst.DNSTest.performALookup` -For example (taken from nettets/examples/example_dnst.py): +For example (taken from :mod:`nettests.examples.example_dnst`): :: @@ -507,4 +504,4 @@ Report format test_started: 1354801371.980114 ... -For a more complex example, see: `DNS Tamper Test <https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/blocking/d…>`_ +For a more complex example, see: :mod:`nettests.blocking.dnstamper`

1 0

[ooni-probe/master] Merge pull request #16 from aagbsn/read_tor_path_from_config
by art＠torproject.org 07 Dec '12

07 Dec '12

commit 87a30994789e699782552d7342033a2359e400d7 Merge: 4648dd5 3fd67d2 Author: Arturo Filastò <hellais(a)gmail.com> Date: Thu Dec 6 09:43:58 2012 -0800 Merge pull request #16 from aagbsn/read_tor_path_from_config call launch_tor with the tor_binary from config ooni/runner.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)

1 0