November 2012 - tor-commits - lists.torproject.org

[ooni-probe/master] Fix links to test description pages
by art＠torproject.org 29 Nov '12

29 Nov '12

commit c64ae8d3c657f553e343d1a351a2d7f3b67bdfa3 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Nov 29 19:22:27 2012 +0100 Fix links to test description pages --- docs/source/index.rst | 18 +++++++++--------- 1 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 7803782..31a7f6e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -48,27 +48,27 @@ respectively. Content Blocking Tests ...................... - * ::`DNSTamper tests/dnstamper` + * `DNSTamper <tests/dnstamper.html>`_ - * ::`HTTP Requests tests/http_requests` + * `HTTP Requests <tests/http_requests.html>`_ - * ::`TCP Connect tests/tcpconnect` + * `TCP Connect <tests/tcpconnect.html>`_ Traffic Manipulation Tests .......................... - * ::`HTTP Invalid Request Line: tests/http_invalid_request_line` + * `HTTP Invalid Request Line: <tests/http_invalid_request_line.html>`_ - * ::`DNS Spoof: tests/dnsspoof` + * `DNS Spoof: <tests/dnsspoof.html>`_ - * ::`HTTP Header Field Manipulation: tests/http_header_field_manipulation` + * `HTTP Header Field Manipulation: <tests/http_header_field_manipulation>`_ - * ::`Traceroute: tests/traceroute` + * `Traceroute: <tests/traceroute.html>`_ - * ::`HTTP Host: tests/http_host` + * `HTTP Host: <tests/http_host.html>`_ - * ::`Daphne: tests/daphne` + * `Daphne: <tests/daphne.html>`_ Other tests ...........

1 0

[ooni-probe/master] Rename documentation for HTTP Invalid Request Line
by art＠torproject.org 29 Nov '12

29 Nov '12

commit a8872467a8a38e445db19548fb9d89cb8b576d66 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Nov 29 17:19:13 2012 +0100 Rename documentation for HTTP Invalid Request Line --- docs/source/tests/http_invalid_request_line.rst | 164 +++++++++++++++++++++++ docs/source/tests/http_invalid_requests.rst | 164 ----------------------- 2 files changed, 164 insertions(+), 164 deletions(-) diff --git a/docs/source/tests/http_invalid_request_line.rst b/docs/source/tests/http_invalid_request_line.rst new file mode 100644 index 0000000..f14a18d --- /dev/null +++ b/docs/source/tests/http_invalid_request_line.rst @@ -0,0 +1,164 @@ +Details +======= + +*Test Name*: HTTP Invalid Request Line + +*Current version*: 0.1.3 + +*NetTest*: HTTP Invalid Request Line (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/manipulati…) + +*Test Helper*: TCPEchoHelper (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/testhelpers/t…) + +*Test Type*: Traffic Manipulation + +*Requires Root*: No + +*WARNING*: This test is more dangerous to run than any other one and you +should do it only if you know what you are doing. + +Description +=========== + +The goal of this test is to do some very basic and not very noisy fuzzing +on the HTTP request line. We generate a series of requests that are not +valid HTTP requests. + +The remote backend runs a TCP echo server. If the response from the backend +does not match with what we have sent then we say that tampering is occuring. + +The idea behind this is that certain transparent HTTP proxies may not be +properly parsing the HTTP request line. + +Unless elsewhere stated 'Xx'\*N refers to N\*2 random upper or lowercase ascii +letters or numbers ('XxXx' will be 4). + +What we fuzz is the following: + +test random invalid method +************************** + +This sends random 4 letter HTTP request method. + +The request on the wire will look like this: + +:: + XxXxX / HTTP/1.1\n\r + + +test random invalid field count +****************************** + +This generates a request that looks like this: + +:: + XxXxX XxXxX XxXxX XxXxX + + This may trigger some bugs in the HTTP parsers of transparent HTTP + proxies. + + +test random big request method +****************************** + +This generates a request that looks like this: + +:: + + Xx*512 / HTTP/1.1 + + +test random invalid version number +********************************** + +This generates a request that looks like this: + +:: + GET / HTTP/XxX + +This attemps to trigger bugs in the parsing of the HTTP version number, that +is usually being split on the `.`. + +How to run the test +=================== + +`./bin/ooniprobe nettests/manipulation/http_invalid_request_line.py -b <address of backend>` + +*address of the backend* is the hostname or IP address of a backend that runs +a TCP echo server on port 80. + +Sample report +============= + +From running: + +`./bin/ooniprobe nettests/manipulation/http_invalid_request_line.py -b 127.0.0.1 -p 57002` + +:: + + ########################################### + # OONI Probe Report for HTTP Invalid Request Line test + # Thu Nov 29 16:41:15 2012 + ########################################### + --- + options: + collector: null + help: 0 + logfile: null + pcapfile: null + reportfile: null + resume: 0 + subargs: [-b, 127.0.0.1, -p, '57002'] + test: nettests/manipulation/http_invalid_request_line.py + probe_asn: null + probe_cc: null + probe_ip: 127.0.0.1 + software_name: ooniprobe + software_version: 0.0.7.1-alpha + start_time: 1354200075.0 + test_name: HTTP Invalid Request Line + test_version: 0.1.3 + ... + --- + input: null + report: + received: ["L3F6 / HTTP/1.1\n\r"] + sent: ["L3F6 / HTTP/1.1\n\r"] + tampering: false + test_name: test_random_invalid_method + test_runtime: 5.011919021606445 + test_started: 1354203675.481258 + ... + --- + input: null + report: + received: ["GET / HTTP/Ayo\n\r"] + sent: ["GET / HTTP/Ayo\n\r"] + tampering: false + test_name: test_random_invalid_version_number + test_runtime: 5.019288063049316 + test_started: 1354203675.48221 + ... + --- + input: null + report: + received: ["RZRcu OyLtu Jtu2T cs0ER\n\r"] + sent: ["RZRcu OyLtu Jtu2T cs0ER\n\r"] + tampering: false + test_name: test_random_invalid_field_count + test_runtime: 5.022854804992676 + test_started: 1354203675.483127 + ... + --- + input: null + report: + received: ["iB2HrwJeB512y5CrAAaIpETZ5RMprmGO4RCex24Kmqkjguc2XsOrGXR3qJIERw1IX13uGWVs1kOd96Y0zsR4ufcktGFnP0gYakTq7GA63rNNOmlG9tNXABnBEHfkHYhrwdrewAKdqZ2lGls66NBY7fbL9xkOsHBjXq7TkvS8MOeTb74wcxKymp1NLOa8u7C8XP9qpafQtWBrC4dkEVjppjlFyetg1tu8zomDUCBx9y2tB411d8lw7WOSMfDiQWG327aVanxaOj9WgZ2u2eu0595UiJxZxMk1LYa96vHvmB1DrX0DoJUkUg2fcEOlia3pVoFfcpbd3TQ0GwEoewU3F48Qvpf2AuOIcPgrYa1XszLCyUoToygc5J6WgIH7f1phsEpmmZ6my9KChZdnazc7mNbCwLB3Z1wMwcoFW1XuDvAhTr8OYoY17360SYkAqEBhYh8uiD4xdIq0T0KJzsJW5wghjCMjRjFyfk3wyDXaPLp7vkeeqbA3FNFatFQlUCTIkqqqM3EjAojynvCDVyrThGlmsauS9Ejhc9TaDojeT3s8HZY3KIDHwCRvpGgpwFie2NyhzmwY2y4YFMcnGXT2jlwsVE1K2fZZ7yGbKqbciuh5Q32JNvQBdGy3N63PWyZIxWhfythHOVBE9GXAr99DqEXUeiRkfNer5aVRt87rDBuLx13IrbCE6runHdoZEq0iXs7IySfw5qqt00fExbaN9UlSDyxyYIrIymrSTSUWWprTqiezn1toQZpxEl53pv4ZnLx00CSMYxWTzMDNSHLpovdAI9A27ugY2uHmPETKHbaLsTzNRtNiZAS9WidDi0oAxdICB47dnJA2CXvBaXeq2nsRasgQ9qqNl0RbTC8vhXgIq8nZZQn0iehXsP5mPXshpALyfAhoLwiChRif8FODkKdAgciLviTvyadpecb6HqRVfM8rLzMPkuXT2I21yB7NVQQLraJOeDO TvmtsV2gw2I2ON8xKtHO78VQfo357yBryndTQTRTk9FPnijwcwusn\ + \ / HTTP/1.1\n\r"] + sent: ["iB2HrwJeB512y5CrAAaIpETZ5RMprmGO4RCex24Kmqkjguc2XsOrGXR3qJIERw1IX13uGWVs1kOd96Y0zsR4ufcktGFnP0gYakTq7GA63rNNOmlG9tNXABnBEHfkHYhrwdrewAKdqZ2lGls66NBY7fbL9xkOsHBjXq7TkvS8MOeTb74wcxKymp1NLOa8u7C8XP9qpafQtWBrC4dkEVjppjlFyetg1tu8zomDUCBx9y2tB411d8lw7WOSMfDiQWG327aVanxaOj9WgZ2u2eu0595UiJxZxMk1LYa96vHvmB1DrX0DoJUkUg2fcEOlia3pVoFfcpbd3TQ0GwEoewU3F48Qvpf2AuOIcPgrYa1XszLCyUoToygc5J6WgIH7f1phsEpmmZ6my9KChZdnazc7mNbCwLB3Z1wMwcoFW1XuDvAhTr8OYoY17360SYkAqEBhYh8uiD4xdIq0T0KJzsJW5wghjCMjRjFyfk3wyDXaPLp7vkeeqbA3FNFatFQlUCTIkqqqM3EjAojynvCDVyrThGlmsauS9Ejhc9TaDojeT3s8HZY3KIDHwCRvpGgpwFie2NyhzmwY2y4YFMcnGXT2jlwsVE1K2fZZ7yGbKqbciuh5Q32JNvQBdGy3N63PWyZIxWhfythHOVBE9GXAr99DqEXUeiRkfNer5aVRt87rDBuLx13IrbCE6runHdoZEq0iXs7IySfw5qqt00fExbaN9UlSDyxyYIrIymrSTSUWWprTqiezn1toQZpxEl53pv4ZnLx00CSMYxWTzMDNSHLpovdAI9A27ugY2uHmPETKHbaLsTzNRtNiZAS9WidDi0oAxdICB47dnJA2CXvBaXeq2nsRasgQ9qqNl0RbTC8vhXgIq8nZZQn0iehXsP5mPXshpALyfAhoLwiChRif8FODkKdAgciLviTvyadpecb6HqRVfM8rLzMPkuXT2I21yB7NVQQLraJOeDOTvmt sV2gw2I2ON8xKtHO78VQfo357yBryndTQTRTk9FPnijwcwusn\ + \ / HTTP/1.1\n\r"] + tampering: false + test_name: test_random_big_request_method + test_runtime: 5.026142120361328 + test_started: 1354203675.484211 + ... + + diff --git a/docs/source/tests/http_invalid_requests.rst b/docs/source/tests/http_invalid_requests.rst deleted file mode 100644 index f14a18d..0000000 --- a/docs/source/tests/http_invalid_requests.rst +++ /dev/null @@ -1,164 +0,0 @@ -Details -======= - -*Test Name*: HTTP Invalid Request Line - -*Current version*: 0.1.3 - -*NetTest*: HTTP Invalid Request Line (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/manipulati…) - -*Test Helper*: TCPEchoHelper (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/testhelpers/t…) - -*Test Type*: Traffic Manipulation - -*Requires Root*: No - -*WARNING*: This test is more dangerous to run than any other one and you -should do it only if you know what you are doing. - -Description -=========== - -The goal of this test is to do some very basic and not very noisy fuzzing -on the HTTP request line. We generate a series of requests that are not -valid HTTP requests. - -The remote backend runs a TCP echo server. If the response from the backend -does not match with what we have sent then we say that tampering is occuring. - -The idea behind this is that certain transparent HTTP proxies may not be -properly parsing the HTTP request line. - -Unless elsewhere stated 'Xx'\*N refers to N\*2 random upper or lowercase ascii -letters or numbers ('XxXx' will be 4). - -What we fuzz is the following: - -test random invalid method -************************** - -This sends random 4 letter HTTP request method. - -The request on the wire will look like this: - -:: - XxXxX / HTTP/1.1\n\r - - -test random invalid field count -****************************** - -This generates a request that looks like this: - -:: - XxXxX XxXxX XxXxX XxXxX - - This may trigger some bugs in the HTTP parsers of transparent HTTP - proxies. - - -test random big request method -****************************** - -This generates a request that looks like this: - -:: - - Xx*512 / HTTP/1.1 - - -test random invalid version number -********************************** - -This generates a request that looks like this: - -:: - GET / HTTP/XxX - -This attemps to trigger bugs in the parsing of the HTTP version number, that -is usually being split on the `.`. - -How to run the test -=================== - -`./bin/ooniprobe nettests/manipulation/http_invalid_request_line.py -b <address of backend>` - -*address of the backend* is the hostname or IP address of a backend that runs -a TCP echo server on port 80. - -Sample report -============= - -From running: - -`./bin/ooniprobe nettests/manipulation/http_invalid_request_line.py -b 127.0.0.1 -p 57002` - -:: - - ########################################### - # OONI Probe Report for HTTP Invalid Request Line test - # Thu Nov 29 16:41:15 2012 - ########################################### - --- - options: - collector: null - help: 0 - logfile: null - pcapfile: null - reportfile: null - resume: 0 - subargs: [-b, 127.0.0.1, -p, '57002'] - test: nettests/manipulation/http_invalid_request_line.py - probe_asn: null - probe_cc: null - probe_ip: 127.0.0.1 - software_name: ooniprobe - software_version: 0.0.7.1-alpha - start_time: 1354200075.0 - test_name: HTTP Invalid Request Line - test_version: 0.1.3 - ... - --- - input: null - report: - received: ["L3F6 / HTTP/1.1\n\r"] - sent: ["L3F6 / HTTP/1.1\n\r"] - tampering: false - test_name: test_random_invalid_method - test_runtime: 5.011919021606445 - test_started: 1354203675.481258 - ... - --- - input: null - report: - received: ["GET / HTTP/Ayo\n\r"] - sent: ["GET / HTTP/Ayo\n\r"] - tampering: false - test_name: test_random_invalid_version_number - test_runtime: 5.019288063049316 - test_started: 1354203675.48221 - ... - --- - input: null - report: - received: ["RZRcu OyLtu Jtu2T cs0ER\n\r"] - sent: ["RZRcu OyLtu Jtu2T cs0ER\n\r"] - tampering: false - test_name: test_random_invalid_field_count - test_runtime: 5.022854804992676 - test_started: 1354203675.483127 - ... - --- - input: null - report: - received: ["iB2HrwJeB512y5CrAAaIpETZ5RMprmGO4RCex24Kmqkjguc2XsOrGXR3qJIERw1IX13uGWVs1kOd96Y0zsR4ufcktGFnP0gYakTq7GA63rNNOmlG9tNXABnBEHfkHYhrwdrewAKdqZ2lGls66NBY7fbL9xkOsHBjXq7TkvS8MOeTb74wcxKymp1NLOa8u7C8XP9qpafQtWBrC4dkEVjppjlFyetg1tu8zomDUCBx9y2tB411d8lw7WOSMfDiQWG327aVanxaOj9WgZ2u2eu0595UiJxZxMk1LYa96vHvmB1DrX0DoJUkUg2fcEOlia3pVoFfcpbd3TQ0GwEoewU3F48Qvpf2AuOIcPgrYa1XszLCyUoToygc5J6WgIH7f1phsEpmmZ6my9KChZdnazc7mNbCwLB3Z1wMwcoFW1XuDvAhTr8OYoY17360SYkAqEBhYh8uiD4xdIq0T0KJzsJW5wghjCMjRjFyfk3wyDXaPLp7vkeeqbA3FNFatFQlUCTIkqqqM3EjAojynvCDVyrThGlmsauS9Ejhc9TaDojeT3s8HZY3KIDHwCRvpGgpwFie2NyhzmwY2y4YFMcnGXT2jlwsVE1K2fZZ7yGbKqbciuh5Q32JNvQBdGy3N63PWyZIxWhfythHOVBE9GXAr99DqEXUeiRkfNer5aVRt87rDBuLx13IrbCE6runHdoZEq0iXs7IySfw5qqt00fExbaN9UlSDyxyYIrIymrSTSUWWprTqiezn1toQZpxEl53pv4ZnLx00CSMYxWTzMDNSHLpovdAI9A27ugY2uHmPETKHbaLsTzNRtNiZAS9WidDi0oAxdICB47dnJA2CXvBaXeq2nsRasgQ9qqNl0RbTC8vhXgIq8nZZQn0iehXsP5mPXshpALyfAhoLwiChRif8FODkKdAgciLviTvyadpecb6HqRVfM8rLzMPkuXT2I21yB7NVQQLraJOeDO TvmtsV2gw2I2ON8xKtHO78VQfo357yBryndTQTRTk9FPnijwcwusn\ - \ / HTTP/1.1\n\r"] - sent: ["iB2HrwJeB512y5CrAAaIpETZ5RMprmGO4RCex24Kmqkjguc2XsOrGXR3qJIERw1IX13uGWVs1kOd96Y0zsR4ufcktGFnP0gYakTq7GA63rNNOmlG9tNXABnBEHfkHYhrwdrewAKdqZ2lGls66NBY7fbL9xkOsHBjXq7TkvS8MOeTb74wcxKymp1NLOa8u7C8XP9qpafQtWBrC4dkEVjppjlFyetg1tu8zomDUCBx9y2tB411d8lw7WOSMfDiQWG327aVanxaOj9WgZ2u2eu0595UiJxZxMk1LYa96vHvmB1DrX0DoJUkUg2fcEOlia3pVoFfcpbd3TQ0GwEoewU3F48Qvpf2AuOIcPgrYa1XszLCyUoToygc5J6WgIH7f1phsEpmmZ6my9KChZdnazc7mNbCwLB3Z1wMwcoFW1XuDvAhTr8OYoY17360SYkAqEBhYh8uiD4xdIq0T0KJzsJW5wghjCMjRjFyfk3wyDXaPLp7vkeeqbA3FNFatFQlUCTIkqqqM3EjAojynvCDVyrThGlmsauS9Ejhc9TaDojeT3s8HZY3KIDHwCRvpGgpwFie2NyhzmwY2y4YFMcnGXT2jlwsVE1K2fZZ7yGbKqbciuh5Q32JNvQBdGy3N63PWyZIxWhfythHOVBE9GXAr99DqEXUeiRkfNer5aVRt87rDBuLx13IrbCE6runHdoZEq0iXs7IySfw5qqt00fExbaN9UlSDyxyYIrIymrSTSUWWprTqiezn1toQZpxEl53pv4ZnLx00CSMYxWTzMDNSHLpovdAI9A27ugY2uHmPETKHbaLsTzNRtNiZAS9WidDi0oAxdICB47dnJA2CXvBaXeq2nsRasgQ9qqNl0RbTC8vhXgIq8nZZQn0iehXsP5mPXshpALyfAhoLwiChRif8FODkKdAgciLviTvyadpecb6HqRVfM8rLzMPkuXT2I21yB7NVQQLraJOeDOTvmt sV2gw2I2ON8xKtHO78VQfo357yBryndTQTRTk9FPnijwcwusn\ - \ / HTTP/1.1\n\r"] - tampering: false - test_name: test_random_big_request_method - test_runtime: 5.026142120361328 - test_started: 1354203675.484211 - ... - -

1 0

[ooni-probe/master] Log to error log file when we get malformed Headers sent to the backend
by art＠torproject.org 29 Nov '12

29 Nov '12

commit 27286ce77673ae99b7e7acc52e42e39fdb97b240 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Nov 29 17:51:11 2012 +0100 Log to error log file when we get malformed Headers sent to the backend * Python's typesafe, but it's still good practice --- oonib/testhelpers/http_helpers.py | 11 ++++++++--- 1 files changed, 8 insertions(+), 3 deletions(-) diff --git a/oonib/testhelpers/http_helpers.py b/oonib/testhelpers/http_helpers.py index 1fa0ccb..6415291 100644 --- a/oonib/testhelpers/http_helpers.py +++ b/oonib/testhelpers/http_helpers.py @@ -71,8 +71,12 @@ class SimpleHTTPChannel(basic.LineReceiver, policies.TimeoutMixin): self.__header = line def headerReceived(self, line): - header, data = line.split(':', 1) - self.headers.append((header, data.strip())) + try: + header, data = line.split(':', 1) + self.headers.append((header, data.strip())) + except: + log.err("Got malformed HTTP Header request field") + log.err("%s" % line) def allHeadersReceived(self): headers_dict = {} @@ -85,8 +89,9 @@ class SimpleHTTPChannel(basic.LineReceiver, policies.TimeoutMixin): 'request_line': self.requestLine, 'headers_dict': headers_dict } + json_response = json.dumps(response) self.transport.write('HTTP/1.1 200 OK\r\n\r\n') - self.transport.write(json.dumps(response)) + self.transport.write('%s' % json_response) self.transport.loseConnection()

1 0

[ooni-probe/master] Port the test page to the index of the documentation
by art＠torproject.org 29 Nov '12

29 Nov '12

commit c195aa60653ed1ebfd9623525148ed9177e80af1 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Nov 29 19:13:24 2012 +0100 Port the test page to the index of the documentation * Only include the tests that we plan supporting in the first release --- docs/source/index.rst | 110 ++++++++++++++++++++++++++++- nettests/experimental/daphn3.py | 119 -------------------------------- nettests/manipulation/captiveportal.py | 57 ++++++++-------- nettests/manipulation/daphne.py | 119 ++++++++++++++++++++++++++++++++ oonib/testhelpers/http_helpers.py | 2 +- 5 files changed, 254 insertions(+), 153 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 389df2a..7803782 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,19 +3,121 @@ Welcome to the OONI developer documentation =========================================== -If you are looking for how to start running the ooniprobe tool see: +ooniprobe is tool for performing internet censorship measurements. Our goal is +to achieve a command data format and set of methodologies for conducting +censorship related research. + +If you are a user interesting in running the ooniprobe command line tool see: https://github.com/hellais/ooni-probe#getting-started +The two main software components of ooniprobe are ooniprobe and oonib. + +ooniprobe +********* + +Is the tool that volunteers and researches interested in contributing data to +the project should be running. + +ooniprobe allows the user to select what test should be run and what backend +should be used for storing the test report and/or assisting them in the running +of the test. + +ooniprobe tests are divided into two categories: **Traffic Manipulation** and +**Content Blocking**. + +**Traffic Manipulation** tests aim to detect the presence of some sort of +tampering with the internet traffic between the probe and a remote test helper +backend. As such they usually require the selection of a oonib backend +component for running the test. + +**Content Blocking** are aimed at enumerating the kind of content that is +blocked from the probes network point of view. As such they usually require to +have specified an input list for running the test. + +Core ooniprobe Tests +-------------------- + +The source for `Content blocking tests +<https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/nettests/blocking>`_ +and `Traffic Manipulation tests +<https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/nettests/blocking>`_ +can be found in the nettests/blocking and nettests/manipulation directories +respectively. + +Content Blocking Tests +...................... + + * ::`DNSTamper tests/dnstamper` + + * ::`HTTP Requests tests/http_requests` + + * ::`TCP Connect tests/tcpconnect` + + +Traffic Manipulation Tests +.......................... + + * ::`HTTP Invalid Request Line: tests/http_invalid_request_line` + + * ::`DNS Spoof: tests/dnsspoof` + + * ::`HTTP Header Field Manipulation: tests/http_header_field_manipulation` + + * ::`Traceroute: tests/traceroute` + + * ::`HTTP Host: tests/http_host` + + * ::`Daphne: tests/daphne` + +Other tests +........... + +We also have some other tests that are currently not fully supported or still +being experimented with. + +You can find these in ::`nettests/experimental: +https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/nettests/experimental`. + +Tests that don't do a measurement but are useful for scanning can be found in +::`nettests/scanning: +https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/nettests/scanning`. + +Tests that involve running third party tools may be found in +::`nettests/third_party: +https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/nettests/third_party`. + +oonib +***** + +This is the server side component of ooniprobe. It will store that data +collected from ooniprobes and it will run a series of Test Helpers that assist +`Traffic Manipulation`_ in performing their measurements. + +Test Helpers +............ + +The currently implemented test helpers are the following: + + * ::`SSL Test Helpers: + https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/oonib/testhelpers/s…. + + * ::`HTTP Test Helpers: + https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/oonib/testhelpers/h…. + + * ::`TCP Test Helpers: + https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/oonib/testhelpers/t…. + + * ::`DNS Test Helpers: + https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/oonib/testhelpers/d…. -Contents -******** +More developer documentation +***************************** .. toctree:: :maxdepth: 2 :glob: - tests/* oonib install writing_tests diff --git a/nettests/experimental/daphn3.py b/nettests/experimental/daphn3.py deleted file mode 100644 index 09279fa..0000000 --- a/nettests/experimental/daphn3.py +++ /dev/null @@ -1,119 +0,0 @@ -# -*- encoding: utf-8 -*- -from twisted.python import usage -from twisted.internet import protocol, endpoints, reactor - -from ooni import nettest -from ooni.kit import daphn3 -from ooni.utils import log - -class Daphn3ClientProtocol(daphn3.Daphn3Protocol): - def nextStep(self): - log.debug("Moving on to next step in the state walk") - self.current_data_received = 0 - if self.current_step >= (len(self.steps) - 1): - log.msg("Reached the end of the state machine") - log.msg("Censorship fingerpint bisected!") - step_idx, mutation_idx = self.factory.mutation - log.msg("step_idx: %s | mutation_id: %s" % (step_idx, mutation_idx)) - #self.transport.loseConnection() - if self.report: - self.report['mutation_idx'] = mutation_idx - self.report['step_idx'] = step_idx - self.d.callback(None) - return - else: - self.current_step += 1 - if self._current_step_role() == self.role: - # We need to send more data because we are again responsible for - # doing so. - self.sendPayload() - - -class Daphn3ClientFactory(protocol.ClientFactory): - protocol = daphn3.Daphn3Protocol - mutation = [0,0] - steps = None - - def buildProtocol(self, addr): - p = self.protocol() - p.steps = self.steps - p.factory = self - return p - - def startedConnecting(self, connector): - log.msg("Started connecting %s" % connector) - - def clientConnectionFailed(self, reason, connector): - log.err("We failed connecting the the OONIB") - log.err("Cannot perform test. Perhaps it got blocked?") - log.err("Please report this to tor-assistants(a)torproject.org") - - def clientConnectionLost(self, reason, connector): - log.err("Daphn3 client connection lost") - print reason - -class daphn3Args(usage.Options): - optParameters = [ - ['host', 'h', '127.0.0.1', 'Target Hostname'], - ['port', 'p', 57003, 'Target port number']] - - optFlags = [['pcap', 'c', 'Specify that the input file is a pcap file'], - ['yaml', 'y', 'Specify that the input file is a YAML file (default)']] - -class daphn3Test(nettest.NetTestCase): - - name = "Daphn3" - usageOptions = daphn3Args - inputFile = ['file', 'f', None, - 'Specify the pcap or YAML file to be used as input to the test'] - - #requiredOptions = ['file'] - - steps = None - - def inputProcessor(self, filename): - """ - step_idx is the step in the packet exchange - ex. - [.X.] are packets sent by a client or a server - - client: [.1.] [.3.] [.4.] - server: [.2.] [.5.] - - mutation_idx: is the sub index of the packet as in the byte of the - packet at the step_idx that is to be mutated - - """ - if self.localOptions['pcap']: - daphn3Steps = daphn3.read_pcap(filename) - else: - daphn3Steps = daphn3.read_yaml(filename) - log.debug("Loaded these steps %s" % daphn3Steps) - yield daphn3Steps - - def test_daphn3(self): - host = self.localOptions['host'] - port = int(self.localOptions['port']) - - def failure(failure): - log.msg("Failed to connect") - self.report['censored'] = True - self.report['mutation'] = 0 - raise Exception("Error in connection, perhaps the backend is censored") - return - - def success(protocol): - log.msg("Successfully connected") - protocol.sendPayload() - return protocol.d - - log.msg("Connecting to %s:%s" % (host, port)) - endpoint = endpoints.TCP4ClientEndpoint(reactor, host, port) - daphn3_factory = Daphn3ClientFactory() - daphn3_factory.steps = self.input - daphn3_factory.report = self.report - d = endpoint.connect(daphn3_factory) - d.addErrback(failure) - d.addCallback(success) - return d - diff --git a/nettests/manipulation/captiveportal.py b/nettests/manipulation/captiveportal.py index be8da27..082960e 100644 --- a/nettests/manipulation/captiveportal.py +++ b/nettests/manipulation/captiveportal.py @@ -1,34 +1,33 @@ # -*- coding: utf-8 -*- -""" - captiveportal - ************* - - This test is a collection of tests to detect the presence of a - captive portal. Code is taken, in part, from the old ooni-probe, - which was written by Jacob Appelbaum and Arturo Filastò. - - This module performs multiple tests that match specific vendor captive - portal tests. This is a basic internet captive portal filter tester written - for RECon 2011. - - Read the following URLs to understand the captive portal detection process - for various vendors: - - http://technet.microsoft.com/en-us/library/cc766017%28WS.10%29.aspx - http://blog.superuser.com/2011/05/16/windows-7-network-awareness/ - http://isc.sans.org/diary.html?storyid=10312& - http://src.chromium.org/viewvc/chrome?view=rev&revision=74608 - http://code.google.com/p/chromium-os/issues/detail?3281ttp, - http://crbug.com/52489 - http://crbug.com/71736 - https://bugzilla.mozilla.org/show_bug.cgi?id=562917 - https://bugzilla.mozilla.org/show_bug.cgi?id=603505 - http://lists.w3.org/Archives/Public/ietf-http-wg/2011JanMar/0086.html - http://tools.ietf.org/html/draft-nottingham-http-portal-02 - - :copyright: (c) 2012 Isis Lovecruft - :license: see LICENSE for more details -""" +# captiveportal +# ************* +# +# This test is a collection of tests to detect the presence of a +# captive portal. Code is taken, in part, from the old ooni-probe, +# which was written by Jacob Appelbaum and Arturo Filastò. +# +# This module performs multiple tests that match specific vendor captive +# portal tests. This is a basic internet captive portal filter tester written +# for RECon 2011. +# +# Read the following URLs to understand the captive portal detection process +# for various vendors: +# +# http://technet.microsoft.com/en-us/library/cc766017%28WS.10%29.aspx +# http://blog.superuser.com/2011/05/16/windows-7-network-awareness/ +# http://isc.sans.org/diary.html?storyid=10312& +# http://src.chromium.org/viewvc/chrome?view=rev&revision=74608 +# http://code.google.com/p/chromium-os/issues/detail?3281ttp, +# http://crbug.com/52489 +# http://crbug.com/71736 +# https://bugzilla.mozilla.org/show_bug.cgi?id=562917 +# https://bugzilla.mozilla.org/show_bug.cgi?id=603505 +# http://lists.w3.org/Archives/Public/ietf-http-wg/2011JanMar/0086.html +# http://tools.ietf.org/html/draft-nottingham-http-portal-02 +# +# :authors: Jacob Appelbaum, Arturo Filastò, Isis Lovecruft +# :license: see LICENSE for more details + import base64 import os import random diff --git a/nettests/manipulation/daphne.py b/nettests/manipulation/daphne.py new file mode 100644 index 0000000..09279fa --- /dev/null +++ b/nettests/manipulation/daphne.py @@ -0,0 +1,119 @@ +# -*- encoding: utf-8 -*- +from twisted.python import usage +from twisted.internet import protocol, endpoints, reactor + +from ooni import nettest +from ooni.kit import daphn3 +from ooni.utils import log + +class Daphn3ClientProtocol(daphn3.Daphn3Protocol): + def nextStep(self): + log.debug("Moving on to next step in the state walk") + self.current_data_received = 0 + if self.current_step >= (len(self.steps) - 1): + log.msg("Reached the end of the state machine") + log.msg("Censorship fingerpint bisected!") + step_idx, mutation_idx = self.factory.mutation + log.msg("step_idx: %s | mutation_id: %s" % (step_idx, mutation_idx)) + #self.transport.loseConnection() + if self.report: + self.report['mutation_idx'] = mutation_idx + self.report['step_idx'] = step_idx + self.d.callback(None) + return + else: + self.current_step += 1 + if self._current_step_role() == self.role: + # We need to send more data because we are again responsible for + # doing so. + self.sendPayload() + + +class Daphn3ClientFactory(protocol.ClientFactory): + protocol = daphn3.Daphn3Protocol + mutation = [0,0] + steps = None + + def buildProtocol(self, addr): + p = self.protocol() + p.steps = self.steps + p.factory = self + return p + + def startedConnecting(self, connector): + log.msg("Started connecting %s" % connector) + + def clientConnectionFailed(self, reason, connector): + log.err("We failed connecting the the OONIB") + log.err("Cannot perform test. Perhaps it got blocked?") + log.err("Please report this to tor-assistants(a)torproject.org") + + def clientConnectionLost(self, reason, connector): + log.err("Daphn3 client connection lost") + print reason + +class daphn3Args(usage.Options): + optParameters = [ + ['host', 'h', '127.0.0.1', 'Target Hostname'], + ['port', 'p', 57003, 'Target port number']] + + optFlags = [['pcap', 'c', 'Specify that the input file is a pcap file'], + ['yaml', 'y', 'Specify that the input file is a YAML file (default)']] + +class daphn3Test(nettest.NetTestCase): + + name = "Daphn3" + usageOptions = daphn3Args + inputFile = ['file', 'f', None, + 'Specify the pcap or YAML file to be used as input to the test'] + + #requiredOptions = ['file'] + + steps = None + + def inputProcessor(self, filename): + """ + step_idx is the step in the packet exchange + ex. + [.X.] are packets sent by a client or a server + + client: [.1.] [.3.] [.4.] + server: [.2.] [.5.] + + mutation_idx: is the sub index of the packet as in the byte of the + packet at the step_idx that is to be mutated + + """ + if self.localOptions['pcap']: + daphn3Steps = daphn3.read_pcap(filename) + else: + daphn3Steps = daphn3.read_yaml(filename) + log.debug("Loaded these steps %s" % daphn3Steps) + yield daphn3Steps + + def test_daphn3(self): + host = self.localOptions['host'] + port = int(self.localOptions['port']) + + def failure(failure): + log.msg("Failed to connect") + self.report['censored'] = True + self.report['mutation'] = 0 + raise Exception("Error in connection, perhaps the backend is censored") + return + + def success(protocol): + log.msg("Successfully connected") + protocol.sendPayload() + return protocol.d + + log.msg("Connecting to %s:%s" % (host, port)) + endpoint = endpoints.TCP4ClientEndpoint(reactor, host, port) + daphn3_factory = Daphn3ClientFactory() + daphn3_factory.steps = self.input + daphn3_factory.report = self.report + d = endpoint.connect(daphn3_factory) + d.addErrback(failure) + d.addCallback(success) + return d + diff --git a/oonib/testhelpers/http_helpers.py b/oonib/testhelpers/http_helpers.py index 6415291..3a76b9a 100644 --- a/oonib/testhelpers/http_helpers.py +++ b/oonib/testhelpers/http_helpers.py @@ -26,7 +26,7 @@ class SimpleHTTPChannel(basic.LineReceiver, policies.TimeoutMixin): The returned JSON dict looks like so: { - 'request_headers': + 'request_headers': [['User-Agent', 'IE6'], ['Content-Length', 200]] 'request_line': 'GET / HTTP/1.1'

1 0

r25923: {website} link to the overall projects page. (website/trunk/en)
by Andrew Lewman 29 Nov '12

29 Nov '12

Author: phobos Date: 2012-11-29 16:25:02 +0000 (Thu, 29 Nov 2012) New Revision: 25923 Modified: website/trunk/en/index.wml Log: link to the overall projects page. Modified: website/trunk/en/index.wml =================================================================== --- website/trunk/en/index.wml 2012-11-29 16:21:07 UTC (rev 25922) +++ website/trunk/en/index.wml 2012-11-29 16:25:02 UTC (rev 25923) @@ -147,6 +147,7 @@ </td> </tr> </table> + <span class="continue"><a href="<page projects/projects>">Learn more about our projects »</a></span> </div>  </div>

1 0

r25922: {website} fix mistaken \ breaking a comment in the wml. (website/trunk/projects/en)
by Andrew Lewman 29 Nov '12

29 Nov '12

Author: phobos Date: 2012-11-29 16:21:07 +0000 (Thu, 29 Nov 2012) New Revision: 25922 Modified: website/trunk/projects/en/projects.wml Log: fix mistaken \ breaking a comment in the wml. Modified: website/trunk/projects/en/projects.wml =================================================================== --- website/trunk/projects/en/projects.wml 2012-11-28 15:45:11 UTC (rev 25921) +++ website/trunk/projects/en/projects.wml 2012-11-29 16:21:07 UTC (rev 25922) @@ -1,4 +1,4 @@ -/## translation metadata +## translation metadata # Revision: $Revision$ # Translation-Priority: 3-low

1 0

[translation/liveusb-creator] Update translations for liveusb-creator
by translation＠torproject.org 29 Nov '12

29 Nov '12

commit b928b112cdef9501d52a7407f9b8d84ed1421e8d Author: Translation commit bot <translation(a)torproject.org> Date: Thu Nov 29 16:15:55 2012 +0000 Update translations for liveusb-creator --- eu/eu.po | 14 +++++++------- 1 files changed, 7 insertions(+), 7 deletions(-) diff --git a/eu/eu.po b/eu/eu.po index fa101de..a24e6c5 100644 --- a/eu/eu.po +++ b/eu/eu.po @@ -10,8 +10,8 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2012-04-13 14:16+0200\n" -"PO-Revision-Date: 2012-11-23 18:31+0000\n" -"Last-Translator: totorika93 <totorika93(a)gmail.com>\n" +"PO-Revision-Date: 2012-11-29 16:07+0000\n" +"Last-Translator: Antxon Baldarra <baldarra(a)euskalerria.org>\n" "Language-Team: LANGUAGE <LL(a)li.org>\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" @@ -38,7 +38,7 @@ msgstr "Gailua txikiegia da: gutxienez %s MiBekoa izan behar da." msgid "" "Error: Cannot set the label or obtain the UUID of your device. Unable to " "continue." -msgstr "" +msgstr "Akatsa: Ezin da etiketa ezarri edo zure gailuaren UUIDa eskuratu. Ezin da jarraitu." #: ../liveusb/gui.py:276 #, python-format @@ -54,11 +54,11 @@ msgid "" "Warning: This tool needs to be run as an Administrator. To do this, right " "click on the icon and open the Properties. Under the Compatibility tab, " "check the \"Run this program as an administrator\" box." -msgstr "" +msgstr "Abisua: Tresna hau Administratzaile bezala exekutatu behar da. Hau egiteko, sakatu eskuinarekin ikonoan eta ireki Propietateak. Bateragarritasuna fitxan, markatu \"Programa hau administratzale bezala exekutatu\" kutxa." #: ../liveusb/gui.py:413 msgid "Unable to find any USB drive" -msgstr "" +msgstr "Ezin da USB gailurik bilatu." #: ../liveusb/gui.py:521 msgid "" @@ -73,7 +73,7 @@ msgstr "%(device)s gailuan ez dago leku askerik." #: ../liveusb/gui.py:533 msgid "Partition is FAT16; Restricting overlay size to 2G" -msgstr "" +msgstr "Partizioa FAT16 da; Gainjarri neurria 2Gtara mugatzen" #: ../liveusb/gui.py:577 msgid "Persistent Storage" @@ -235,7 +235,7 @@ msgstr "" #: ../liveusb/creator.py:497 msgid "Removing existing Live OS" -msgstr "" +msgstr "Dagoen Live OS kentzen" #: ../liveusb/creator.py:506 ../liveusb/creator.py:517 #, python-format

1 0

[ooni-probe/master] Write documentation for HTTP Invalid Requests test
by art＠torproject.org 29 Nov '12

29 Nov '12

commit d67eecfd05f03002e3a9334db82ab2d8c2658368 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Nov 29 17:09:34 2012 +0100 Write documentation for HTTP Invalid Requests test --- docs/source/tests/http_invalid_requests.rst | 164 ++++++++++++++++++++ nettests/manipulation/http_invalid_request_line.py | 19 ++- ooni/templates/tcpt.py | 2 +- requirements.txt | 2 +- 4 files changed, 177 insertions(+), 10 deletions(-) diff --git a/docs/source/tests/http_invalid_requests.rst b/docs/source/tests/http_invalid_requests.rst new file mode 100644 index 0000000..f14a18d --- /dev/null +++ b/docs/source/tests/http_invalid_requests.rst @@ -0,0 +1,164 @@ +Details +======= + +*Test Name*: HTTP Invalid Request Line + +*Current version*: 0.1.3 + +*NetTest*: HTTP Invalid Request Line (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/manipulati…) + +*Test Helper*: TCPEchoHelper (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/testhelpers/t…) + +*Test Type*: Traffic Manipulation + +*Requires Root*: No + +*WARNING*: This test is more dangerous to run than any other one and you +should do it only if you know what you are doing. + +Description +=========== + +The goal of this test is to do some very basic and not very noisy fuzzing +on the HTTP request line. We generate a series of requests that are not +valid HTTP requests. + +The remote backend runs a TCP echo server. If the response from the backend +does not match with what we have sent then we say that tampering is occuring. + +The idea behind this is that certain transparent HTTP proxies may not be +properly parsing the HTTP request line. + +Unless elsewhere stated 'Xx'\*N refers to N\*2 random upper or lowercase ascii +letters or numbers ('XxXx' will be 4). + +What we fuzz is the following: + +test random invalid method +************************** + +This sends random 4 letter HTTP request method. + +The request on the wire will look like this: + +:: + XxXxX / HTTP/1.1\n\r + + +test random invalid field count +****************************** + +This generates a request that looks like this: + +:: + XxXxX XxXxX XxXxX XxXxX + + This may trigger some bugs in the HTTP parsers of transparent HTTP + proxies. + + +test random big request method +****************************** + +This generates a request that looks like this: + +:: + + Xx*512 / HTTP/1.1 + + +test random invalid version number +********************************** + +This generates a request that looks like this: + +:: + GET / HTTP/XxX + +This attemps to trigger bugs in the parsing of the HTTP version number, that +is usually being split on the `.`. + +How to run the test +=================== + +`./bin/ooniprobe nettests/manipulation/http_invalid_request_line.py -b <address of backend>` + +*address of the backend* is the hostname or IP address of a backend that runs +a TCP echo server on port 80. + +Sample report +============= + +From running: + +`./bin/ooniprobe nettests/manipulation/http_invalid_request_line.py -b 127.0.0.1 -p 57002` + +:: + + ########################################### + # OONI Probe Report for HTTP Invalid Request Line test + # Thu Nov 29 16:41:15 2012 + ########################################### + --- + options: + collector: null + help: 0 + logfile: null + pcapfile: null + reportfile: null + resume: 0 + subargs: [-b, 127.0.0.1, -p, '57002'] + test: nettests/manipulation/http_invalid_request_line.py + probe_asn: null + probe_cc: null + probe_ip: 127.0.0.1 + software_name: ooniprobe + software_version: 0.0.7.1-alpha + start_time: 1354200075.0 + test_name: HTTP Invalid Request Line + test_version: 0.1.3 + ... + --- + input: null + report: + received: ["L3F6 / HTTP/1.1\n\r"] + sent: ["L3F6 / HTTP/1.1\n\r"] + tampering: false + test_name: test_random_invalid_method + test_runtime: 5.011919021606445 + test_started: 1354203675.481258 + ... + --- + input: null + report: + received: ["GET / HTTP/Ayo\n\r"] + sent: ["GET / HTTP/Ayo\n\r"] + tampering: false + test_name: test_random_invalid_version_number + test_runtime: 5.019288063049316 + test_started: 1354203675.48221 + ... + --- + input: null + report: + received: ["RZRcu OyLtu Jtu2T cs0ER\n\r"] + sent: ["RZRcu OyLtu Jtu2T cs0ER\n\r"] + tampering: false + test_name: test_random_invalid_field_count + test_runtime: 5.022854804992676 + test_started: 1354203675.483127 + ... + --- + input: null + report: + received: ["iB2HrwJeB512y5CrAAaIpETZ5RMprmGO4RCex24Kmqkjguc2XsOrGXR3qJIERw1IX13uGWVs1kOd96Y0zsR4ufcktGFnP0gYakTq7GA63rNNOmlG9tNXABnBEHfkHYhrwdrewAKdqZ2lGls66NBY7fbL9xkOsHBjXq7TkvS8MOeTb74wcxKymp1NLOa8u7C8XP9qpafQtWBrC4dkEVjppjlFyetg1tu8zomDUCBx9y2tB411d8lw7WOSMfDiQWG327aVanxaOj9WgZ2u2eu0595UiJxZxMk1LYa96vHvmB1DrX0DoJUkUg2fcEOlia3pVoFfcpbd3TQ0GwEoewU3F48Qvpf2AuOIcPgrYa1XszLCyUoToygc5J6WgIH7f1phsEpmmZ6my9KChZdnazc7mNbCwLB3Z1wMwcoFW1XuDvAhTr8OYoY17360SYkAqEBhYh8uiD4xdIq0T0KJzsJW5wghjCMjRjFyfk3wyDXaPLp7vkeeqbA3FNFatFQlUCTIkqqqM3EjAojynvCDVyrThGlmsauS9Ejhc9TaDojeT3s8HZY3KIDHwCRvpGgpwFie2NyhzmwY2y4YFMcnGXT2jlwsVE1K2fZZ7yGbKqbciuh5Q32JNvQBdGy3N63PWyZIxWhfythHOVBE9GXAr99DqEXUeiRkfNer5aVRt87rDBuLx13IrbCE6runHdoZEq0iXs7IySfw5qqt00fExbaN9UlSDyxyYIrIymrSTSUWWprTqiezn1toQZpxEl53pv4ZnLx00CSMYxWTzMDNSHLpovdAI9A27ugY2uHmPETKHbaLsTzNRtNiZAS9WidDi0oAxdICB47dnJA2CXvBaXeq2nsRasgQ9qqNl0RbTC8vhXgIq8nZZQn0iehXsP5mPXshpALyfAhoLwiChRif8FODkKdAgciLviTvyadpecb6HqRVfM8rLzMPkuXT2I21yB7NVQQLraJOeDO TvmtsV2gw2I2ON8xKtHO78VQfo357yBryndTQTRTk9FPnijwcwusn\ + \ / HTTP/1.1\n\r"] + sent: ["iB2HrwJeB512y5CrAAaIpETZ5RMprmGO4RCex24Kmqkjguc2XsOrGXR3qJIERw1IX13uGWVs1kOd96Y0zsR4ufcktGFnP0gYakTq7GA63rNNOmlG9tNXABnBEHfkHYhrwdrewAKdqZ2lGls66NBY7fbL9xkOsHBjXq7TkvS8MOeTb74wcxKymp1NLOa8u7C8XP9qpafQtWBrC4dkEVjppjlFyetg1tu8zomDUCBx9y2tB411d8lw7WOSMfDiQWG327aVanxaOj9WgZ2u2eu0595UiJxZxMk1LYa96vHvmB1DrX0DoJUkUg2fcEOlia3pVoFfcpbd3TQ0GwEoewU3F48Qvpf2AuOIcPgrYa1XszLCyUoToygc5J6WgIH7f1phsEpmmZ6my9KChZdnazc7mNbCwLB3Z1wMwcoFW1XuDvAhTr8OYoY17360SYkAqEBhYh8uiD4xdIq0T0KJzsJW5wghjCMjRjFyfk3wyDXaPLp7vkeeqbA3FNFatFQlUCTIkqqqM3EjAojynvCDVyrThGlmsauS9Ejhc9TaDojeT3s8HZY3KIDHwCRvpGgpwFie2NyhzmwY2y4YFMcnGXT2jlwsVE1K2fZZ7yGbKqbciuh5Q32JNvQBdGy3N63PWyZIxWhfythHOVBE9GXAr99DqEXUeiRkfNer5aVRt87rDBuLx13IrbCE6runHdoZEq0iXs7IySfw5qqt00fExbaN9UlSDyxyYIrIymrSTSUWWprTqiezn1toQZpxEl53pv4ZnLx00CSMYxWTzMDNSHLpovdAI9A27ugY2uHmPETKHbaLsTzNRtNiZAS9WidDi0oAxdICB47dnJA2CXvBaXeq2nsRasgQ9qqNl0RbTC8vhXgIq8nZZQn0iehXsP5mPXshpALyfAhoLwiChRif8FODkKdAgciLviTvyadpecb6HqRVfM8rLzMPkuXT2I21yB7NVQQLraJOeDOTvmt sV2gw2I2ON8xKtHO78VQfo357yBryndTQTRTk9FPnijwcwusn\ + \ / HTTP/1.1\n\r"] + tampering: false + test_name: test_random_big_request_method + test_runtime: 5.026142120361328 + test_started: 1354203675.484211 + ... + + diff --git a/nettests/manipulation/http_invalid_request_line.py b/nettests/manipulation/http_invalid_request_line.py index 00ab24d..01626ab 100644 --- a/nettests/manipulation/http_invalid_request_line.py +++ b/nettests/manipulation/http_invalid_request_line.py @@ -1,12 +1,14 @@ # -*- encoding: utf-8 -*- from twisted.python import usage -from ooni.utils import randomStr +from ooni.utils import log +from ooni.utils import randomStr, randomSTR from ooni.templates import tcpt class UsageOptions(usage.Options): optParameters = [['backend', 'b', '127.0.0.1', - 'The OONI backend that runs a TCP echo server']] + 'The OONI backend that runs a TCP echo server'], + ['backendport', 'p', 80, 'Specify the port that the TCP echo server is running (should only be set for debugging)']] class HTTPInvalidRequestLine(tcpt.TCPTest): """ @@ -14,10 +16,10 @@ class HTTPInvalidRequestLine(tcpt.TCPTest): on the HTTP request line. We generate a series of requests that are not valid HTTP requests. - Unless elsewhere stated 'Xx'*N refers to N*2 random upper or lowercase ascii - letters or numbers ('XxXx' will be 4). + Unless elsewhere stated 'Xx'*N refers to N*2 random upper or lowercase + ascii letters or numbers ('XxXx' will be 4). """ - name = "HTTP Invalid Requests" + name = "HTTP Invalid Request Line" version = "0.1.3" authors = "Arturo Filastò" @@ -28,14 +30,15 @@ class HTTPInvalidRequestLine(tcpt.TCPTest): requiredOptions = ['backend'] def setUp(self): - self.port = 80 + self.port = int(self.localOptions['backendport']) self.address = self.localOptions['backend'] def check_for_manipulation(self, response, payload): + log.debug("Checking if %s == %s" % (response, payload)) if response != payload: self.report['tampering'] = True else: - self.report['tampering'] = 'unknown' + self.report['tampering'] = False def test_random_invalid_method(self): """ @@ -57,7 +60,7 @@ class HTTPInvalidRequestLine(tcpt.TCPTest): Proxy-Connection: close """ - payload = randomStr(10) + " / HTTP/1.1\n\r" + payload = randomSTR(4) + " / HTTP/1.1\n\r" d = self.sendPayload(payload) d.addCallback(self.check_for_manipulation, payload) diff --git a/ooni/templates/tcpt.py b/ooni/templates/tcpt.py index 26f38ed..323ead8 100644 --- a/ooni/templates/tcpt.py +++ b/ooni/templates/tcpt.py @@ -68,7 +68,7 @@ class TCPTest(NetTestCase): self.report['received'].append(proto.received_data) proto.transport.loseConnection() log.debug("Closing connection") - d1.callback(self.report['received']) + d1.callback(proto.received_data) def timedOut(proto): self.report['failure'] = 'tcp_timed_out_error' diff --git a/requirements.txt b/requirements.txt index 4ddb915..ce2e721 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,7 +14,7 @@ repoze.sphinx.autointerface>=0.7.1 https://hg.secdev.org/scapy/archive/tip.zip#egg=scapy storm>=0.19 transaction>=1.3.0 -txtorcon +txtorcon>=0.7 wsgiref>=0.1.2 zope.component>=4.0.0 zope.event>=4.0.0

1 0

[ooni-probe/master] Make getDiff take as required argument a TrueHeaders object instead of a dict
by art＠torproject.org 29 Nov '12

29 Nov '12

commit a3025bf24be234c6fbb6b9fad251ad5fa0ef4e6f Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Nov 29 13:13:46 2012 +0100 Make getDiff take as required argument a TrueHeaders object instead of a dict --- ooni/templates/httpt.py | 3 ++- ooni/utils/txagentwithsocks.py | 20 +++++++++++++------- tests/test_trueheaders.py | 8 ++++---- 3 files changed, 19 insertions(+), 12 deletions(-) diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index dcd8ff9..804a3e4 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -115,9 +115,10 @@ class HTTPTest(NetTestCase): failure (instance): An instance of :class:twisted.internet.failure.Failure """ log.debug("Adding %s to report" % request) + request_headers = TrueHeaders(request['headers']) request_response = { 'request': { - 'headers': request['headers'], + 'headers': list(request_headers.getAllRawHeaders()), 'body': request['body'], 'url': request['url'], 'method': request['method'] diff --git a/ooni/utils/txagentwithsocks.py b/ooni/utils/txagentwithsocks.py index ffaa733..7c34288 100644 --- a/ooni/utils/txagentwithsocks.py +++ b/ooni/utils/txagentwithsocks.py @@ -149,18 +149,25 @@ class TrueHeaders(http_headers.Headers): self._rawHeaders[name.lower()]['name'] = name self._rawHeaders[name.lower()]['values'] = values - def getDiff(self, header_dict, ignore=[]): + def getDiff(self, headers, ignore=[]): """ - ignore: specify a list of header fields to ignore - Returns a set containing the header names that are not present in - header_dict or not present in self. + Args: + + headers: a TrueHeaders object + + ignore: specify a list of header fields to ignore + + Returns: + + a set containing the header names that are not present in + header_dict or not present in self. """ diff = set() field_names = [] headers_a = copy(self) - headers_b = TrueHeaders(header_dict) + headers_b = copy(headers) for name in ignore: try: del headers_a._rawHeaders[name.lower()] @@ -176,8 +183,7 @@ class TrueHeaders(http_headers.Headers): field_names.append(k) for name in field_names: - if self.getRawHeaders(name) and \ - name in header_dict: + if self.getRawHeaders(name) and headers.getRawHeaders(name): pass else: diff.add(name) diff --git a/tests/test_trueheaders.py b/tests/test_trueheaders.py index 33521b8..9ac0a27 100644 --- a/tests/test_trueheaders.py +++ b/tests/test_trueheaders.py @@ -23,18 +23,18 @@ dummy_headers_dict3 = { class TestTrueHeaders(unittest.TestCase): def test_names_match(self): th = TrueHeaders(dummy_headers_dict) - self.assertEqual(th.getDiff(dummy_headers_dict), set()) + self.assertEqual(th.getDiff(TrueHeaders(dummy_headers_dict)), set()) def test_names_not_match(self): th = TrueHeaders(dummy_headers_dict) - self.assertEqual(th.getDiff(dummy_headers_dict2), set(['Header3'])) + self.assertEqual(th.getDiff(TrueHeaders(dummy_headers_dict2)), set(['Header3'])) th = TrueHeaders(dummy_headers_dict3) - self.assertEqual(th.getDiff(dummy_headers_dict2), set(['Header3', 'Header4'])) + self.assertEqual(th.getDiff(TrueHeaders(dummy_headers_dict2)), set(['Header3', 'Header4'])) def test_names_match_expect_ignore(self): th = TrueHeaders(dummy_headers_dict) - self.assertEqual(th.getDiff(dummy_headers_dict2, ignore=['Header3']), set()) + self.assertEqual(th.getDiff(TrueHeaders(dummy_headers_dict2), ignore=['Header3']), set())

1 0

[ooni-probe/master] Add documentation for DNS Tamper test
by art＠torproject.org 29 Nov '12

29 Nov '12

commit 340e4b932f0817f552f12c2900039020fe2eba63 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Nov 29 12:26:50 2012 +0100 Add documentation for DNS Tamper test --- docs/source/tests/dnstamper.rst | 229 +++++++++++++++++++++++---------------- 1 files changed, 137 insertions(+), 92 deletions(-) diff --git a/docs/source/tests/dnstamper.rst b/docs/source/tests/dnstamper.rst index d37afa8..b97131b 100644 --- a/docs/source/tests/dnstamper.rst +++ b/docs/source/tests/dnstamper.rst @@ -3,11 +3,11 @@ Details *Test Name*: DNS Tamper -*Current version*: 0.3 +*Current version*: 0.4 -*NetTest*: DNSTamperTest (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/core/dnsta…) +*NetTest*: DNS Tamper Test (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/nettests/blocking/d…) -*Test Helper*: DNSTestHelper (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/testhelpers/d…) +*Test Helper*: DNS Test Helper (https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/testhelpers/d…) *Test Type*: Content Blocking @@ -28,7 +28,9 @@ youtube, etc. How to run the test =================== -`./bin/ooniprobe nettests/core/dnstamper.py -t <test resolvers file> -f <input file> -b IP:PORT` +`./bin/ooniprobe nettests/core/dnstamper.py [-t <test resolvers>|-T <test resolver file>-f <input file> -b IP:PORT` + +*test resolvers* is a single test resolver (IP address) *test resolvers file* is a file containing the IP addresses of the resolvers to test for censorship, one per line. @@ -44,92 +46,135 @@ From running: :: - ########################################### - # OONI Probe Report for DNS tamper test - # Tue Nov 20 20:38:54 2012 - ########################################### - --- - {probe_asn: null, probe_cc: null, probe_ip: 127.0.0.1, software_name: ooniprobe, software_version: 0.0.7.1-alpha, - start_time: 1353436734.0, test_name: DNS tamper, test_version: '0.3'} - ... - --- - input: torproject.org - report: - control_resolver: &id001 [8.8.8.8, 53] - queries: - - addrs: [86.59.30.40, 38.229.72.14, 38.229.72.16, 82.195.75.101] - answers: - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=86.59.30.40 - ttl=142>] - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=38.229.72.14 - ttl=142>] - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=38.229.72.16 - ttl=142>] - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=82.195.75.101 - ttl=142>] - query: '[Query(''torproject.org'', 1, 1)]' - query_type: A - resolver: *id001 - - addrs: [86.59.30.40, 38.229.72.14, 38.229.72.16, 82.195.75.101] - answers: - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=86.59.30.40 - ttl=142>] - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=38.229.72.14 - ttl=142>] - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=38.229.72.16 - ttl=142>] - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=82.195.75.101 - ttl=142>] - query: '[Query(''torproject.org'', 1, 1)]' - query_type: A - resolver: [8.8.8.8, 53] - - addrs: [86.59.30.40, 38.229.72.14, 38.229.72.16, 82.195.75.101] - answers: - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=86.59.30.40 - ttl=142>] - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=38.229.72.14 - ttl=142>] - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=38.229.72.16 - ttl=142>] - - [<RR name=torproject.org type=A class=IN ttl=142s auth=False>, <A address=82.195.75.101 - ttl=142>] - query: '[Query(''torproject.org'', 1, 1)]' - query_type: A - resolver: [8.8.4.4, 53] - tampering: {8.8.4.4: false, 8.8.8.8: false} - test_resolvers: [8.8.8.8, 8.8.4.4] - test_name: test_a_queries - test_started: 1353440334.075345 - ... - --- - input: ooni.nu - report: - control_resolver: &id001 [8.8.8.8, 53] - queries: - - addrs: [178.79.139.176] - answers: - - [<RR name=ooni.nu type=A class=IN ttl=1478s auth=False>, <A address=178.79.139.176 - ttl=1478>] - query: '[Query(''ooni.nu'', 1, 1)]' - query_type: A - resolver: *id001 - - addrs: [178.79.139.176] - answers: - - [<RR name=ooni.nu type=A class=IN ttl=1478s auth=False>, <A address=178.79.139.176 - ttl=1478>] - query: '[Query(''ooni.nu'', 1, 1)]' - query_type: A - resolver: [8.8.8.8, 53] - - addrs: [178.79.139.176] - answers: - - [<RR name=ooni.nu type=A class=IN ttl=1478s auth=False>, <A address=178.79.139.176 - ttl=1478>] - query: '[Query(''ooni.nu'', 1, 1)]' - query_type: A - resolver: [8.8.4.4, 53] - tampering: {8.8.4.4: false, 8.8.8.8: false} - test_resolvers: [8.8.8.8, 8.8.4.4] - test_name: test_a_queries - test_started: 1353440334.077116 - ... + ########################################### + # OONI Probe Report for DNS tamper test + # Thu Nov 29 12:17:19 2012 + ########################################### + --- + options: + collector: null + help: 0 + logfile: null + pcapfile: null + reportfile: null + resume: 0 + subargs: [-t, 8.8.8.8, -f, test_inputs/dns_tamper_file.txt] + test: nettests/blocking/dnstamper.py + probe_asn: null + probe_cc: null + probe_ip: 127.0.0.1 + software_name: ooniprobe + software_version: 0.0.7.1-alpha + start_time: 1354184239.0 + test_name: DNS tamper + test_version: '0.4' + ... + --- + input: torproject.org + report: + control_resolver: &id001 [8.8.8.8, 53] + queries: + - addrs: [86.59.30.40, 38.229.72.14, 38.229.72.16, 82.195.75.101] + answers: + - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=86.59.30.40 + ttl=91>] + - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.14 + ttl=91>] + - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.16 + ttl=91>] + - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=82.195.75.101 + ttl=91>] + query: '[Query(''torproject.org'', 1, 1)]' + query_type: A + resolver: *id001 + - addrs: [86.59.30.40, 38.229.72.14, 38.229.72.16, 82.195.75.101] + answers: + - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=86.59.30.40 + ttl=91>] + - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.14 + ttl=91>] + - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=38.229.72.16 + ttl=91>] + - [<RR name=torproject.org type=A class=IN ttl=91s auth=False>, <A address=82.195.75.101 + ttl=91>] + query: '[Query(''torproject.org'', 1, 1)]' + query_type: A + resolver: [8.8.8.8, 53] + tampering: {8.8.8.8: false} + test_name: test_a_lookup + test_runtime: 0.0733950138092041 + test_started: 1354187839.508863 + ... + --- + input: google.com + report: + control_resolver: &id001 [8.8.8.8, 53] + queries: + - addrs: [173.194.69.100, 173.194.69.139, 173.194.69.113, 173.194.69.101, 173.194.69.138, + 173.194.69.102] + answers: + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.100 + ttl=54>] + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.139 + ttl=54>] + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.113 + ttl=54>] + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.101 + ttl=54>] + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.138 + ttl=54>] + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.102 + ttl=54>] + query: '[Query(''google.com'', 1, 1)]' + query_type: A + resolver: *id001 + - addrs: [173.194.69.100, 173.194.69.139, 173.194.69.113, 173.194.69.101, 173.194.69.138, + 173.194.69.102] + answers: + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.100 + ttl=54>] + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.139 + ttl=54>] + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.113 + ttl=54>] + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.101 + ttl=54>] + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.138 + ttl=54>] + - [<RR name=google.com type=A class=IN ttl=54s auth=False>, <A address=173.194.69.102 + ttl=54>] + query: '[Query(''google.com'', 1, 1)]' + query_type: A + resolver: [8.8.8.8, 53] + tampering: {8.8.8.8: false} + test_name: test_a_lookup + test_runtime: 0.08325004577636719 + test_started: 1354187839.51091 + ... + --- + input: measurementlab.net + report: + control_resolver: &id001 [8.8.8.8, 53] + queries: + - addrs: [72.249.86.184] + answers: + - [<RR name=measurementlab.net type=A class=IN ttl=600s auth=False>, <A address=72.249.86.184 + ttl=600>] + query: '[Query(''measurementlab.net'', 1, 1)]' + query_type: A + resolver: *id001 + - addrs: [72.249.86.184] + answers: + - [<RR name=measurementlab.net type=A class=IN ttl=600s auth=False>, <A address=72.249.86.184 + ttl=600>] + query: '[Query(''measurementlab.net'', 1, 1)]' + query_type: A + resolver: [8.8.8.8, 53] + tampering: {8.8.8.8: false} + test_name: test_a_lookup + test_runtime: 0.2565779685974121 + test_started: 1354187839.512434 + ... + +Notes: Query is the string repsentation of :class:twisted.names.dns.Query

1 0