- tor-commits - lists.torproject.org

[spec/master] Merge pull request #49 from TheTorProject/feature/df-0.2
by art＠torproject.org 09 May '16

09 May '16

commit 3d3bd21cf45cb070296208bccd00e6e6b30fc90b Merge: 744b8d5 7c82e5d Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Apr 5 14:38:51 2016 +0200 Merge pull request #49 from TheTorProject/feature/df-0.2 Feature/df 0.2 Release-Procedure.md | 18 +- data-formats/df-000-base.md | 174 +- data-formats/df-001-httpt.md | 328 ++-- data-formats/df-002-dnst.md | 270 ++- data-formats/df-003-scapy.md | 663 ++++++- data-formats/df-004-tcpt.md | 63 + oonib.md | 78 +- test-specs/ts-001-bridget.md | 2 + test-specs/ts-002-dns-consistency.md | 247 +++ test-specs/ts-002-dnsconsistency.md | 178 -- test-specs/ts-003-http-requests.md | 321 ++-- test-specs/ts-004-http-host.md | 234 +++ test-specs/ts-004-httphost.md | 257 --- test-specs/ts-005-dns-spoof.md | 151 ++ test-specs/ts-005-dnsspoof.md | 149 -- test-specs/ts-006-header-field-manipulation.md | 186 +- test-specs/ts-007-http-invalid-request-line.md | 129 +- test-specs/ts-008-tcp-connect.md | 96 + test-specs/ts-008-tcpconnect.md | 98 - test-specs/ts-009-multi-port-traceroute.md | 2384 ------------------------ test-specs/ts-009-multi-protocol-traceroute.md | 619 ++++++ test-specs/ts-010-captive-portal.md | 125 +- test-specs/ts-011-bridge-reachability.md | 31 +- test-specs/ts-012-dns-injection.md | 2 +- 24 files changed, 3034 insertions(+), 3769 deletions(-)

1 0

[spec/master] Adjust oonib specification to the test_start_time field
by art＠torproject.org 09 May '16

09 May '16

commit 7c82e5d4ad86a19f828179dfe2048b5ac92f4272 Author: Arturo Filastò <arturo(a)filasto.net> Date: Thu Mar 17 15:59:20 2016 +0100 Adjust oonib specification to the test_start_time field --- oonib.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/oonib.md b/oonib.md index 90d7b99..036aa7d 100644 --- a/oonib.md +++ b/oonib.md @@ -103,9 +103,9 @@ The HTTP request it performs is: `string` that must be either "json" or "yaml" to identify the format of the content. - 'start_time': + 'test_start_time': NEW since oonibackend 1.3 - `float` seconds since epoch in UTC of when the measurement was started. + `string` timestamp in UTC of when the test was started using the format "%Y-%m-%d %H:%M:%S" 'input_hashes': (optional) `list` of hex encoded sha256sum of the contents

1 0

[spec/master] formatting and filename fixes
by art＠torproject.org 09 May '16

09 May '16

commit 29f56af6ba8cfbc6c8d8d4f5729f8c30bf384111 Author: Joe Landers <joe(a)joelanders.net> Date: Tue Apr 5 13:22:44 2016 +0200 formatting and filename fixes --- test-specs/ts-001-bridget.md | 2 +- test-specs/ts-005-dns-spoof.md | 131 ++++++++--------- test-specs/ts-012-lantern.md | 111 --------------- test-specs/ts-013-lantern.md | 111 +++++++++++++++ test-specs/ts-013-meek-fronted-requests.md | 111 --------------- test-specs/ts-014-meek-fronted-requests.md | 111 +++++++++++++++ test-specs/ts-014-psiphon.md | 146 ------------------- test-specs/ts-015-openvpn.md | 218 ---------------------------- test-specs/ts-015-psiphon.md | 148 +++++++++++++++++++ test-specs/ts-016-openvpn.md | 220 +++++++++++++++++++++++++++++ 10 files changed, 657 insertions(+), 652 deletions(-) diff --git a/test-specs/ts-001-bridget.md b/test-specs/ts-001-bridget.md index f800df5..e2b9f47 100644 --- a/test-specs/ts-001-bridget.md +++ b/test-specs/ts-001-bridget.md @@ -55,7 +55,7 @@ Note: This test is deprecated # Privacy considerations - * Bridge location should not be devolved to adversaries. + * Bridge location should not be revealed to adversaries. * If possible, the fact that a scan is running should be difficult to detect. # Packet capture considerations diff --git a/test-specs/ts-005-dns-spoof.md b/test-specs/ts-005-dns-spoof.md index 417dc8b..94f0f3a 100644 --- a/test-specs/ts-005-dns-spoof.md +++ b/test-specs/ts-005-dns-spoof.md @@ -77,72 +77,73 @@ Whether or not DNS spoofing is occurring for a particular FQDN. ## Example output sample - - ########################################### - # OONI Probe Report for dns_spoof (0.0.1) - # Wed Sep 25 15:39:32 2013 - ########################################### - --- - input_hashes: [] - options: [-r, '10.211.0.10:53', -h, google.com] - probe_asn: AS2819 - probe_cc: CZ - probe_ip: 127.0.0.1 - software_name: ooniprobe - software_version: 1.0.0-rc3 - start_time: 1380116372.573729 - test_name: dns_spoof - test_version: 0.0.1 - ... - --- - answer_flags: [ipsrc] +``` +########################################### +# OONI Probe Report for dns_spoof (0.0.1) +# Wed Sep 25 15:39:32 2013 +########################################### +--- +input_hashes: [] +options: [-r, '10.211.0.10:53', -h, google.com] +probe_asn: AS2819 +probe_cc: CZ +probe_ip: 127.0.0.1 +software_name: ooniprobe +software_version: 1.0.0-rc3 +start_time: 1380116372.573729 +test_name: dns_spoof +test_version: 0.0.1 +... +--- +answer_flags: [ipsrc] +answered_packets: +- - raw_packet: !!binary | + RbgA6OumAAAyEY4+CAgICH8AAAEANQA1ANSshgAAgYAAAQALAAAAAAZnb29nbGUDY29tAAABAAEG + Z29vZ2xlA2NvbQAAAQABAAAA4AAErcIs5QZnb29nbGUDY29tAAABAAEAAADgAAStwizkBmdvb2ds + ZQNjb20AAAEAAQAAAOAABK3CLOYGZ29vZ2xlA2NvbQAAAQABAAAA4AAErcIs6QZnb29nbGUDY29t + AAABAAEAAADgAAStwizoBmdvb2dsZQNjb20AAAEAAQAAAOAABK3CLOcGZ29vZ2xlA2NvbQAAAQAB + AAAA4AAErcIs4gZnb29nbGUDY29tAAABAAEAAADgAAStwizjBmdvb2dsZQNjb20AAAEAAQAAAOAA + BK3CLOAGZ29vZ2xlA2NvbQAAAQABAAAA4AAErcIs4QZnb29nbGUDY29tAAABAAEAAADgAAStwizu + summary: 'IP / UDP / DNS Ans "173.194.44.229" ' +- - raw_packet: !!binary | + RbgA6J0DAABAEdQUCtMACn8AAAEANQA1ANSxxAAAgYAAAQALAAAAAAZnb29nbGUDY29tAAABAAEG + Z29vZ2xlA2NvbQAAAQABAAAA3wAErcIs5wZnb29nbGUDY29tAAABAAEAAADfAAStwizoBmdvb2ds + ZQNjb20AAAEAAQAAAN8ABK3CLOkGZ29vZ2xlA2NvbQAAAQABAAAA3wAErcIs5gZnb29nbGUDY29t + AAABAAEAAADfAAStwizkBmdvb2dsZQNjb20AAAEAAQAAAN8ABK3CLOUGZ29vZ2xlA2NvbQAAAQAB + AAAA3wAErcIs7gZnb29nbGUDY29tAAABAAEAAADfAAStwizhBmdvb2dsZQNjb20AAAEAAQAAAN8A + BK3CLOAGZ29vZ2xlA2NvbQAAAQABAAAA3wAErcIs4wZnb29nbGUDY29tAAABAAEAAADfAAStwizi + summary: 'IP / UDP / DNS Ans "173.194.44.231" ' +input: null +sent_packets: +- - raw_packet: !!binary | + RQAAOAABAABAEeujfwAAAQgICAgANQA1ACRccgAAAQAAAQAAAAAAAAZnb29nbGUDY29tAAABAAE= + summary: 'IP / UDP / DNS Qry "google.com" ' +- - raw_packet: !!binary | + RQAAOAABAABAEfDWfwAAAQrTAAoANQA1ACRhpQAAAQAAAQAAAAAAAAZnb29nbGUDY29tAAABAAE= + summary: 'IP / UDP / DNS Qry "google.com" ' +spoofing: false +test_a_lookup: + answered_packets: + - raw_packet: !!binary | + RbgA6J0DAABAEdQUCtMACn8AAAEANQA1ANSxxAAAgYAAAQALAAAAAAZnb29nbGUDY29tAAABAAEG + Z29vZ2xlA2NvbQAAAQABAAAA3wAErcIs5wZnb29nbGUDY29tAAABAAEAAADfAAStwizoBmdvb2ds + ZQNjb20AAAEAAQAAAN8ABK3CLOkGZ29vZ2xlA2NvbQAAAQABAAAA3wAErcIs5gZnb29nbGUDY29t + AAABAAEAAADfAAStwizkBmdvb2dsZQNjb20AAAEAAQAAAN8ABK3CLOUGZ29vZ2xlA2NvbQAAAQAB + AAAA3wAErcIs7gZnb29nbGUDY29tAAABAAEAAADfAAStwizhBmdvb2dsZQNjb20AAAEAAQAAAN8A + BK3CLOAGZ29vZ2xlA2NvbQAAAQABAAAA3wAErcIs4wZnb29nbGUDY29tAAABAAEAAADfAAStwizi + summary: 'IP / UDP / DNS Ans "173.194.44.231" ' +test_control_a_lookup: answered_packets: - - - raw_packet: !!binary | - RbgA6OumAAAyEY4+CAgICH8AAAEANQA1ANSshgAAgYAAAQALAAAAAAZnb29nbGUDY29tAAABAAEG - Z29vZ2xlA2NvbQAAAQABAAAA4AAErcIs5QZnb29nbGUDY29tAAABAAEAAADgAAStwizkBmdvb2ds - ZQNjb20AAAEAAQAAAOAABK3CLOYGZ29vZ2xlA2NvbQAAAQABAAAA4AAErcIs6QZnb29nbGUDY29t - AAABAAEAAADgAAStwizoBmdvb2dsZQNjb20AAAEAAQAAAOAABK3CLOcGZ29vZ2xlA2NvbQAAAQAB - AAAA4AAErcIs4gZnb29nbGUDY29tAAABAAEAAADgAAStwizjBmdvb2dsZQNjb20AAAEAAQAAAOAA - BK3CLOAGZ29vZ2xlA2NvbQAAAQABAAAA4AAErcIs4QZnb29nbGUDY29tAAABAAEAAADgAAStwizu - summary: 'IP / UDP / DNS Ans "173.194.44.229" ' - - - raw_packet: !!binary | - RbgA6J0DAABAEdQUCtMACn8AAAEANQA1ANSxxAAAgYAAAQALAAAAAAZnb29nbGUDY29tAAABAAEG - Z29vZ2xlA2NvbQAAAQABAAAA3wAErcIs5wZnb29nbGUDY29tAAABAAEAAADfAAStwizoBmdvb2ds - ZQNjb20AAAEAAQAAAN8ABK3CLOkGZ29vZ2xlA2NvbQAAAQABAAAA3wAErcIs5gZnb29nbGUDY29t - AAABAAEAAADfAAStwizkBmdvb2dsZQNjb20AAAEAAQAAAN8ABK3CLOUGZ29vZ2xlA2NvbQAAAQAB - AAAA3wAErcIs7gZnb29nbGUDY29tAAABAAEAAADfAAStwizhBmdvb2dsZQNjb20AAAEAAQAAAN8A - BK3CLOAGZ29vZ2xlA2NvbQAAAQABAAAA3wAErcIs4wZnb29nbGUDY29tAAABAAEAAADfAAStwizi - summary: 'IP / UDP / DNS Ans "173.194.44.231" ' - input: null - sent_packets: - - - raw_packet: !!binary | - RQAAOAABAABAEeujfwAAAQgICAgANQA1ACRccgAAAQAAAQAAAAAAAAZnb29nbGUDY29tAAABAAE= - summary: 'IP / UDP / DNS Qry "google.com" ' - - - raw_packet: !!binary | - RQAAOAABAABAEfDWfwAAAQrTAAoANQA1ACRhpQAAAQAAAQAAAAAAAAZnb29nbGUDY29tAAABAAE= - summary: 'IP / UDP / DNS Qry "google.com" ' - spoofing: false - test_a_lookup: - answered_packets: - - raw_packet: !!binary | - RbgA6J0DAABAEdQUCtMACn8AAAEANQA1ANSxxAAAgYAAAQALAAAAAAZnb29nbGUDY29tAAABAAEG - Z29vZ2xlA2NvbQAAAQABAAAA3wAErcIs5wZnb29nbGUDY29tAAABAAEAAADfAAStwizoBmdvb2ds - ZQNjb20AAAEAAQAAAN8ABK3CLOkGZ29vZ2xlA2NvbQAAAQABAAAA3wAErcIs5gZnb29nbGUDY29t - AAABAAEAAADfAAStwizkBmdvb2dsZQNjb20AAAEAAQAAAN8ABK3CLOUGZ29vZ2xlA2NvbQAAAQAB - AAAA3wAErcIs7gZnb29nbGUDY29tAAABAAEAAADfAAStwizhBmdvb2dsZQNjb20AAAEAAQAAAN8A - BK3CLOAGZ29vZ2xlA2NvbQAAAQABAAAA3wAErcIs4wZnb29nbGUDY29tAAABAAEAAADfAAStwizi - summary: 'IP / UDP / DNS Ans "173.194.44.231" ' - test_control_a_lookup: - answered_packets: - - raw_packet: !!binary | - RbgA6OumAAAyEY4+CAgICH8AAAEANQA1ANSshgAAgYAAAQALAAAAAAZnb29nbGUDY29tAAABAAEG - Z29vZ2xlA2NvbQAAAQABAAAA4AAErcIs5QZnb29nbGUDY29tAAABAAEAAADgAAStwizkBmdvb2ds - ZQNjb20AAAEAAQAAAOAABK3CLOYGZ29vZ2xlA2NvbQAAAQABAAAA4AAErcIs6QZnb29nbGUDY29t - AAABAAEAAADgAAStwizoBmdvb2dsZQNjb20AAAEAAQAAAOAABK3CLOcGZ29vZ2xlA2NvbQAAAQAB - AAAA4AAErcIs4gZnb29nbGUDY29tAAABAAEAAADgAAStwizjBmdvb2dsZQNjb20AAAEAAQAAAOAA - BK3CLOAGZ29vZ2xlA2NvbQAAAQABAAAA4AAErcIs4QZnb29nbGUDY29tAAABAAEAAADgAAStwizu - summary: 'IP / UDP / DNS Ans "173.194.44.229" ' - ... + - raw_packet: !!binary | + RbgA6OumAAAyEY4+CAgICH8AAAEANQA1ANSshgAAgYAAAQALAAAAAAZnb29nbGUDY29tAAABAAEG + Z29vZ2xlA2NvbQAAAQABAAAA4AAErcIs5QZnb29nbGUDY29tAAABAAEAAADgAAStwizkBmdvb2ds + ZQNjb20AAAEAAQAAAOAABK3CLOYGZ29vZ2xlA2NvbQAAAQABAAAA4AAErcIs6QZnb29nbGUDY29t + AAABAAEAAADgAAStwizoBmdvb2dsZQNjb20AAAEAAQAAAOAABK3CLOcGZ29vZ2xlA2NvbQAAAQAB + AAAA4AAErcIs4gZnb29nbGUDY29tAAABAAEAAADgAAStwizjBmdvb2dsZQNjb20AAAEAAQAAAOAA + BK3CLOAGZ29vZ2xlA2NvbQAAAQABAAAA4AAErcIs4QZnb29nbGUDY29tAAABAAEAAADgAAStwizu + summary: 'IP / UDP / DNS Ans "173.194.44.229" ' +... +``` # Privacy considerations diff --git a/test-specs/ts-012-lantern.md b/test-specs/ts-012-lantern.md deleted file mode 100644 index 508bdfc..0000000 --- a/test-specs/ts-012-lantern.md +++ /dev/null @@ -1,111 +0,0 @@ -# Specification version number - -2015-04-03-000 - -# Specification name - -Lantern Test - -# Test preconditions - -Downloaded or compiled the "lantern" binary and made executable and in -the users PATH environment variable. - -# Expected impact - -Ability to measure whether Lantern is working from the given network vantage point. - -# Expected inputs - -A single URL to fetch, supplied by command line argument "--url (-u)". To test -Lantern, it must be a URL from the whitelisted set. -See: https://github.com/getlantern/flashlight/blob/master/genconfig/proxiedsites/ - -# Test description - -This test launches Lantern in --headless mode, and parses output to determine -if it has bootstrapped. After bootstrap, it fetches the URL supplied by the ---url option using Lanterns http proxy interface listening on 127.0.0.1.8787. - -The specific string used to determine bootstrap from Lantern output in version -"2.0.10" is "Connected to proxy on localhost" from standard output. - -# Expected output - -## Parent data format - -None. - -## Required output data - -success: -**boolean** The bootstrap status of Lantern (success or failure). - -lantern --headless: -**dictionary** the parent key of Lanterns output that contains the keys stdout and stderr - -stdout: -**string** Output produced by Lanterns standard output. - -stderr: -**string** Error produced by Lanterns standard error. - -body: -**string** The page body of a successful HTTP request. - -failure: -**string** If failure, then the corresponding failure message. - - -## Data specification version number - -## Semantics - -'success' - True or False - whether Lantern has bootstrapped. -'body' - http page body if successfully requested. -'failure' - optional, present if there is a failure. -'lantern --headless': - 'stdout' - Contents of standard output produced by Lantern. - 'stderr' - Contents of standard error produced by Lantern. - - -## Possible conclusions - -We can determine whether or not Lantern is able to bootstrap, according to its output. -We can determine whether or not a given URL is reachable via Lantern. - -## Example output sample -``` ---- -input_hashes: [] -options: [-u, google.com] -probe_asn: AS1234 -probe_cc: US -probe_city: null -probe_ip: 127.0.0.1 -software_name: ooniprobe -software_version: 1.2.3-rc1 -start_time: 1428344311.0 -test_name: lantern_circumvention_tool_test -test_version: 0.0.1 -... ---- -body: "<HTML><HEAD><meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\ - >\n<TITLE>301 Moved</TITLE></HEAD><BODY>\n<H1>301 Moved</H1>\nThe document has moved\n\ - <A HREF=\"http://www.google.com/\">here</A>.\r\n</BODY></HTML>\r\n" -bootstrapped: true -input: null -lantern --headless: {exit_reason: process_done, stderr: '', stdout: ''} -``` - -## Expected Post-processing efforts - -# Privacy considerations - -Lantern does not seek to provide anonymity. Lantern contains tracking analytics -software and may connect directly to Lantern-provided proxy endpoints, or use -fronted domains via Content Delivery Networks (CDNs) as a data channel. - -# Packet capture considerations - -This test does not capture packets by default. diff --git a/test-specs/ts-013-lantern.md b/test-specs/ts-013-lantern.md new file mode 100644 index 0000000..508bdfc --- /dev/null +++ b/test-specs/ts-013-lantern.md @@ -0,0 +1,111 @@ +# Specification version number + +2015-04-03-000 + +# Specification name + +Lantern Test + +# Test preconditions + +Downloaded or compiled the "lantern" binary and made executable and in +the users PATH environment variable. + +# Expected impact + +Ability to measure whether Lantern is working from the given network vantage point. + +# Expected inputs + +A single URL to fetch, supplied by command line argument "--url (-u)". To test +Lantern, it must be a URL from the whitelisted set. +See: https://github.com/getlantern/flashlight/blob/master/genconfig/proxiedsites/ + +# Test description + +This test launches Lantern in --headless mode, and parses output to determine +if it has bootstrapped. After bootstrap, it fetches the URL supplied by the +--url option using Lanterns http proxy interface listening on 127.0.0.1.8787. + +The specific string used to determine bootstrap from Lantern output in version +"2.0.10" is "Connected to proxy on localhost" from standard output. + +# Expected output + +## Parent data format + +None. + +## Required output data + +success: +**boolean** The bootstrap status of Lantern (success or failure). + +lantern --headless: +**dictionary** the parent key of Lanterns output that contains the keys stdout and stderr + +stdout: +**string** Output produced by Lanterns standard output. + +stderr: +**string** Error produced by Lanterns standard error. + +body: +**string** The page body of a successful HTTP request. + +failure: +**string** If failure, then the corresponding failure message. + + +## Data specification version number + +## Semantics + +'success' - True or False - whether Lantern has bootstrapped. +'body' - http page body if successfully requested. +'failure' - optional, present if there is a failure. +'lantern --headless': + 'stdout' - Contents of standard output produced by Lantern. + 'stderr' - Contents of standard error produced by Lantern. + + +## Possible conclusions + +We can determine whether or not Lantern is able to bootstrap, according to its output. +We can determine whether or not a given URL is reachable via Lantern. + +## Example output sample +``` +--- +input_hashes: [] +options: [-u, google.com] +probe_asn: AS1234 +probe_cc: US +probe_city: null +probe_ip: 127.0.0.1 +software_name: ooniprobe +software_version: 1.2.3-rc1 +start_time: 1428344311.0 +test_name: lantern_circumvention_tool_test +test_version: 0.0.1 +... +--- +body: "<HTML><HEAD><meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\ + >\n<TITLE>301 Moved</TITLE></HEAD><BODY>\n<H1>301 Moved</H1>\nThe document has moved\n\ + <A HREF=\"http://www.google.com/\">here</A>.\r\n</BODY></HTML>\r\n" +bootstrapped: true +input: null +lantern --headless: {exit_reason: process_done, stderr: '', stdout: ''} +``` + +## Expected Post-processing efforts + +# Privacy considerations + +Lantern does not seek to provide anonymity. Lantern contains tracking analytics +software and may connect directly to Lantern-provided proxy endpoints, or use +fronted domains via Content Delivery Networks (CDNs) as a data channel. + +# Packet capture considerations + +This test does not capture packets by default. diff --git a/test-specs/ts-013-meek-fronted-requests.md b/test-specs/ts-013-meek-fronted-requests.md deleted file mode 100644 index f78066e..0000000 --- a/test-specs/ts-013-meek-fronted-requests.md +++ /dev/null @@ -1,111 +0,0 @@ -# Specification version number - -2015-04-01-000 - -# Specification name - -Meek Fronted Request Test - -# Test preconditions - -* An internet connection - -# Expected impact - -Ability to detect uncensored fronted domains that can transmit requests to the -"inside" meek-server via an intermediary web service. - -# Expected inputs - -## Import document or import data format - -A list of domain names (fronted domains) and host headers (meek-server) of the -intermediary web service. - -## Semantics - -The input document may contain a domain name and a host header combination per -line separated by colon in the format: - - DomainName:HostHeader - -Example: - - www.google.com:meek-reflect.appspot.com - a0.awsstatic.com:d2zfqthxsdq309.cloudfront.net - -# Test description - -Performs a HTTP GET request over TLS (HTTPS) to a list of fronted domains with -the Host Header of the "inside" meek-server. For diagnostic purposes the -meek-server handles a GET request and respond with: "I’m just a happy little -web server.\n". The GET request is sent over TLS to the root of the fronted -domain with the Host Header option of the desired meek-server host. - - -# Expected output - -## Parent data format - -df-001-httpt-000 - -## Required output data - -* The domain name and host header used in the measurement - (DomainName:HostHeader) - -* The requests that have been made - -* The received responses - -* If the meek server is blocked or unreachable - -## Semantics - -success: - **boolean** indicates if an HTTPS GET response to the meek server is - successfull - -## Possible conclusions - -If the fronted request/response to the meek server is successful. - -## Example output sample - -``` -agent: agent -input: ajax.aspnetcdn.com:az668014.vo.msecnd.net -requests: -- request: - body: null - headers: - - - Host - - [az668014.vo.msecnd.net] - method: GET - tor: {is_tor: false} - url: https://ajax.aspnetcdn.com - response: - body: "I\u2019m just a happy little web server.\n" - code: 200 - headers: - - - Content-Length - - ['38'] - - - X-Cache - - [HIT] - - - X-Powered-By - - [ASP.NET] - - - Accept-Ranges - - [bytes] - - - Server - - [ECAcc (fcn/40C4)] - - - Last-Modified - - ['Wed, 01 Apr 2015 09:25:13 GMT'] - - - Connection - - [close] - - - Date - - ['Wed, 01 Apr 2015 10:01:37 GMT'] - - - Content-Type - - [text/plain; charset=utf-8] -socksproxy: null -success: true -``` diff --git a/test-specs/ts-014-meek-fronted-requests.md b/test-specs/ts-014-meek-fronted-requests.md new file mode 100644 index 0000000..f78066e --- /dev/null +++ b/test-specs/ts-014-meek-fronted-requests.md @@ -0,0 +1,111 @@ +# Specification version number + +2015-04-01-000 + +# Specification name + +Meek Fronted Request Test + +# Test preconditions + +* An internet connection + +# Expected impact + +Ability to detect uncensored fronted domains that can transmit requests to the +"inside" meek-server via an intermediary web service. + +# Expected inputs + +## Import document or import data format + +A list of domain names (fronted domains) and host headers (meek-server) of the +intermediary web service. + +## Semantics + +The input document may contain a domain name and a host header combination per +line separated by colon in the format: + + DomainName:HostHeader + +Example: + + www.google.com:meek-reflect.appspot.com + a0.awsstatic.com:d2zfqthxsdq309.cloudfront.net + +# Test description + +Performs a HTTP GET request over TLS (HTTPS) to a list of fronted domains with +the Host Header of the "inside" meek-server. For diagnostic purposes the +meek-server handles a GET request and respond with: "I’m just a happy little +web server.\n". The GET request is sent over TLS to the root of the fronted +domain with the Host Header option of the desired meek-server host. + + +# Expected output + +## Parent data format + +df-001-httpt-000 + +## Required output data + +* The domain name and host header used in the measurement + (DomainName:HostHeader) + +* The requests that have been made + +* The received responses + +* If the meek server is blocked or unreachable + +## Semantics + +success: + **boolean** indicates if an HTTPS GET response to the meek server is + successfull + +## Possible conclusions + +If the fronted request/response to the meek server is successful. + +## Example output sample + +``` +agent: agent +input: ajax.aspnetcdn.com:az668014.vo.msecnd.net +requests: +- request: + body: null + headers: + - - Host + - [az668014.vo.msecnd.net] + method: GET + tor: {is_tor: false} + url: https://ajax.aspnetcdn.com + response: + body: "I\u2019m just a happy little web server.\n" + code: 200 + headers: + - - Content-Length + - ['38'] + - - X-Cache + - [HIT] + - - X-Powered-By + - [ASP.NET] + - - Accept-Ranges + - [bytes] + - - Server + - [ECAcc (fcn/40C4)] + - - Last-Modified + - ['Wed, 01 Apr 2015 09:25:13 GMT'] + - - Connection + - [close] + - - Date + - ['Wed, 01 Apr 2015 10:01:37 GMT'] + - - Content-Type + - [text/plain; charset=utf-8] +socksproxy: null +success: true +``` diff --git a/test-specs/ts-014-psiphon.md b/test-specs/ts-014-psiphon.md deleted file mode 100644 index 7bc6c04..0000000 --- a/test-specs/ts-014-psiphon.md +++ /dev/null @@ -1,146 +0,0 @@ -# Specification version number - -2015-10-11-000 - -# Specification name - -Psiphon Test - -# Test preconditions - -Have psiphon-circumvention-system (including psiphon-circumvention-system/pyclient/psi_client.py) cloned in the home of the user that runs ooni or somewhere else accessible to the user that runs ooni. - -# Expected impact - -Ability to measure whether Psiphon is working from the given network vantage point. - -# Expected inputs - -Optionally: -A single URL to fetch, supplied by command line argument "--url (-u)". -Psiphon path, specified by the command line argument "--psiphonpath (-p)" -The ip:port that Psiphon will use for the SOCKS proxy, with the command line argument "--socksproxy (-s)" - -# Test description - -This test first check that the Psiphon path exists, then launches Psiphon and parses output to determine if it has bootstrapped. After bootstrap, it fetches google.com (or other URL specified by the --url argument) using Psiphons SOCKS proxy listening on 127.0.0.1:1080 (or otherwise specified by the --socksproxy argument). - -The specific string used to determine bootstrap from Psiphon output in version -"0.0.1" is "Press Ctrl-C to terminate." from standard output. - -# Expected output - -## Parent data format - -The following keys from df-001-httpt.md are used when Psiphon bootstraps: requests, socksproxy, agent. -When Psiphon is not installed or does not bootstrap, only agent and socksproxy are used. - -## Required output data - -psiphon_installed: -**boolean** Whether Psiphon client is found or not (success or failure). - -success: -**boolean** The bootstrap status of Psiphon (success or failure). - -/tmp/<temporary file>: -**dictionary** the parent key of Psiphon's output that contains the keys stdout and stderr and exit_reason - -stdout: -**string** Output produced by Psiphon's standard output. - -stderr: -**string** Error produced by Psiphon's standard error. - -## Data specification version number - -## Semantics - -'psiphon_installed' - True or False - whether Psiphon is found. -'success' - True or False - whether Psiphon has bootstrapped. -'body' - http page body if successfully requested. -'/tmp/<temporary file>': - 'stdout' - Contents of standard output produced by Psiphon. - 'stderr' - Contents of standard error produced by Psiphon. - - -## Possible conclusions - -We can determine whether or not Psiphon is found. -We can determine whether or not Psiphon is able to bootstrap, according to its output. -We can determine whether or not a given URL is reachable via Psiphon. - -## Example output sample -``` ---- -input_hashes: [] -options: [-u, google.com] -probe_asn: AS0 -probe_cc: ZZ -probe_city: null -probe_ip: 127.0.0.1 -report_id: 4dAHr0ceNDBmw5lUQ7pBoxqgyUSfP873Qj1zv5VyElnSSTXwcsLYeCv69DsUjb94 -software_name: ooniprobe -software_version: 1.3.1 -start_time: 1444686051.0 -test_helpers: {} -test_name: psiphon_test -test_version: 0.0.1 -... ---- -/tmp/tmplKg8K3: {exit_reason: process_done, stderr: '', stdout: "./ssh is not a valid\ - \ executable. Using standard ssh.\r\n\r\nYour SOCKS proxy is now running at 127.0.0.1:1080\r\ - \n\r\nPress Ctrl-C to terminate.\r\nTerminating...\r\nConnection closed\r\n"} -agent: agent -input: null -psiphon_installed: true -requests: -- request: - body: null - headers: [] - method: GET - tor: {is_tor: false} - url: http://google.com - response: - body: "<HTML><HEAD><meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\ - >\n<TITLE>301 Moved</TITLE></HEAD><BODY>\n<H1>301 Moved</H1>\nThe document has\ - \ moved\n<A HREF=\"http://www.google.com/\">here</A>.\r\n</BODY></HTML>\r\n" - code: 301 - headers: - - - Content-Length - - ['219'] - - - X-XSS-Protection - - [1; mode=block] - - - Expires - - ['Wed, 11 Nov 2015 21:40:58 UTC'] - - - Server - - [gws] - - - Connection - - [close] - - - Location - - ['http://www.google.com/'] - - - Cache-Control - - ['public, max-age=2592000'] - - - Date - - ['Mon, 12 Oct 2015 21:40:58 UTC'] - - - X-Frame-Options - - [SAMEORIGIN] - - - Content-Type - - [text/html; charset=UTF-8] -socksproxy: 127.0.0.1:1080 -test_runtime: 7.373162031173706 -test_start_time: 1444686052.0 -... -``` - -## Expected Post-processing efforts - -# Privacy considerations - -Psiphon does not seek to provide anonymity. -An adversary can observe that a user is connecting to Psiphon servers. -Psiphon servers can also determine the users location. - -# Packet capture considerations - -This test does not capture packets by default. diff --git a/test-specs/ts-015-openvpn.md b/test-specs/ts-015-openvpn.md deleted file mode 100644 index 84ed367..0000000 --- a/test-specs/ts-015-openvpn.md +++ /dev/null @@ -1,218 +0,0 @@ -# Specification version number - -2015-10-11-000 - -# Specification name - -OpenVPN Test - -# Test preconditions - -Have OpenVPN installed and configured to work with at least one server and privileges to run the test as root. - -# Expected impact - -Ability to measure whether OpenVPN is working from the given network vantage point. - -# Expected inputs - -A single URL to fetch, supplied by command line argument "--url (-u)". -OpenVPN configuration file, specified by the command line argument "--openvpn-config (-c)" - -# Test description - -This test first launches OpenVPN and parses output to determine if it has bootstrapped. After bootstrap, it fetches the URL specified by the --url argument using OpenVPN. - -The specific string used to determine bootstrap from OpenVPN output in version -"0.0.1" is "Initialization Sequence Completed" from standard output. - -# Expected output - -## Parent data format - -None. - -## Required output data - -success: -**boolean** The bootstrap status of OpenVPN (success or failure). - -OpenVPN_linux --headless: -**dictionary** the parent key of OpenVPNs output that contains the keys stdout and stderr - -stdout: -**string** Output produced by OpenVPNs standard output. - -stderr: -**string** Error produced by OpenVPNs standard error. - -body: -**string** The page body of a successful HTTP request. - -failure: -**string** If failure, then the corresponding failure message. - -## Data specification version number - -## Semantics - -'success' - True or False - whether OpenVPN has bootstrapped. -'body' - http page body if successfully requested. -'failure' - optional, present if there is a failure. -'l/usr/sbin/openvpn --config configfile': - 'stdout' - Contents of standard output produced by OpenVPN. - 'stderr' - Contents of standard error produced by OpenVPN. - -## Possible conclusions - -We can determine whether or not OpenVPN is able to bootstrap, according to its output. -We can determine whether or not a given URL is reachable via OpenVPN. - -## Example output sample -``` ---- -input_hashes: [] -options: [-c, openvpnconfigfile.ovpn, -u, ''] -probe_asn: AS0 -probe_cc: ZZ -probe_city: null -probe_ip: 127.0.0.1 -report_id: nqvK7YrK6J5Di7BiWDwPUBfyKcbLoVWeU4DgnxTzzKWMQABvhC2l3q6aLUwF0CA9 -software_name: ooniprobe -software_version: 1.3.1 -start_time: 1444925440.0 -test_helpers: {} -test_name: test_openvpn_circumvent -test_version: 0.0.1 -... ---- -/usr/sbin/openvpn --config /pathtoopenvpnconfigfile/openvpnconfigfile.ovpn: { - exit_reason: process_done, stderr: '', stdout: 'Thu Oct 15 20:10:40 2015 OpenVPN - 2.3.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] - [IPv6] built on Dec 1 2014 - - Thu Oct 15 20:10:40 2015 WARNING: file ''/tmp/openvpn.txt'' is group or others - accessible - - Thu Oct 15 20:10:40 2015 Control Channel Authentication: tls-auth using INLINE - static key file - - Thu Oct 15 20:10:40 2015 Attempting to establish TCP connection with [AF_INET]10.0.0.10:993 - [nonblock] - - Thu Oct 15 20:10:41 2015 TCP connection established with [AF_INET]10.0.0.10:993 - - Thu Oct 15 20:10:41 2015 TCPv4_CLIENT link local: [undef] - - Thu Oct 15 20:10:41 2015 TCPv4_CLIENT link remote: [AF_INET]10.0.0.10:993 - - Thu Oct 15 20:10:41 2015 WARNING: this configuration may cache passwords in memory - -- use the auth-nocache option to prevent this - - Thu Oct 15 20:10:46 2015 [server] Peer Connection Initiated with [AF_INET]10.0.0.10:993 - - Thu Oct 15 20:10:48 2015 Options error: Unrecognized option or missing parameter(s) - in [PUSH-OPTIONS]:3: dhcp (2.3.2) - - Thu Oct 15 20:10:48 2015 TUN/TAP device tun0 opened - - Thu Oct 15 20:10:48 2015 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0 - - Thu Oct 15 20:10:48 2015 /sbin/ip link set dev tun0 up mtu 1500 - - Thu Oct 15 20:10:48 2015 /sbin/ip addr add dev tun0 local 10.10.0.34 peer 10.10.0.33 - - Thu Oct 15 20:10:48 2015 Initialization Sequence Completed - - '} -body: "<?xml version=\"1.0\"?>\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"\ - \n \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\n<html xmlns=\"http://www.w3.org/1999/xhtml\"\ - >\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\"\ - \ />\n<title>This is a Tor Exit Router</title>\n\n\n\n</head>\n<body>\n\n<p style=\"text-align:center; font-size:xx-large; font-weight:bold\"\ - >This is a\nTor Exit Router</p>\n\n<p>\nMost likely you are accessing this website\ - \ because you had some issue with\nthe traffic coming from this IP. This router\ - \ is part of the <a\nhref=\"https://www.torproject.org/\">Tor Anonymity Network</a>,\ - \ which is\ndedicated to <a href=\"https://www.torproject.org/about/overview\">providing\n\ - privacy</a> to people who need it most: average computer users. This\nrouter IP\ - \ should be generating no other traffic, unless it has been\ncompromised.</p>\n\n\ - <p style=\"text-align:center\">\n<a href=\"https://www.torproject.org/about/overview\"\ - >\n<img src=\"how_tor_works_thumb.png\" alt=\"How Tor works\" style=\"border-style:none\"\ - />\n</a></p>\n\n<p>\nTor sees use by <a href=\"https://www.torproject.org/about/torusers\"\ - >many\nimportant segments of the population</a>, including whistle blowers,\njournalists,\ - \ Chinese dissidents skirting the Great Firewall and oppressive\ncensorship, abuse\ - \ victims, stalker targets, the US military, and law\nenforcement, just to name\ - \ a few. While Tor is not designed for malicious\ncomputer users, it is true that\ - \ they can use the network for malicious ends.\nIn reality however, the actual amount\ - \ of <a\nhref=\"https://www.torproject.org/docs/faq-abuse\">abuse</a> is quite low.\ - \ This\nis largely because criminals and hackers have significantly better access\ - \ to\nprivacy and anonymity than do the regular users whom they prey upon. Criminals\n\ - can and do <a\nhref=\"http://voices.washingtonpost.com/securityfix/2008/08/web_fraud_20_tools.htm…"\ - >build,\nsell, and trade</a> far larger and <a\nhref=\"http://voices.washingtonpost.com/securityfix/2008/08/web_fraud_20_distribut…"\ - >more\npowerful networks</a> than Tor on a daily basis. Thus, in the mind of this\n\ - operator, the social need for easily accessible censorship-resistant private,\n\ - anonymous communication trumps the risk of unskilled bad actors, who are\nalmost\ - \ always more easily uncovered by traditional police work than by\nextensive monitoring\ - \ and surveillance anyway.</p>\n\n<p>\nIn terms of applicable law, the best way\ - \ to understand Tor is to consider it a\nnetwork of routers operating as common\ - \ carriers, much like the Internet\nbackbone. However, unlike the Internet backbone\ - \ routers, Tor routers\nexplicitly do not contain identifiable routing information\ - \ about the source of\na packet, and no single Tor node can determine both the origin\ - \ and destination\nof a given transmission.</p>\n\n<p>\nAs such, there is little\ - \ the operator of this router can do to help you track\nthe connection further.\ - \ This router maintains no logs of any of the Tor\ntraffic, so there is little that\ - \ can be done to trace either legitimate or\nillegitimate traffic (or to filter\ - \ one from the other). Attempts to\nseize this router will accomplish nothing.</p>\n\ - \n\n\n<p>\nIf you are a representative of a company\ - \ who feels that this router is being\nused to violate the DMCA, please be aware\ - \ that this machine does not host or\ncontain any illegal content. Also be aware\ - \ that network infrastructure\nmaintainers are not liable for the type of content\ - \ that passes over their\nequipment, in accordance with <a\nhref=\"http://www.law.cornell.edu/uscode/text/17/512\"\ - >DMCA\n\"safe harbor\" provisions</a>. In other words, you will have just as much\ - \ luck\nsending a takedown notice to the Internet backbone providers. Please consult\n\ - <a href=\"https://www.torproject.org/eff/tor-dmca-response\">EFF's prepared\nresponse</a>\ - \ for more information on this matter.</p>\n\n<p>For more information, please consult\ - \ the following documentation:</p>\n\n<ol>\n<li><a href=\"https://www.torproject.org/about/overview\"\ - >Tor Overview</a></li>\n<li><a href=\"https://www.torproject.org/docs/faq-abuse\"\ - >Tor Abuse FAQ</a></li>\n<li><a href=\"https://www.torproject.org/eff/tor-legal-faq\"\ - >Tor Legal FAQ</a></li>\n</ol>\n\n<p>\nThat being said, if you still have a complaint\ - \ about the router, you may\nemail the <a href=\"mailto:tor@openvpnconfigfile.ie\">maintainer</a>.\ - \ If\ncomplaints are related to a particular service that is being abused, I will\n\ - consider removing that service from my exit policy, which would prevent my\nrouter\ - \ from allowing that traffic to exit through it. I can only do this on an\nIP+destination\ - \ port basis, however. Common P2P ports are\nalready blocked.</p>\n\n<p>\nYou also\ - \ have the option of blocking this IP address and others on\nthe Tor network if\ - \ you so desire. The Tor project provides a <a\nhref=\"https://check.torproject.org/cgi-bin/TorBulkExitList.py\"\ - >web service</a>\nto fetch a list of all IP addresses of Tor exit nodes that allow\ - \ exiting to a\nspecified IP:port combination, and an official <a\nhref=\"https://www.torproject.org/tordnsel/dist/\"\ - >DNSRBL</a> is also available to\ndetermine if a given IP address is actually a\ - \ Tor exit server. Please\nbe considerate\nwhen using these options. It would be\ - \ unfortunate to deny all Tor users access\nto your site indefinitely simply because\ - \ of a few bad apples.</p>\n\n<p style=\"text-align:center; margin-bottom: 0.5em\"\ - >Exit Node provided by:<p>\n<h2 style=\"text-align: center\"><a style=\"color: black\"\ - \ href=\"http://www.openvpnconfigfile.ie\">DU Pirate Party</a><h2>\n\n</body>\n</html>\n" -input: null -success: true -test_runtime: 8.374207019805908 -test_start_time: 1444925440.0 -... -``` - -## Expected Post-processing efforts - -# Privacy considerations - -OpenVPN does not seek to provide anonymity. -An adversary can observe that a user is connecting to OpenVPN servers. -OpenVPN servers can also determine the users location. - -# Packet capture considerations - -This test does not capture packets by default. diff --git a/test-specs/ts-015-psiphon.md b/test-specs/ts-015-psiphon.md new file mode 100644 index 0000000..799fc21 --- /dev/null +++ b/test-specs/ts-015-psiphon.md @@ -0,0 +1,148 @@ +# Specification version number + +2015-10-11-000 + +# Specification name + +Psiphon Test + +# Test preconditions + +Have psiphon-circumvention-system (including psiphon-circumvention-system/pyclient/psi_client.py) cloned in the home of the user that runs ooni or somewhere else accessible to the user that runs ooni. + +# Expected impact + +Ability to measure whether Psiphon is working from the given network vantage point. + +# Expected inputs + +Optionally: +A single URL to fetch, supplied by command line argument "--url (-u)". +Psiphon path, specified by the command line argument "--psiphonpath (-p)" +The ip:port that Psiphon will use for the SOCKS proxy, with the command line argument "--socksproxy (-s)" + +# Test description + +This test first check that the Psiphon path exists, then launches Psiphon and parses output to determine if it has bootstrapped. After bootstrap, it fetches google.com (or other URL specified by the --url argument) using Psiphons SOCKS proxy listening on 127.0.0.1:1080 (or otherwise specified by the --socksproxy argument). + +The specific string used to determine bootstrap from Psiphon output in version +"0.0.1" is "Press Ctrl-C to terminate." from standard output. + +# Expected output + +## Parent data format + +The following keys from df-001-httpt.md are used when Psiphon bootstraps: requests, socksproxy, agent. +When Psiphon is not installed or does not bootstrap, only agent and socksproxy are used. + +## Required output data + +psiphon_installed: +**boolean** Whether Psiphon client is found or not (success or failure). + +success: +**boolean** The bootstrap status of Psiphon (success or failure). + +/tmp/<temporary file>: +**dictionary** the parent key of Psiphon's output that contains the keys stdout and stderr and exit_reason + +stdout: +**string** Output produced by Psiphon's standard output. + +stderr: +**string** Error produced by Psiphon's standard error. + +## Data specification version number + +## Semantics + +``` +'psiphon_installed' - True or False - whether Psiphon is found. +'success' - True or False - whether Psiphon has bootstrapped. +'body' - http page body if successfully requested. +'/tmp/<temporary file>': + 'stdout' - Contents of standard output produced by Psiphon. + 'stderr' - Contents of standard error produced by Psiphon. +``` + + +## Possible conclusions + +We can determine whether or not Psiphon is found. +We can determine whether or not Psiphon is able to bootstrap, according to its output. +We can determine whether or not a given URL is reachable via Psiphon. + +## Example output sample +``` +--- +input_hashes: [] +options: [-u, google.com] +probe_asn: AS0 +probe_cc: ZZ +probe_city: null +probe_ip: 127.0.0.1 +report_id: 4dAHr0ceNDBmw5lUQ7pBoxqgyUSfP873Qj1zv5VyElnSSTXwcsLYeCv69DsUjb94 +software_name: ooniprobe +software_version: 1.3.1 +start_time: 1444686051.0 +test_helpers: {} +test_name: psiphon_test +test_version: 0.0.1 +... +--- +/tmp/tmplKg8K3: {exit_reason: process_done, stderr: '', stdout: "./ssh is not a valid\ + \ executable. Using standard ssh.\r\n\r\nYour SOCKS proxy is now running at 127.0.0.1:1080\r\ + \n\r\nPress Ctrl-C to terminate.\r\nTerminating...\r\nConnection closed\r\n"} +agent: agent +input: null +psiphon_installed: true +requests: +- request: + body: null + headers: [] + method: GET + tor: {is_tor: false} + url: http://google.com + response: + body: "<HTML><HEAD><meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\ + >\n<TITLE>301 Moved</TITLE></HEAD><BODY>\n<H1>301 Moved</H1>\nThe document has\ + \ moved\n<A HREF=\"http://www.google.com/\">here</A>.\r\n</BODY></HTML>\r\n" + code: 301 + headers: + - - Content-Length + - ['219'] + - - X-XSS-Protection + - [1; mode=block] + - - Expires + - ['Wed, 11 Nov 2015 21:40:58 UTC'] + - - Server + - [gws] + - - Connection + - [close] + - - Location + - ['http://www.google.com/'] + - - Cache-Control + - ['public, max-age=2592000'] + - - Date + - ['Mon, 12 Oct 2015 21:40:58 UTC'] + - - X-Frame-Options + - [SAMEORIGIN] + - - Content-Type + - [text/html; charset=UTF-8] +socksproxy: 127.0.0.1:1080 +test_runtime: 7.373162031173706 +test_start_time: 1444686052.0 +... +``` + +## Expected Post-processing efforts + +# Privacy considerations + +Psiphon does not seek to provide anonymity. +An adversary can observe that a user is connecting to Psiphon servers. +Psiphon servers can also determine the users location. + +# Packet capture considerations + +This test does not capture packets by default. diff --git a/test-specs/ts-016-openvpn.md b/test-specs/ts-016-openvpn.md new file mode 100644 index 0000000..0346f4a --- /dev/null +++ b/test-specs/ts-016-openvpn.md @@ -0,0 +1,220 @@ +# Specification version number + +2015-10-11-000 + +# Specification name + +OpenVPN Test + +# Test preconditions + +Have OpenVPN installed and configured to work with at least one server and privileges to run the test as root. + +# Expected impact + +Ability to measure whether OpenVPN is working from the given network vantage point. + +# Expected inputs + +A single URL to fetch, supplied by command line argument "--url (-u)". +OpenVPN configuration file, specified by the command line argument "--openvpn-config (-c)" + +# Test description + +This test first launches OpenVPN and parses output to determine if it has bootstrapped. After bootstrap, it fetches the URL specified by the --url argument using OpenVPN. + +The specific string used to determine bootstrap from OpenVPN output in version +"0.0.1" is "Initialization Sequence Completed" from standard output. + +# Expected output + +## Parent data format + +None. + +## Required output data + +success: +**boolean** The bootstrap status of OpenVPN (success or failure). + +OpenVPN_linux --headless: +**dictionary** the parent key of OpenVPNs output that contains the keys stdout and stderr + +stdout: +**string** Output produced by OpenVPNs standard output. + +stderr: +**string** Error produced by OpenVPNs standard error. + +body: +**string** The page body of a successful HTTP request. + +failure: +**string** If failure, then the corresponding failure message. + +## Data specification version number + +## Semantics + +``` +'success' - True or False - whether OpenVPN has bootstrapped. +'body' - http page body if successfully requested. +'failure' - optional, present if there is a failure. +'l/usr/sbin/openvpn --config configfile': + 'stdout' - Contents of standard output produced by OpenVPN. + 'stderr' - Contents of standard error produced by OpenVPN. +``` + +## Possible conclusions + +We can determine whether or not OpenVPN is able to bootstrap, according to its output. +We can determine whether or not a given URL is reachable via OpenVPN. + +## Example output sample +``` +--- +input_hashes: [] +options: [-c, openvpnconfigfile.ovpn, -u, ''] +probe_asn: AS0 +probe_cc: ZZ +probe_city: null +probe_ip: 127.0.0.1 +report_id: nqvK7YrK6J5Di7BiWDwPUBfyKcbLoVWeU4DgnxTzzKWMQABvhC2l3q6aLUwF0CA9 +software_name: ooniprobe +software_version: 1.3.1 +start_time: 1444925440.0 +test_helpers: {} +test_name: test_openvpn_circumvent +test_version: 0.0.1 +... +--- +/usr/sbin/openvpn --config /pathtoopenvpnconfigfile/openvpnconfigfile.ovpn: { + exit_reason: process_done, stderr: '', stdout: 'Thu Oct 15 20:10:40 2015 OpenVPN + 2.3.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] + [IPv6] built on Dec 1 2014 + + Thu Oct 15 20:10:40 2015 WARNING: file ''/tmp/openvpn.txt'' is group or others + accessible + + Thu Oct 15 20:10:40 2015 Control Channel Authentication: tls-auth using INLINE + static key file + + Thu Oct 15 20:10:40 2015 Attempting to establish TCP connection with [AF_INET]10.0.0.10:993 + [nonblock] + + Thu Oct 15 20:10:41 2015 TCP connection established with [AF_INET]10.0.0.10:993 + + Thu Oct 15 20:10:41 2015 TCPv4_CLIENT link local: [undef] + + Thu Oct 15 20:10:41 2015 TCPv4_CLIENT link remote: [AF_INET]10.0.0.10:993 + + Thu Oct 15 20:10:41 2015 WARNING: this configuration may cache passwords in memory + -- use the auth-nocache option to prevent this + + Thu Oct 15 20:10:46 2015 [server] Peer Connection Initiated with [AF_INET]10.0.0.10:993 + + Thu Oct 15 20:10:48 2015 Options error: Unrecognized option or missing parameter(s) + in [PUSH-OPTIONS]:3: dhcp (2.3.2) + + Thu Oct 15 20:10:48 2015 TUN/TAP device tun0 opened + + Thu Oct 15 20:10:48 2015 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0 + + Thu Oct 15 20:10:48 2015 /sbin/ip link set dev tun0 up mtu 1500 + + Thu Oct 15 20:10:48 2015 /sbin/ip addr add dev tun0 local 10.10.0.34 peer 10.10.0.33 + + Thu Oct 15 20:10:48 2015 Initialization Sequence Completed + + '} +body: "<?xml version=\"1.0\"?>\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"\ + \n \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\n<html xmlns=\"http://www.w3.org/1999/xhtml\"\ + >\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\"\ + \ />\n<title>This is a Tor Exit Router</title>\n\n\n\n</head>\n<body>\n\n<p style=\"text-align:center; font-size:xx-large; font-weight:bold\"\ + >This is a\nTor Exit Router</p>\n\n<p>\nMost likely you are accessing this website\ + \ because you had some issue with\nthe traffic coming from this IP. This router\ + \ is part of the <a\nhref=\"https://www.torproject.org/\">Tor Anonymity Network</a>,\ + \ which is\ndedicated to <a href=\"https://www.torproject.org/about/overview\">providing\n\ + privacy</a> to people who need it most: average computer users. This\nrouter IP\ + \ should be generating no other traffic, unless it has been\ncompromised.</p>\n\n\ + <p style=\"text-align:center\">\n<a href=\"https://www.torproject.org/about/overview\"\ + >\n<img src=\"how_tor_works_thumb.png\" alt=\"How Tor works\" style=\"border-style:none\"\ + />\n</a></p>\n\n<p>\nTor sees use by <a href=\"https://www.torproject.org/about/torusers\"\ + >many\nimportant segments of the population</a>, including whistle blowers,\njournalists,\ + \ Chinese dissidents skirting the Great Firewall and oppressive\ncensorship, abuse\ + \ victims, stalker targets, the US military, and law\nenforcement, just to name\ + \ a few. While Tor is not designed for malicious\ncomputer users, it is true that\ + \ they can use the network for malicious ends.\nIn reality however, the actual amount\ + \ of <a\nhref=\"https://www.torproject.org/docs/faq-abuse\">abuse</a> is quite low.\ + \ This\nis largely because criminals and hackers have significantly better access\ + \ to\nprivacy and anonymity than do the regular users whom they prey upon. Criminals\n\ + can and do <a\nhref=\"http://voices.washingtonpost.com/securityfix/2008/08/web_fraud_20_tools.htm…"\ + >build,\nsell, and trade</a> far larger and <a\nhref=\"http://voices.washingtonpost.com/securityfix/2008/08/web_fraud_20_distribut…"\ + >more\npowerful networks</a> than Tor on a daily basis. Thus, in the mind of this\n\ + operator, the social need for easily accessible censorship-resistant private,\n\ + anonymous communication trumps the risk of unskilled bad actors, who are\nalmost\ + \ always more easily uncovered by traditional police work than by\nextensive monitoring\ + \ and surveillance anyway.</p>\n\n<p>\nIn terms of applicable law, the best way\ + \ to understand Tor is to consider it a\nnetwork of routers operating as common\ + \ carriers, much like the Internet\nbackbone. However, unlike the Internet backbone\ + \ routers, Tor routers\nexplicitly do not contain identifiable routing information\ + \ about the source of\na packet, and no single Tor node can determine both the origin\ + \ and destination\nof a given transmission.</p>\n\n<p>\nAs such, there is little\ + \ the operator of this router can do to help you track\nthe connection further.\ + \ This router maintains no logs of any of the Tor\ntraffic, so there is little that\ + \ can be done to trace either legitimate or\nillegitimate traffic (or to filter\ + \ one from the other). Attempts to\nseize this router will accomplish nothing.</p>\n\ + \n\n\n<p>\nIf you are a representative of a company\ + \ who feels that this router is being\nused to violate the DMCA, please be aware\ + \ that this machine does not host or\ncontain any illegal content. Also be aware\ + \ that network infrastructure\nmaintainers are not liable for the type of content\ + \ that passes over their\nequipment, in accordance with <a\nhref=\"http://www.law.cornell.edu/uscode/text/17/512\"\ + >DMCA\n\"safe harbor\" provisions</a>. In other words, you will have just as much\ + \ luck\nsending a takedown notice to the Internet backbone providers. Please consult\n\ + <a href=\"https://www.torproject.org/eff/tor-dmca-response\">EFF's prepared\nresponse</a>\ + \ for more information on this matter.</p>\n\n<p>For more information, please consult\ + \ the following documentation:</p>\n\n<ol>\n<li><a href=\"https://www.torproject.org/about/overview\"\ + >Tor Overview</a></li>\n<li><a href=\"https://www.torproject.org/docs/faq-abuse\"\ + >Tor Abuse FAQ</a></li>\n<li><a href=\"https://www.torproject.org/eff/tor-legal-faq\"\ + >Tor Legal FAQ</a></li>\n</ol>\n\n<p>\nThat being said, if you still have a complaint\ + \ about the router, you may\nemail the <a href=\"mailto:tor@openvpnconfigfile.ie\">maintainer</a>.\ + \ If\ncomplaints are related to a particular service that is being abused, I will\n\ + consider removing that service from my exit policy, which would prevent my\nrouter\ + \ from allowing that traffic to exit through it. I can only do this on an\nIP+destination\ + \ port basis, however. Common P2P ports are\nalready blocked.</p>\n\n<p>\nYou also\ + \ have the option of blocking this IP address and others on\nthe Tor network if\ + \ you so desire. The Tor project provides a <a\nhref=\"https://check.torproject.org/cgi-bin/TorBulkExitList.py\"\ + >web service</a>\nto fetch a list of all IP addresses of Tor exit nodes that allow\ + \ exiting to a\nspecified IP:port combination, and an official <a\nhref=\"https://www.torproject.org/tordnsel/dist/\"\ + >DNSRBL</a> is also available to\ndetermine if a given IP address is actually a\ + \ Tor exit server. Please\nbe considerate\nwhen using these options. It would be\ + \ unfortunate to deny all Tor users access\nto your site indefinitely simply because\ + \ of a few bad apples.</p>\n\n<p style=\"text-align:center; margin-bottom: 0.5em\"\ + >Exit Node provided by:<p>\n<h2 style=\"text-align: center\"><a style=\"color: black\"\ + \ href=\"http://www.openvpnconfigfile.ie\">DU Pirate Party</a><h2>\n\n</body>\n</html>\n" +input: null +success: true +test_runtime: 8.374207019805908 +test_start_time: 1444925440.0 +... +``` + +## Expected Post-processing efforts + +# Privacy considerations + +OpenVPN does not seek to provide anonymity. +An adversary can observe that a user is connecting to OpenVPN servers. +OpenVPN servers can also determine the users location. + +# Packet capture considerations + +This test does not capture packets by default.

1 0

[spec/master] Update to reflect normalisations performed in pipeline the specs of the base and the HTTP data formats
by art＠torproject.org 09 May '16

09 May '16

commit 2e519c8da7946467e3615f05d0da1a12be920fe8 Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Jan 27 12:04:53 2016 +0100 Update to reflect normalisations performed in pipeline the specs of the base and the HTTP data formats --- data-formats/df-000-base.md | 166 +++++++++++----------- data-formats/df-001-httpt.md | 328 ++++++++++++++++++++++++++++--------------- 2 files changed, 301 insertions(+), 193 deletions(-) diff --git a/data-formats/df-000-base.md b/data-formats/df-000-base.md index d0d5378..c1431e3 100644 --- a/data-formats/df-000-base.md +++ b/data-formats/df-000-base.md @@ -14,113 +14,116 @@ In this directory shall go only the data format specifications of Test templates. The Test specific data formats should go in the specification of the test. -All data produced from ooniprobe tests is in YAML formatted. +Data produced by the ooniprobe client can be either in YAML or JSON format. -Every test that is interested in reporting with ooniprobe MUST use such data -format. +YAML is used when writing reports to the users filesystem, while JSON is used +as a format for the published processed reports. # Base test data format This specification is of the basic data format common to all ooniprobe test outputs. -Data Format Version: df-000-base-001 +Every entry contains the following common fields. The `test_keys` key will +contain instead all the keys that are specific to the test in question. + +The JSON data format is made up of a series of JSON documents separated by +newline characters. + +Data Format Version: 0.2.0 ## Specification - ########################################### - # OONI Probe Report for HTTP Requests test - # Wed Jan 30 21:03:56 2013 - ########################################### - --- - options: - A dict containing the keys and values of options passed to the test. +``` +{ + "input": "If the test takes an input this will contain the value of it" + " these can be for example URLs, hostnames, IPs, etc.", - probe_asn: - The AS Number of the probe (prefixed by AS, ex. AS1234) or null if includeasn is set to false. + "input_hashes": "A list of the SHA256 hash of encoded " + "as hex of the inputs to this test.", - probe_cc: - The two letter country code of the probe or null if inlcudecc is set to false. + "id": "This is an identifier of this particular measurement", - probe_ip: - The IPv4 address of the probe or null if includeip is set to false. + "bucket_date": "A date in the format of %Y-%m-%d that indicates " + "when the report was processed by the data pipeline" - software_name: - The name of the software that has generated such report (ex. ooniprobe). + "data_format_version": "0.1.0|0.2.0", - software_version: - The version of the software that has generated such report (ex. 0.0.10). + "report_filename": "{bucket_date}/{timestamp as '%Y%m%dT%h%M%sZ'}-{probe_cc}-{probe_asn}-{test_name}-{report_id}-{data_format_version}-{probe|backend}.json", - start_time: - The time at which the test was started in seconds since epoch. + "options": ["A list of options passed to the test as command line arguments"], - test_name: - The name of the test that such report is for (ex. HTTP Requests). + "probe_asn": "The AS Number of the probe (prefixed by AS, ex. AS1234) " + "or AS0 if includeasn is set to false.", - test_version: - The version of the test that such report is for (ex. 0.0.10). + "probe_cc": "The two letter country code of the probe or ZZ if " + "inlcudecountry is set to false.", - data_format_version: - The version string of the data format being used by the test (ex. httpt-000) - - report_id: - A 64 character mixed case string that is generated by the client used to identify the report. + "probe_ip": "The IPv4 address of the probe or 127.0.0.1 if " + "includeip is set to false.", - test_helpers: - A dictionary with as keys the names of the options and values the addresses of the test helpers used - ... + "probe_ip": "The name of the city of the probe or null if " + "includecity is set to false.", -# Example output + "report_id": "20140130T111423Z_ELNkuajQzUWfktBupbfZUxseQDczEvEaIhtciykhoLSuiNiCCV", - ########################################### - # OONI Probe Report for HTTP Invalid Request Line test - # Mon Jan 28 21:33:59 2013 - ########################################### - --- - options: - collector: null - help: 0 - logfile: null - parallelism: '10' - pcapfile: null - reportfile: null - resume: 0 - subargs: [-b, 'http://93.95.227.200'] - test: nettests/manipulation/http_invalid_request_line.py - probe_asn: null - probe_cc: null - probe_ip: null - software_name: ooniprobe - software_version: 0.0.10 - start_time: 1359401639.0 - test_name: HTTP Invalid Request Line - test_version: 0.1.3 - test_helpers: {backend: "http://93.95.227.200"} - report_id: xxxxxxxXXXxXXXxxxxxxxxxxxxXXXxXXXxsxxxXXXxXXXxxxXXXxXXXxxxXXXxX - ... - - -# Report Entry data format - -Every iteration over an input given to a test will produce a Report Entry. - -A Report Entry is a YAML Stream as specified here: -http://www.yaml.org/spec/1.2/spec.html#id2801681 - -Here are specified the keys that will always be present inside of every report -entry. + "software_name": "The name of the software that has generated " + "such report (ex. ooniprobe)", -## Specification + "software_version": "The version of the software used to generate this report", + + "backend_version": "The version of the backend that collected this measurement", + + "test_helpers": null, -input: - The item we this specific test instance is referring to. null in case no - input is being iterated over. + "test_name": "The name of the test that generated " + "this measurement (ex. http_requests)", -test_runtime: - `float` the runtime of the test + "test_version": "", -test_start_time: - `float` seconds since epoch from the starting of the test. + "test_runtime": null, + + "test_start_time": "Timestamp of when the measurement was performed in " + "UTC time coordinates (ex. 2015-08-24 12:02:23)", + + "test_keys": { + "The keys that are specific to the test" + } +} +``` + +# Example output + +``` +{ + "bucket_date": "2015-11-22", + "data_format_version": "0.2.0", + "id": "07873c37-9441-47e3-93b8-94db10444c64", + "input": "http://example.com/", + "options": [ + "-f", + "37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1" + ], + "probe_asn": "AS0", + "probe_cc": "CH", + "probe_ip": "127.0.0.1", + "report_filename": "2015-11-22/20151122T103202Z-CH-AS0-http_requests-XsQk40qrhgvJEdbXAUFzYjbbGCBuEsc1UV5RAAFXo4hysiUo3qyTfo4NTr7MjiwN-0.1.0-probe.json", + "report_id": "XsQk40qrhgvJEdbXAUFzYjbbGCBuEsc1UV5RAAFXo4hysiUo3qyTfo4NTr7MjiwN", + "software_name": "ooniprobe", + "software_version": "1.3.1", + "backend_version": "1.1.4", + "test_helpers": {}, + "input_hashes": [ + "37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1" + ], + "test_name": "http_requests", + "test_runtime": 0.1842639446, + "test_start_time": "2015-11-22 10:32:02", + "test_version": "0.2.4" + "test_keys": { + }, +} +``` # Error strings @@ -164,4 +167,3 @@ error_string: * This will be the error message if the task has timed out: `task_timed_out` * Every other failure: 'unknown_failure %s' % str(failure.value) - diff --git a/data-formats/df-001-httpt.md b/data-formats/df-001-httpt.md index ed8fb63..8479a6d 100644 --- a/data-formats/df-001-httpt.md +++ b/data-formats/df-001-httpt.md @@ -1,6 +1,6 @@ # HTTPTest template data format -Data Format Version: df-001-httpt-000 +Data Format Version: 0.2.0 This is the specification of the data format that every test that is based on ooni.templates.httpt.HTTPTest shall be using. @@ -10,116 +10,222 @@ data format. ## Specification - --- - requests: - - request: - headers: - `dict` the headers of the request - body: - `string` the body of the response - - url: - `string` the URL of the request being made (if prefixed with 's' it means - the request was made via the Tor SOCKS proxy) - - method: - `string` the HTTP method being used - - response: - headers: - `dict` the headers of the response - body: - `string` the body of the response - - code: - `int` the response status code - - failure: - `string` (optional) this will be set if an error was returned. - For a list of error messages see the Error strings section of - df-000-base.md. - - - request: - etc. etc. - - socksproxy: - null if no socks proxy was used for this request or an IP port - combination (as a string) if a SOCKS proxy was used. - - agent: - either 'agent' if 30X redirects should not be followed or 'redirect' if - they should be followed. - - ... +``` +"agent": "agent|redirect depending on weither the client " + "will ignore 30X redirects or follow them.", + +"socksproxy": "null | IP:PORT of the socksproxy to be used to " + "perform the experiment requests on", + +"requests": [ + { + "failure": "This will contain an error string for why the " + "request failed or null if no failure occurred", + + "request": { + "body": "If the request of the client contains some payload it " + "will be in here, otherwise it is set to null", + + "headers": { + "Header-Name": "Header-Value" + }, + + "method": "GET|POST|PUT", + "tor": { + "exit_ip": "The address of the Tor exit used for the request or " + "null if Tor was not used or the test was run with an older version of ooniprobe.", + + "exit_name": "The name of the Tor exit used for the request or " + "null if Tor was not used or the test was run with an older version of ooniprobe.", + + "is_tor": "true|false depending on wether or not " + "this request was done over Tor or not." + }, + "url": "The URL of the request that has been performed." + }, + "response": { + "body": "The body of the response or null if not response was found. If the response is binary " + "then this will be a dictionary containing the format in which the binary data is encoded and " + "the encoded data (ex. {"format": "base64", "data": "AQI="}). " + "Currently the only type of format supported is base64.", + + "headers": { + "Header-Name": "Header-Value" + } + }, + "response_length": null + } +] +``` ## Example output - input: http://google.com/ - agent: agent - requests: - - request: - body: null - headers: - - - User-Agent - - - &id001 [Mozilla/4.0 (compatible; MSIE 5.0; Windows NT 5.1; .NET CLR 1.1.4322), - 'Internet Explorer 5, Windows XP'] - method: GET - url: http://google.com/ - response: - body: '' - code: 301 - headers: - - - Content-Length - - ['219'] - - - X-XSS-Protection - - [1; mode=block] - - - Expires - - ['Tue, 29 Jan 2013 14:29:19 GMT'] - - - Server - - [gws] - - - Connection - - [close] - - - Location - - ['http://www.google.com/'] - - - Cache-Control - - ['public, max-age=2592000'] - - - Date - - ['Sun, 30 Dec 2012 14:29:19 GMT'] - - - X-Frame-Options - - [SAMEORIGIN] - - - Content-Type - - [text/html; charset=UTF-8] - - request: - body: null - headers: - - - User-Agent - - - *id001 - method: GET - url: shttp://google.com/ - response: - body: '' - code: 301 - headers: - - - Content-Length - - ['219'] - - - X-XSS-Protection - - [1; mode=block] - - - Expires - - ['Tue, 29 Jan 2013 14:29:20 GMT'] - - - Server - - [gws] - - - Connection - - [close] - - - Location - - ['http://www.google.com/'] - - - Cache-Control - - ['public, max-age=2592000'] - - - Date - - ['Sun, 30 Dec 2012 14:29:20 GMT'] - - - X-Frame-Options - - [SAMEORIGIN] - - - Content-Type - - [text/html; charset=UTF-8] - socksproxy: null - - +``` +{ + "bucket_date": "2015-11-22", + "data_format_version": "0.2.0", + "id": "07873c37-9441-47e3-93b8-94db10444c64", + "input": "http://googleusercontent.com/", + "options": [ + "-f", + "37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1" + ], + "probe_asn": "AS0", + "probe_cc": "CH", + "probe_ip": "127.0.0.1", + "report_filename": "2015-11-22/20151122T103202Z-CH-AS0-http_requests-XsQk40qrhgvJEdbXAUFzYjbbGCBuEsc1UV5RAAFXo4hysiUo3qyTfo4NTr7MjiwN-0.1.0-probe.json", + "report_id": "XsQk40qrhgvJEdbXAUFzYjbbGCBuEsc1UV5RAAFXo4hysiUo3qyTfo4NTr7MjiwN", + "software_name": "ooniprobe", + "software_version": "1.3.1", + "test_helpers": {}, + "backend_version": "1.1.4", + "input_hashes": [ + "37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1" + ], + "probe_city": null, + "test_name": "http_requests", + "test_runtime": 0.1842639446, + "test_start_time": "2015-11-22 10:32:02", + "test_version": "0.2.4" + "test_keys": { + "agent": "agent", + "body_length_match": null, + "body_proportion": null, + "control_failure": "socks_host_unreachable", + "experiment_failure": "dns_lookup_error", + "factor": 0.8, + "headers_diff": null, + "headers_match": null, + "requests": [ + { + "failure": "dns_lookup_error", + "request": { + "body": null, + "headers": { + "User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6" + }, + "method": "GET", + "tor": { + "exit_ip": false, + "exit_name": false, + "is_tor": false + }, + "url": "http://googleusercontent.com/" + }, + "response": { + "body": null, + "headers": {} + }, + "response_length": null + }, + { + "failure": "socks_host_unreachable", + "request": { + "body": null, + "headers": { + "User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7" + }, + "method": "GET", + "tor": { + "exit_ip": null, + "exit_name": null, + "is_tor": true + }, + "url": "http://googleusercontent.com/" + }, + "response": { + "body": null, + "headers": {} + }, + "response_length": null + }, + { + "failure": "dns_lookup_error", + "request": { + "body": null, + "headers": { + "User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6" + }, + "method": "GET", + "tor": { + "exit_ip": null, + "exit_name": null, + "is_tor": false + }, + "url": "http://googleusercontent.com/" + }, + "response": { + "body": null, + "headers": {} + }, + "response_length": null + }, + { + "failure": "dns_lookup_error", + "request": { + "body": null, + "headers": { + "User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6" + }, + "method": "GET", + "tor": { + "exit_ip": null, + "exit_name": null, + "is_tor": false + }, + "url": "http://googleusercontent.com/" + }, + "response": { + "body": null, + "headers": {} + }, + "response_length": null + }, + { + "failure": "socks_host_unreachable", + "request": { + "body": null, + "headers": { + "User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6" + }, + "method": "GET", + "tor": { + "exit_ip": null, + "exit_name": null, + "is_tor": true + }, + "url": "http://googleusercontent.com/" + }, + "response": { + "body": null, + "headers": {} + }, + "response_length": null + }, + { + "failure": "socks_host_unreachable", + "request": { + "body": null, + "headers": { + "User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6" + }, + "method": "GET", + "tor": { + "exit_ip": null, + "exit_name": null, + "is_tor": true + }, + "url": "http://googleusercontent.com/" + }, + "response": { + "body": null, + "headers": {} + }, + "response_length": null + } + ], + "socksproxy": null, + "start_time": 1448184722.0 + } +} +```

1 0

[spec/master] Update the specifications of DNS and Scapy
by art＠torproject.org 09 May '16

09 May '16

commit 7c59c2fb301e79803db8ff5fb339e45d37e8da77 Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Jan 27 13:04:55 2016 +0100 Update the specifications of DNS and Scapy --- data-formats/df-002-dnst.md | 268 ++++++++++++------ data-formats/df-003-scapy.md | 661 ++++++++++++++++++++++++++++++++++++------- 2 files changed, 748 insertions(+), 181 deletions(-) diff --git a/data-formats/df-002-dnst.md b/data-formats/df-002-dnst.md index a3e7909..4e1ac26 100644 --- a/data-formats/df-002-dnst.md +++ b/data-formats/df-002-dnst.md @@ -1,6 +1,6 @@ # DNSTest template data format -Data Format Version: df-001-dnst-000 +Data Format Version: 0.2.0 This is the specification of the data format that every test that is based on ooni.templates.dnst.DNSTest shall be using. @@ -10,97 +10,193 @@ data format. ## Specification - queries: - `list` of DNS query and answers for this testing session +``` +"queries": [ + { + "failure": "null or a string representing the failure", -Every DNS query is formatted as follows: + "hostname": + "The hostname used in the DNS query. For PTR lookups this will be the reversed " + "ipv4 address in the form: xxx.xxx.xxx.xxx.in-addr.arpa.", - addrs: - `list` present if the DNS query is question is of A type and will include - the IPv4 addresses of the answer. + "query_type": "A | MX | PTR | SOA", - answers: - - [`string` representation of the Resource Record of the answer (repr), - `string` representation of the answer payload (python repr)] - - [etc. etc.] + "resolver_hostname": + "The hostname of the DNS resolver being used", - query: - `string` that is the representation (python repr) of the DNS querie(s) - that where performed. - This string can be passed to an instance of `twisted.names.client.Resolver` - and it will reproduce the DNS query of this report. - The Query object is described in `twisted.names.dns.Query`. + "resolver_port": + "The port of the DNS resolver being used" - query_type: - `string` the name of the DNS query that is being performed. + "answers": [ + { + "answer_type": "A | MX | PTR | SOA | CNAME", - resolver: [`string` resolver IP address, `int` resolver port number] + "ipv4": + "(only for A answers) The dotted quad of the IPV4 address", -## Example output + "ttl": "The ttl of the Returned resource record", + + "hostname": + "(only for PTR, SOA and CNAME answers)" + "For PTR lookups it is the hostname that points back to the queried IP." + "For CNAME lookups it is the hostname of the alias for the IP." + "For SOA answers it is the hostname of the nameserver " + "responsible for the data of this zone." + + "responsible_name": + "(only for SOA answers) The mailbox of the person responsible " + "for this zone." + + "serial_number": + "(only for SOA answers) Version number of the original copy of " + "the zone." + + "refresh_interval": + "(only for SOA answers) The time interval before which this " + "zone should be refreshed." + + "retry_interval": + "(only for SOA answers) The time interval that should be " + "elapsed before the zone should be retried in case of failure." - input: example.com - queries: - - addrs: [203.0.113.2] - answers: - - [<RR name=example.com type=A class=IN ttl=2562s auth=False>, <A address=203.0.113.2 - ttl=2562>] - query: '[Query(''example.com'', 1, 1)]' - query_type: A - resolver: *id001 - - addrs: [203.0.113.2] - answers: - - [<RR name=example.com type=A class=IN ttl=7200s auth=False>, <A address=203.0.113.2 - ttl=7200>] - query: '[Query(''example.com'', 1, 1)]' - query_type: A - resolver: [203.0.113.3, 53] - - addrs: [203.0.113.2] - answers: - - [<RR name=example.com type=A class=IN ttl=4007s auth=False>, <A address=203.0.113.2 - ttl=4007>] - query: '[Query(''example.com'', 1, 1)]' - query_type: A - resolver: [203.0.113.4, 53] - - addrs: [203.0.113.2] - answers: - - [<RR name=example.com type=A class=IN ttl=7200s auth=False>, <A address=203.0.113.2 - ttl=7200>] - query: '[Query(''example.com'', 1, 1)]' - query_type: A - resolver: [203.0.113.5, 53] - - addrs: [203.0.113.2] - answers: - - [<RR name=example.com type=A class=IN ttl=7200s auth=False>, <A address=203.0.113.2 - ttl=7200>] - query: '[Query(''example.com'', 1, 1)]' - query_type: A - resolver: [203.0.113.6, 53] - - addrs: [203.0.113.2] - answers: - - [<RR name=example.com type=A class=IN ttl=7200s auth=False>, <A address=203.0.113.2 - ttl=7200>] - query: '[Query(''example.com'', 1, 1)]' - query_type: A - resolver: [203.0.113.7, 53] - - addrs: [203.0.113.2] - answers: - - [<RR name=example.com type=A class=IN ttl=7200s auth=False>, <A address=203.0.113.2 - ttl=7200>] - query: '[Query(''example.com'', 1, 1)]' - query_type: A - resolver: [203.0.113.8, 53] - - query: '[Query(''example.com'', 1, 1)]' - query_type: A - resolver: [203.0.113.9, 53] - - query: '[Query(''example.com'', 1, 1)]' - query_type: A - resolver: [203.0.113.1, 53] - - failure: deferred_timeout_error - query: '[Query(''example.com'', 1, 1)]' - query_type: A - resolver: [203.0.113.10, 53] - test_name: test_a_lookup - test_runtime: 2.6424369812011719 - test_started: 1357167921.0688701 - ... + "minimum_ttl": + "(only for SOA answers) The minimum ttl to be exported with " + "any record from this zone." + + "expiration_limit": + "(only for SOA answers) The time after which this zone should " + "no longer be authoritative." + } + ], + + + } +] +``` + +## Example output +``` +{ + "bucket_date": "2015-11-23", + "data_format_version": "0.2.0", + "id": "e6f6257f-e7c2-48fa-8345-b7ed055ab1d2", + "input": "198.175.124.185", + "options": [ + "-f", + "citizenlab-urls-global.txt", + "-T", + "dns-server-jo.txt" + ], + "probe_asn": "AS8376", + "probe_cc": "JO", + "probe_ip": "127.0.0.1", + "report_filename": "2015-11-23/20151123T161428Z-JO-AS8376-dns_consistency-F1KI1WusW4c1T6OGyQDgHJjaSQ1bfpqV2G39bpmuHiLAJnse8R1F44vdRuTz6nO4-0.1.0-probe.json", + "report_id": "F1KI1WusW4c1T6OGyQDgHJjaSQ1bfpqV2G39bpmuHiLAJnse8R1F44vdRuTz6nO4", + "software_name": "ooniprobe", + "software_version": "1.3.1", + "test_helpers": { + "backend": "8.8.8.8:53" + }, + "input_hashes": [ + "0055f0881fba857d8b48123017d7aec83014e89f057e44b66107f657ec5e2eab" + ], + "probe_city": null, + "backend_version": "1.1.4", + "test_keys": { + "control_resolver": "8.8.8.8:53", + "errors": { + "212.118.0.1": "no_answer", + "212.118.0.2": "no_answer", + "212.38.128.3": "dns_lookup_error", + "217.144.6.6": "no_answer", + "8.8.8.8:53": "no_answer", + "80.90.160.135": "dns_lookup_error", + "80.90.160.172": "dns_lookup_error", + "81.28.112.2": "dns_lookup_error" + }, + "failed": [ + "80.90.160.172", + "8.8.8.8:53", + "212.118.0.2", + "80.90.160.135", + "212.38.128.3", + "212.118.0.1", + "217.144.6.6", + "81.28.112.2" + ], + "queries": [ + { + "answers": [], + "failure": "no_answer", + "hostname": "198.175.124.185", + "query_type": "A", + "resolver_hostname": "8.8.8.8", + "resolver_port": 53 + }, + { + "answers": [], + "failure": "deferred_timeout_error", + "hostname": "198.175.124.185", + "query_type": "A", + "resolver_hostname": "212.38.128.3", + "resolver_port": 53 + }, + { + "answers": [], + "failure": "no_answer", + "hostname": "198.175.124.185", + "query_type": "A", + "resolver_hostname": "217.144.6.6", + "resolver_port": 53 + }, + { + "answers": [], + "failure": "deferred_timeout_error", + "hostname": "198.175.124.185", + "query_type": "A", + "resolver_hostname": "81.28.112.2", + "resolver_port": 53 + }, + { + "answers": [], + "failure": "deferred_timeout_error", + "hostname": "198.175.124.185", + "query_type": "A", + "resolver_hostname": "80.90.160.135", + "resolver_port": 53 + }, + { + "answers": [], + "failure": "deferred_timeout_error", + "hostname": "198.175.124.185", + "query_type": "A", + "resolver_hostname": "80.90.160.172", + "resolver_port": 53 + }, + { + "answers": [], + "failure": "no_answer", + "hostname": "198.175.124.185", + "query_type": "A", + "resolver_hostname": "212.118.0.1", + "resolver_port": 53 + }, + { + "answers": [], + "failure": "no_answer", + "hostname": "198.175.124.185", + "query_type": "A", + "resolver_hostname": "212.118.0.2", + "resolver_port": 53 + } + ], + "start_time": 1448291668.0, + "successful": [] + }, + "test_name": "dns_consistency", + "test_runtime": 0.0837070942, + "test_start_time": "2015-11-23 16:14:28", + "test_version": "0.6" +} +``` diff --git a/data-formats/df-003-scapy.md b/data-formats/df-003-scapy.md index c1db4fb..cae1972 100644 --- a/data-formats/df-003-scapy.md +++ b/data-formats/df-003-scapy.md @@ -1,113 +1,586 @@ # ScapyTest data format -Data Format Version: df-003-scapy-000 +Data Format Version: 0.2.0 This specifies the data format for tests that are based on ooni.templates.scapyt.ScapyTest. ## Specification - answer_flags: - `list` of options that are set for determining how to understand if a - received packet is answer to a sent packet (these only apply to ICMP - messages) +**sent_packets** are the packets that have been generated and sent over the +network by the probe. - 'ipsrc' means that we check to see if the src and destination ports in - the ICMP IP citation match. +**answered_packets** are the packets that match up as answers according to the +rules defined in the **answers_flags** options. - 'ipid' means that we look at the IPID in the response to match it up - to sent packets. +``` +{ + "answer_flags": [ + "List of options that are set for determining how to understand if a" + "received packet is answer to a sent packet (these only apply to ICMP" + "messages)" - 'seqack' means that we check if TCP sequence number and ACK match in - the ICMP citation when processing TCP inside of ICMP. + "'ipsrc' means that we check to see if the src and destination ports in" + "the ICMP IP citation match." - sent_packets: - This contains the list of packets that have been sent. + "'ipid' means that we look at the IPID in the response to match it up" + "to sent packets." - - raw_packet: - `string` BASE64Encoding of packet the packet inclusive of IP - headers + "'seqack' means that we check if TCP sequence number and ACK match in" + "the ICMP citation when processing TCP inside of ICMP." - summary: - `string` human readable representation of the packet as is the - output of scapy.packet.summary(). - - - etc. etc. - - answered_packets: - This contains the list of packets that are considered to be the - answer of the above sent packets. - - - raw_packet: - `string` BASE64Encoding of packet the packet inclusive of IP - headers - - summary: - `string` human readable representation of the packet as is the - output of scapy.packet.summary(). - - - etc. etc. + ], + "answered_packets": [ + { + "raw_packet": { + "data": "Encoding of packet the packet inclusive of IP header. " + "The type of encoding is specified in the format field." + "format": "The encoding of the data field. Currently only supports base64." + }, + "summary": + "A human readable representation of the packet as is the output " + "of repr on the scapy.packet object." + }, + ], + "sent_packets": [ + { + "raw_packet": { + "data": "Encoding of packet the packet inclusive of IP header. " + "The type of encoding is specified in the format field." + "format": "The encoding of the data field. Currently only supports base64." + }, + "summary": + "A human readable representation of the packet as is the output " + "of repr on the scapy.packet object." + }, + ], +} +``` ## Example output - answer_flags: [ipsrc] - answered_packets: - - - raw_packet: !!binary | - RQAAqI08AACAATC61Lw2eX8AAAELACiDAAAAAEUAABzCJ0AAARGEQMCosIAuppOaG2gBuwAIr1EA - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIACQ/gAIAQFM9wEB - summary: IP / ICMP 212.188.54.121 > 127.0.0.1 time-exceeded ttl-zero-during-transit - / IPerror / UDPerror / Padding - - - raw_packet: !!binary | - RQAAqI09AACAAUoF1LwdLX8AAAELACiDAAAAAEUAABzCJ0AAARGEQMCosIAuppOaG2gBuwAIr1EA - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIACL1wAIAQFSHgEB - summary: IP / ICMP 212.188.29.45 > 127.0.0.1 time-exceeded ttl-zero-during-transit - / IPerror / UDPerror / Padding - - - raw_packet: !!binary | - RQAAOI0+AACAAUn71Lwdpn8AAAELACiDAAAAAEUAABzCJ0AAARGEQMCosIAuppOaG2gBuwAIr1E= - summary: IP / ICMP 212.188.29.166 > 127.0.0.1 time-exceeded ttl-zero-during-transit - / IPerror / UDPerror - - - raw_packet: !!binary | - RQAAqI0/AACAAUrm1LwcSn8AAAELACiDAAAAAEUAABzCJ0AAARGEQMCosIAuppOaG2gBuwAIr1EA - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIACLtwAIAQFOPgUB - summary: IP / ICMP 212.188.28.74 > 127.0.0.1 time-exceeded ttl-zero-during-transit - / IPerror / UDPerror / Padding - - - raw_packet: !!binary | - RQAAOI1BAACAAT4U1Lwpin8AAAELACiDAAAAAEUAABzCJ0AAARGEQMCosIAuppOaG2gBuwAIr1E= - summary: IP / ICMP 212.188.41.138 > 127.0.0.1 time-exceeded ttl-zero-during-transit - / IPerror / UDPerror - - - raw_packet: !!binary | - RQAAOI1CAACAAYzr2cfVpn8AAAELACiDAAAAAEUAABzCJ0AAARGEQMCosIAuppOaG2gBuwAIr1E= - summary: IP / ICMP 217.199.213.166 > 127.0.0.1 time-exceeded ttl-zero-during-transit - / IPerror / UDPerror - - - raw_packet: !!binary | - RQAAOI1DAACAAXqrLqaTB38AAAELACiDAAAAAEUAABzCJ0AAARGEQMCosIAuppOaG2gBuwAIr1E= - summary: IP / ICMP 46.166.147.7 > 127.0.0.1 time-exceeded ttl-zero-during-transit - / IPerror / UDPerror - sent_packets: - - - raw_packet: !!binary | - RQAAHH1NAAABEftCfwAAAS6mk5obaAG7AAiheQ== - summary: IP / UDP 127.0.0.1:7016 > 46.166.147.154:https - - - raw_packet: !!binary | - RQAAHG71AAACEQibfwAAAS6mk5obaAG7AAiheQ== - summary: IP / UDP 127.0.0.1:7016 > 46.166.147.154:https - - - raw_packet: !!binary | - RQAAHAUlAAADEXFrfwAAAS6mk5obaAG7AAiheQ== - summary: IP / UDP 127.0.0.1:7016 > 46.166.147.154:https - - - raw_packet: !!binary | - RQAAHBqUAAAEEVr8fwAAAS6mk5obaAG7AAiheQ== - summary: IP / UDP 127.0.0.1:7016 > 46.166.147.154:https - - - raw_packet: !!binary | - RQAAHPuMAAAFEXkDfwAAAS6mk5obaAG7AAiheQ== - summary: IP / UDP 127.0.0.1:7016 > 46.166.147.154:https - - - raw_packet: !!binary | - RQAAHK4cAAAGEcVzfwAAAS6mk5obaAG7AAiheQ== - summary: IP / UDP 127.0.0.1:7016 > 46.166.147.154:https - - - raw_packet: !!binary | - RQAAHFyiAAAHERXufwAAAS6mk5obaAG7AAiheQ== - summary: IP / UDP 127.0.0.1:7016 > 46.166.147.154:https +``` +{ + "bucket_date": "2015-11-13", + "data_format_version": "0.2.0", + "id": "ea9f588b-49ca-4261-babf-0111307877c4", + "input": null, + "options": [], + "probe_asn": "AS8048", + "probe_cc": "VE", + "probe_ip": "127.0.0.1", + "report_filename": "2015-11-13/20151113T104654Z-VE-AS8048-multi_protocol_traceroute-iq0hFnbx4ex7bAvysJBenb9uJAuh7LT02BhDrvIhK6Lpwe7PdCvJ7BHFzm4voYe7-0.1.0-probe.json", + "report_id": "iq0hFnbx4ex7bAvysJBenb9uJAuh7LT02BhDrvIhK6Lpwe7PdCvJ7BHFzm4voYe7", + "software_name": "ooniprobe", + "software_version": "1.3.1", + "test_helpers": { + "backend": "213.138.109.232" + }, + "backend_version": "1.1.4", + "input_hashes": [], + "probe_city": null, + "test_keys": { + "answer_flags": [ + "ipsrc" + ], + "answered_packets": [ + { + "raw_packet": { + "data": "RcAAOFqPAAAwAeD/1Ypt6AoAAQQDA6VKAAAAAEUAABxEnwAAAREmvAoAAQTVim3oXIYAFgAI+w0=", + "format": "base64" + }, + "summary": "[<IP version=4L ihl=5L tos=0xc0 len=56 id=23183 flags= frag=0L ttl=48 proto=icmp chksum=0xe0ff src=213.138.109.232 dst=10.0.1.4 options=[] |<ICMP type=dest-unreach code=port-unreachable chksum=0xa54a unused=0 |<IPerror version=4L ihl=5L tos=0x0 len=28 id=17567 flags= frag=0L ttl=1 proto=udp chksum=0x26bc src=10.0.1.4 dst=213.138.109.232 options=[] |<UDPerror sport=23686 dport=ssh len=8 chksum=0xfb0d |>>>>]" + }, + { + "raw_packet": { + "data": "RQAAOBGSAAD/AZQuCgABAQoAAQQLAEOQAAAAAEUAABzCWAAAARGpAgoAAQTVim3ocGMAFwAIQO0=", + "format": "base64" + }, + "summary": "[<IP version=4L ihl=5L tos=0x0 len=56 id=4498 flags= frag=0L ttl=255 proto=icmp chksum=0x942e src=10.0.1.1 dst=10.0.1.4 options=[] |<ICMP type=time-exceeded code=ttl-zero-during-transit chksum=0x4390 unused=0 |<IPerror version=4L ihl=5L tos=0x0 len=28 id=49752 flags= frag=0L ttl=1 proto=udp chksum=0xa902 src=10.0.1.4 dst=213.138.109.232 options=[] |<UDPerror sport=28771 dport=telnet len=8 chksum=0x40ed |>>>>]" + }, + { + "raw_packet": { + "data": "RcAAOD57AADvAetXW986TwoAAQQLAJ1NAAAAAEUAABwySQAAARE5EgoAAQTVim3ogEgANQAI1yw=", + "format": "base64" + }, + "summary": "[<IP version=4L ihl=5L tos=0xc0 len=56 id=15995 flags= frag=0L ttl=239 proto=icmp chksum=0xeb57 src=91.223.58.79 dst=10.0.1.4 options=[] |<ICMP type=time-exceeded code=ttl-zero-during-transit chksum=0x9d4d unused=0 |<IPerror version=4L ihl=5L tos=0x0 len=28 id=12873 flags= frag=0L ttl=1 proto=udp chksum=0x3912 src=10.0.1.4 dst=213.138.109.232 options=[] |<UDPerror sport=32840 dport=domain len=8 chksum=0xd72c |>>>>]" + }, + { + "raw_packet": { + "data": "RcAAOCLEAAAwARjL1Ypt6AoAAQQDA6VKAAAAAEUAABwa6wAACxFGcAoAAQTVim3oZO4AFgAI8qU=", + "format": "base64" + }, + "summary": "[<IP version=4L ihl=5L tos=0xc0 len=56 id=8900 flags= frag=0L ttl=48 proto=icmp chksum=0x18cb src=213.138.109.232 dst=10.0.1.4 options=[] |<ICMP type=dest-unreach code=port-unreachable chksum=0xa54a unused=0 |<IPerror version=4L ihl=5L tos=0x0 len=28 id=6891 flags= frag=0L ttl=11 proto=udp chksum=0x4670 src=10.0.1.4 dst=213.138.109.232 options=[] |<UDPerror sport=25838 dport=ssh len=8 chksum=0xf2a5 |>>>>]" + }, + { + "raw_packet": { + "data": "RQAAOBGYAAD/AZQoCgABAQoAAQQLAEOQAAAAAEUAABxnSAAAAREEEwoAAQTVim3oU6f//wAIXcA=", + "format": "base64" + }, + "summary": "[<IP version=4L ihl=5L tos=0x0 len=56 id=4504 flags= frag=0L ttl=255 proto=icmp chksum=0x9428 src=10.0.1.1 dst=10.0.1.4 options=[] |<ICMP type=time-exceeded code=ttl-zero-during-transit chksum=0x4390 unused=0 |<IPerror version=4L ihl=5L tos=0x0 len=28 id=26440 flags= frag=0L ttl=1 proto=udp chksum=0x413 src=10.0.1.4 dst=213.138.109.232 options=[] |<UDPerror sport=21415 dport=65535 len=8 chksum=0x5dc0 |>>>>]" + }, + { + "raw_packet": { + "data": "RcAAOEz9AADvAdzVW986TwoAAQQLAJ1NAAAAAEUAABxiIAAAAREJOwoAAQTVim3o9N4fkAAIQzs=", + "format": "base64" + }, + "summary": "[<IP version=4L ihl=5L tos=0xc0 len=56 id=19709 flags= frag=0L ttl=239 proto=icmp chksum=0xdcd5 src=91.223.58.79 dst=10.0.1.4 options=[] |<ICMP type=time-exceeded code=ttl-zero-during-transit chksum=0x9d4d unused=0 |<IPerror version=4L ihl=5L tos=0x0 len=28 id=25120 flags= frag=0L ttl=1 proto=udp chksum=0x93b src=10.0.1.4 dst=213.138.109.232 options=[] |<UDPerror sport=62686 dport=http_alt len=8 chksum=0x433b |>>>>]" + }, + { + "raw_packet": { + "data": "RcAAOHZYAADvAbN6W986TwoAAQQLAJ1NAAAAAEUAABwZDgAAARFSTQoAAQTVim3ou3kAUAAIm+A=", + "format": "base64" + }, + "summary": "[<IP version=4L ihl=5L tos=0xc0 len=56 id=30296 flags= frag=0L ttl=239 proto=icmp chksum=0xb37a src=91.223.58.79 dst=10.0.1.4 options=[] |<ICMP type=time-exceeded code=ttl-zero-during-transit chksum=0x9d4d unused=0 |<IPerror version=4L ihl=5L tos=0x0 len=28 id=6414 flags= frag=0L ttl=1 proto=udp chksum=0x524d src=10.0.1.4 dst=213.138.109.232 options=[] |<UDPerror sport=47993 dport=http len=8 chksum=0x9be0 |>>>>]" + }, + { + "raw_packet": { + "data": "RQAAqAJHAAD5AalDCpYAMQoAAQQLAH/9AAAABEUAABxL8AAAAhEeawoAAQTVim3oxtsAFgAIkLiqqgAWAAjHuqqqABYACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAD5AwAIAQED8+H/", + "format": "base64" + }, + "summary": "[<IP version=4L ihl=5L tos=0x0 len=168 id=583 flags= frag=0L ttl=249 proto=icmp chksum=0xa943 src=10.150.0.49 dst=10.0.1.4 options=[] |<ICMP type=time-exceeded code=ttl-zero-during-transit chksum=0x7ffd unused=4 |<IPerror version=4L ihl=5L tos=0x0 len=28 id=19440 flags= frag=0L ttl=2 proto=udp chksum=0x1e6b src=10.0.1.4 dst=213.138.109.232 options=[] |<UDPerror sport=50907 dport=ssh len=8 chksum=0x90b8 |<Padding load='\\xaa\\xaa\\x00\\x16\\x00\\x08\\xc7\\xba\\xaa\\xaa\\x00\\x16\\x00\\x08\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00 \\x00\\xf9\\x03\\x00\\x08\\x01\\x01\\x03\ \xf3\\xe1\\xff' |>>>>>]" + }, + { + "raw_packet": { + "data": "RQAAOFaJAAD4AVoOx6g/gQoAAQQLAJ1NAAAAAEUAABw/wAAAARErmwoAAQTVim3o3yEAFgAIeHI=", + "format": "base64" + }, + "summary": "[<IP version=4L ihl=5L tos=0x0 len=56 id=22153 flags= frag=0L ttl=248 proto=icmp chksum=0x5a0e src=199.168.63.129 dst=10.0.1.4 options=[] |<ICMP type=time-exceeded code=ttl-zero-during-transit chksum=0x9d4d unused=0 |<IPerror version=4L ihl=5L tos=0x0 len=28 id=16320 flags= frag=0L ttl=1 proto=udp chksum=0x2b9b src=10.0.1.4 dst=213.138.109.232 options=[] |<UDPerror sport=57121 dport=ssh len=8 chksum=0x7872 |>>>>]" + } + ], + "sent_packets": [ + { + "raw_packet": { + "data": "RQAAKIc9AAABBuQcCgABBNWKbeiRQgAAAACVawAAAABQAiAAGr4AAA==", + "format": "base64" + }, + "summary": "[<IP id=34621 frag=0 ttl=1 proto=tcp dst=213.138.109.232 |<TCP sport=37186 dport=0 seq=38251 flags=S |>>]" + }, + { + "raw_packet": { + "data": "RQAAKADhAAABBmp5CgABBNWKbehsSAAWAACbxwAAAABQAiAAOUYAAA==", + "format": "base64" + }, + "summary": "[<IP id=225 frag=0 ttl=1 proto=tcp dst=213.138.109.232 |<TCP sport=27720 dport=ssh seq=39879 flags=S |>>]" + }, + { + "raw_packet": { + "data": "RQAAKO9JAAABBnwQCgABBNWKbeg1bwAXAAApTwAAAABQAiAA4pYAAA==", + "format": "base64" + }, + "summary": "[<IP id=61257 frag=0 ttl=1 proto=tcp dst=213.138.109.232 |<TCP sport=13679 dport=telnet seq=10575 flags=S |>>]" + }, + { + "raw_packet": { + "data": "RQAAKNKoAAABBpixCgABBNWKbeixRwA1AABobwAAAABQAiAAJ4AAAA==", + "format": "base64" + }, + "summary": "[<IP id=53928 frag=0 ttl=1 proto=tcp dst=213.138.109.232 |<TCP sport=45383 dport=domain seq=26735 flags=S |>>]" + }, + { + "raw_packet": { + "data": "RQAAKPzTAAABBm6GCgABBNWKbejCJwBQAACA3wAAAABQAiAA/hQAAA==", + "format": "base64" + }, + "summary": "[<IP id=64723 frag=0 ttl=1 proto=tcp dst=213.138.109.232 |<TCP sport=49703 dport=http seq=32991 flags=S |>>]" + }, + { + "raw_packet": { + "data": "RQAAKNzQAAABBo6JCgABBNWKbejZyAB7AAD+NQAAAABQAiAAaPIAAA==", + "format": "base64" + }, + "summary": "[<IP id=56528 frag=0 ttl=1 proto=tcp dst=213.138.109.232 |<TCP sport=55752 dport=ntp seq=65077 flags=S |>>]" + } + ], + "test_icmp_traceroute": { + "hops": [ + { + "address": "141.136.98.238", + "rtt": 2.039386034, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2439029217, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4453678131, + "ttl": 12 + } + ] + }, + "test_tcp_traceroute": { + "hops_0": [ + { + "address": "141.136.98.238", + "rtt": 2.0464248657, + "sport": 34854, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2470018864, + "sport": 10415, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4521570206, + "sport": 0, + "ttl": 12 + } + ], + "hops_123": [ + { + "address": "141.136.98.238", + "rtt": 2.0381779671, + "sport": 50988, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.240860939, + "sport": 7867, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4443869591, + "sport": 123, + "ttl": 12 + } + ], + "hops_22": [ + { + "address": "141.136.98.238", + "rtt": 2.0354959965, + "sport": 36980, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2357189655, + "sport": 26447, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4381670952, + "sport": 22, + "ttl": 12 + } + ], + "hops_23": [ + { + "address": "141.136.98.238", + "rtt": 2.0388379097, + "sport": 4690, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2384901047, + "sport": 51177, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4421479702, + "sport": 23, + "ttl": 12 + } + ], + "hops_443": [ + { + "address": "141.136.98.238", + "rtt": 2.0386300087, + "sport": 11218, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.240696907, + "sport": 26319, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4430809021, + "sport": 443, + "ttl": 12 + } + ], + "hops_53": [ + { + "address": "141.136.98.238", + "rtt": 2.0381169319, + "sport": 36901, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2402248383, + "sport": 60900, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4425189495, + "sport": 53, + "ttl": 12 + } + ], + "hops_65535": [ + { + "address": "141.136.98.238", + "rtt": 2.0387971401, + "sport": 57248, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2455801964, + "sport": 52285, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4457411766, + "sport": 65535, + "ttl": 12 + } + ], + "hops_80": [ + { + "address": "141.136.108.130", + "rtt": 1.8421959877, + "sport": 23745, + "ttl": 9 + }, + { + "address": "141.136.98.238", + "rtt": 2.0382699966, + "sport": 22618, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2403969765, + "sport": 45566, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4452321529, + "sport": 80, + "ttl": 12 + } + ], + "hops_8080": [ + { + "address": "141.136.98.238", + "rtt": 2.0389461517, + "sport": 57892, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2410969734, + "sport": 51593, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4440560341, + "sport": 8080, + "ttl": 12 + } + ] + }, + "test_udp_traceroute": { + "hops_0": [], + "hops_123": [ + { + "address": "141.136.108.130", + "rtt": 1.920017004, + "sport": 57583, + "ttl": 9 + }, + { + "address": "91.223.58.79", + "rtt": 2.24168396, + "sport": 18721, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4463579655, + "sport": 25517, + "ttl": 12 + } + ], + "hops_22": [ + { + "address": "141.136.108.130", + "rtt": 1.9136769772, + "sport": 45625, + "ttl": 9 + }, + { + "address": "141.136.98.238", + "rtt": 2.0372998714, + "sport": 50609, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2419588566, + "sport": 14160, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4428777695, + "sport": 23686, + "ttl": 12 + } + ], + "hops_23": [ + { + "address": "141.136.108.130", + "rtt": 1.9202280045, + "sport": 58718, + "ttl": 9 + }, + { + "address": "141.136.98.238", + "rtt": 2.0369830132, + "sport": 39310, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2406070232, + "sport": 9348, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4436769485, + "sport": 56507, + "ttl": 12 + } + ], + "hops_443": [ + { + "address": "141.136.108.130", + "rtt": 1.9194400311, + "sport": 61062, + "ttl": 9 + }, + { + "address": "141.136.98.238", + "rtt": 2.0356152058, + "sport": 57814, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2425971031, + "sport": 36493, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4469130039, + "sport": 20708, + "ttl": 12 + } + ], + "hops_53": [ + { + "address": "141.136.108.130", + "rtt": 1.9192700386, + "sport": 63309, + "ttl": 9 + }, + { + "address": "141.136.98.238", + "rtt": 2.0371799469, + "sport": 44259, + "ttl": 10 + }, + { + "address": "91.223.58.79", + "rtt": 2.2417399883, + "sport": 32840, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4441540241, + "sport": 6946, + "ttl": 12 + } + ], + "hops_65535": [ + { + "address": "141.136.108.130", + "rtt": 1.9188029766, + "sport": 16637, + "ttl": 9 + }, + { + "address": "91.223.58.79", + "rtt": 2.2441849709, + "sport": 56282, + "ttl": 11 + } + ], + "hops_80": [ + { + "address": "141.136.108.130", + "rtt": 1.9194018841, + "sport": 26666, + "ttl": 9 + }, + { + "address": "91.223.58.79", + "rtt": 2.2409110069, + "sport": 47993, + "ttl": 11 + }, + { + "address": "213.138.109.232", + "rtt": 2.4452769756, + "sport": 43930, + "ttl": 12 + } + ], + "hops_8080": [ + { + "address": "141.136.108.130", + "rtt": 1.9188380241, + "sport": 53083, + "ttl": 9 + }, + { + "address": "91.223.58.79", + "rtt": 2.2436709404, + "sport": 62686, + "ttl": 11 + } + ] + } + }, + "test_name": "multi_protocol_traceroute", + "test_runtime": 57.7884280682, + "test_start_time": "2015-11-13 10:46:54", + "test_version": "0.3" +} +``` ## Privacy considerations @@ -121,5 +594,3 @@ the packet they are referring to and it will contain the non anonymized user IP address). On this specific issue there is an open ticket here: https://trac.torproject.org/projects/tor/ticket/7933. - -

1 0

[spec/master] Add measurement_start_time field
by art＠torproject.org 09 May '16

09 May '16

commit 83e6e769b2ba3c91a81c521c35073654745c12a7 Author: Arturo Filastò <arturo(a)filasto.net> Date: Thu Mar 17 14:36:04 2016 +0100 Add measurement_start_time field --- data-formats/df-000-base.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/data-formats/df-000-base.md b/data-formats/df-000-base.md index aab0f1c..f20aa21 100644 --- a/data-formats/df-000-base.md +++ b/data-formats/df-000-base.md @@ -83,8 +83,12 @@ Data Format Version: 0.2.0 "test_runtime": null, - "test_start_time": "Timestamp of when the measurement was performed in " - "UTC time coordinates (ex. 2015-08-24 12:02:23)", + "measurement_start_time": "Timestamp of when the measurement was performed in " + "UTC time coordinates (ex. 2015-08-24 12:02:23) (Note: ooniprobe <= 1.4.0 generates skewed timing information)", + + "test_start_time": "Timestamp of when the test was started in " + "UTC time coordinates (ex. 2015-08-24 12:02:23) (Note: ooniprobe <= 1.4.0 generates skewed timing information)", + "test_keys": { "The keys that are specific to the test"

1 0

[spec/master] Update my PGP key to be the latest one
by art＠torproject.org 09 May '16

09 May '16

commit a42b44a57b73e62c7dabc66c6566e0423fa42f5c Author: Arturo Filastò <arturo(a)filasto.net> Date: Thu Mar 17 14:36:17 2016 +0100 Update my PGP key to be the latest one --- Release-Procedure.md | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/Release-Procedure.md b/Release-Procedure.md index f19d44c..6256130 100644 --- a/Release-Procedure.md +++ b/Release-Procedure.md @@ -177,16 +177,12 @@ All tags should be incremental and they will be signed with either of the following GPG keys: ``` -pub 4096R/150FE210 2011-10-23 [expires: 2014-11-24] - Key fingerprint = 46E5 EF37 DE26 4EA6 8DCF 53EA E3A2 1297 150F E210 - uid Arturo Filastò <art(a)torproject.org> - uid Arturo Filastò <arturo(a)filasto.net> - uid Arturo Filastò <arturo.filasto(a)logioshermes.org> - uid Arturo Filastò <art(a)fuffa.org> - uid Arturo Filastò <art(a)globaleaks.org> - uid Arturo Filastò <art(a)baculo.org> - uid Arturo Filastò <hellais(a)gmail.com> - sub 4096R/F423B27C 2011-10-23 [expires: 2014-11-24] +pub 3072D/702287F4 2015-10-19 [expires: 2018-05-06] + Key fingerprint = 67EF 3966 5099 86E9 6ACE E84E 5D67 CD18 7022 87F4 +uid [ultimate] Arturo Filastò <arturo(a)filasto.net> +uid [ultimate] Arturo Filastò <art(a)fuffa.org> +uid [ultimate] Arturo Filastò <art(a)torproject.org> +sub 2752g/C58FC4EE 2015-10-19 [expires: 2018-05-06] ``` ## Versioning scheme @@ -198,7 +194,7 @@ All OONI tools follow [semantic versioning](http://semver.org/). To upload a new version to pypi run: ``` -python setup.py sdist upload -s +python setup.py sdist upload -s --identity=702287F4 ``` # Build systems & environments

1 0

[spec/master] Merge pull request #53 from TheTorProject/feature/df-0.2
by art＠torproject.org 09 May '16

09 May '16

commit b11344272c179e786810baefa9089a39039d36de Merge: 3d3bd21 ef8b64b Author: anadahz <anadahz(a)users.noreply.github.com> Date: Wed Apr 6 22:34:40 2016 +0000 Merge pull request #53 from TheTorProject/feature/df-0.2 Update data format to include all missing details. data-formats/df-000-base.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)

1 0

[spec/master] Update data format to include all missing details.
by art＠torproject.org 09 May '16

09 May '16

commit ef8b64b40dfa22303914ae00ef66eea0a815709d Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Apr 5 17:11:16 2016 +0200 Update data format to include all missing details. --- data-formats/df-000-base.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/data-formats/df-000-base.md b/data-formats/df-000-base.md index f20aa21..c72b3af 100644 --- a/data-formats/df-000-base.md +++ b/data-formats/df-000-base.md @@ -49,6 +49,9 @@ Data Format Version: 0.2.0 "data_format_version": "0.1.0|0.2.0", + "annotations": "Optional user provided annotations to the report that " + "provide metadata to this measurement", + "report_filename": "{bucket_date}/{timestamp as '%Y%m%dT%h%M%sZ'}-{probe_cc}-{probe_asn}-{test_name}-{report_id}-{data_format_version}-{probe|backend}.json", "options": ["A list of options passed to the test as command line arguments"], @@ -74,7 +77,7 @@ Data Format Version: 0.2.0 "backend_version": "The version of the backend that collected this measurement", - "test_helpers": null, + "test_helpers": {"test_helper_name": "The address of the test helper used"}, "test_name": "The name of the test that generated " "this measurement (ex. http_requests)",

1 0

[spec/master] Merge pull request #52 from TheTorProject/joe/formatting-fixes
by art＠torproject.org 09 May '16

09 May '16

commit aa17e0f06cc09b7b5aa456c03fffb5bf4c561ea8 Merge: b113442 29f56af Author: anadahz <anadahz(a)users.noreply.github.com> Date: Tue Apr 12 23:27:36 2016 +0000 Merge pull request #52 from TheTorProject/joe/formatting-fixes formatting and filename fixes test-specs/ts-001-bridget.md | 2 +- test-specs/ts-005-dns-spoof.md | 131 ++++++++--------- test-specs/ts-012-lantern.md | 111 --------------- test-specs/ts-013-lantern.md | 111 +++++++++++++++ test-specs/ts-013-meek-fronted-requests.md | 111 --------------- test-specs/ts-014-meek-fronted-requests.md | 111 +++++++++++++++ test-specs/ts-014-psiphon.md | 146 ------------------- test-specs/ts-015-openvpn.md | 218 ---------------------------- test-specs/ts-015-psiphon.md | 148 +++++++++++++++++++ test-specs/ts-016-openvpn.md | 220 +++++++++++++++++++++++++++++ 10 files changed, 657 insertions(+), 652 deletions(-)

1 0