April 2015 - tor-commits - lists.torproject.org

[spec/master] Add meek fronted requests test specification
by art＠torproject.org 15 Apr '15

15 Apr '15

commit b2308768e776189c385dffc5a394ec4bcfbb73cd Author: V <auser@0> Date: Sun Apr 5 18:30:06 2015 +0200 Add meek fronted requests test specification --- test-specs/ts-013-meek_fronted_requests.md | 108 ++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) diff --git a/test-specs/ts-013-meek_fronted_requests.md b/test-specs/ts-013-meek_fronted_requests.md new file mode 100644 index 0000000..f211cc4 --- /dev/null +++ b/test-specs/ts-013-meek_fronted_requests.md @@ -0,0 +1,108 @@ +# Specification version number + +2015-04-01-000 + +# Specification name + +Meek Fronted Request Test + +# Test preconditions + +* An internet connection + +# Expected impact + +Ability to detect uncensored fronted domains that can transmit requests to the +"inside" meek-server via an intermediary web service. + +# Expected inputs + +## Import document or import data format + +A list of domain names (fronted domains) and host headers (meek-server) of the +intermediary web service. + +## Semantics + +The input document may contain a domain name and a host header combination per +line separated by colon in the format: + + DomainName:HostHeader + +Example: + + www.google.com:meek-reflect.appspot.com + a0.awsstatic.com:d2zfqthxsdq309.cloudfront.net + +# Test description + +Performs a HTTP GET request to a list of fronted domains with the Host +Header of the "inside" meek-server. The meek-server handles a GET request and +response with: "I’m just a happy little web server.\n". + +# Expected output + +## Parent data format + +df-001-httpt-000 + +## Required output data + +* The domain name and host header used in the measurement + (DomainName:HostHeader) + +* The requests that have been made + +* The received responses + +* If the meek request is blocked + +## Semantics + +censored: + **boolean** indicates if an HTTP GET response to the meek server is + successfull + +## Possible conclusions + +If the fronted request/response to the meek server is successful. + +## Example output sample + +``` +agent: agent +censored: false +input: ajax.aspnetcdn.com:az668014.vo.msecnd.net +requests: +- request: + body: null + headers: + - - Host + - [az668014.vo.msecnd.net] + method: GET + tor: {is_tor: false} + url: https://ajax.aspnetcdn.com + response: + body: "I\u2019m just a happy little web server.\n" + code: 200 + headers: + - - Content-Length + - ['38'] + - - X-Cache + - [HIT] + - - X-Powered-By + - [ASP.NET] + - - Accept-Ranges + - [bytes] + - - Server + - [ECAcc (fcn/40C4)] + - - Last-Modified + - ['Wed, 01 Apr 2015 09:25:13 GMT'] + - - Connection + - [close] + - - Date + - ['Wed, 01 Apr 2015 10:01:37 GMT'] + - - Content-Type + - [text/plain; charset=utf-8] +socksproxy: null +```

1 0

[spec/master] Review of policy aware changes
by art＠torproject.org 15 Apr '15

15 Apr '15

commit 65c049b0217493193477ee2545f2b7a47f8e01f1 Author: kudrom <kudrom(a)riseup.net> Date: Mon Aug 25 23:32:58 2014 +0200 Review of policy aware changes --- oonib.md | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 68 insertions(+), 4 deletions(-) diff --git a/oonib.md b/oonib.md index 7f31b78..81d83a3 100644 --- a/oonib.md +++ b/oonib.md @@ -571,10 +571,14 @@ This is the service that will allow the ooni-probe to discover collectors that r ### POST /bouncer -This will return a list of required collectors and test helpers. +This will return a list of required collectors. -Client sends: +Currently a client has two ways to request the required collectors. +The first and old way is to provide a bunch of test-helpers, oonib then will return the identity of +the collectors and the addresses of machines that can run the requested test helpers. + +This way, the client sends: ``` { 'test-helpers': [ @@ -583,7 +587,7 @@ Client sends: } ``` -Bouncer replies: +And the bouncer replies: ``` { @@ -607,7 +611,67 @@ Bouncer replies: If it is not possible to find a test helper for one of the required test helpers an error message is returned: - ``` {'error': 'test-helper-not-found'} ``` + +The second and policy respectful way is to provide a bunch of nettests, then the bouncer +will return the identity of the machines that: + 1. Can run the required test-helpers + 2. Can send back to the client a set of required inputs + 3. Can collect the report of the nettest + +If a known collector to the bouncer cannot provide any of the three previous requirements, +the collector won't be sent back to the client in the bouncer request. + +If no policy respectful collector can be returned to the client, a default one will be sent +when the bouncer knows the identity of a collector that accepts any request, which is +expressed as a lack of any policy for that collector in the oonib.conf file. + +Both the test-helpers and the input-hashed attributes are optional for the client's request, +but the nettest's name is mandatory. + +This way the client sends: +``` +{ + 'net-tests': [ + { + 'test-helpers': ['required test helper', ...], + 'input-hashed': ['required input id', ...], + 'name': 'name of nettest', + 'version': 'version of nettest' + }, + .... + ] +} +``` + +And the bouncer replies: +``` +{ + 'net-tests': [ + { + 'test-helpers': [ + {'requested test helper': 'address'}, + ... + ], + 'input-hashed': ['requested input id', ...], + 'name': 'name of nettest', + 'version': 'version of nettest', + 'collector': 'address' + }, + .... + ] +} +``` + +If it's not possible to find a collector for the request, oonib will respond with: +``` +{'error': 'collector-not-found'} +``` + +Both ways to communicate with the bouncer are explicitly set with the proper JSON object in the +payload of the POST request. If an invalid JSON object is provided, oonib will respond with: +``` +{'error': 'test-helpers-or-net-test-key-missing'} +```

1 0

[spec/master] Update data format with recent changes to it.
by art＠torproject.org 15 Apr '15

15 Apr '15

commit 73aa34f1123af603fab00d00b5cd1806c9152c45 Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Oct 17 10:32:45 2014 +0200 Update data format with recent changes to it. --- data-formats/df-000-base.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/data-formats/df-000-base.md b/data-formats/df-000-base.md index 24651d1..812edf0 100644 --- a/data-formats/df-000-base.md +++ b/data-formats/df-000-base.md @@ -108,14 +108,14 @@ input: The item we this specific test instance is referring to. null in case no input is being iterated over. -test_name: +test_method: `string` the name of the test method this test is referring to (that is the class method of a NetTestCase) test_runtime: `float` the runtime of the test -test_started: +test_start_time: `float` seconds since epoch from the starting of the test. # Error strings

1 0

[spec/master] Update release procedure with ooniinput and ooniresources commands.
by art＠torproject.org 15 Apr '15

15 Apr '15

commit e651ae72392ee7edebdad5e33ea94c6b15850b5a Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Oct 9 09:34:41 2014 +0200 Update release procedure with ooniinput and ooniresources commands. --- Release-Procedure.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/Release-Procedure.md b/Release-Procedure.md index 89afb5f..f19d44c 100644 --- a/Release-Procedure.md +++ b/Release-Procedure.md @@ -105,6 +105,22 @@ ooniprobe -n manipulation/http_header_field_manipulation ooniprobe -n blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1 ``` +### Testing oonideckgen + +This command must produce a deck in the current working directory. + +``` +oonideckgen --country-code IT +``` + +### Testing ooniresource + +This command must update the resources. + +``` +ooniresources --update-geoip --update-inputs +``` + ### Testing vagrant installation To setup and test ooniprobe on a fresh ubuntu installation with

1 0

[spec/master] Merge pull request #31 from kudrom/feature/updated_oonib_policy_aware
by art＠torproject.org 15 Apr '15

15 Apr '15

commit 4525ff6ba4fa9f7f264a7dd46955dc26c36d78c0 Merge: abd539e 197063b Author: Arturo Filastò <arturo(a)filasto.net> Date: Mon Sep 1 12:32:18 2014 +0200 Merge pull request #31 from kudrom/feature/updated_oonib_policy_aware Review of policy aware changes oonib.md | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 4 deletions(-)

1 0

[spec/master] Improved RESTfulness of the bouncer API.
by art＠torproject.org 15 Apr '15

15 Apr '15

commit 197063b7afca942374f6c0ae3c4f66995cb285ff Author: kudrom <kudrom(a)riseup.net> Date: Fri Aug 29 15:59:49 2014 +0200 Improved RESTfulness of the bouncer API. --- oonib.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/oonib.md b/oonib.md index 81d83a3..e653b02 100644 --- a/oonib.md +++ b/oonib.md @@ -575,6 +575,8 @@ This will return a list of required collectors. Currently a client has two ways to request the required collectors. +#### POST /bouncer/test-helpers + The first and old way is to provide a bunch of test-helpers, oonib then will return the identity of the collectors and the addresses of machines that can run the requested test helpers. @@ -615,6 +617,8 @@ helpers an error message is returned: {'error': 'test-helper-not-found'} ``` +#### POST /bouncer/net-tests + The second and policy respectful way is to provide a bunch of nettests, then the bouncer will return the identity of the machines that: 1. Can run the required test-helpers @@ -669,9 +673,3 @@ If it's not possible to find a collector for the request, oonib will respond wit ``` {'error': 'collector-not-found'} ``` - -Both ways to communicate with the bouncer are explicitly set with the proper JSON object in the -payload of the POST request. If an invalid JSON object is provided, oonib will respond with: -``` -{'error': 'test-helpers-or-net-test-key-missing'} -```

1 0

[spec/master] Remove not used test_method
by art＠torproject.org 15 Apr '15

15 Apr '15

commit f90fd4a2965177431264b4c441de7d872e4381c7 Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Oct 17 11:05:40 2014 +0200 Remove not used test_method --- data-formats/df-000-base.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/data-formats/df-000-base.md b/data-formats/df-000-base.md index 812edf0..8ca41c8 100644 --- a/data-formats/df-000-base.md +++ b/data-formats/df-000-base.md @@ -108,10 +108,6 @@ input: The item we this specific test instance is referring to. null in case no input is being iterated over. -test_method: - `string` the name of the test method this test is referring to (that is the - class method of a NetTestCase) - test_runtime: `float` the runtime of the test

1 0

[spec/master] Merge branch 'master' of github.com:TheTorProject/ooni-spec
by art＠torproject.org 15 Apr '15

15 Apr '15

commit 9ad8dca517efce0c669c39fffe5672c6d66f604a Merge: e651ae7 4525ff6 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Oct 14 12:37:06 2014 +0200 Merge branch 'master' of github.com:TheTorProject/ooni-spec * 'master' of github.com:TheTorProject/ooni-spec: Improved RESTfulness of the bouncer API. Review of policy aware changes oonib.md | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 4 deletions(-)

1 0

[spec/master] Initial import of taw TCP and UDP echo helpers
by art＠torproject.org 15 Apr '15

15 Apr '15

commit d04ab0477ae4feb9e038f99661b677f3d01c1289 Author: Iain R. Learmonth <irl(a)fsfe.org> Date: Mon Dec 8 15:45:21 2014 +0000 Initial import of taw TCP and UDP echo helpers --- test-helpers/th-004-raw-tcp-echo.md | 67 +++++++++++++++++++++++++++++++++++ test-helpers/th-005-raw-udp-echo.md | 54 ++++++++++++++++++++++++++++ 2 files changed, 121 insertions(+) diff --git a/test-helpers/th-004-raw-tcp-echo.md b/test-helpers/th-004-raw-tcp-echo.md new file mode 100644 index 0000000..8b9cd54 --- /dev/null +++ b/test-helpers/th-004-raw-tcp-echo.md @@ -0,0 +1,67 @@ +# Specification version number + +2014-12-08-000 + +# Specification name + +Raw TCP Echo Test Helper + +# Helper description + +The Raw TCPEcho Test Helper listens on a TCP port for incoming connections. +When a connection is initiated, it accepts the connection, reads a dummy HTTP +request and sends an HTTP reply containing an encoding of the IP packets that +made up the TCP 3 way handshake [[Wikipedia-3WHS][]] which can be compared +against the packets sent by an ooniprobe test. + +# Helper preconditions + + * An Internet connection + * An Internet-Reachable TCP Port (preferably port 80) + * No known middleboxes rewriting IP or TCP headers in unexpected ways between + the helper and the transit ISP + +# Expected impact + +Ability to help an ooni-probe client determine if the IP or TCP headers have +been modified in transit, possibly indicating that the connection is being +transparently proxied. + +# Expected inputs + + * An HTTP 1.1 [[RFC2616][]] request that should not include any "dubious" Host: + header or keywords that might attract special treatment + +By emulating an HTTP session, a protocol known to censorship systems and +protocol-enhancing proxies, it will increase the chances to see intereference +occuring over using another protocol that may simply be passed through. + +# Expected output + + * An HTTP 1.1 [[RFC2616][]] reply that includes as its body an encoding of the + packets that made up the TCP 3 way handshake [[Wikipedia-3WHS][]] from the + vantage point of the helper + +The encoding chosen could be simply a binary PCAP trace, a base64 encoded PCAP +trace or even a JSON encoding of a dissector output as long as the dissector is +sufficiently verbose to maximise the usefulness of the helper. + +MIME types like text/html should be avoided as these have a stronger potential +for being rewritten on the wire. + +# Possible conclusions + +Possible conclusions that could be drawn from tests using this helper are: + + * The presence of a transparent proxy + * The presence of network address translation + * The bleaching of bits in the IP and TCP header by badly implemented middleboxes + +# Notes + +An implementation of this test helper is currently being worked on by Iain R. +Learmonth <<irl(a)fsfe.org>> using Scapy. + +[RFC2616]: http://tools.ietf.org/html/rfc2616 +[Wikipedia-3WHS]: http://en.wikipedia.org/wiki/Transmission_Control_Protocol#Connection_estab… + diff --git a/test-helpers/th-005-raw-udp-echo.md b/test-helpers/th-005-raw-udp-echo.md new file mode 100644 index 0000000..3414076 --- /dev/null +++ b/test-helpers/th-005-raw-udp-echo.md @@ -0,0 +1,54 @@ +# Specification version number + +2014-12-08-000 + +# Specification name + +Raw UDP Echo Test Helper + +# Helper description + +The Raw UDP Echo Test Helper listens on a UDP port for incoming packets. When a +packet is recieved, it sends a UDP packet in reply with the original packet's +header as its payload. + +# Helper preconditions + + * An Internet connection + * An Internet-Reachable UDP Port + * No known middleboxes rewriting UDP headers in unexpected ways between + the helper and the transit ISP + +# Expected impact + +Ability to help an ooni-probe client determine if the UDP header is being +rewritten and if packets are being truncated when UDP-lite is used with a +shorter checksum coverage than the full length of the packet using the UDP +protocol number in the IP header. + +# Expected inputs + + * A UDP packet addressed to the helper + * The destination port should not be one that might attract special treatment + * The payload should not contain keywords that might attract special treatment + +# Expected output + + * A UDP packet with the original incoming packet as its payload addressed to + return to the ooniprobe client + +# Possible conclusions + +Possible conclusions that could be drawn from tests using this helper are: + + * Middleboxes are truncating UDP-lite packets when the UDP protocol number is + used + * Middleboxes are discarding UDP-lite packets with the checksum does not match + for the full length of the packet + * Source and destination ports are being rewritten + +# Notes + +An implementation of this test helper is currently being worked on by Iain R. +Learmonth <<irl(a)fsfe.org>> using Scapy. +

1 0

[spec/master] Merge pull request #32 from irl/master
by art＠torproject.org 15 Apr '15

15 Apr '15

commit b0591bd0fc64c2029e38d16ecba940e8f6b15965 Merge: 8ee0cef d04ab04 Author: Arturo Filastò <arturo(a)filasto.net> Date: Thu Dec 11 23:26:06 2014 +0100 Merge pull request #32 from irl/master Initial import of raw TCP and UDP echo helpers test-helpers/th-004-raw-tcp-echo.md | 67 +++++++++++++++++++++++++++++++++++ test-helpers/th-005-raw-udp-echo.md | 54 ++++++++++++++++++++++++++++ 2 files changed, 121 insertions(+)

1 0