June 2013 - tor-commits - lists.torproject.org

[ooni-probe/develop] Merge branch 'master' into experimental-tests
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 9cad53b7a979e5c8478f51dca32a191c2b31e0a1 Merge: 02aff59 5f568cb Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Feb 26 15:40:18 2013 +0100 Merge branch 'master' into experimental-tests * master: (32 commits) Add sudo to apt-key Add note on wheezy and natty fix text where we called a SOCKS port an orport fixup decks/before_i_commit.testdeck to actually run against master add mkdir Add default config file remove python-… [View More]pcapy add missing ; Use Tor mirror of scapy-02-25-2013-tip.zip Add wheezy support for arma to test Make the final output more generic Update README.md to explain easy setup steps Add stable mirror to requirements.txt Set a default collector Catch 302 redirect loops and log.err not log.warn Remove dependency on _WrappingFactory Remove some superfluous requirements from the requirements.txt Make sure to import Gen, SetGen and MTU even when pypcap is and libdnet are not installed Rename DNS Tamper Test to DNS Consistency Test Apply patch from dma in #8009 #8008 ... .gitignore | 4 + README.md | 73 ++++++++--- apt.key | Bin 0 -> 6624 bytes collector | 1 + decks/before_i_commit.testdeck | 33 +++-- docs/source/index.rst | 2 +- docs/source/nettests/nettests.blocking.rst | 4 +- docs/source/tests/dnsconsistency.rst | 180 ++++++++++++++++++++++++++++ docs/source/tests/dnsspoof.rst | 2 +- docs/source/tests/dnstamper.rst | 180 ---------------------------- docs/source/writing_tests.rst | 6 +- inputs/README | 2 +- nettests/blocking/dnsconsistency.py | 173 ++++++++++++++++++++++++++ nettests/blocking/dnstamper.py | 166 ------------------------- nettests/manipulation/captiveportal.py | 21 ++-- nettests/manipulation/traceroute.py | 2 +- ooni/oonicli.py | 3 +- ooni/runner.py | 9 ++ ooni/utils/txagentwithsocks.py | 2 +- ooni/utils/txscapy.py | 2 +- oonib/requirements.txt | 8 +- ooniprobe.conf.sample | 2 +- requirements.txt | 28 +++-- setup-dependencies.sh | 83 +++++++++++++ 24 files changed, 577 insertions(+), 409 deletions(-) [View Less]

1 0

[ooni-probe/develop] Uniform transparent_http_proxy keys in report
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 0c49af0fef382cef3dfa8da5b3288f526d5b40bb Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Feb 26 17:57:55 2013 +0100 Uniform transparent_http_proxy keys in report --- nettests/manipulation/http_host.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/nettests/manipulation/http_host.py b/nettests/manipulation/http_host.py index 38b0fff..d95d836 100644 --- a/nettests/manipulation/http_host.py +++ b/nettests/manipulation/http_host.py @@ -121,7 +… [View More]

1 0

[ooni-probe/develop] Bump up the version of the HTTP Host test.
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit f85dbd4a4433a478e12ce3921ce072a2b7019eb9 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Feb 26 16:29:05 2013 +0100 Bump up the version of the HTTP Host test. * Always perform a connection directly to the oonib test helper * Always include the 'censored' key (default to False) --- nettests/experimental/http_filtering_bypassing.py | 2 +- nettests/manipulation/http_host.py | 39 ++++++++++++--------- 2 files changed, 23 insertions(+), 18 … [View More]deletions(-) diff --git a/nettests/experimental/http_filtering_bypassing.py b/nettests/experimental/http_filtering_bypassing.py index a82994a..dc103db 100644 --- a/nettests/experimental/http_filtering_bypassing.py +++ b/nettests/experimental/http_filtering_bypassing.py @@ -66,7 +66,7 @@ class HTTPFilteringBypass(tcpt.TCPTest): return d def test_fuzzy_match_blocking(self): - hostname_field = randomStr(10) + self.input ++ randomStr(10) + hostname_field = randomStr(10) + self.input + randomStr(10) payload = "GET / HTTP/1.1\n\r" payload += "Host: %s\n\r" % hostname_field diff --git a/nettests/manipulation/http_host.py b/nettests/manipulation/http_host.py index 1665992..38b0fff 100644 --- a/nettests/manipulation/http_host.py +++ b/nettests/manipulation/http_host.py @@ -15,10 +15,12 @@ from ooni.utils import log from ooni.templates import httpt class UsageOptions(usage.Options): - optParameters = [['backend', 'b', 'http://127.0.0.1:57001', - 'URL of the test backend to use'], - ['content', 'c', None, - 'The file to read from containing the content of a block page']] + optParameters = [['backend', 'b', 'http://127.0.0.1:57001', + 'URL of the test backend to use. Should be \ + listening on port 80 and be a \ + HTTPReturnJSONHeadersHelper'], + ['content', 'c', None, 'The file to read \ + from containing the content of a block page']] class HTTPHost(httpt.HTTPTest): """ @@ -33,7 +35,7 @@ class HTTPHost(httpt.HTTPTest): """ name = "HTTP Host" author = "Arturo Filastò" - version = "0.2.1" + version = "0.2.3" randomizeUA = False usageOptions = UsageOptions @@ -46,25 +48,25 @@ class HTTPHost(httpt.HTTPTest): def test_filtering_prepend_newline_to_method(self): headers = {} headers["Host"] = [self.input] - return self.doRequest('http://'+self.input, method="\nGET", + return self.doRequest(self.localOptions['backend'], method="\nGET", headers=headers) def test_filtering_add_tab_to_host(self): headers = {} headers["Host"] = [self.input + '\t'] - return self.doRequest('http://'+self.input, + return self.doRequest(self.localOptions['backend'], headers=headers) def test_filtering_of_subdomain(self): headers = {} headers["Host"] = [randomStr(10) + '.' + self.input] - return self.doRequest('http://'+self.input, + return self.doRequest(self.localOptions['backend'], headers=headers) def test_filtering_via_fuzzy_matching(self): headers = {} headers["Host"] = [randomStr(10) + self.input + randomStr(10)] - return self.doRequest('http://'+self.input, + return self.doRequest(self.localOptions['backend'], headers=headers) def test_send_host_header(self): @@ -89,7 +91,6 @@ class HTTPHost(httpt.HTTPTest): """ if self.localOptions['content']: self.report['censored'] = True - censorship_page = open(self.localOptions['content']) response_page = iter(body.split("\n")) @@ -100,6 +101,8 @@ class HTTPHost(httpt.HTTPTest): break censorship_page.close() + else: + self.report['censored'] = None def processResponseBody(self, body): """ @@ -110,13 +113,14 @@ class HTTPHost(httpt.HTTPTest): # If we don't see a json array we know that something is wrong for # sure if not body.startswith("{"): + log.msg("This does not appear to be JSON") self.report['transparent_http_proxy'] = True self.check_for_censorship(body) return try: content = json.loads(body) except: - log.debug("The json does not parse, this is not what we expected") + log.msg("The json does not parse, this is not what we expected") self.report['trans_http_proxy'] = True self.check_for_censorship(body) return @@ -124,13 +128,14 @@ class HTTPHost(httpt.HTTPTest): # We base the determination of the presence of a transparent HTTP # proxy on the basis of the response containing the json that is to be # returned by a HTTP Request Test Helper - if 'request_method' in content and \ - 'request_uri' in content and \ - 'request_headers' in content: - log.debug("Found the keys I expected in %s" % content) + if 'request_headers' in content and \ + 'request_line' in content and \ + 'headers_dict' in content: + log.msg("Found the keys I expected in %s" % content) self.report['trans_http_proxy'] = False + self.report['censored'] = False else: - log.debug("Did not find the keys I expected in %s" % content) + log.msg("Did not find the keys I expected in %s" % content) self.report['trans_http_proxy'] = True + self.check_for_censorship(body) - self.check_for_censorship(body) [View Less]

1 0

[ooni-probe/develop] Add an example test that inherits from the HTTP and DNS test template
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 0c99be6979ace45d9d46758aa9c1e1741fbb69b9 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Feb 26 23:34:01 2013 +0100 Add an example test that inherits from the HTTP and DNS test template --- nettests/examples/example_dns_http.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/nettests/examples/example_dns_http.py b/nettests/examples/example_dns_http.py new file mode 100644 index 0000000..9b76775 --- /dev/null +++ b/nettests/examples/example_dns_http.py @… [View More]

1 0

[ooni-probe/develop] Fix bug #7999 allowing tests to inherit from multiple test templates
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 1e351c9bc86f89ad022fb1e580182acbb4778dee Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Feb 26 23:33:37 2013 +0100 Fix bug #7999 allowing tests to inherit from multiple test templates --- ooni/templates/dnst.py | 10 ++++++---- ooni/templates/httpt.py | 8 +++++--- ooni/templates/scapyt.py | 2 ++ ooni/templates/tcpt.py | 2 ++ 4 files changed, 15 insertions(+), 7 deletions(-) diff --git a/ooni/templates/dnst.py b/ooni/templates/dnst.py index 08d660a..… [View More]caf4d04 100644 --- a/ooni/templates/dnst.py +++ b/ooni/templates/dnst.py @@ -28,6 +28,8 @@ class DNSTest(NetTestCase): queryTimeout = [1] def _setUp(self): + super(DNSTest, self)._setUp() + self.report['queries'] = [] def performPTRLookup(self, address, dns_server): @@ -51,7 +53,7 @@ class DNSTest(NetTestCase): name = str(answer.payload.name) answers.append(representAnswer(answer)) - self.addToReport(query, resolver=dns_server, + DNSTest.addToReport(self, query, resolver=dns_server, query_type = 'PTR', answers=answers, name=name) return name @@ -59,7 +61,7 @@ class DNSTest(NetTestCase): log.err("Failed to perform lookup") log.exception(failure) failure.trap(gaierror, TimeoutError) - self.addToReport(query, resolver=dns_server, + DNSTest.addToReport(self, query, resolver=dns_server, query_type = 'PTR', failure=failure) return None @@ -89,13 +91,13 @@ class DNSTest(NetTestCase): addrs.append(addr) answers.append(representAnswer(answer)) - self.addToReport(query, resolver=dns_server, query_type='A', + DNSTest.addToReport(self, query, resolver=dns_server, query_type='A', answers=answers, addrs=addrs) return addrs def gotError(failure): failure.trap(gaierror, TimeoutError) - self.addToReport(query, resolver=dns_server, query_type='A', + DNSTest.addToReport(self, query, resolver=dns_server, query_type='A', failure=failure) return failure diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index a265f4c..5e6bc5d 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -54,6 +54,8 @@ class HTTPTest(NetTestCase): responses = [] def _setUp(self): + super(HTTPTest, self)._setUp() + try: import OpenSSL except: @@ -134,7 +136,7 @@ class HTTPTest(NetTestCase): def _processResponseBody(self, response_body, request, response, body_processor): log.debug("Processing response body") - self.addToReport(request, response, response_body) + HTTPTest.addToReport(self, request, response, response_body) if body_processor: body_processor(response_body) else: @@ -205,7 +207,7 @@ class HTTPTest(NetTestCase): """ if not response: log.err("Got no response for request %s" % request) - self.addToReport(request, response) + HTTPTest.addToReport(self, request, response) return else: log.debug("Got response %s" % response) @@ -304,7 +306,7 @@ class HTTPTest(NetTestCase): def errback(failure, request): failure_string = handleAllFailures(failure) log.err("Error performing %s" % request) - self.addToReport(request, failure_string=failure_string) + HTTPTest.addToReport(self, request, failure_string=failure_string) return failure d = agent.request(request['method'], request['url'], headers, diff --git a/ooni/templates/scapyt.py b/ooni/templates/scapyt.py index 5f6c7c8..cadc793 100644 --- a/ooni/templates/scapyt.py +++ b/ooni/templates/scapyt.py @@ -37,6 +37,8 @@ class BaseScapyTest(NetTestCase): ] def _setUp(self): + super(BaseScapyTest, self)._setUp() + if not config.scapyFactory: log.debug("Scapy factoring not set, registering it.") config.scapyFactory = ScapyFactory(config.advanced.interface) diff --git a/ooni/templates/tcpt.py b/ooni/templates/tcpt.py index 323ead8..1acd6bb 100644 --- a/ooni/templates/tcpt.py +++ b/ooni/templates/tcpt.py @@ -57,6 +57,8 @@ class TCPTest(NetTestCase): port = None def _setUp(self): + super(TCPTest, self)._setUp() + self.report['sent'] = [] self.report['received'] = [] [View Less]

1 0

[ooni-probe/develop] Ported tls_handshake.py to be a nettest, rather than an external script.
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 348befd12ca6e1f0e0deb75dff43e1994f098325 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed Jan 23 19:34:30 2013 +0000 Ported tls_handshake.py to be a nettest, rather than an external script. --- nettests/experimental/tls_handshake.py | 231 ++++++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py new file mode 100644 index 0000000..d909603 --- /dev/null +++ b/… [View More]nettests/experimental/tls_handshake.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python +# -*- encoding: utf-8 -*- +""" + tls_handshake.py + ---------------- + This file contains test cases for determining if a TLS handshake completes + successfully, including ways to test if a TLS handshake which uses Mozilla + Firefox's current ciphersuite list completes. + + These NetTestCases are a rewrite of a script contributed by Hackerberry + Finn, in order to fit into OONI's core network tests. + + @authors: Isis Agora Lovecruft <isis(a)torproject.org>, + Hackerberry Finn <anon@localhost> + @license: see included LICENSE file +""" + +import os +import socket +from socket import error as serror +import struct +import sys + +from ipaddr import IPAddress +from OpenSSL import SSL +from twisted.internet import defer +from twisted.python import usage + +from ooni import nettest, config +from ooni.utils import log + +## For a way to obtain the current version of Firefox's default ciphersuite +## list, see https://trac.torproject.org/projects/tor/attachment/ticket/4744/ +## and the attached file "get_mozilla_files.py". +## +## Note, however, that doing so requires the source code to the version of +## firefox that you wish to emulate. + +firefox_ciphers = ["ECDHE-ECDSA-AES256-SHA", + "ECDHE-RSA-AES256-SHA", + "DHE-RSA-CAMELLIA256-SHA", + "DHE-DSS-CAMELLIA256-SHA", + "DHE-RSA-AES256-SHA", + "DHE-DSS-AES256-SHA", + "ECDH-ECDSA-AES256-CBC-SHA", + "ECDH-RSA-AES256-CBC-SHA", + "CAMELLIA256-SHA", + "AES256-SHA", + "ECDHE-ECDSA-RC4-SHA", + "ECDHE-ECDSA-AES128-SHA", + "ECDHE-RSA-RC4-SHA", + "ECDHE-RSA-AES128-SHA", + "DHE-RSA-CAMELLIA128-SHA", + "DHE-DSS-CAMELLIA128-SHA",] + + +class UsageOptions(usage.Options): + optParameters = [ + ['host', 'h', None, 'Remote host IP address (v4/v6)'], + ['port', 'p', None, + "Use this port for all hosts, regardless of port specified in file"], + ['ciphersuite', 'c', None , + 'File containing ciphersuite list, one per line'],] + optFlags = [ + ['ssl2', '2', 'Use SSLv2'], + ['ssl3', '3', 'Use SSLv3'], + ['tls1', 't', 'Use TLSv1'],] + +class TLSHandshakeTest(nettest.NetTestCase): + """ + An ooniprobe NetTestCase for determining if we can complete a TLS handshake + with a remote host. + """ + name = 'tls-handshake' + author = 'Isis Lovecruft <isis(a)torproject.org>' + description = 'A test to determing if we can complete a TLS hankshake.' + version = '0.0.1' + + requiresRoot = False + usageOptions = UsageOptions + + inputFile = ['file', 'f', None, 'List of <IP>:<PORT>s to test'] + + def setUp(self, *args, **kwargs): + if self.localOptions: + options = self.localOptions + self.ciphers = [] + self.methods = [] + + ## check that we're actually testing an IP:PORT, else exit + ## gracefully: + if not (options['host'] and options['port']) \ + and not options['file']: + sys.exit("Need --host and --port, or --file!") + + ## set the SSL/TLS method to use: + for method in ['ssl2', 'ssl3', 'tls1']: + if options[method]: + self.methods.append(method) + + ## if we weren't given a file with a list of ciphersuites to use, + ## then use the firefox default list: + if not options['ciphersuite']: + self.ciphers = firefox_ciphers + else: + if os.path.isfile(options['ciphersuite']): + with open(options['ciphersuite']) as cipherfile: + for line in cipherfile.readlines(): + self.ciphers.append(line.strip()) + self.ciphersuite = ":".join(self.ciphers) + + if hasattr(config.advanced, 'default_timeout'): + timeout = config.advanced.default_timeout + else: + timeout = 10 ## default the timeout to 10 seconds + socket.setdefaulttimeout(timeout) + self.timeout = struct.pack('ll', int(timeout), 0) + + def splitInput(self, input): + addr, port = input.strip().rsplit(':', 1) + if self.localOptions['port']: + port = self.localOptions['port'] + return (str(addr), int(port)) + + def inputProcessor(self, file=None): + if os.path.isfile(file): + with open(file) as fh: + for line in fh.readlines(): + if line.startswith('#'): + continue + yield self.splitInput(line) + + def buildSocket(self, addr): + ip = IPAddress(addr) ## learn if we're IPv4 or IPv6 + if ip.version == 4: + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + elif ip.version == 6: + s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM) + return s + + def getContext(self): + if 'ssl2' in self.methods: + if not 'ssl3' in self.methods: + context = SSL.Context(SSL.SSLv2_METHOD) + else: + context = SSL.Context(SSL.SSLv23_METHOD) + elif 'ssl3' in self.methods: + context = SSL.Context(SSL.SSLv3_METHOD) + elif 'tls1' in self.methods: + context = SSL.Context(SSL.TLSv1_METHOD) + else: + raise Exception("No SSL/TLS method chosen!") + context.set_cipher_list(self.ciphersuite) + return context + + def test_tlsv1_handshake(self): + + def makeConnection(addr, port): + socket = self.buildSocket(addr) + context = self.getContext() + + connection = SSL.Connection(context, socket) + + try: + connection.connect((addr, port)) + except serror, se: + if se.message.find("[Errno 101]"): + connection.shutdown() + log.err(se) + else: + connection.setblocking(1) + connection.setsockopt(socket.SOL_SOCKET, socket.SO_RCVTIMEO, + self.timeout) + log.msg("Connected to %s" % connection.getpeername()) + log.msg("Connection state: %s " % connection.state_string()) + return connection + + def doHandshake(connection): + try: + connection.do_handshake() + except SSL.WantReadError(): + log.msg("Timeout exceeded.") + connection.shutdown() + else: + log.msg("State: %s" % connection.state_string()) + log.msg("Transmitted %d bytes" % connection.send("o\r\n")) + try: + recvstr = connection.recv(1024) + except SSL.WantReadError: + log.msg("Timeout exceeded") + connection.shutdown() + else: + log.msg("Received: %s" % recvstr) + return connection + + def handshakeSucceeded(connection): + if connection: + host, port = connection.getpeername() + self.report['host'] = host + self.report['port'] = port + self.report['state'] = connection.state_string() + + def handshakeFailed(connection, addr, port): + if connection is None: + self.report['host'] = addr + self.report['port'] = port + self.report['state'] = 'FAILED' + else: + return handshakeSucceeded(connection) + + addr, port = self.input + connection = defer.maybeDeferred(makeConnection, addr, port) + connection.addCallback(doHandshake) + connection.addErrback(log.err) + connection.addCallback(handshakeSucceeded) + connection.addErrback(handshakeFailed) + + return connection + +## XXX clean me up +## old function from anonymous contribution: (saved mostly for reference of the +## success string) +## +#def checkBridgeConnection(host, port) +# cipher_arg = ":".join(ciphers) +# cmd = ["openssl", "s_client", "-connect", "%s:%s" % (host,port)] +# cmd += ["-cipher", cipher_arg] +# proc = subprocess.Popen(cmd, stdout=PIPE, stderr=PIPE,stdin=PIPE) +# out, error = proc.communicate() +# success = "Cipher is DHE-RSA-AES256-SHA" in out +# return success [View Less]

1 0

[ooni-probe/develop] Use ssh config file
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 7f8f366d556cac02e7d92baddc019286cc789b25 Author: Arturo Filastò <art(a)fuffa.org> Date: Sat Mar 2 16:47:45 2013 +0100 Use ssh config file --- docs/fabfile.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/fabfile.py b/docs/fabfile.py index d0461fd..4e15993 100644 --- a/docs/fabfile.py +++ b/docs/fabfile.py @@ -1,8 +1,10 @@ import os -from fabric.api import run +from fabric.api import run, env from fabric.context_managers import settings … [View More]

1 0

[ooni-probe/develop] Update README.md
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit cb12b185c5c01f891109282ed75b87a6ba842c5a Author: samsmith <github.com(a)msmith.net> Date: Mon Mar 4 16:01:07 2013 +0000 Update README.md tcpdump seems to be required on a clean debian (6.0.7) install. (this change may also need making in other places) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 13102d3..b49a69b 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ Basic system … [View More]

1 0

[ooni-probe/develop] Fix #8329, txscapy.PcapDummyWriter.write() has wrong signature method.
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit b203c2e26985aa22bd8179da01fbdb642a12fc11 Author: Isis Lovecruft <isis(a)torproject.org> Date: Sun Feb 24 22:45:12 2013 +0000 Fix #8329, txscapy.PcapDummyWriter.write() has wrong signature method. --- ooni/utils/txscapy.py | 1 + 1 file changed, 1 insertion(+) diff --git a/ooni/utils/txscapy.py b/ooni/utils/txscapy.py index 87c0b4a..a6cacb1 100644 --- a/ooni/utils/txscapy.py +++ b/ooni/utils/txscapy.py @@ -64,6 +64,7 @@ else: def __init__(self, pcap_filename, *… [View More]

1 0

[ooni-probe/develop] Add Exception classes for conditions when the network is not reachable and
by isis＠torproject.org 06 Jun '13

06 Jun '13

commit 0dc64291dcf0c4ad6459d20b33123fc9cbe71ed6 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 01:45:54 2013 +0000 Add Exception classes for conditions when the network is not reachable and when a TLS/SSL context was not chosen from the test commandline options. --- nettests/experimental/tls_handshake.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index … [View More]

1 0