April 2013 - tor-commits - lists.torproject.org

[ooni-probe/master] Merge branch 'master' of github.com:TheTorProject/ooni-probe
by art＠torproject.org 30 Apr '13

30 Apr '13

commit 865ec32d230e18321df3ab006b8eb5fff747f9e0 Merge: df06507 4fdbd62 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Apr 30 15:01:06 2013 +0200 Merge branch 'master' of github.com:TheTorProject/ooni-probe * 'master' of github.com:TheTorProject/ooni-probe: (61 commits) Fix tls_handshake input processing Fix import conflict Set self.input to self.host if no inputFile, and update several log messages. Reformat docstrings for several HandshakeTest functions to Sphinx format. Rename HandshakeTest.test_tlsv1_handshake() to HandshakeTest.test_handshake(). Cleanup docstrings for getX509Name() and getPublicKey() functions. Add Sphinx docstring for getPeerCert() function. Move to call IPAddress explicitly from ipaddr.IPAdress. Add self.host to the inputProcessor(). If no ciphersuite list is provided, explain that we're using firefox's. Fix bug where conflicting SSL context choices was not explained to the user. Fix #8440, set the self.host attribute when given the --host option. Create a host attribute for storing the --host option, and set self.ciphers. Make 'context' be a class-level attribute and give it a Sphinx :attr: docstring. Bump the version number on HandshakeTest, and rename the usageOptions attribute. Rename TLSHandshakeTest to HandshakeTest, since it's not only TLS. Remove UsageOptions to HandshakeOptions, and make arg help strings more informative. Fix HostUnreachable docstring to have proper Sphinx formatting. Add dictionary of pre-defined error messages to SSLContextError. Update docstring for SSLContextError. ... .gitignore | 2 + nettests/experimental/tls_handshake.py | 794 ++++++++++++++++++++++++++++++++ ooni/utils/txscapy.py | 1 + 3 files changed, 797 insertions(+), 0 deletions(-)

1 0

[ooni-probe/master] Fix method connectionFailed to handle socket timeouts.
by art＠torproject.org 30 Apr '13

30 Apr '13

commit 61cea5561d9a22dd9009331ca16b7506a3c24d7f Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 17:42:31 2013 +0000 Fix method connectionFailed to handle socket timeouts. --- nettests/experimental/tls_handshake.py | 33 +++++++++++++++++++------------ 1 files changed, 20 insertions(+), 13 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index 4c42017..610cdff 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -292,22 +292,29 @@ class TLSHandshakeTest(nettest.NetTestCase): specifying the host port, i.e. ('1.1.1.1', 443) """ addr, port = host - if isinstance(connection, IOError): - ## On some *nix distros, /dev/random is 0600 root:root and we get - ## a permissions error when trying to read - if connection.message.find("[Errno 13]"): - raise NotRootError( - "%s" % connection.message.split("[Errno 13]", 1)[1]) - - if isinstance(connection, socket_error): - if connection.message.find("[Errno 101]"): - raise HostUnreachableError( - "Host unreachable: %s:%s" % (addr, port)) - - log.err(connection) + + if not isinstance(connection, SSL.Connection): + if isinstance(connection, IOError): + ## On some *nix distros, /dev/random is 0600 root:root and + ## we get a permissions error when trying to read + if connection.message.find("[Errno 13]"): + raise NotRootError( + "%s" % connection.message.split("[Errno 13]", 1)[1]) + elif isinstance(connection, socket_error): + if connection.message.find("[Errno 101]"): + raise HostUnreachableError( + "Host unreachable: %s:%s" % (addr, port)) + elif isinstance(connection, Exception): + log.debug("connectionFailed: got Exception:") + log.err("Connection failed with reason: %s" + % connection.message) + else: + log.err("Connection failed with reason: %s" % str(connection)) + self.report['host'] = addr self.report['port'] = port self.report['state'] = 'CONNECTION_FAILED' + return connection def connectionSucceeded(connection, host, timeout):

1 0

[ooni-probe/master] Changed log.msg to log.debug when we can't get the peer name in
by art＠torproject.org 30 Apr '13

30 Apr '13

commit 3461ad38a1e817daffa7d79f873db989d5e8efe9 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 17:43:26 2013 +0000 Changed log.msg to log.debug when we can't get the peer name in connectionSucceeded(). --- nettests/experimental/tls_handshake.py | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index 610cdff..d9d805f 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -345,8 +345,8 @@ class TLSHandshakeTest(nettest.NetTestCase): log.msg("Connected to %s" % peer_name) else: log.debug("Couldn't get peer name from connection: %s" % host) - log.msg("Connected to: %s" % host) - log.msg("Connection state: %s " % connection.state_string()) + log.msg("Connected to %s" % host) + log.debug("Connection state: %s " % connection.state_string()) return connection @@ -423,6 +423,7 @@ class TLSHandshakeTest(nettest.NetTestCase): else: log.debug("connectionShutdown: expected connection, got %s" % connection.__repr__()) + return connection def handleWantRead(connection):

1 0

[ooni-probe/master] Fix method doHandshake() to callback to handleWantRead() and handleWantWrite().
by art＠torproject.org 30 Apr '13

30 Apr '13

commit 27292c9c868694ca19e2d2b9aa9c32220f9d89cf Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 17:57:05 2013 +0000 Fix method doHandshake() to callback to handleWantRead() and handleWantWrite(). --- nettests/experimental/tls_handshake.py | 26 +++++++++++++++++++------- 1 files changed, 19 insertions(+), 7 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index 3c9a033..e443242 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -553,17 +553,29 @@ class TLSHandshakeTest(nettest.NetTestCase): """ peername, peerport = connection.getpeername() - log.msg("Attempting handshake: %s" % peername) - connection.do_handshake() - log.debug("State: %s" % connection.state_string()) - if connection.state_string() == \ - 'SSL negotiation finished successfully': + try: + log.msg("Attempting handshake: %s" % peername) + connection.do_handshake() + except OpenSSL.SSL.WantReadError() as wre: + self.state = connection.state_string() + log.debug("Handshake state: %s" % self.state) + log.debug("doHandshake: WantReadError on first handshake attempt.") + connection = handleWantRead(connection) + except OpenSSL.SSL.WantWriteError() as wwe: + self.state = connection.state_string() + log.debug("Handshake state: %s" % self.state) + log.debug("doHandshake: WantWriteError on first handshake attempt.") + connection = handleWantWrite(connection) + else: + self.state = connection.state_string() + + if self.state == 'SSL negotiation finished successfully': ## jump to handshakeSuccessful and get certchain return connection - else: sent = connection.send("o\r\n") - log.debug("State: %s" % connection.state_string()) + self.state = connection.state_string() + log.debug("Handshake state: %s" % self.state) log.debug("Transmitted %d bytes" % sent) _read_buffer = connection.pending()

1 0

[ooni-probe/master] Set tls_handshake.state to be connection.state_string so that it gets updated
by art＠torproject.org 30 Apr '13

30 Apr '13

commit 2ca25a6d0d2a7320e1f508411bf6c2dea077d5c4 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 17:51:32 2013 +0000 Set tls_handshake.state to be connection.state_string so that it gets updated and we always have the latest state accessible in failure and success callbacks. --- nettests/experimental/tls_handshake.py | 17 +++++++++++++---- 1 files changed, 13 insertions(+), 4 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index d9d805f..3e754ae 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -482,6 +482,7 @@ class TLSHandshakeTest(nettest.NetTestCase): """ try: while connection.want_read(): + self.state = connection.state_string() log.debug("Connection to %s HAS want_read" % host) _read_buffer = connection.pending() log.debug("Rereading %d bytes..." % _read_buffer) @@ -490,11 +491,13 @@ class TLSHandshakeTest(nettest.NetTestCase): log.debug("Received %d bytes" % rereceived) log.debug("State: %s" % connection.state_string()) else: + self.state = connection.state_string() peername, peerport = connection.getpeername() log.debug("Connection to %s:%s DOES NOT HAVE want_read" % (peername, peerport)) log.debug("State: %s" % connection.state_string()) except SSL.WantWriteError, wwe: + self.state = connection.state_string() log.debug("Got WantWriteError while handling want_read") log.debug("WantWriteError: %s" % wwe.message) log.debug("Switching to handleWantWrite()...") @@ -507,12 +510,14 @@ class TLSHandshakeTest(nettest.NetTestCase): """ try: while connection.want_write(): + self.state = connection.state_string() log.debug("Connection to %s HAS want_write" % host) sleep(1) resent = connection.send("o\r\n") log.debug("Sent: %d" % resent) log.debug("State: %s" % connection.state_string()) except SSL.WantReadError, wre: + self.state = connection.state_string() log.debug("Got WantReadError while handling want_write") log.debug("WantReadError: %s" % wre.message) log.debug("Switching to handleWantRead()...") @@ -565,23 +570,27 @@ class TLSHandshakeTest(nettest.NetTestCase): received = connection.recv(int(_read_buffer)) except SSL.WantReadError, wre: if connection.want_read(): + self.state = connection.state_string() connection = handleWantRead(connection) else: ## if we still have an SSL_ERROR_WANT_READ, then try ## to renegotiate + self.state = connection.state_string() connection = connectionRenegotiate(connection, connection.getpeername(), wre.message) except SSL.WantWriteError, wwe: - log.debug("State: %s" % connection.state_string()) + self.state = connection.state_string() + log.debug("Handshake state: %s" % self.state) if connection.want_write(): connection = handleWantWrite(connection) else: - log.msg("Connection to %s:%s timed out." - % (peername, str(peerport))) + raise ConnectionTimeout("Connection to %s:%d timed out." + % (peername, peerport)) else: log.msg("Received: %s" % received) - log.debug("State: %s" % connection.state_string()) + self.state = connection.state_string() + log.debug("Handshake state: %s" % self.state) return connection

1 0

[ooni-probe/master] Make a note on the socket option settings for blocking.
by art＠torproject.org 30 Apr '13

30 Apr '13

commit 578c6703106c81171e4af5ad45fca22fe8979d76 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 17:56:29 2013 +0000 Make a note on the socket option settings for blocking. --- nettests/experimental/tls_handshake.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index fbd60d8..3c9a033 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -325,7 +325,10 @@ class TLSHandshakeTest(nettest.NetTestCase): @param connection: A :class:`OpenSSL.SSL.Connection` object. @param host: A tuple of the host IP and port, i.e. ('1.1.1.1', 443). """ + ## xxx TODO to get this to work with a non-blocking socket, see how + ## twisted.internet.tcp.Client handles socket objects. connection.setblocking(1) + ## Set the timeout on the connection: ## ## We want to set SO_RCVTIMEO and SO_SNDTIMEO, which both are

1 0

[ooni-probe/master] Appease asn's penchant for borderline obsessive compulsive docstrings.
by art＠torproject.org 30 Apr '13

30 Apr '13

commit a67c418beb80ba8f3c8e81ddc906472fc30b3274 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 17:52:51 2013 +0000 Appease asn's penchant for borderline obsessive compulsive docstrings. * Notice, also, how the last sentence is less than 70 chars. :) --- nettests/experimental/tls_handshake.py | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index 3e754ae..fbd60d8 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -603,8 +603,6 @@ class TLSHandshakeTest(nettest.NetTestCase): >>> server_cert.get_pubkey() <OpenSSL.crypto.PKey at 0x4985d28> >>> pk = server_cert.get_pubkey() - >>> pk.check - <function check> >>> pk.check() Segmentation fault

1 0

[ooni-probe/master] Switch to using threads.deferToThread() in method deferMakeConnection().
by art＠torproject.org 30 Apr '13

30 Apr '13

commit f69452eac1cf5334ab84caab260cbfcaf3adc2dd Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 18:35:49 2013 +0000 Switch to using threads.deferToThread() in method deferMakeConnection(). --- nettests/experimental/tls_handshake.py | 24 ++++++++++++++++-------- 1 files changed, 16 insertions(+), 8 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index 48902b3..ddef435 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -696,15 +696,23 @@ class TLSHandshakeTest(nettest.NetTestCase): self.report['host'] = host self.report['port'] = port - @defer.inlineCallbacks - def deferMakeConnection(host): - connection = yield makeConnection(host) - if isinstance(connection, Failure) \ - or isinstance(connection, Exception): - failed = connectionFailed(connection, host) - defer.returnValue(failed) + if isinstance(connection, Exception) \ + or isinstance(connection, ConnectionTimeout): + log.msg("Handshake failed with reason: %s" % connection.message) + self.report['state'] = connection.message + elif isinstance(connection, failure.Failure): + log.msg("Handshake failed with reason: Socket %s" + % connection.getErrorMessage()) + self.report['state'] = connection.getErrorMessage() + ctmo = connection.trap(ConnectionTimeout) + if ctmo == ConnectionTimeout: + connection.cleanFailure() else: - defer.returnValue(connection) + log.msg("Handshake failed with reason: %s" % str(connection)) + if not 'state' in self.report.keys(): + self.report['state'] = str(connection) + + return None connection = deferMakeConnection(self.input) connection.addCallbacks(connectionSucceeded, connectionFailed,

1 0

[ooni-probe/master] Add call to new deferMakeConnection() to test_tls_handshake().
by art＠torproject.org 30 Apr '13

30 Apr '13

commit 5e1f148a6171efddc96afdd6930978387a052cb5 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 18:37:18 2013 +0000 Add call to new deferMakeConnection() to test_tls_handshake(). --- nettests/experimental/tls_handshake.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index ddef435..1b66b45 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -714,6 +714,9 @@ class TLSHandshakeTest(nettest.NetTestCase): return None + def deferMakeConnection(host): + return threads.deferToThread(makeConnection, self.input) + connection = deferMakeConnection(self.input) connection.addCallbacks(connectionSucceeded, connectionFailed, callbackArgs=[self.input, self.timeout],

1 0

[ooni-probe/master] Remove erroneous state string report from handshakeSuccessful().
by art＠torproject.org 30 Apr '13

30 Apr '13

commit c2d3d3813dac2c337cac059315c990b57767abc1 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 18:00:17 2013 +0000 Remove erroneous state string report from handshakeSuccessful(). --- nettests/experimental/tls_handshake.py | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index 47e5091..48902b3 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -692,11 +692,9 @@ class TLSHandshakeTest(nettest.NetTestCase): """ addr, port = host log.msg("Handshake with %s:%d failed!" % host) + self.report['host'] = host self.report['port'] = port - self.report['state'] = "HANDSHAKE_FAILED" - ## xxx do we need this? - #return connection @defer.inlineCallbacks def deferMakeConnection(host):

1 0