- tor-commits - lists.torproject.org

[translation/vidalia_alpha] Update translations for vidalia_alpha
by translation＠torproject.org 23 Nov '12

23 Nov '12

commit 4349e0a78740b7538dfcb2c7705b93488cf14274 Author: Translation commit bot <translation(a)torproject.org> Date: Fri Nov 23 05:45:17 2012 +0000 Update translations for vidalia_alpha --- es/vidalia_es.po | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/es/vidalia_es.po b/es/vidalia_es.po index 62999d3..21ab572 100644 --- a/es/vidalia_es.po +++ b/es/vidalia_es.po @@ -9,7 +9,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2012-03-21 17:46+0000\n" -"PO-Revision-Date: 2012-11-23 05:12+0000\n" +"PO-Revision-Date: 2012-11-23 05:26+0000\n" "Last-Translator: strel <strelnic(a)gmail.com>\n" "Language-Team: Spanish (http://www.transifex.com/projects/p/torproject/language/es/)\n" "MIME-Version: 1.0\n" @@ -3666,11 +3666,11 @@ msgstr "Éxito" msgctxt "UPNPControl" msgid "No UPnP-enabled devices found" -msgstr "No se encontraron dispositivos UpnP habilitados" +msgstr "No se encontraron dispositivos con UPnP activo" msgctxt "UPNPControl" msgid "No valid UPnP-enabled Internet gateway devices found" -msgstr "No se encontraron dispositivos UPnP de puertas de enlace de Internet válidos" +msgstr "No se encontraron dispositivos válidos de salida a Internet (gateway) con UPnP activo" msgctxt "UPNPControl" msgid "WSAStartup failed"

1 0

[translation/vidalia_alpha_completed] Update translations for vidalia_alpha_completed
by translation＠torproject.org 23 Nov '12

23 Nov '12

commit 434c23ee51053ca49723418013304b3a48624bbc Author: Translation commit bot <translation(a)torproject.org> Date: Fri Nov 23 05:15:23 2012 +0000 Update translations for vidalia_alpha_completed --- es/vidalia_es.po | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/es/vidalia_es.po b/es/vidalia_es.po index 45fb5a7..62999d3 100644 --- a/es/vidalia_es.po +++ b/es/vidalia_es.po @@ -9,7 +9,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2012-03-21 17:46+0000\n" -"PO-Revision-Date: 2012-11-22 19:03+0000\n" +"PO-Revision-Date: 2012-11-23 05:12+0000\n" "Last-Translator: strel <strelnic(a)gmail.com>\n" "Language-Team: Spanish (http://www.transifex.com/projects/p/torproject/language/es/)\n" "MIME-Version: 1.0\n" @@ -3598,7 +3598,7 @@ msgstr "No se pudo iniciar el servicio Tor." msgctxt "TorSettings" msgid "Failed to hash the control password." -msgstr "Fallo al obtener hash de la contraseña de control." +msgstr "Fallo al obtener hash de la contraseña de control (del servicio Tor)" msgctxt "TorrcDialog" msgid "Editing torrc"

1 0

[translation/vidalia_alpha] Update translations for vidalia_alpha
by translation＠torproject.org 23 Nov '12

23 Nov '12

commit 3f0acac901023b2e8bcc3529600d73b6d7096d70 Author: Translation commit bot <translation(a)torproject.org> Date: Fri Nov 23 05:15:18 2012 +0000 Update translations for vidalia_alpha --- es/vidalia_es.po | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/es/vidalia_es.po b/es/vidalia_es.po index 45fb5a7..62999d3 100644 --- a/es/vidalia_es.po +++ b/es/vidalia_es.po @@ -9,7 +9,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2012-03-21 17:46+0000\n" -"PO-Revision-Date: 2012-11-22 19:03+0000\n" +"PO-Revision-Date: 2012-11-23 05:12+0000\n" "Last-Translator: strel <strelnic(a)gmail.com>\n" "Language-Team: Spanish (http://www.transifex.com/projects/p/torproject/language/es/)\n" "MIME-Version: 1.0\n" @@ -3598,7 +3598,7 @@ msgstr "No se pudo iniciar el servicio Tor." msgctxt "TorSettings" msgid "Failed to hash the control password." -msgstr "Fallo al obtener hash de la contraseña de control." +msgstr "Fallo al obtener hash de la contraseña de control (del servicio Tor)" msgctxt "TorrcDialog" msgid "Editing torrc"

1 0

[ooni-probe/master] Get HTTP Requests test in a working state.
by art＠torproject.org 22 Nov '12

22 Nov '12

commit 01d80a2abc235fedbd2944500e259e537fd46c45 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Nov 22 23:23:07 2012 +0100 Get HTTP Requests test in a working state. * Performs random capitalization of a set of static HTTP Header fields and measures on the backend if they are as expected. * XXX the detection logic needs some more work * XXX We need to keep track of the order in which we send the headers from the HTTP Agent. Perhaps it makes sense to implement a specific HTTP Agent that does not implement the full HTTP protocol like what we did for the backend component. --- nettests/core/http_requests.py | 90 +++++++++++++++++++++++++++++++----- ooni/templates/httpt.py | 5 +- ooni/utils/txagentwithsocks.py | 30 +++++++------ oonib/testhelpers/http_helpers.py | 9 +++- 4 files changed, 102 insertions(+), 32 deletions(-) diff --git a/nettests/core/http_requests.py b/nettests/core/http_requests.py index 0f3a2d8..5d67070 100644 --- a/nettests/core/http_requests.py +++ b/nettests/core/http_requests.py @@ -8,22 +8,26 @@ import json from twisted.python import usage -from ooni.utils import log, net +from ooni.utils import log, net, randomStr from ooni.templates import httpt def random_capitalization(string): output = "" + original_string = string string = string.swapcase() for i in range(len(string)): if random.randint(0, 1): output += string[i].swapcase() else: output += string[i] - return output + if original_string == output: + return random_capitalization(output) + else: + return output class UsageOptions(usage.Options): optParameters = [ - ['backend', 'b', None, + ['backend', 'b', 'http://127.0.0.1:57001', 'URL of the backend to use for sending the requests'], ['headers', 'h', None, 'Specify a yaml formatted file from which to read the request headers to send'] @@ -36,8 +40,9 @@ class HTTPRequests(httpt.HTTPTest): """ name = "HTTP Requests" author = "Arturo Filastò" - version = 0.1 + version = "0.1.1" + randomizeUA = False usageOptions = UsageOptions requiredOptions = ['backend'] @@ -49,12 +54,69 @@ class HTTPRequests(httpt.HTTPTest): raise Exception("No backend specified") def processResponseBody(self, data): + self.check_for_tampering(data) + + def check_for_tampering(self, data): + """ + Here we do checks to verify if the request we made has been tampered + with. We have 3 categories of tampering: + + * **total** when the response is not a json object and therefore we were not + able to reach the ooniprobe test backend + + * **request_line_capitalization** when the HTTP Request line (e.x. GET / + HTTP/1.1) does not match the capitalization we set. + + * **header_field_number** when the number of headers we sent does not match + with the ones the backend received + + * **header_name_capitalization** when the header field names do not match + those that we sent. + + * **header_field_value** when the header field value does not match with the + one we transmitted. + """ + self.report['tampering'] = {'total': False, + 'request_line_capitalization': False, + 'header_name_capitalization': False, + 'header_field_value': False, + 'header_field_number': False + } + try: response = json.loads(data) except ValueError: - self.report['tampering'] = True - - # XXX add checks for validation of sent headers + self.report['tampering']['total'] = True + return + + requestLine = "%s / HTTP/1.1" % self.request_method + if response['request_line'] != requestLine: + self.report['tampering']['request_line_capitalization'] = True + + # We compare against length -1 because the response will also contain + # the Connection: close header since we do not do persistent + # connections + if len(self.request_headers) != (len(response['headers_dict']) - 1): + self.report['tampering']['header_field_number'] = True + + for header, value in self.request_headers.items(): + # XXX this still needs some work + # in particular if the response headers are of different length or + # some extra headers get added in the response (so the lengths + # match), we will get header_name_capitalization set to true, while + # the actual tampering is the addition of an extraneous header + # field. + if header == "Connection": + # Ignore Connection header + continue + try: + response_value = response['headers_dict'][header] + if response_value != value[0]: + log.msg("Tampering detected because %s != %s" % (response_value, value[0])) + self.report['tampering']['header_field_value'] = True + except KeyError: + log.msg("Tampering detected because %s not in %s" % (header, response['headers_dict'])) + self.report['tampering']['header_name_capitalization'] = True def get_headers(self): headers = {} @@ -69,11 +131,13 @@ class HTTPRequests(httpt.HTTPTest): headers = yaml.load(content) return headers else: - headers = {"User-Agent": [random.choice(net.userAgents)], + headers = {"User-Agent": [random.choice(net.userAgents)[0]], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Accept-Encoding": ["gzip,deflate,sdch"], "Accept-Language": ["en-US,en;q=0.8"], - "Accept-Charset": ["ISO-8859-1,utf-8;q=0.7,*;q=0.3"]} + "Accept-Charset": ["ISO-8859-1,utf-8;q=0.7,*;q=0.3"], + "Host": [randomStr(15)+'.com'] + } return headers def get_random_caps_headers(self): @@ -90,25 +154,25 @@ class HTTPRequests(httpt.HTTPTest): return self.doRequest(self.url, self.request_method, headers=self.request_headers) - def a_test_get_random_capitalization(self): + def test_get_random_capitalization(self): self.request_method = random_capitalization("GET") self.request_headers = self.get_random_caps_headers() return self.doRequest(self.url, self.request_method, headers=self.request_headers) - def a_test_post(self): + def test_post(self): self.request_method = "POST" self.request_headers = self.get_headers() return self.doRequest(self.url, self.request_method, headers=self.request_headers) - def a_test_post_random_capitalization(self): + def test_post_random_capitalization(self): self.request_method = random_capitalization("POST") self.request_headers = self.get_random_caps_headers() return self.doRequest(self.url, self.request_method, headers=self.request_headers) - def a_test_put(self): + def test_put(self): self.request_method = "PUT" self.request_headers = self.get_headers() return self.doRequest(self.url, self.request_method, diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index 46bde28..e166263 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -15,7 +15,6 @@ from twisted.internet import reactor from twisted.internet.error import ConnectionRefusedError from twisted.web._newclient import Request -from twisted.web.http_headers import Headers from ooni.nettest import NetTestCase from ooni.utils import log @@ -23,7 +22,7 @@ from ooni import config from ooni.utils.net import BodyReceiver, StringProducer, userAgents -from ooni.utils.txagentwithsocks import Agent, SOCKSError +from ooni.utils.txagentwithsocks import Agent, SOCKSError, TrueHeaders class HTTPTest(NetTestCase): """ @@ -195,7 +194,7 @@ class HTTPTest(NetTestCase): else: body_producer = None - headers = Headers(request['headers']) + headers = TrueHeaders(request['headers']) def errback(failure): failure.trap(ConnectionRefusedError, SOCKSError) diff --git a/ooni/utils/txagentwithsocks.py b/ooni/utils/txagentwithsocks.py index fc1c17c..05da54f 100644 --- a/ooni/utils/txagentwithsocks.py +++ b/ooni/utils/txagentwithsocks.py @@ -50,13 +50,13 @@ class SOCKSv5ClientProtocol(_WrappingProtocol): errcode = ord(data[1]) self._connectedDeferred.errback(SOCKSError(errcode)) - + return self.ready = True self._wrappedProtocol.transport = self.transport self._wrappedProtocol.connectionMade() - + self._connectedDeferred.callback(self._wrappedProtocol) def connectionMade(self): @@ -81,7 +81,7 @@ class SOCKSv5ClientProtocol(_WrappingProtocol): class SOCKSv5ClientFactory(_WrappingFactory): protocol = SOCKSv5ClientProtocol - + def __init__(self, wrappedFactory, host, port): _WrappingFactory.__init__(self, wrappedFactory) self._host, self._port = host, port @@ -119,7 +119,7 @@ class SOCKS5ClientEndpoint(object): except: return defer.fail() -class Headers(http_headers.Headers): +class TrueHeaders(http_headers.Headers): def __init__(self, rawHeaders=None): self._rawHeaders = dict() if rawHeaders is not None: @@ -141,12 +141,13 @@ class Headers(http_headers.Headers): def getRawHeaders(self, name, default=None): if name.lower() in self._rawHeaders: - return self._rawHeaders[name.lower()]["values"] + return self._rawHeaders[name.lower()]['values'] return default + class HTTPClientParser(_newclient.HTTPClientParser): def connectionMade(self): - self.headers = Headers() - self.connHeaders = Headers() + self.headers = TrueHeaders() + self.connHeaders = TrueHeaders() self.state = STATUS self._partialHeader = None @@ -224,31 +225,32 @@ class Agent(client.Agent): return TCP4ClientEndpoint(self._reactor, host, port, **kwargs) elif scheme == 'shttp': return SOCKS5ClientEndpoint(self._reactor, self._sockshost, - self._socksport, host, port, **kwargs) + self._socksport, host, port, **kwargs) elif scheme == 'httpo': return SOCKS5ClientEndpoint(self._reactor, self._sockshost, - self._socksport, host, port, **kwargs) + self._socksport, host, port, **kwargs) elif scheme == 'https': return SSL4ClientEndpoint(self._reactor, host, port, - self._wrapContextFactory(host, port), - **kwargs) + self._wrapContextFactory(host, port), **kwargs) else: raise SchemeNotSupported("Unsupported scheme: %r" % (scheme,)) def _requestWithEndpoint(self, key, endpoint, method, parsedURI, headers, bodyProducer, requestPath): if headers is None: - headers = Headers() + headers = TrueHeaders() if not headers.hasHeader('host'): headers = headers.copy() headers.addRawHeader( - 'host', self._computeHostValue(parsedURI.scheme, parsedURI.host, - parsedURI.port)) + 'host', self._computeHostValue(parsedURI.scheme, + parsedURI.host, parsedURI.port)) d = self._pool.getConnection(key, endpoint) + print headers._rawHeaders def cbConnected(proto): return proto.request( Request(method, requestPath, headers, bodyProducer, persistent=self._pool.persistent)) d.addCallback(cbConnected) return d + diff --git a/oonib/testhelpers/http_helpers.py b/oonib/testhelpers/http_helpers.py index cbd6caa..b384216 100644 --- a/oonib/testhelpers/http_helpers.py +++ b/oonib/testhelpers/http_helpers.py @@ -72,12 +72,17 @@ class SimpleHTTPChannel(basic.LineReceiver, policies.TimeoutMixin): def headerReceived(self, line): header, data = line.split(':', 1) - self.headers.append((header, data)) + self.headers.append((header, data.strip())) def allHeadersReceived(self): + headers_dict = {} + for k, v in self.headers: + headers_dict[k] = v response = {'request_headers': self.headers, - 'request_line': self.requestLine + 'request_line': self.requestLine, + 'headers_dict': headers_dict } + self.transport.write('HTTP/1.1 200 OK\r\n\r\n') self.transport.write(json.dumps(response)) self.transport.loseConnection()

1 0

[ooni-probe/master] Implement a bit more of HTTP so that the response if properly parsed by HTTP clients
by art＠torproject.org 22 Nov '12

22 Nov '12

commit dfc33ae6d9c6ed97182d5b886b3704c8e865ccf7 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Nov 22 21:38:16 2012 +0100 Implement a bit more of HTTP so that the response if properly parsed by HTTP clients --- nettests/core/http_requests.py | 18 +++++++++--------- 1 files changed, 9 insertions(+), 9 deletions(-) diff --git a/nettests/core/http_requests.py b/nettests/core/http_requests.py index 7b27b94..0f3a2d8 100644 --- a/nettests/core/http_requests.py +++ b/nettests/core/http_requests.py @@ -49,11 +49,11 @@ class HTTPRequests(httpt.HTTPTest): raise Exception("No backend specified") def processResponseBody(self, data): - response = json.loads(data) - if response['request_method'] != self.request_method: + try: + response = json.loads(data) + except ValueError: self.report['tampering'] = True - else: - self.report['tampering'] = False + # XXX add checks for validation of sent headers def get_headers(self): @@ -87,7 +87,7 @@ class HTTPRequests(httpt.HTTPTest): def test_get(self): self.request_method = "GET" self.request_headers = self.get_random_caps_headers() - return self.doRequest(self.url, self.request_method, + return self.doRequest(self.url, self.request_method, headers=self.request_headers) def a_test_get_random_capitalization(self): @@ -99,25 +99,25 @@ class HTTPRequests(httpt.HTTPTest): def a_test_post(self): self.request_method = "POST" self.request_headers = self.get_headers() - return self.doRequest(self.url, self.request_method, + return self.doRequest(self.url, self.request_method, headers=self.request_headers) def a_test_post_random_capitalization(self): self.request_method = random_capitalization("POST") self.request_headers = self.get_random_caps_headers() - return self.doRequest(self.url, self.request_method, + return self.doRequest(self.url, self.request_method, headers=self.request_headers) def a_test_put(self): self.request_method = "PUT" self.request_headers = self.get_headers() - return self.doRequest(self.url, self.request_method, + return self.doRequest(self.url, self.request_method, headers=self.request_headers) def test_put_random_capitalization(self): self.request_method = random_capitalization("PUT") self.request_headers = self.get_random_caps_headers() - return self.doRequest(self.url, self.request_method, + return self.doRequest(self.url, self.request_method, headers=self.request_headers)

1 0

[ooni-probe/master] Move txagentwithsocks to utils
by art＠torproject.org 22 Nov '12

22 Nov '12

commit 837924eec54fec42036db11987c698ad3744c41b Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Nov 22 21:14:50 2012 +0100 Move txagentwithsocks to utils --- nettests/core/http_requests.py | 2 +- ooni/lib/txagentwithsocks.py | 254 ---------------------------------------- ooni/reporter.py | 2 +- ooni/templates/httpt.py | 2 +- ooni/utils/txagentwithsocks.py | 254 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 257 insertions(+), 257 deletions(-) diff --git a/nettests/core/http_requests.py b/nettests/core/http_requests.py index 7a2f5ef..7b27b94 100644 --- a/nettests/core/http_requests.py +++ b/nettests/core/http_requests.py @@ -93,7 +93,7 @@ class HTTPRequests(httpt.HTTPTest): def a_test_get_random_capitalization(self): self.request_method = random_capitalization("GET") self.request_headers = self.get_random_caps_headers() - return self.doRequest(self.url, self.request_method, + return self.doRequest(self.url, self.request_method, headers=self.request_headers) def a_test_post(self): diff --git a/ooni/lib/txagentwithsocks.py b/ooni/lib/txagentwithsocks.py deleted file mode 100644 index fc1c17c..0000000 --- a/ooni/lib/txagentwithsocks.py +++ /dev/null @@ -1,254 +0,0 @@ -# -*- encoding: utf-8 -*- -# -# :authors: Giovanni Pellerano -# :licence: see LICENSE - -import struct - -from zope.interface import implements -from twisted.web import client, _newclient, http_headers -from twisted.web._newclient import Request, RequestNotSent, RequestGenerationFailed, TransportProxyProducer, STATUS -from twisted.internet import protocol -from twisted.internet.protocol import ClientFactory, Protocol -from twisted.internet.endpoints import TCP4ClientEndpoint, SSL4ClientEndpoint, _WrappingProtocol, _WrappingFactory -from twisted.internet import interfaces, defer -from twisted.internet.defer import Deferred, succeed, fail, maybeDeferred - -class SOCKSError(Exception): - def __init__(self, value): - Exception.__init__(self) - self.code = value - -class SOCKSv5ClientProtocol(_WrappingProtocol): - state = 0 - - def __init__(self, connectedDeferred, wrappedProtocol, host, port): - _WrappingProtocol.__init__(self, connectedDeferred, wrappedProtocol) - self._host = host - self._port = port - self.ready = False - - def socks_state_0(self, data): - # error state - self._connectedDeferred.errback(SOCKSError(0x00)) - return - - def socks_state_1(self, data): - if data != "\x05\x00": - self._connectedDeferred.errback(SOCKSError(0x00)) - return - - # Anonymous access allowed - let's issue connect - self.transport.write(struct.pack("!BBBBB", 5, 1, 0, 3, - len(self._host)) + - self._host + - struct.pack("!H", self._port)) - - def socks_state_2(self, data): - if data[:2] != "\x05\x00": - # Anonymous access denied - - errcode = ord(data[1]) - self._connectedDeferred.errback(SOCKSError(errcode)) - - return - - self.ready = True - self._wrappedProtocol.transport = self.transport - self._wrappedProtocol.connectionMade() - - self._connectedDeferred.callback(self._wrappedProtocol) - - def connectionMade(self): - # We implement only Anonymous access - self.transport.write(struct.pack("!BB", 5, len("\x00")) + "\x00") - - self.state = self.state + 1 - - def write(self, data): - if self.ready: - self.transport.write(data) - else: - self.buf.append(data) - - def dataReceived(self, data): - if self.state != 3: - getattr(self, 'socks_state_%s' % (self.state), - self.socks_state_0)(data) - self.state = self.state + 1 - else: - self._wrappedProtocol.dataReceived(data) - -class SOCKSv5ClientFactory(_WrappingFactory): - protocol = SOCKSv5ClientProtocol - - def __init__(self, wrappedFactory, host, port): - _WrappingFactory.__init__(self, wrappedFactory) - self._host, self._port = host, port - - def buildProtocol(self, addr): - try: - proto = self._wrappedFactory.buildProtocol(addr) - except: - self._onConnection.errback() - else: - return self.protocol(self._onConnection, proto, - self._host, self._port) - -class SOCKS5ClientEndpoint(object): - implements(interfaces.IStreamClientEndpoint) - - def __init__(self, reactor, sockshost, socksport, - host, port, timeout=30, bindAddress=None): - - self._reactor = reactor - self._sockshost = sockshost - self._socksport = socksport - self._host = host - self._port = port - self._timeout = timeout - self._bindAddress = bindAddress - - def connect(self, protocolFactory): - try: - wf = SOCKSv5ClientFactory(protocolFactory, self._host, self._port) - self._reactor.connectTCP( - self._sockshost, self._socksport, wf, - timeout=self._timeout, bindAddress=self._bindAddress) - return wf._onConnection - except: - return defer.fail() - -class Headers(http_headers.Headers): - def __init__(self, rawHeaders=None): - self._rawHeaders = dict() - if rawHeaders is not None: - for name, values in rawHeaders.iteritems(): - if type(values) is list: - self.setRawHeaders(name, values[:]) - elif type(values) is dict: - self._rawHeaders[name.lower()] = values - - def setRawHeaders(self, name, values): - if name.lower() not in self._rawHeaders: - self._rawHeaders[name.lower()] = dict() - self._rawHeaders[name.lower()]['name'] = name - self._rawHeaders[name.lower()]['values'] = values - - def getAllRawHeaders(self): - for k, v in self._rawHeaders.iteritems(): - yield v['name'], v['values'] - - def getRawHeaders(self, name, default=None): - if name.lower() in self._rawHeaders: - return self._rawHeaders[name.lower()]["values"] - return default -class HTTPClientParser(_newclient.HTTPClientParser): - def connectionMade(self): - self.headers = Headers() - self.connHeaders = Headers() - self.state = STATUS - self._partialHeader = None - - def headerReceived(self, name, value): - if self.isConnectionControlHeader(name): - headers = self.connHeaders - else: - headers = self.headers - headers.addRawHeader(name, value) - - -class HTTP11ClientProtocol(_newclient.HTTP11ClientProtocol): - def request(self, request): - if self._state != 'QUIESCENT': - return fail(RequestNotSent()) - - self._state = 'TRANSMITTING' - _requestDeferred = maybeDeferred(request.writeTo, self.transport) - self._finishedRequest = Deferred() - - self._currentRequest = request - - self._transportProxy = TransportProxyProducer(self.transport) - self._parser = HTTPClientParser(request, self._finishResponse) - self._parser.makeConnection(self._transportProxy) - self._responseDeferred = self._parser._responseDeferred - - def cbRequestWrotten(ignored): - if self._state == 'TRANSMITTING': - self._state = 'WAITING' - self._responseDeferred.chainDeferred(self._finishedRequest) - - def ebRequestWriting(err): - if self._state == 'TRANSMITTING': - self._state = 'GENERATION_FAILED' - self.transport.loseConnection() - self._finishedRequest.errback( - Failure(RequestGenerationFailed([err]))) - else: - log.err(err, 'Error writing request, but not in valid state ' - 'to finalize request: %s' % self._state) - - _requestDeferred.addCallbacks(cbRequestWrotten, ebRequestWriting) - - return self._finishedRequest - -class _HTTP11ClientFactory(client._HTTP11ClientFactory): - def buildProtocol(self, addr): - return HTTP11ClientProtocol(self._quiescentCallback) - -class HTTPConnectionPool(client.HTTPConnectionPool): - _factory = _HTTP11ClientFactory - -class Agent(client.Agent): - def __init__(self, reactor, - contextFactory=client.WebClientContextFactory(), - connectTimeout=None, bindAddress=None, - pool=None, sockshost=None, socksport=None): - if pool is None: - pool = HTTPConnectionPool(reactor, False) - self._reactor = reactor - self._pool = pool - self._contextFactory = contextFactory - self._connectTimeout = connectTimeout - self._bindAddress = bindAddress - self._sockshost = sockshost - self._socksport = socksport - - def _getEndpoint(self, scheme, host, port): - kwargs = {} - if self._connectTimeout is not None: - kwargs['timeout'] = self._connectTimeout - kwargs['bindAddress'] = self._bindAddress - if scheme == 'http': - return TCP4ClientEndpoint(self._reactor, host, port, **kwargs) - elif scheme == 'shttp': - return SOCKS5ClientEndpoint(self._reactor, self._sockshost, - self._socksport, host, port, **kwargs) - elif scheme == 'httpo': - return SOCKS5ClientEndpoint(self._reactor, self._sockshost, - self._socksport, host, port, **kwargs) - elif scheme == 'https': - return SSL4ClientEndpoint(self._reactor, host, port, - self._wrapContextFactory(host, port), - **kwargs) - else: - raise SchemeNotSupported("Unsupported scheme: %r" % (scheme,)) - - def _requestWithEndpoint(self, key, endpoint, method, parsedURI, - headers, bodyProducer, requestPath): - if headers is None: - headers = Headers() - if not headers.hasHeader('host'): - headers = headers.copy() - headers.addRawHeader( - 'host', self._computeHostValue(parsedURI.scheme, parsedURI.host, - parsedURI.port)) - - d = self._pool.getConnection(key, endpoint) - def cbConnected(proto): - return proto.request( - Request(method, requestPath, headers, bodyProducer, - persistent=self._pool.persistent)) - d.addCallback(cbConnected) - return d diff --git a/ooni/reporter.py b/ooni/reporter.py index 265cd93..8df2264 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -178,7 +178,7 @@ class OONIBTestDetailsLookupFailed(Exception): class OONIBReporter(OReporter): def __init__(self, backend_url): - from ooni.lib.txagentwithsocks import Agent + from ooni.utils.txagentwithsocks import Agent from twisted.internet import reactor try: self.agent = Agent(reactor, sockshost="127.0.0.1", diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index 9bf41e8..46bde28 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -23,7 +23,7 @@ from ooni import config from ooni.utils.net import BodyReceiver, StringProducer, userAgents -from ooni.lib.txagentwithsocks import Agent, SOCKSError +from ooni.utils.txagentwithsocks import Agent, SOCKSError class HTTPTest(NetTestCase): """ diff --git a/ooni/utils/txagentwithsocks.py b/ooni/utils/txagentwithsocks.py new file mode 100644 index 0000000..fc1c17c --- /dev/null +++ b/ooni/utils/txagentwithsocks.py @@ -0,0 +1,254 @@ +# -*- encoding: utf-8 -*- +# +# :authors: Giovanni Pellerano +# :licence: see LICENSE + +import struct + +from zope.interface import implements +from twisted.web import client, _newclient, http_headers +from twisted.web._newclient import Request, RequestNotSent, RequestGenerationFailed, TransportProxyProducer, STATUS +from twisted.internet import protocol +from twisted.internet.protocol import ClientFactory, Protocol +from twisted.internet.endpoints import TCP4ClientEndpoint, SSL4ClientEndpoint, _WrappingProtocol, _WrappingFactory +from twisted.internet import interfaces, defer +from twisted.internet.defer import Deferred, succeed, fail, maybeDeferred + +class SOCKSError(Exception): + def __init__(self, value): + Exception.__init__(self) + self.code = value + +class SOCKSv5ClientProtocol(_WrappingProtocol): + state = 0 + + def __init__(self, connectedDeferred, wrappedProtocol, host, port): + _WrappingProtocol.__init__(self, connectedDeferred, wrappedProtocol) + self._host = host + self._port = port + self.ready = False + + def socks_state_0(self, data): + # error state + self._connectedDeferred.errback(SOCKSError(0x00)) + return + + def socks_state_1(self, data): + if data != "\x05\x00": + self._connectedDeferred.errback(SOCKSError(0x00)) + return + + # Anonymous access allowed - let's issue connect + self.transport.write(struct.pack("!BBBBB", 5, 1, 0, 3, + len(self._host)) + + self._host + + struct.pack("!H", self._port)) + + def socks_state_2(self, data): + if data[:2] != "\x05\x00": + # Anonymous access denied + + errcode = ord(data[1]) + self._connectedDeferred.errback(SOCKSError(errcode)) + + return + + self.ready = True + self._wrappedProtocol.transport = self.transport + self._wrappedProtocol.connectionMade() + + self._connectedDeferred.callback(self._wrappedProtocol) + + def connectionMade(self): + # We implement only Anonymous access + self.transport.write(struct.pack("!BB", 5, len("\x00")) + "\x00") + + self.state = self.state + 1 + + def write(self, data): + if self.ready: + self.transport.write(data) + else: + self.buf.append(data) + + def dataReceived(self, data): + if self.state != 3: + getattr(self, 'socks_state_%s' % (self.state), + self.socks_state_0)(data) + self.state = self.state + 1 + else: + self._wrappedProtocol.dataReceived(data) + +class SOCKSv5ClientFactory(_WrappingFactory): + protocol = SOCKSv5ClientProtocol + + def __init__(self, wrappedFactory, host, port): + _WrappingFactory.__init__(self, wrappedFactory) + self._host, self._port = host, port + + def buildProtocol(self, addr): + try: + proto = self._wrappedFactory.buildProtocol(addr) + except: + self._onConnection.errback() + else: + return self.protocol(self._onConnection, proto, + self._host, self._port) + +class SOCKS5ClientEndpoint(object): + implements(interfaces.IStreamClientEndpoint) + + def __init__(self, reactor, sockshost, socksport, + host, port, timeout=30, bindAddress=None): + + self._reactor = reactor + self._sockshost = sockshost + self._socksport = socksport + self._host = host + self._port = port + self._timeout = timeout + self._bindAddress = bindAddress + + def connect(self, protocolFactory): + try: + wf = SOCKSv5ClientFactory(protocolFactory, self._host, self._port) + self._reactor.connectTCP( + self._sockshost, self._socksport, wf, + timeout=self._timeout, bindAddress=self._bindAddress) + return wf._onConnection + except: + return defer.fail() + +class Headers(http_headers.Headers): + def __init__(self, rawHeaders=None): + self._rawHeaders = dict() + if rawHeaders is not None: + for name, values in rawHeaders.iteritems(): + if type(values) is list: + self.setRawHeaders(name, values[:]) + elif type(values) is dict: + self._rawHeaders[name.lower()] = values + + def setRawHeaders(self, name, values): + if name.lower() not in self._rawHeaders: + self._rawHeaders[name.lower()] = dict() + self._rawHeaders[name.lower()]['name'] = name + self._rawHeaders[name.lower()]['values'] = values + + def getAllRawHeaders(self): + for k, v in self._rawHeaders.iteritems(): + yield v['name'], v['values'] + + def getRawHeaders(self, name, default=None): + if name.lower() in self._rawHeaders: + return self._rawHeaders[name.lower()]["values"] + return default +class HTTPClientParser(_newclient.HTTPClientParser): + def connectionMade(self): + self.headers = Headers() + self.connHeaders = Headers() + self.state = STATUS + self._partialHeader = None + + def headerReceived(self, name, value): + if self.isConnectionControlHeader(name): + headers = self.connHeaders + else: + headers = self.headers + headers.addRawHeader(name, value) + + +class HTTP11ClientProtocol(_newclient.HTTP11ClientProtocol): + def request(self, request): + if self._state != 'QUIESCENT': + return fail(RequestNotSent()) + + self._state = 'TRANSMITTING' + _requestDeferred = maybeDeferred(request.writeTo, self.transport) + self._finishedRequest = Deferred() + + self._currentRequest = request + + self._transportProxy = TransportProxyProducer(self.transport) + self._parser = HTTPClientParser(request, self._finishResponse) + self._parser.makeConnection(self._transportProxy) + self._responseDeferred = self._parser._responseDeferred + + def cbRequestWrotten(ignored): + if self._state == 'TRANSMITTING': + self._state = 'WAITING' + self._responseDeferred.chainDeferred(self._finishedRequest) + + def ebRequestWriting(err): + if self._state == 'TRANSMITTING': + self._state = 'GENERATION_FAILED' + self.transport.loseConnection() + self._finishedRequest.errback( + Failure(RequestGenerationFailed([err]))) + else: + log.err(err, 'Error writing request, but not in valid state ' + 'to finalize request: %s' % self._state) + + _requestDeferred.addCallbacks(cbRequestWrotten, ebRequestWriting) + + return self._finishedRequest + +class _HTTP11ClientFactory(client._HTTP11ClientFactory): + def buildProtocol(self, addr): + return HTTP11ClientProtocol(self._quiescentCallback) + +class HTTPConnectionPool(client.HTTPConnectionPool): + _factory = _HTTP11ClientFactory + +class Agent(client.Agent): + def __init__(self, reactor, + contextFactory=client.WebClientContextFactory(), + connectTimeout=None, bindAddress=None, + pool=None, sockshost=None, socksport=None): + if pool is None: + pool = HTTPConnectionPool(reactor, False) + self._reactor = reactor + self._pool = pool + self._contextFactory = contextFactory + self._connectTimeout = connectTimeout + self._bindAddress = bindAddress + self._sockshost = sockshost + self._socksport = socksport + + def _getEndpoint(self, scheme, host, port): + kwargs = {} + if self._connectTimeout is not None: + kwargs['timeout'] = self._connectTimeout + kwargs['bindAddress'] = self._bindAddress + if scheme == 'http': + return TCP4ClientEndpoint(self._reactor, host, port, **kwargs) + elif scheme == 'shttp': + return SOCKS5ClientEndpoint(self._reactor, self._sockshost, + self._socksport, host, port, **kwargs) + elif scheme == 'httpo': + return SOCKS5ClientEndpoint(self._reactor, self._sockshost, + self._socksport, host, port, **kwargs) + elif scheme == 'https': + return SSL4ClientEndpoint(self._reactor, host, port, + self._wrapContextFactory(host, port), + **kwargs) + else: + raise SchemeNotSupported("Unsupported scheme: %r" % (scheme,)) + + def _requestWithEndpoint(self, key, endpoint, method, parsedURI, + headers, bodyProducer, requestPath): + if headers is None: + headers = Headers() + if not headers.hasHeader('host'): + headers = headers.copy() + headers.addRawHeader( + 'host', self._computeHostValue(parsedURI.scheme, parsedURI.host, + parsedURI.port)) + + d = self._pool.getConnection(key, endpoint) + def cbConnected(proto): + return proto.request( + Request(method, requestPath, headers, bodyProducer, + persistent=self._pool.persistent)) + d.addCallback(cbConnected) + return d

1 0

[ooni-probe/master] Implement very simple HTTP Request backend that does only the part of HTTP we need for testing
by art＠torproject.org 22 Nov '12

22 Nov '12

commit 4522dc005ea3f9c5379d44cf58c3696c62b129a2 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Nov 22 21:05:22 2012 +0100 Implement very simple HTTP Request backend that does only the part of HTTP we need for testing --- oonib/oonibackend.py | 2 +- oonib/testhelpers/http_helpers.py | 95 ++++++++++++++++++++++++++++++------- 2 files changed, 79 insertions(+), 18 deletions(-) diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py index c712bc4..c747bfd 100644 --- a/oonib/oonibackend.py +++ b/oonib/oonibackend.py @@ -68,7 +68,7 @@ if config.helpers.http_return_request.port: print "Starting HTTP return request helper on %s" % config.helpers.http_return_request.port http_return_request_helper = internet.TCPServer( int(config.helpers.http_return_request.port), - http_helpers.HTTPReturnJSONHeadersHelper) + http_helpers.HTTPReturnJSONHeadersHelper()) http_return_request_helper.setServiceParent(serviceCollection) reactor.addSystemEventTrigger('after', 'shutdown', db_threadpool.stop) diff --git a/oonib/testhelpers/http_helpers.py b/oonib/testhelpers/http_helpers.py index 7001fa6..cbd6caa 100644 --- a/oonib/testhelpers/http_helpers.py +++ b/oonib/testhelpers/http_helpers.py @@ -10,11 +10,86 @@ from twisted.web.microdom import escape from cyclone.web import RequestHandler, Application -class HTTPTrapAll(RequestHandler): +from twisted.protocols import policies, basic +from twisted.web.http import Request + +class SimpleHTTPChannel(basic.LineReceiver, policies.TimeoutMixin): """ - Master class to be used to trap all the HTTP methods and make capitalized - requests pass. + This is a simplified version of twisted.web.http.HTTPChannel to overcome + header lowercase normalization. It does not actually implement the HTTP + protocol, but only the subset of it that we need for testing. + + What this HTTP channel currently does is process the HTTP Request Line and + the Request Headers and returns them in a JSON datastructure in the order + we received them. + + The returned JSON dict looks like so: + + { + 'request_headers': + [['User-Agent', 'IE6'], ['Content-Length', 200]] + 'request_line': + 'GET / HTTP/1.1' + } """ + requestFactory = Request + __first_line = 1 + __header = '' + __content = None + + length = 0 + maxHeaders = 500 + requestLine = '' + headers = [] + + timeOut = 60 * 60 * 12 + + def __init__(self): + self.requests = [] + + def connectionMade(self): + self.setTimeout(self.timeOut) + + def lineReceived(self, line): + if self.__first_line: + self.requestLine = line + self.__first_line = 0 + elif line == '': + # We have reached the end of the headers. + if self.__header: + self.headerReceived(self.__header) + self.__header = '' + self.allHeadersReceived() + self.setRawMode() + elif line[0] in ' \t': + # This is to support header field value folding over multiple lines + # as specified by rfc2616. + self.__header = self.__header+'\n'+line + else: + if self.__header: + self.headerReceived(self.__header) + self.__header = line + + def headerReceived(self, line): + header, data = line.split(':', 1) + self.headers.append((header, data)) + + def allHeadersReceived(self): + response = {'request_headers': self.headers, + 'request_line': self.requestLine + } + self.transport.write(json.dumps(response)) + self.transport.loseConnection() + + +class HTTPReturnJSONHeadersHelper(protocol.ServerFactory): + protocol = SimpleHTTPChannel + def buildProtocol(self, addr): + p = self.protocol() + p.headers = [] + return p + +class HTTPTrapAll(RequestHandler): def _execute(self, transforms, *args, **kwargs): self._transforms = transforms defer.maybeDeferred(self.prepare).addCallbacks( @@ -33,20 +108,6 @@ class HTTPTrapAll(RequestHandler): d.addCallbacks(self._execute_success, self._execute_failure) self.notifyFinish().addCallback(self.on_connection_close) -class HTTPReturnJSONHeaders(HTTPTrapAll): - def all(self): - # XXX make sure that the request headers are in the correct order - submitted_data = {'request_body': self.request.body, - 'request_headers': self.request.headers, - 'request_uri': self.request.uri, - 'request_method': self.request.method} - response = json.dumps(submitted_data) - self.write(response) - -HTTPReturnJSONHeadersHelper = Application([ - (r"/*", HTTPReturnJSONHeaders) -]) - class HTTPRandomPage(HTTPTrapAll): """

1 0

[translation/vidalia_alpha_completed] Update translations for vidalia_alpha_completed
by translation＠torproject.org 22 Nov '12

22 Nov '12

commit 3fcb6511c0d15ff89c85459abc55ed1224932740 Author: Translation commit bot <translation(a)torproject.org> Date: Thu Nov 22 19:15:21 2012 +0000 Update translations for vidalia_alpha_completed --- es/vidalia_es.po | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/es/vidalia_es.po b/es/vidalia_es.po index 7dedc12..45fb5a7 100644 --- a/es/vidalia_es.po +++ b/es/vidalia_es.po @@ -9,7 +9,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2012-03-21 17:46+0000\n" -"PO-Revision-Date: 2012-11-22 18:40+0000\n" +"PO-Revision-Date: 2012-11-22 19:03+0000\n" "Last-Translator: strel <strelnic(a)gmail.com>\n" "Language-Team: Spanish (http://www.transifex.com/projects/p/torproject/language/es/)\n" "MIME-Version: 1.0\n" @@ -3566,7 +3566,7 @@ msgctxt "TorControl" msgid "" "Vidalia has not started Tor. You need to stop Tor through the interface you " "started it." -msgstr "Vidalia no ha iniciado Tor. Ha de parar el proceso de Tor a través de la interfaz con la que lo inició." +msgstr "Vidalia no ha iniciado Tor. Ha de detener el proceso de Tor a través de la interfaz con la que lo inició." msgctxt "TorControl" msgid "Start failed: %1" @@ -3586,19 +3586,19 @@ msgstr "Desconectado" msgctxt "TorProcess" msgid "Process %1 failed to stop. [%2]" -msgstr "Proceso %1 no pudo detener [%2]" +msgstr "No se pudo detener el proceso %1 [%2]" msgctxt "TorService" msgid "The Tor service is not installed." -msgstr "El servicio Tor no esta instalado." +msgstr "El servicio Tor no está instalado." msgctxt "TorService" msgid "Unable to start the Tor service." -msgstr "No se puede comenzar el servicio Tor." +msgstr "No se pudo iniciar el servicio Tor." msgctxt "TorSettings" msgid "Failed to hash the control password." -msgstr "Falló al obtener hash de la contraseña de control." +msgstr "Fallo al obtener hash de la contraseña de control." msgctxt "TorrcDialog" msgid "Editing torrc"

1 0

[translation/vidalia_alpha] Update translations for vidalia_alpha
by translation＠torproject.org 22 Nov '12

22 Nov '12

commit a45b348875525a46293673901de8a62c542d5a56 Author: Translation commit bot <translation(a)torproject.org> Date: Thu Nov 22 19:15:18 2012 +0000 Update translations for vidalia_alpha --- es/vidalia_es.po | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/es/vidalia_es.po b/es/vidalia_es.po index 7dedc12..45fb5a7 100644 --- a/es/vidalia_es.po +++ b/es/vidalia_es.po @@ -9,7 +9,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2012-03-21 17:46+0000\n" -"PO-Revision-Date: 2012-11-22 18:40+0000\n" +"PO-Revision-Date: 2012-11-22 19:03+0000\n" "Last-Translator: strel <strelnic(a)gmail.com>\n" "Language-Team: Spanish (http://www.transifex.com/projects/p/torproject/language/es/)\n" "MIME-Version: 1.0\n" @@ -3566,7 +3566,7 @@ msgctxt "TorControl" msgid "" "Vidalia has not started Tor. You need to stop Tor through the interface you " "started it." -msgstr "Vidalia no ha iniciado Tor. Ha de parar el proceso de Tor a través de la interfaz con la que lo inició." +msgstr "Vidalia no ha iniciado Tor. Ha de detener el proceso de Tor a través de la interfaz con la que lo inició." msgctxt "TorControl" msgid "Start failed: %1" @@ -3586,19 +3586,19 @@ msgstr "Desconectado" msgctxt "TorProcess" msgid "Process %1 failed to stop. [%2]" -msgstr "Proceso %1 no pudo detener [%2]" +msgstr "No se pudo detener el proceso %1 [%2]" msgctxt "TorService" msgid "The Tor service is not installed." -msgstr "El servicio Tor no esta instalado." +msgstr "El servicio Tor no está instalado." msgctxt "TorService" msgid "Unable to start the Tor service." -msgstr "No se puede comenzar el servicio Tor." +msgstr "No se pudo iniciar el servicio Tor." msgctxt "TorSettings" msgid "Failed to hash the control password." -msgstr "Falló al obtener hash de la contraseña de control." +msgstr "Fallo al obtener hash de la contraseña de control." msgctxt "TorrcDialog" msgid "Editing torrc"

1 0

[translation/vidalia_alpha_completed] Update translations for vidalia_alpha_completed
by translation＠torproject.org 22 Nov '12

22 Nov '12

commit 0e60b1cd28823e259518f9509e9afac00527150f Author: Translation commit bot <translation(a)torproject.org> Date: Thu Nov 22 18:45:20 2012 +0000 Update translations for vidalia_alpha_completed --- es/vidalia_es.po | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/es/vidalia_es.po b/es/vidalia_es.po index f769fbb..7dedc12 100644 --- a/es/vidalia_es.po +++ b/es/vidalia_es.po @@ -9,7 +9,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2012-03-21 17:46+0000\n" -"PO-Revision-Date: 2012-11-22 17:53+0000\n" +"PO-Revision-Date: 2012-11-22 18:40+0000\n" "Last-Translator: strel <strelnic(a)gmail.com>\n" "Language-Team: Spanish (http://www.transifex.com/projects/p/torproject/language/es/)\n" "MIME-Version: 1.0\n" @@ -3544,7 +3544,7 @@ msgstr "Abrir" msgctxt "Stream" msgid "Failed" -msgstr "Falló" +msgstr "Fallo" msgctxt "Stream" msgid "Closed" @@ -3552,7 +3552,7 @@ msgstr "Cerrado" msgctxt "Stream" msgid "Retrying" -msgstr "Intentando de nuevo" +msgstr "Reintentando" msgctxt "Stream" msgid "Remapped" @@ -3566,7 +3566,7 @@ msgctxt "TorControl" msgid "" "Vidalia has not started Tor. You need to stop Tor through the interface you " "started it." -msgstr "Vidalia no ha iniciado Tor. Tiene que parar el proceso activo Tor a través de la interfaz con la que lo inició." +msgstr "Vidalia no ha iniciado Tor. Ha de parar el proceso de Tor a través de la interfaz con la que lo inició." msgctxt "TorControl" msgid "Start failed: %1"

1 0