November 2012 - tor-commits - lists.torproject.org

[ooni-probe/master] Tells pip to install scapy from the website
by art＠torproject.org 16 Nov '12

16 Nov '12

commit 8d09d4fb3b926a834029729031c44576328d17dd Author: Flavio Amieiro <amieiro.flavio(a)gmail.com> Date: Thu Nov 15 14:33:53 2012 -0200 Tells pip to install scapy from the website --- oonib/requirements.txt | 2 +- requirements.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/oonib/requirements.txt b/oonib/requirements.txt index f9d7541..c94ae08 100644 --- a/oonib/requirements.txt +++ b/oonib/requirements.txt @@ -14,7 +14,7 @@ numpy==1.6.2 pyOpenSSL==0.13 pygeoip==0.2.5 repoze.sphinx.autointerface==0.7.1 -scapy==2.1.0 +https://hg.secdev.org/scapy/archive/tip.zip#egg=scapy storm==0.19 transaction==1.3.0 txtorcon==0.7 diff --git a/requirements.txt b/requirements.txt index e9a7bfc..33ecf44 100644 --- a/requirements.txt +++ b/requirements.txt @@ -11,7 +11,7 @@ ipaddr>=2.1.10 pyOpenSSL>=0.13 pygeoip>=0.2.5 repoze.sphinx.autointerface>=0.7.1 -scapy>=2.1.0 +https://hg.secdev.org/scapy/archive/tip.zip#egg=scapy storm>=0.19 transaction>=1.3.0 txtorcon>=0.6

1 0

[ooni-probe/master] Use the latest txtorcon in pypi
by art＠torproject.org 16 Nov '12

16 Nov '12

commit 4716166dbf9b5e62e3c14576577b0e402f73bbb9 Author: Flavio Amieiro <amieiro.flavio(a)gmail.com> Date: Thu Nov 15 14:35:29 2012 -0200 Use the latest txtorcon in pypi Currently, txtorcon in pypi is 0.6, oonib/requirements.txt required 0.7. --- oonib/requirements.txt | 2 +- requirements.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/oonib/requirements.txt b/oonib/requirements.txt index c94ae08..e84e681 100644 --- a/oonib/requirements.txt +++ b/oonib/requirements.txt @@ -17,7 +17,7 @@ repoze.sphinx.autointerface==0.7.1 https://hg.secdev.org/scapy/archive/tip.zip#egg=scapy storm==0.19 transaction==1.3.0 -txtorcon==0.7 +txtorcon wsgiref==0.1.2 zope.component==4.0.0 zope.event==4.0.0 diff --git a/requirements.txt b/requirements.txt index 33ecf44..0848cca 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,7 +14,7 @@ repoze.sphinx.autointerface>=0.7.1 https://hg.secdev.org/scapy/archive/tip.zip#egg=scapy storm>=0.19 transaction>=1.3.0 -txtorcon>=0.6 +txtorcon wsgiref>=0.1.2 zope.component>=4.0.0 zope.event>=4.0.0

1 0

[ooni-probe/master] Merge remote-tracking branch 'flavioamiero/fix-requirements' into requirements
by art＠torproject.org 16 Nov '12

16 Nov '12

commit e670b5b218506ebce7d157248bde9f91f392aa8c Merge: a48ca0f 4716166 Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Nov 16 10:26:52 2012 +0100 Merge remote-tracking branch 'flavioamiero/fix-requirements' into requirements * flavioamiero/fix-requirements: Use the latest txtorcon in pypi Tells pip to install scapy from the website Conflicts: oonib/requirements.txt oonib/requirements.txt | 2 ++ requirements.txt | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-)

1 0

[ooni-probe/master] Merge branch 'master' of ssh://git-rw.torproject.org/ooni-probe
by art＠torproject.org 16 Nov '12

16 Nov '12

commit a48ca0f2cd8a415787ae0145ad716b3e107b92e3 Merge: d482b29 7affab1 Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Nov 16 10:14:11 2012 +0100 Merge branch 'master' of ssh://git-rw.torproject.org/ooni-probe * 'master' of ssh://git-rw.torproject.org/ooni-probe: * Adding back in a few changes that got lost in an octopus battle. * Updated oonib requirements. * Rewrite of echo. * Minor change to Makefile. * Moved iface and ipaddr discovery function to ooni.utils.net. * Updated oonib requirements. * Rewrite of echo. * Minor change to Makefile. * Moved iface and ipaddr discovery function to ooni.utils.net. Makefile | 28 +++-- nettests/bridge_reachability/echo.py | 198 +++++++++++++------------------- ooni/utils/net.py | 206 +++++++++++++++++++++++++++++---- oonib/requirements.txt | 2 - 4 files changed, 278 insertions(+), 156 deletions(-)

1 0

[ooni-probe/master] implemented a SOCKv5 endpoint using the same _WrappingFactory used for TCP and SSL.
by art＠torproject.org 16 Nov '12

16 Nov '12

commit 4fad239939f250cdbd192e842ebcee3d3cbbebee Author: Giovanni `evilaliv3` Pellerano <giovanni.pellerano(a)evilaliv3.org> Date: Thu Nov 15 23:58:05 2012 +0100 implemented a SOCKv5 endpoint using the same _WrappingFactory used for TCP and SSL. Twisted Agent class extended to use SOCKv5 andpoint to permit HTTP connections throught Tor (shttp:// standing for socksified http) --- ooni/lib/txagentwithsocks.py | 203 ++++++++++++++++++++++++++++++++++++++++++ ooni/templates/httpt.py | 15 ++-- 2 files changed, 212 insertions(+), 6 deletions(-) diff --git a/ooni/lib/txagentwithsocks.py b/ooni/lib/txagentwithsocks.py new file mode 100644 index 0000000..fecd9fc --- /dev/null +++ b/ooni/lib/txagentwithsocks.py @@ -0,0 +1,203 @@ +# -*- encoding: utf-8 -*- +# +# :authors: Giovanni Pellerano +# :licence: see LICENSE + +import struct + +from zope.interface import implements +from twisted.web import client +from twisted.internet.protocol import ClientFactory, Protocol +from twisted.internet.endpoints import TCP4ClientEndpoint, SSL4ClientEndpoint, _WrappingProtocol, _WrappingFactory +from twisted.internet import interfaces, defer + +class SOCKSError(Exception): + def __init__(self, value): + Exception.__init__(self) + self.code = value + +class SOCKSv5ClientProtocol(_WrappingProtocol): + state = 0 + + def __init__(self, connectedDeferred, wrappedProtocol, host, port): + _WrappingProtocol.__init__(self, connectedDeferred, wrappedProtocol) + self._host = host + self._port = port + self.ready = False + self.buf = [] + + def socks_state_0(self, data): + # error state + self._connectedDeferred.errback(SOCKSError(0x00)) + return + + def socks_state_1(self, data): + if data != "\x05\x00": + self._connectedDeferred.errback(SOCKSError(0x00)) + return + + # Anonymous access allowed - let's issue connect + self.transport.write(struct.pack("!BBBBB", 5, 1, 0, 3, + len(self._host)) + + self._host + + struct.pack("!H", self._port)) + + def socks_state_2(self, data): + if data[:2] != "\x05\x00": + # Anonymous access denied + + errcode = ord(data[1]) + self._connectedDeferred.errback(SOCKSError(errcode)) + + return + + self.ready = True + self._wrappedProtocol.transport = self.transport + self._wrappedProtocol.connectionMade() + + if self.buf != []: + self.transport.write(''.join(self.buf)) + self.buf = [] + + self._connectedDeferred.callback(self._wrappedProtocol) + + def connectionMade(self): + # We implement only Anonymous access + self.transport.write(struct.pack("!BB", 5, len("\x00")) + "\x00") + + self.state = self.state + 1 + + def connectionLost(self, reason): + pass + + def write(self, data): + if self.ready: + self.transport.write(data) + else: + self.buf.append(data) + + def dataReceived(self, data): + if self.state != 3: + getattr(self, 'socks_state_%s' % (self.state), + self.socks_state_0)(data) + self.state = self.state + 1 + else: + self._wrappedProtocol.dataReceived(data) + +class SOCKSv5ClientFactory(_WrappingFactory): + protocol = SOCKSv5ClientProtocol + + def __init__(self, wrappedFactory, host, port): + _WrappingFactory.__init__(self, wrappedFactory) + self._host, self._port = host, port + + def buildProtocol(self, addr): + """ + Proxy C{buildProtocol} to our C{self._wrappedFactory} or errback + the C{self._onConnection} L{Deferred}. + + @return: An instance of L{_WrappingProtocol} or C{None} + """'' + try: + proto = self._wrappedFactory.buildProtocol(addr) + except: + self._onConnection.errback() + else: + print self._host + return self.protocol(self._onConnection, proto, + self._host, self._port) + +class SOCKS5ClientEndpoint(object): + """ + TCP client endpoint with an IPv4 configuration. + """ + implements(interfaces.IStreamClientEndpoint) + + def __init__(self, reactor, sockhost, sockport, + host, port, timeout=30,bindAddress=None): + """ + @param reactor: An L{IReactorTCP} provider + + @param host: A hostname, used when connecting + @type host: str + + @param port: The port number, used when connecting + @type port: int + + @param timeout: The number of seconds to wait before assuming the + connection has failed. + @type timeout: int + + @param bindAddress: A (host, port) tuple of local address to bind to, + or None. + @type bindAddress: tuple + """ + self._reactor = reactor + self._sockhost = sockhost + self._sockport = sockport + self._host = host + self._port = port + self._timeout = timeout + self._bindAddress = bindAddress + + def connect(self, protocolFactory): + """ + Implement L{IStreamClientEndpoint.connect} to connect via TCP. + """ + try: + wf = SOCKSv5ClientFactory(protocolFactory, self._host, self._port) + self._reactor.connectTCP( + self._sockhost, self._sockport, wf, + timeout=self._timeout, bindAddress=self._bindAddress) + return wf._onConnection + except: + return defer.fail() + +class Agent(client.Agent): + def __init__(self, reactor, + contextFactory=client.WebClientContextFactory(), + connectTimeout=None, bindAddress=None, + pool=None, sockhost=None, sockport=None): + if pool is None: + pool = client.HTTPConnectionPool(reactor, False) + self._reactor = reactor + self._pool = pool + self._contextFactory = contextFactory + self._connectTimeout = connectTimeout + self._bindAddress = bindAddress + self._sockhost = sockhost + self._sockport = sockport + + def _getEndpoint(self, scheme, host, port): + """ + Get an endpoint for the given host and port, using a transport + selected based on scheme. + + @param scheme: A string like C{'http'} or C{'https'} (the only two + supported values) to use to determine how to establish the + connection. + + @param host: A C{str} giving the hostname which will be connected to in + order to issue a request. + + @param port: An C{int} giving the port number the connection will be + on. + + @return: An endpoint which can be used to connect to given address. + """ + kwargs = {} + if self._connectTimeout is not None: + kwargs['timeout'] = self._connectTimeout + kwargs['bindAddress'] = self._bindAddress + if scheme == 'http': + return TCP4ClientEndpoint(self._reactor, host, port, **kwargs) + elif scheme == 'shttp': + return SOCKS5ClientEndpoint(self._reactor, self._sockhost, + self._sockport, host, port, **kwargs) + elif scheme == 'https': + return SSL4ClientEndpoint(self._reactor, host, port, + self._wrapContextFactory(host, port), + **kwargs) + else: + raise SchemeNotSupported("Unsupported scheme: %r" % (scheme,)) + diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index ee6af90..011ff5b 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -4,15 +4,13 @@ # :licence: see LICENSE import copy import random - -from zope.interface import implements +import struct from twisted.python import usage from twisted.plugin import IPlugin from twisted.internet import protocol, defer from twisted.internet.ssl import ClientContextFactory -from twisted.web.client import Agent from twisted.internet import reactor from twisted.internet.error import ConnectionRefusedError @@ -23,6 +21,8 @@ from ooni.utils import log from ooni.utils.net import BodyReceiver, StringProducer, userAgents +from ooni.lib.txagentwithsocks import Agent, SOCKSError + class HTTPTest(NetTestCase): """ A utility class for dealing with HTTP based testing. It provides methods to @@ -47,7 +47,7 @@ class HTTPTest(NetTestCase): log.err("Warning! pyOpenSSL is not installed. https websites will" "not work") - self.agent = Agent(reactor) + self.agent = Agent(reactor, sockhost="127.0.0.1", sockport=9050) if self.followRedirects: try: @@ -131,8 +131,11 @@ class HTTPTest(NetTestCase): d = self.build_request(url, method, headers, body) def errback(failure): - failure.trap(ConnectionRefusedError) - log.err("Connection refused. The backend may be down") + failure.trap(ConnectionRefusedError, SOCKSError) + if type(failure.value) is ConnectionRefusedError: + log.err("Connection refused. The backend may be down") + else: + log.err("Sock error. The SOCK proxy may be down") self.report["failure"] = str(failure.value) def finished(data):

1 0

[ooni-probe/master] Add documentation on usage of the SOCKS5 Tor socks client
by art＠torproject.org 16 Nov '12

16 Nov '12

commit 502f2375633bd04166057ce4660562ac2f677365 Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Nov 16 10:11:33 2012 +0100 Add documentation on usage of the SOCKS5 Tor socks client --- ooni/templates/httpt.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index 011ff5b..b5849d3 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -31,6 +31,10 @@ class HTTPTest(NetTestCase): processResponseHeader that are invoked once the headers have been received and once the request body has been received. + To perform requests over Tor you will have to use the special URL schema + "shttp". For example to request / on example.com you will have to do + specify as URL "shttp://example.com/". + XXX all of this requires some refactoring. """ name = "HTTP Test"

1 0

[ooni-probe/master] Parametrize tor socksport for usage with modified HTTP Agent
by art＠torproject.org 16 Nov '12

16 Nov '12

commit d482b29e85a157419fce4d728d74a7b4b6c017be Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Nov 16 10:13:16 2012 +0100 Parametrize tor socksport for usage with modified HTTP Agent --- ooni/templates/httpt.py | 6 +++++- ooniprobe.conf | 1 + 2 files changed, 6 insertions(+), 1 deletions(-) diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py index b5849d3..3a4201c 100644 --- a/ooni/templates/httpt.py +++ b/ooni/templates/httpt.py @@ -16,8 +16,10 @@ from twisted.internet.error import ConnectionRefusedError from twisted.web._newclient import Request from twisted.web.http_headers import Headers + from ooni.nettest import NetTestCase from ooni.utils import log +from ooni import config from ooni.utils.net import BodyReceiver, StringProducer, userAgents @@ -51,7 +53,9 @@ class HTTPTest(NetTestCase): log.err("Warning! pyOpenSSL is not installed. https websites will" "not work") - self.agent = Agent(reactor, sockhost="127.0.0.1", sockport=9050) + self.agent = Agent(reactor, + sockhost="127.0.0.1", + sockport=config.advanced.tor_socksport) if self.followRedirects: try: diff --git a/ooniprobe.conf b/ooniprobe.conf index 191d809..2e5992a 100644 --- a/ooniprobe.conf +++ b/ooniprobe.conf @@ -24,4 +24,5 @@ advanced: geoip_data_dir: /home/x/code/networking/ooni-probe/data/ debug: true threadpool_size: 10 + tor_socksport: 9050

1 0

[torbutton/master] Bug 6156: Raise check limit to 1.5 hours.
by mikeperry＠torproject.org 16 Nov '12

16 Nov '12

commit ad92f4ab86768493593aef53885d3ae81864b899 Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Thu Nov 15 19:56:08 2012 -0800 Bug 6156: Raise check limit to 1.5 hours. In retrospect, 15 minutes is probably a little too often. --- src/chrome/content/torbutton.js | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/chrome/content/torbutton.js b/src/chrome/content/torbutton.js index e305e4e..93d7968 100644 --- a/src/chrome/content/torbutton.js +++ b/src/chrome/content/torbutton.js @@ -970,7 +970,7 @@ function torbutton_do_async_versioncheck() { // Suppress update check if done recently. const kLastCheckPref = "extension.torbutton.lastUpdateCheck"; - const kMinSecsBetweenChecks = 15 * 60; // 15 minutes + const kMinSecsBetweenChecks = 90 * 60; // 1.5 hours var now = Date.now() / 1000; var lastCheckTime; try {

1 0

[torbutton/master] Bug 6096: Perform version check every time there's a new tab.
by mikeperry＠torproject.org 16 Nov '12

16 Nov '12

commit 1f5c7cceac380503605ad56e231824ef5c78c560 Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Thu Nov 15 22:12:36 2012 -0800 Bug 6096: Perform version check every time there's a new tab. This is limited by the timer from #6156. --- src/chrome/content/torbutton.js | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/src/chrome/content/torbutton.js b/src/chrome/content/torbutton.js index 93d7968..75fe7ab 100644 --- a/src/chrome/content/torbutton.js +++ b/src/chrome/content/torbutton.js @@ -4269,6 +4269,11 @@ function torbutton_new_tab(event) //var browser = event.currentTarget; var browser = gBrowser.getBrowserForTab(event.target); + /* Perform the version check on new tab, module timer */ + if (!tor_tag) { // tor is enabled... + torbutton_do_async_versioncheck(); + } + torbutton_tag_new_browser(browser, tor_tag, no_plugins); // XXX: This is possibly slightly the wrong place to do this check,

1 0

[tor/master] Give useful warning when both IPv4 and IPv6 are disabled on a socksport
by nickm＠torproject.org 16 Nov '12

16 Nov '12

commit ecb619d96bf22150a926c233b3ddc5d82b3dc032 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Nov 15 22:58:54 2012 -0500 Give useful warning when both IPv4 and IPv6 are disabled on a socksport --- src/or/config.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/src/or/config.c b/src/or/config.c index f7a803f..206ccc8 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -4777,6 +4777,12 @@ parse_port_config(smartlist_t *out, else got_zero_port = 1; + if (ipv4_traffic == 0 && ipv6_traffic == 0) { + log_warn(LD_CONFIG, "You have a %sPort entry with both IPv4 and " + "IPv6 disabled; that won't work.", portname); + goto err; + } + if (out && port) { port_cfg_t *cfg = tor_malloc_zero(sizeof(port_cfg_t)); tor_addr_copy(&cfg->addr, &addr);

1 0