May 2016 - tor-commits - lists.torproject.org

[oonib/master] Document configuration (#78)
by art＠torproject.org 30 May '16

30 May '16

commit 0396a55a27e9923537a6cd3d743d37009ef4715b Author: Joe Landers <joe(a)joelanders.net> Date: Thu May 26 12:49:23 2016 +0200 Document configuration (#78) * Update README.rst * remove old configuration option --- README.rst | 17 +++++++++++++++++ oonib.conf.example | 1 - 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 63a242c..a73d776 100644 --- a/README.rst +++ b/README.rst @@ -107,6 +107,23 @@ this would simply be ``'report'``) and ``tor_datadir`` (where you would like the spawned Tor process to keep its data). If you compiled Tor yourself, you'll likely want to specify it for the ``tor_binary`` option. +Configure bouncer and collector endpoints +......................................... + +The bouncer and collector are HTTP applications ("protocols" in twisted terminology) that can be configured to run on top of plain TCP, TLS, or onion service endpoints. +Here is an example of the relevant part of the configuration:: + bouncer_endpoints: + - {type: tls, port: 10443, cert: "private/ssl-key-and-cert.pem"} + - {type: tcp, port: 10080} + - {type: onion, hsdir: "/some/private/bouncer"} + + collector_endpoints: + - {type: tls, port: 11443, cert: "private/ssl-key-and-cert.pem"} + - {type: onion, hsdir: "/some/private/collector"} + +`scripts/gen-ssl-key-cert.sh` in this repo contains the openssl command to generate a self-signed certificate which you can use for the tls endpoint. +txtorcon will use the hostname/private_key from the configured hsdir to start an onion service, or generate a new key if hsdir is empty. + Generate self signed certs for OONIB .................................... If you want to use the HTTPS test helper, you will need to create a diff --git a/oonib.conf.example b/oonib.conf.example index 95df448..1925d81 100644 --- a/oonib.conf.example +++ b/oonib.conf.example @@ -27,7 +27,6 @@ main: debug: false stale_time: 3600 - tor_hidden_service: false tor_datadir: null bouncer_endpoints:

1 0

[oonib/master] [hotfix] Only create bouncer endpoints when it's configured
by art＠torproject.org 30 May '16

30 May '16

commit f03f8055d44d3bbb7cbcfb789946ba67a94e1e52 Author: Arturo Filastò <arturo(a)filasto.net> Date: Thu May 26 13:49:28 2016 +0200 [hotfix] Only create bouncer endpoints when it's configured --- oonib/oonibackend.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py index 33ea92d..601a972 100644 --- a/oonib/oonibackend.py +++ b/oonib/oonibackend.py @@ -163,9 +163,12 @@ if config.main.tor_hidden_service and \ config.main.collector_endpoints = [ {'type': 'onion', 'hsdir': collector_hsdir} ] for endpoint_config in config.main.get('bouncer_endpoints', []): - print "Starting bouncer with config %s" % endpoint_config - endpoint = getEndpoint(endpoint_config) - createService(endpoint, 'bouncer', endpoint_config) + if config.main.bouncer_file: + print "Starting bouncer with config %s" % endpoint_config + endpoint = getEndpoint(endpoint_config) + createService(endpoint, 'bouncer', endpoint_config) + else: + print "No bouncer configured" for endpoint_config in config.main.get('collector_endpoints', []): print "Starting collector with config %s" % endpoint_config

1 0

[oonib/master] Download openssl with wget as well to fix SSL errors on MLAB
by art＠torproject.org 30 May '16

30 May '16

commit 94d34521e9ec7f6a61b1d486dbc53506cb9034cb Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue May 10 19:06:44 2016 +0200 Download openssl with wget as well to fix SSL errors on MLAB --- scripts/build_tor2web_tor.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/build_tor2web_tor.sh b/scripts/build_tor2web_tor.sh index 080a878..c9d7d1b 100755 --- a/scripts/build_tor2web_tor.sh +++ b/scripts/build_tor2web_tor.sh @@ -12,12 +12,12 @@ SCRIPT_ROOT=`pwd` # Package URLS CURL_URLS="\ http://zlib.net/zlib-$ZLIB_VERSION.tar.gz -https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz.asc -https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz https://github.com/downloads/libevent/libevent/libevent-$LIBEVENT_VERSION.t… https://github.com/downloads/libevent/libevent/libevent-$LIBEVENT_VERSION.t…" WGET_URLS="\ +https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz.asc +https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz https://dist.torproject.org/tor-$TOR_VERSION.tar.gz https://dist.torproject.org/tor-$TOR_VERSION.tar.gz.asc"

1 0

[oonib/master] remove echo server i used for debugging
by art＠torproject.org 30 May '16

30 May '16

commit 8e359977af6f92829a27eb45fe4be0a4cda0435a Author: Joe Landers <joe(a)joelanders.net> Date: Fri May 6 12:57:21 2016 +0200 remove echo server i used for debugging --- oonib/oonibackend.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py index 1c6a23a..0b8c4ec 100644 --- a/oonib/oonibackend.py +++ b/oonib/oonibackend.py @@ -135,10 +135,6 @@ def getEndpoint(endpoint_config): else: raise Exception("unknown endpoint type") -class Echo(protocol.Protocol): - def dataReceived(self, data): - self.transport.write('server echoes ' + data) - def createService(endpoint, role, endpoint_config): if role == 'bouncer': factory = ooniBouncer

1 0

[oonib/master] maintain old behaviour
by art＠torproject.org 30 May '16

30 May '16

commit 5fa24c3364c4349995d3bb1ca900dc54e546f5c4 Author: Joe Landers <joe(a)joelanders.net> Date: Fri May 6 13:11:33 2016 +0200 maintain old behaviour --- oonib/oonibackend.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py index 0b8c4ec..1e77d6b 100644 --- a/oonib/oonibackend.py +++ b/oonib/oonibackend.py @@ -154,15 +154,20 @@ if config.main.tor_hidden_service: torconfig = TorConfig() configTor(torconfig) -if config.main.bouncer_endpoints: - for endpoint_config in config.main.bouncer_endpoints: - print "Starting bouncer with config %s" % endpoint_config - endpoint = getEndpoint(endpoint_config) - createService(endpoint, 'bouncer', endpoint_config) - -if config.main.bouncer_endpoints: - for endpoint_config in config.main.collector_endpoints: - print "Starting collector with config %s" % endpoint_config - endpoint = getEndpoint(endpoint_config) - createService(endpoint, 'collector', endpoint_config) +# this is to ensure same behaviour with an old config file +if config.main.bouncer_endpoints is None: + config.main.bouncer_endpoints = [ {'type': 'onion', 'hsdir': 'bouncer'} ] + +if config.main.collector_endpoints is None: + config.main.collector_endpoints = [ {'type': 'onion', 'hsdir': 'collector'} ] + +for endpoint_config in config.main.bouncer_endpoints: + print "Starting bouncer with config %s" % endpoint_config + endpoint = getEndpoint(endpoint_config) + createService(endpoint, 'bouncer', endpoint_config) + +for endpoint_config in config.main.collector_endpoints: + print "Starting collector with config %s" % endpoint_config + endpoint = getEndpoint(endpoint_config) + createService(endpoint, 'collector', endpoint_config)

1 0

[oonib/master] easily swappable {TCP, TLS, onion} endpoints
by art＠torproject.org 30 May '16

30 May '16

commit 8ef4dc35b303d0b1e7d3e5ee0cc8a0f3e2eac240 Author: Joe Landers <joe(a)joelanders.net> Date: Wed May 4 22:45:37 2016 +0200 easily swappable {TCP, TLS, onion} endpoints --- gen-ssl-key-cert.sh | 13 ++++++ oonib.conf.example | 13 +++++- oonib/onion.py | 10 +---- oonib/oonibackend.py | 125 ++++++++++++++++++++++++++++----------------------- 4 files changed, 93 insertions(+), 68 deletions(-) diff --git a/gen-ssl-key-cert.sh b/gen-ssl-key-cert.sh new file mode 100755 index 0000000..2a50766 --- /dev/null +++ b/gen-ssl-key-cert.sh @@ -0,0 +1,13 @@ +#!/bin/sh +set -eu + +if [ ! -d private ] +then + mkdir private +fi + +openssl req -x509 -newkey rsa:2048 \ + -keyout private/ssl-key.pem -out private/ssl-cert.pem \ + -days 400 -nodes -subj '/CN=selfie' + +cat private/ssl-key.pem private/ssh-cert.pem > private/ssl-key-and-cert.pem diff --git a/oonib.conf.example b/oonib.conf.example index 7821dc1..0a5f4a5 100644 --- a/oonib.conf.example +++ b/oonib.conf.example @@ -7,7 +7,6 @@ main: bouncer_file: data/bouncer.yaml logfile: null - tor_datadir: null database_uri: 'sqlite://oonib_test_db.db' db_threadpool_size: 10 tor_binary: null @@ -28,7 +27,17 @@ main: debug: false stale_time: 3600 - tor_hidden_service: true + tor_hidden_service: false + tor_datadir: test_datadir + + bouncer_endpoints: + - {type: tls, port: 10443, cert: "private/ssl-key-and-cert.pem"} + - {type: tcp, port: 10080} + - {type: onion, hsdir: bouncer} + + collector_endpoints: + - {type: tls, port: 11443, cert: "private/ssl-key-and-cert.pem"} + report_file_template: '{iso8601_timestamp}-{test_name}-{report_id}-{probe_asn}-{probe_cc}-probe-0.2.0.{ext}' helpers: http-return-json-headers: diff --git a/oonib/onion.py b/oonib/onion.py index c06dc6b..262e672 100644 --- a/oonib/onion.py +++ b/oonib/onion.py @@ -53,7 +53,7 @@ def txSetupFailed(failure): log.err("Setup failed") log.exception(failure) -def startTor(torconfig): +def configTor(torconfig): def updates(prog, tag, summary): print("%d%%: %s" % (prog, summary)) @@ -89,11 +89,3 @@ def startTor(torconfig): config.main.socks_port = socks_port torconfig.save() - - if config.main.tor_binary is not None: - d = launch_tor(torconfig, reactor, - tor_binary=config.main.tor_binary, - progress_updates=updates) - else: - d = launch_tor(torconfig, reactor, progress_updates=updates) - return d diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py index a392b82..1c6a23a 100644 --- a/oonib/oonibackend.py +++ b/oonib/oonibackend.py @@ -11,14 +11,14 @@ from distutils.version import LooseVersion from oonib.api import ooniBackend, ooniBouncer from oonib.config import config -from oonib.onion import startTor +from oonib.onion import configTor from oonib.testhelpers import dns_helpers, ssl_helpers from oonib.testhelpers import http_helpers, tcp_helpers import os from twisted.application import internet, service -from twisted.internet import reactor +from twisted.internet import reactor, endpoints, defer, ssl, protocol from twisted.names import dns from txtorcon import TCPHiddenServiceEndpoint, TorConfig @@ -102,60 +102,71 @@ if config.helpers['http-return-json-headers'].port: multiService.addService(http_return_request_helper) http_return_request_helper.startService() -# add the tor collector service here +def getHSEndpoint(endpoint_config): + hsdir = os.path.join(torconfig.DataDirectory, endpoint_config['hsdir']) + if LooseVersion(txtorcon_version) >= LooseVersion('0.10.0'): + return TCPHiddenServiceEndpoint.global_tor(reactor, + 80, + hidden_service_dir=hsdir) + else: + return TCPHiddenServiceEndpoint.global_tor(reactor, + 80, + data_dir=hsdir) + +def getTCPEndpoint(endpoint_config): + return endpoints.TCP4ServerEndpoint(reactor, endpoint_config['port']) + +def getTLSEndpoint(endpoint_config): + with open(endpoint_config['cert'], 'r') as f: + cert_data = f.read() + certificate = ssl.PrivateCertificate.loadPEM(cert_data) + print certificate.inspect() + return endpoints.SSL4ServerEndpoint(reactor, + endpoint_config['port'], + certificate.options()) + +def getEndpoint(endpoint_config): + if endpoint_config['type'] == 'onion': + return getHSEndpoint(endpoint_config) + elif endpoint_config['type'] == 'tcp': + return getTCPEndpoint(endpoint_config) + elif endpoint_config['type'] == 'tls': + return getTLSEndpoint(endpoint_config) + else: + raise Exception("unknown endpoint type") + +class Echo(protocol.Protocol): + def dataReceived(self, data): + self.transport.write('server echoes ' + data) + +def createService(endpoint, role, endpoint_config): + if role == 'bouncer': + factory = ooniBouncer + elif role == 'collector': + factory = ooniBackend + else: + raise Exception("unknown service type") + + service = internet.StreamServerEndpointService( + endpoint, factory + ) + service.setName("-".join([endpoint_config['type'], role])) + multiService.addService(service) + service.startService() + if config.main.tor_hidden_service: torconfig = TorConfig() - d = startTor(torconfig) - - def getHSEndpoint(data_dir): - if LooseVersion(txtorcon_version) >= LooseVersion('0.10.0'): - return TCPHiddenServiceEndpoint(reactor, - torconfig, - 80, - hidden_service_dir=data_dir) - else: - return TCPHiddenServiceEndpoint(reactor, - torconfig, - 80, - data_dir=data_dir) - - def printOnionEndpoint(endpointService): - print ("Exposed %s Tor hidden service on httpo://%s" % - (endpointService.name, endpointService.endpoint.onion_uri)) - - def addCollector(torControlProtocol): - data_dir = os.path.join(torconfig.DataDirectory, 'collector') - collector_service = internet.StreamServerEndpointService( - getHSEndpoint(data_dir), ooniBackend - ) - collector_service.setName('collector') - multiService.addService(collector_service) - collector_service.startService() - return collector_service - - d.addCallback(addCollector) - d.addCallback(printOnionEndpoint) - - if ooniBouncer: - def addBouncer(torControlProtocol): - data_dir = os.path.join(torconfig.DataDirectory, 'bouncer') - bouncer_service = internet.StreamServerEndpointService( - getHSEndpoint(data_dir), ooniBouncer - ) - bouncer_service.setName('bouncer') - multiService.addService(bouncer_service) - bouncer_service.startService() - return bouncer_service - - d.addCallback(addBouncer) - d.addCallback(printOnionEndpoint) -else: - if ooniBouncer: - bouncer_service = internet.TCPServer(8888, ooniBouncer, - interface="127.0.0.1") - multiService.addService(bouncer_service) - bouncer_service.startService() - collector_service = internet.TCPServer(8889, ooniBackend, - interface="127.0.0.1") - multiService.addService(collector_service) - collector_service.startService() + configTor(torconfig) + +if config.main.bouncer_endpoints: + for endpoint_config in config.main.bouncer_endpoints: + print "Starting bouncer with config %s" % endpoint_config + endpoint = getEndpoint(endpoint_config) + createService(endpoint, 'bouncer', endpoint_config) + +if config.main.bouncer_endpoints: + for endpoint_config in config.main.collector_endpoints: + print "Starting collector with config %s" % endpoint_config + endpoint = getEndpoint(endpoint_config) + createService(endpoint, 'collector', endpoint_config) +

1 0

[oonib/master] remove another print from debugging
by art＠torproject.org 30 May '16

30 May '16

commit 2fa28488fc6dea793da147f9f1bd7a87961ec7a7 Author: Joe Landers <joe(a)joelanders.net> Date: Fri May 6 13:21:31 2016 +0200 remove another print from debugging --- oonib/oonibackend.py | 1 - 1 file changed, 1 deletion(-) diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py index 1e77d6b..9c9a0fa 100644 --- a/oonib/oonibackend.py +++ b/oonib/oonibackend.py @@ -120,7 +120,6 @@ def getTLSEndpoint(endpoint_config): with open(endpoint_config['cert'], 'r') as f: cert_data = f.read() certificate = ssl.PrivateCertificate.loadPEM(cert_data) - print certificate.inspect() return endpoints.SSL4ServerEndpoint(reactor, endpoint_config['port'], certificate.options())

1 0

[oonib/master] Merge branch 'feature/https_endpoint' of github.com:TheTorProject/ooni-backend into feature/https_endpoint
by art＠torproject.org 30 May '16

30 May '16

commit 4141c6a1485a32a7c0e43161201c77150b665a3c Merge: dda3f4d 35ccea7 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri May 13 12:52:28 2016 +0200 Merge branch 'feature/https_endpoint' of github.com:TheTorProject/ooni-backend into feature/https_endpoint * 'feature/https_endpoint' of github.com:TheTorProject/ooni-backend: only assume HS endpoint when tor_hidden_service: true explain + bail out when inconsistent config oonib/oonibackend.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)

1 0

[oonib/master] explain + bail out when inconsistent config
by art＠torproject.org 30 May '16

30 May '16

commit 3bcce2080b6e986c992c9cad9dc4b8e25bebaa10 Author: Joe Landers <joe(a)joelanders.net> Date: Fri May 13 09:32:17 2016 +0200 explain + bail out when inconsistent config --- oonib/oonibackend.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py index 9c9a0fa..3802566 100644 --- a/oonib/oonibackend.py +++ b/oonib/oonibackend.py @@ -103,6 +103,8 @@ if config.helpers['http-return-json-headers'].port: http_return_request_helper.startService() def getHSEndpoint(endpoint_config): + if torconfig is None: + raise Exception("you probably need to set tor_hidden_service: true") hsdir = os.path.join(torconfig.DataDirectory, endpoint_config['hsdir']) if LooseVersion(txtorcon_version) >= LooseVersion('0.10.0'): return TCPHiddenServiceEndpoint.global_tor(reactor, @@ -149,6 +151,7 @@ def createService(endpoint, role, endpoint_config): multiService.addService(service) service.startService() +torconfig = None if config.main.tor_hidden_service: torconfig = TorConfig() configTor(torconfig)

1 0

[oonib/master] Remove unused imports
by art＠torproject.org 30 May '16

30 May '16

commit 0bd12a10549d214fff93c4b11e14b5d62bafc0ea Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri May 13 13:18:05 2016 +0200 Remove unused imports --- oonib/oonibackend.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py index eaf235f..5a994e8 100644 --- a/oonib/oonibackend.py +++ b/oonib/oonibackend.py @@ -18,7 +18,7 @@ from oonib.testhelpers import http_helpers, tcp_helpers import os from twisted.application import internet, service -from twisted.internet import reactor, endpoints, defer, ssl, protocol +from twisted.internet import reactor, endpoints, ssl from twisted.names import dns from txtorcon import TCPHiddenServiceEndpoint, TorConfig

1 0