tor-commits September 2013

tor-commits@lists.torproject.org

17 participants
1101 discussions

r26348: {website} update the blog feed. (website/trunk/en)
by Andrew Lewman 11 Sep '13

11 Sep '13

Author: phobos Date: 2013-09-11 11:06:56 +0000 (Wed, 11 Sep 2013) New Revision: 26348 Modified: website/trunk/en/index.wml Log: update the blog feed. Modified: website/trunk/en/index.wml =================================================================== --- website/trunk/en/index.wml 2013-09-10 16:32:47 UTC (rev 26347) +++ website/trunk/en/index.wml 2013-09-11 11:06:56 UTC (rev 26348) @@ -157,26 +157,26 @@ <table> <tr> <td> - <div class="calendar"><span class="month">Sep</span><br><span class="day">05</span></div> - <p>How to handle millions of new Tor clients. Read the <a href="https://blog.torproject.org/blog/how-to-handle-millions-new-tor-clients">details</a>.</p> + <div class="calendar"><span class="month">Sep</span><br><span class="day">11</span></div> + <p>Tor, NSA, and QUICK ANT Speculation. Read <a href="https://blog.torproject.org/blog/tor-nsa-and-quick-ant-speculation">our thoughts</a>.</p> </td> </tr> <tr> <td> - <div class="calendar"><span class="month">Sep</span><br><span class="day">04</span></div> - <p><a href="https://blog.torproject.org/blog/tor-weekly-news-%E2%80%94-september-4th-20…">Tor Weekly News</a> as of 04 September.</p> + <div class="calendar"><span class="month">Sep</span><br><span class="day">11</span></div> + <p>Learn about the <a href="https://blog.torproject.org/blog/lifecycle-of-a-new-relay">lifecycle of a new relay</a>.</p> </td> </tr> <tr> <td> - <div class="calendar"><span class="month">Aug</span><br><span class="day">20</span></div> - <p>Deterministic Builds Part One: Cyberwar and Global Compromise. Read the <a href="https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and…">blog post</a>.</p> + <div class="calendar"><span class="month">Sep</span><br><span class="day">11</span></div> + <p>New Tor 0.2.4.17-rc packages. Learn <a href="https://blog.torproject.org/blog/new-tor-02417-rc-packages">what's new</a>.</p> </td> </tr> <tr> <td> - <div class="calendar"><span class="month">Aug</span><br><span class="day">19</span></div> - <p>Transparency, openness, and our 2012 financial docs. <a href="https://blog.torproject.org/blog/transparency-openness-and-our-2012-financi…">Read more</a>.</p> + <div class="calendar"><span class="month">Sep</span><br><span class="day">05</span></div> + <p>How to handle millions of new Tor clients. Read the <a href="https://blog.torproject.org/blog/how-to-handle-millions-new-tor-clients">details</a>.</p> </td> </tr> </table>

1 0

[oonib/master] Parametrise the DNS test helper remote resolver.
by art＠torproject.org 11 Sep '13

11 Sep '13

commit 639e23d7341c4a12a3cf5b901d1479e2f4fee38b Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Sep 10 19:52:33 2013 +0200 Parametrise the DNS test helper remote resolver. --- oonib.conf.example | 1 + oonib/testhelpers/dns_helpers.py | 11 ++++++++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/oonib.conf.example b/oonib.conf.example index 323b2af..c25d405 100644 --- a/oonib.conf.example +++ b/oonib.conf.example @@ -50,6 +50,7 @@ helpers: address: null udp_port: 57004 tcp_port: 57005 + resolver_address: '8.8.8.8:53' ssl: address: null diff --git a/oonib/testhelpers/dns_helpers.py b/oonib/testhelpers/dns_helpers.py index dc9c772..8959215 100644 --- a/oonib/testhelpers/dns_helpers.py +++ b/oonib/testhelpers/dns_helpers.py @@ -3,11 +3,20 @@ from twisted.internet import reactor from twisted.names import dns from twisted.names import client, server +from oonib import config + class DNSTestHelper(server.DNSServerFactory): def __init__(self, authorities = None, caches = None, clients = None, verbose = 0): - resolver = client.Resolver(servers=[('8.8.8.8', 53)]) + try: + host, port = config.helpers.dns.split(':') + port = int(port) + # XXX remove this when we have configuration file versioning. + # https://github.com/TheTorProject/ooni-probe/issues/190 + except: + host, port = '8.8.8.8', 53 + resolver = client.Resolver(servers=[(host, port)]) server.DNSServerFactory.__init__(self, authorities = authorities, caches = caches, clients = [resolver], verbose = verbose)

1 0

[oonib/master] Merge pull request #16 from TheTorProject/fix/simplify_filterHelperAddresses
by art＠torproject.org 11 Sep '13

11 Sep '13

commit 1dd52502d756ca560982737a62a40b2a460f16e1 Merge: fb49373 bacb8cc Author: Arturo Filastò <arturo(a)filasto.net> Date: Tue Sep 10 08:28:45 2013 -0700 Merge pull request #16 from TheTorProject/fix/simplify_filterHelperAddresses Simplify filterHelperAddresses oonib/bouncer/handlers.py | 68 +++++++++++++++++++++------------------------ 1 file changed, 32 insertions(+), 36 deletions(-)

1 0

[oonib/master] Return the correct format
by art＠torproject.org 11 Sep '13

11 Sep '13

commit bacb8cc652916859757438ab330fafa695ec72ad Author: aagbsn <aagbsn(a)extc.org> Date: Tue Sep 10 14:26:23 2013 +0200 Return the correct format --- oonib/bouncer/handlers.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/oonib/bouncer/handlers.py b/oonib/bouncer/handlers.py index d33dcfe..c6b249d 100644 --- a/oonib/bouncer/handlers.py +++ b/oonib/bouncer/handlers.py @@ -94,14 +94,15 @@ class Bouncer(object): # If we can, try to pick the same collector. choices = self.getHelperAddresses(helper_name) for item in response.values(): - if item['collector'] in choices.keys(): - choice = choices[item['collector']] + c = item['collector'] + h = choices[c] + if c in choices.keys(): break # Or default to a random selection else: c,h = random.choice(choices.items()) - choice = {'collector': c, 'address': h} - response[helper_name] = choice + + response[helper_name] = {'collector': c, 'address': h} except e.TestHelperNotFound: response = {'error': 'test-helper-not-found'}

1 0

[oonib/master] Select the correct helper
by art＠torproject.org 11 Sep '13

11 Sep '13

commit 83155689f1968c449cba4983fd6e1f556a05b18a Author: aagbsn <aagbsn(a)extc.org> Date: Tue Sep 10 13:50:46 2013 +0200 Select the correct helper --- oonib/bouncer/handlers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oonib/bouncer/handlers.py b/oonib/bouncer/handlers.py index 5033fad..d33dcfe 100644 --- a/oonib/bouncer/handlers.py +++ b/oonib/bouncer/handlers.py @@ -95,7 +95,7 @@ class Bouncer(object): choices = self.getHelperAddresses(helper_name) for item in response.values(): if item['collector'] in choices.keys(): - choice = item + choice = choices[item['collector']] break # Or default to a random selection else:

1 0

[oonib/master] OONIB release candidate 2 version 1.0.0
by art＠torproject.org 11 Sep '13

11 Sep '13

commit fb493733f237ae637d17a754938d40e7f05f1fb9 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 27 17:15:12 2013 +0200 OONIB release candidate 2 version 1.0.0 --- oonib/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oonib/__init__.py b/oonib/__init__.py index e01b7b3..4473fd8 100644 --- a/oonib/__init__.py +++ b/oonib/__init__.py @@ -3,7 +3,7 @@ In here we shall keep track of all variables and objects that should be instantiated only once and be common to pieces of GLBackend code. """ -__version__ = '1.0.0-rc1' +__version__ = '1.0.0-rc2' __all__ = ['Storage', 'randomStr']

1 0

[oonib/master] Should be break; not continue
by art＠torproject.org 11 Sep '13

11 Sep '13

commit 9d61639c0f2783c37bb1037d55d7670bc43e7d01 Author: aagbsn <aagbsn(a)extc.org> Date: Mon Sep 9 14:58:55 2013 +0200 Should be break; not continue --- oonib/bouncer/handlers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oonib/bouncer/handlers.py b/oonib/bouncer/handlers.py index 710eb6c..5033fad 100644 --- a/oonib/bouncer/handlers.py +++ b/oonib/bouncer/handlers.py @@ -96,7 +96,7 @@ class Bouncer(object): for item in response.values(): if item['collector'] in choices.keys(): choice = item - continue + break # Or default to a random selection else: c,h = random.choice(choices.items())

1 0

[oonib/master] Simplify filterHelperAddresses
by art＠torproject.org 11 Sep '13

11 Sep '13

commit c4f3a34ae559419cfc8a6b9e95ab1b60ab480ee9 Author: aagbsn <aagbsn(a)extc.org> Date: Mon Sep 9 14:47:19 2013 +0200 Simplify filterHelperAddresses --- oonib/bouncer/handlers.py | 67 +++++++++++++++++++++------------------------ 1 file changed, 31 insertions(+), 36 deletions(-) diff --git a/oonib/bouncer/handlers.py b/oonib/bouncer/handlers.py index 9e987f0..710eb6c 100644 --- a/oonib/bouncer/handlers.py +++ b/oonib/bouncer/handlers.py @@ -9,6 +9,18 @@ class Bouncer(object): def __init__(self): self.knownHelpers = {} self.updateKnownHelpers() + self.updateKnownCollectors() + + def updateKnownCollectors(self): + """ + Returns the list of all known collectors + """ + self.knownCollectors = [] + with open(config.main.bouncer_file) as f: + bouncerFile = yaml.safe_load(f) + for collectorName, helpers in bouncerFile['collector'].items(): + if collectorName not in self.knownCollectors: + self.knownCollectors.append(collectorName) def updateKnownHelpers(self): with open(config.main.bouncer_file) as f: @@ -76,43 +88,26 @@ class Bouncer(object): if no valid helper was found """ - result = {} - for helper_name in requested_helpers: - for collector, helper_address in self.getHelperAddresses(helper_name).items(): - if collector not in result.keys(): - result[collector] = {} - result[collector][helper_name] = helper_address - - # { - # 'foo.onion': {'some-helper': 'some-address'}, - # 'foo2.onion': {'some-helper': 'some-addres2'} - # } - response = {} - default_collector = None - max_helper_count = 0 - for collector, helpers in result.items(): - if len(helpers) > max_helper_count: - default_collector = collector - else: - continue - - response['default'] = {} - response['default']['collector'] = default_collector - - if not len(result[default_collector]) == len(requested_helpers): - found_helpers = set(result[default_collector].items()) - for missing_helper in found_helpers.difference(requested_helpers): - collector, address = random.choice(self.getHelperAddress(missing_helper).items()) - response[missing_helper] = {} - response[missing_helper]['collector'] = collector - response[missing_helper]['address'] = address - - for name, address in result[default_collector].items(): - response[name] = {} - response[name]['address'] = address - response[name]['collector'] = default_collector - + for helper_name in requested_helpers: + try: + # If we can, try to pick the same collector. + choices = self.getHelperAddresses(helper_name) + for item in response.values(): + if item['collector'] in choices.keys(): + choice = item + continue + # Or default to a random selection + else: + c,h = random.choice(choices.items()) + choice = {'collector': c, 'address': h} + response[helper_name] = choice + + except e.TestHelperNotFound: + response = {'error': 'test-helper-not-found'} + return response + + response['default'] = {'collector': random.choice(self.knownCollectors)} return response class BouncerQueryHandler(OONIBHandler):

1 0

[oonib/master] Remove all nettest API related code.
by art＠torproject.org 11 Sep '13

11 Sep '13

commit f9181caa8d5805cec51b7ce37277f32553c726fa Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 27 15:03:07 2013 +0200 Remove all nettest API related code. --- oonib/nettest/api.py | 10 ---------- oonib/nettest/handlers.py | 27 --------------------------- 2 files changed, 37 deletions(-) diff --git a/oonib/nettest/__init__.py b/oonib/nettest/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/oonib/nettest/api.py b/oonib/nettest/api.py deleted file mode 100644 index 758b31d..0000000 --- a/oonib/nettest/api.py +++ /dev/null @@ -1,10 +0,0 @@ -from cyclone import web -from oonib.nettest import handlers -from oonib import config - -nettestAPI = [ - (r"/nettest", handlers.NetTestListHandler), - (r"/nettest/([a-f0-9]{64})", handlers.NetTestDescHandler), - (r"/nettest/([a-f0-9]{64})/py$", web.StaticFileHandler, {"path": - config.main.nettest_dir}), -] diff --git a/oonib/nettest/handlers.py b/oonib/nettest/handlers.py deleted file mode 100644 index b3ceed4..0000000 --- a/oonib/nettest/handlers.py +++ /dev/null @@ -1,27 +0,0 @@ -import glob -import json -import os -import yaml - -from oonib.handlers import OONIBHandler -from oonib import config, log - -class NetTestDescHandler(OONIBHandler): - def get(self, netTestID): - bn = os.path.basename(netTestID) + ".desc" - try: - with open(os.path.join(config.main.nettest_dir, bn)) as f: - response = {} - netTestDesc = yaml.safe_load(f) - for k in ['name', 'description', 'version', 'author', 'date']: - response[k] = netTestDesc[k] - self.write(response) - except IOError: - log.err("No NetTest Descriptor found for id %s" % netTestID) - self.set_status(404) - self.write({'error': 'missing-nettest'}) - - except Exception, e: - log.err("Invalid NetTest Descriptor found for id %s" % netTestID) - self.set_status(500) - self.write({'error': 'invalid-nettest-descriptor'})

1 0

[oonib/master] Start 2 separate Tor Hidden Services for the bouncer and the collector
by art＠torproject.org 11 Sep '13

11 Sep '13

commit 54889673061b7420ed67c2c42dd66f43ac36d851 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 27 15:12:56 2013 +0200 Start 2 separate Tor Hidden Services for the bouncer and the collector --- oonib/api.py | 3 ++- oonib/runner.py | 35 +++++++++++++++++++++++++++++------ 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/oonib/api.py b/oonib/api.py index 622df4d..f2c48be 100644 --- a/oonib/api.py +++ b/oonib/api.py @@ -8,6 +8,7 @@ from oonib.bouncer.api import bouncerAPI from oonib import config +ooniBouncer = None ooniBackendAPI = [] ooniBackendAPI += reportAPI @@ -21,6 +22,6 @@ if config.main.policy_file: ooniBackendAPI += policyAPI if config.main.bouncer_file: - ooniBackendAPI += bouncerAPI + ooniBouncer = web.Application(bouncerAPI, debug=True) ooniBackend = web.Application(ooniBackendAPI, debug=True) diff --git a/oonib/runner.py b/oonib/runner.py index 294f414..d03aa33 100644 --- a/oonib/runner.py +++ b/oonib/runner.py @@ -21,7 +21,7 @@ from txtorcon import TCPHiddenServiceEndpoint, TorConfig from txtorcon import launch_tor from oonib.report.api import reportAPI -from oonib.api import ooniBackend +from oonib.api import ooniBackend, ooniBouncer from oonib import oonibackend from oonib import config @@ -52,10 +52,29 @@ def setupCollector(tor_process_protocol, datadir): #XXX: set up the various API endpoints, if configured and enabled #XXX: also set up a separate keyed hidden service for collectors to push their status to, if the bouncer is enabled hs_endpoint = TCPHiddenServiceEndpoint(reactor, torconfig, public_port, - data_dir=datadir) - hidden_service = hs_endpoint.listen(ooniBackend) - hidden_service.addCallback(setup_complete) - hidden_service.addErrback(txSetupFailed) + data_dir=os.path.join(datadir, 'collector')) + d = hs_endpoint.listen(ooniBackend) + + d.addCallback(setup_complete) + d.addErrback(txSetupFailed) + + return tor_process_protocol + +def setupBouncer(tor_process_protocol, datadir): + def setup_complete(port): + #XXX: drop some other noise about what API are available on this machine + print("Exposed bouncer Tor hidden service on httpo://%s" + % port.onion_uri) + + torconfig = TorConfig(tor_process_protocol.tor_protocol) + public_port = 80 + + hs_endpoint = TCPHiddenServiceEndpoint(reactor, torconfig, public_port, + data_dir=os.path.join(datadir, 'bouncer')) + + d = hs_endpoint.listen(ooniBouncer) + d.addCallback(setup_complete) + d.addErrback(txSetupFailed) def startTor(): def updates(prog, tag, summary): @@ -87,6 +106,8 @@ def startTor(): else: d = launch_tor(torconfig, reactor, progress_updates=updates) d.addCallback(setupCollector, datadir) + if ooniBouncer: + d.addCallback(setupBouncer, datadir) d.addErrback(txSetupFailed) if platformType == "win32": @@ -110,7 +131,9 @@ else: if config.main.tor_hidden_service: startTor() else: - reactor.listenTCP(8888, ooniBackend, interface="127.0.0.1") + if ooniBouncer: + reactor.listenTCP(8888, ooniBouncer, interface="127.0.0.1") + reactor.listenTCP(8889, ooniBackend, interface="127.0.0.1") self.startReactor(None, self.oldstdout, self.oldstderr) self.removePID(self.config['pidfile']) if os.path.exists(tempfile.gettempdir()):

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits September 2013