October 2012 - tor-commits - lists.torproject.org

[ooni-probe/master] * Added a trick to supress scapy's annoying "No route found for IPv6 destination"
by isis＠torproject.org 11 Oct '12

11 Oct '12

commit 2f126cdfa5f4971e3a670a6b90790cada05b2a89 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Oct 11 22:26:02 2012 +0000 * Added a trick to supress scapy's annoying "No route found for IPv6 destination" warnings. --- ooni/reporter.py | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-) diff --git a/ooni/reporter.py b/ooni/reporter.py index 47b8158..5de2ff1 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -1,8 +1,9 @@ -import time +import copy_reg +import itertools +import logging import sys +import time import yaml -import itertools -import copy_reg from yaml.representer import * from yaml.emitter import * @@ -16,6 +17,9 @@ from twisted.internet import defer from ooni.utils import date, log, geodata try: + ## Get rid of the annoying "No route found for + ## IPv6 destination warnings": + logging.getLogger("scapy.runtime").setLevel(logging.ERROR) from scapy.all import packet except: class FooClass:

1 0

[ooni-probe/master] * Rewrote the runner.LegacyOONITest class and runner.loadTestsAndOptions
by isis＠torproject.org 11 Oct '12

11 Oct '12

commit 270db771f360111efd46329796ec4567fedae0f8 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Oct 11 22:27:52 2012 +0000 * Rewrote the runner.LegacyOONITest class and runner.loadTestsAndOptions function so that they actually run old tests. * Added a weird dynamic method-finding hack to get around old tests being considered a module inside a module, e.g. ooni.plugins.dnstamper.dnstamper.startTest(). This is due to the __init__ structure of that directory, and so the test filename is a module, which ooniprobe.py then calls the test from, which I think makes it into another module or class. Sometimes distributed python modules have this characteristic as well, and the hack seems to work to figure out the name of the test we're supposed to be running and thus pass that to ooniprobe.py correctly. * Added another weird hack to re-add the __base__ builtin attr to subclasses of OONITest, because for some odd reason it was missing in all of them, and both inspect and Twisted expect it to be there. * Moved legacy_reporter and LegacyOONITest classes out of their prior wrapper. * TODO: Check that new-API-style tests still run correctly! * TODO: Some of the legacy tests throw KeyErrors on assets['somekey'] when they expect to find them and don't. The new oonicli should handle these in case the testwriters don't. --- ooni/runner.py | 190 +++++++++++++++++++++++++++++++++++++++----------------- 1 files changed, 133 insertions(+), 57 deletions(-) diff --git a/ooni/runner.py b/ooni/runner.py index 7f1c13e..e4f31ba 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -41,6 +41,101 @@ def isLegacyTest(obj): except TypeError: return False +class legacy_reporter(object): + def __init__(self, report_target): + self.report_target = report_target + + def __call__(self, what): + self.report_target.append(what) + +class LegacyOONITest(nettest.TestCase): + + ## we need bases so that inherited methods get parsed for prefixes too + from ooni.plugoo.tests import OONITest + __bases__ = (OONITest, ) + + def __init__(self, obj, config): + super(LegacyOONITest, self).__init__() + self.originalTest = obj + log.debug("obj: %s" % obj) + log.debug("originalTest: %s" % self.originalTest) + + self.subArgs = (None, ) + if 'subArgs' in config: + self.subArgs = config['subArgs'] + + try: + self.name = self.originalTest.shortName + except: + self.was_named = False + self.name = "LegacyOONITest" + + try: + self.subOptions = self.originalTest.options() + except AttributeError: + if self.was_named is False: + origClass = self.originalTest.__class__ + origClassStr = str(origClass) + fromModule = origClassStr.rsplit('.', 2)[:-1] + #origNamespace = globals()[origClass]() + #origAttr = getattr(origNamespace, fromModule) + log.debug("original class: %s" % origClassStr) + log.debug("from module: %s" % fromModule) + #log.debug("orginal namespace: %s" % origNamespace) + #log.debug("orginal attr: %s" % origAttr) + + def _options_from_name_tag(method_name, + orig_test=self.originalTest): + return orig_test.method_name.options() + + self.subOptions = _options_from_name_tag(fromModule, + self.originalTest) + else: + self.subOptions = None + log.err("That test appears to have a name, but no options!") + + if self.subOptions is not None: + self.subOptions.parseOptions(self.subArgs) + self.local_options = self.subOptions + + self.legacy_test = self.originalTest(None, None, None, None) + ## xxx fix me + #my_test.global_options = config['Options'] + self.legacy_test.local_options = self.subOptions + if self.was_named: + self.legacy_test.name = self.name + else: + self.legacy_test.name = fromModule + self.legacy_test.assets = self.legacy_test.load_assets() + self.legacy_test.report = legacy_reporter({}) + self.legacy_test.initialize() + + inputs = [] + + if len(self.legacy_test.assets.items()) == 0: + inputs.append('internal_asset_handler') + else: + for key, inputs in self.legacy_test.assets.items(): + pass + self.inputs = inputs + + def __getattr__(self, name): + def method(*args): + log.msg("Call to unknown method %s.%s" % (self.originalTest, name)) + if args: + log.msg("Unknown method %s parameters: %s" % str(args)) + return method + + @defer.inlineCallbacks + def test_start_legacy_test(self): + args = {} + for key, inputs in self.legacy_test.assets.items(): + args[key] = inputs + result = yield self.legacy_test.startTest(args) + self.report.update({'result': result}) + ## xxx we need to retVal on the defer.inlineCallbacks, right? + defer.returnValue(self.report) + def adaptLegacyTest(obj, config): """ We take a legacy OONITest class and convert it into a nettest.TestCase. @@ -49,49 +144,7 @@ def adaptLegacyTest(obj, config): XXX perhaps we could implement another extra layer that makes the even older test cases compatible with the new OONI. """ - class legacy_reporter(object): - def __init__(self, report_target): - self.report_target = report_target - - def __call__(self, what): - self.report_target.append(what) - - class LegacyOONITest(nettest.TestCase): - try: - name = obj.shortName - except: - name = "LegacyOONITest" - - originalTest = obj - - subOptions = obj.options() - subOptions.parseOptions(config['subArgs']) - - test_class = obj(None, None, None, None) - test_class.local_options = subOptions - assets = test_class.load_assets() - - inputs = [None] - # XXX here we are only taking assets that are set to one item only. - for key, inputs in assets.items(): - pass - - inputs = inputs - local_options = subOptions - - @defer.inlineCallbacks - def test_start_legacy_test(self): - - self.legacy_report = [] - - my_test = self.originalTest(None, None, None) - my_test.report = legacy_reporter(self.legacy_report) - args = {} - args[self.key] = self.input - result = yield my_test.startTest(args) - self.report['result'] = result - - return LegacyOONITest + return LegacyOONITest(obj, config) def processTest(obj, config): inputFile = obj.inputFile @@ -119,7 +172,6 @@ def processTest(obj, config): except usage.UsageError: options.opt_help() - return obj def findTestClassesFromConfig(config): @@ -151,7 +203,7 @@ def makeTestCases(klass, tests, methodPrefix): cases.append(klass(methodPrefix+test)) return cases -def loadTestsAndOptions(classes): +def loadTestsAndOptions(classes, config): """ Takes a list of classes and returnes their testcases and options. Legacy tests will be adapted. @@ -162,19 +214,29 @@ def loadTestsAndOptions(classes): testCases = [] names = [] + from ooni.runner import LegacyOONITest + _old_klass_type = LegacyOONITest + for klass in classes: + try: - k = klass() - opts = k.getOptions() - options.append(opts) - except AttributeError: - options.append([]) - tests = reflect.prefixedMethodNames(klass, methodPrefix) - if tests: - cases = makeTestCases(klass, tests, methodPrefix) - testCases.append(cases) - else: - options.pop() + assert not isinstance(klass, _old_klass_type) + except: + assert isinstance(klass, _old_klass_type) + #log.debug(type(klass)) + #legacyTest = adaptLegacyTest(klass, config) + klass.test_start_legacy_test() + else: + tests = reflect.prefixedMethodNames(klass, methodPrefix) + if tests: + cases = makeTestCases(klass, tests, methodPrefix) + testCases.append(cases) + try: + k = klass() + opts = k.getOptions() + options.append(opts) + except AttributeError: + options.append([]) return testCases, options @@ -189,7 +251,21 @@ class ORunner(object): self.baseSuite = InputTestSuite self.cases = cases self.options = options - self.inputs = options['inputs'] + + try: + assert len(options) != 0, "Length of options is zero!" + except AssertionError, ae: + self.inputs = [] + log.err(ae) + else: + first = options.pop() + if 'inputs' in first: + self.inputs = options['inputs'] + else: + log.msg("Could not find inputs!") + log.msg("options[0] = %s" % first) + self.inputs = [] + try: reportFile = open(config['reportfile'], 'a+') except:

1 0

[tor/master] Make very sure to handle cells in-order on channels.
by andrea＠torproject.org 11 Oct '12

11 Oct '12

commit a45760b53b6aef609e3d48f9e0abb5e1550e44fb Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Oct 11 00:35:58 2012 -0400 Make very sure to handle cells in-order on channels. Fix on code for 6465, not yet in any release. --- src/or/channel.c | 23 +++++++++++++---------- 1 files changed, 13 insertions(+), 10 deletions(-) diff --git a/src/or/channel.c b/src/or/channel.c index b52db40..6527288 100644 --- a/src/or/channel.c +++ b/src/or/channel.c @@ -2427,6 +2427,8 @@ channel_listener_queue_incoming(channel_listener_t *listener, void channel_process_cells(channel_t *chan) { + smartlist_t *queue; + int putback = 0; tor_assert(chan); tor_assert(chan->state == CHANNEL_STATE_CLOSING || chan->state == CHANNEL_STATE_MAINT || @@ -2442,17 +2444,20 @@ channel_process_cells(channel_t *chan) /* Nothing we can do if we have no cells */ if (!(chan->incoming_queue)) return; + queue = chan->incoming_queue; + chan->incoming_queue = NULL; /* * Process cells until we're done or find one we have no current handler * for. */ - SMARTLIST_FOREACH_BEGIN(chan->incoming_queue, - cell_queue_entry_t *, q) { + SMARTLIST_FOREACH_BEGIN(queue, cell_queue_entry_t *, q) { tor_assert(q); tor_assert(q->type == CELL_QUEUE_FIXED || q->type == CELL_QUEUE_VAR); - if (q->type == CELL_QUEUE_FIXED && + if (putback) { + smartlist_add(chan->incoming_queue, q); + } else if (q->type == CELL_QUEUE_FIXED && chan->cell_handler) { /* Handle a fixed-length cell */ tor_assert(q->u.fixed.cell); @@ -2462,7 +2467,6 @@ channel_process_cells(channel_t *chan) q->u.fixed.cell, chan, U64_PRINTF_ARG(chan->global_identifier)); chan->cell_handler(chan, q->u.fixed.cell); - SMARTLIST_DEL_CURRENT(chan->incoming_queue, q); tor_free(q); } else if (q->type == CELL_QUEUE_VAR && chan->var_cell_handler) { @@ -2474,19 +2478,18 @@ channel_process_cells(channel_t *chan) q->u.var.var_cell, chan, U64_PRINTF_ARG(chan->global_identifier)); chan->var_cell_handler(chan, q->u.var.var_cell); - SMARTLIST_DEL_CURRENT(chan->incoming_queue, q); tor_free(q); } else { /* Can't handle this one */ - break; + if (!chan->incoming_queue) + chan->incoming_queue = smartlist_new(); + smartlist_add(chan->incoming_queue, q); + putback = 1; } } SMARTLIST_FOREACH_END(q); /* If the list is empty, free it */ - if (smartlist_len(chan->incoming_queue) == 0 ) { - smartlist_free(chan->incoming_queue); - chan->incoming_queue = NULL; - } + smartlist_free(queue); } /**

1 0

[torspec/master] Add proposal for consensus bootstrapping.
by nickm＠torproject.org 11 Oct '12

11 Oct '12

commit 3d6232cfa097f9e9102aaeb423472ef486dad730 Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Mon Oct 1 14:38:54 2012 -0700 Add proposal for consensus bootstrapping. --- .../xxx-faster-headless-consensus-bootstrap.txt | 78 ++++++++++++++++++++ 1 files changed, 78 insertions(+), 0 deletions(-) diff --git a/proposals/xxx-faster-headless-consensus-bootstrap.txt b/proposals/xxx-faster-headless-consensus-bootstrap.txt new file mode 100644 index 0000000..2fd34c1 --- /dev/null +++ b/proposals/xxx-faster-headless-consensus-bootstrap.txt @@ -0,0 +1,78 @@ +Title: Faster Headless Consensus Bootstrapping +Author: Mike Perry +Created: 01-10-2012 +Status: Open +Target: 0.2.4.x+ + + +Overview + + This proposal describes a way for clients to fetch the initial consensus + more quickly in situations where some or all of the directory authorities + are unreachable. It is meant to describe a solution for Bug #4483. + +Description + + The core idea is to attempt to establish bootstrap connections in + parallel during the bootstrap process, and download the consensus from the + first connection that completes. + + Connection attempts will be done in batches of five. Only one connection + will be performed to one of the canonical directory authorities. Four + connections will be performed to randomly chosen hard coded directory + mirrors. + + If no connections complete within 5 seconds, another batch of five + connections will be launched. Otherwise, the first connection to complete + will be used to download the consensus document, after which + bootstrapping will proceed as normal. + + If at any time, the total open bootstrap connection attempts exceeds + 25, no new connection attempts are to be launched until existing + connection attempts experience full timeout. + +Fallback Dir Mirror Selection + + The set of hard coded directory mirrors from #572 shall be chosen using + the 50 Guard nodes with the longest uptime. + + The fallback weights will be set using each mirror's fraction of + consensus bandwidth out of the total of all 50 mirrors. + + This list of fallback dir mirrors should be updated with every + major Tor release. + +Implementation Notes + + The implementation of the bootstrap process is unfortunately mixed + in with many types of directory activity. + + The process starts in update_consensus_networkstatus_downloads(), + which initiates a single directory connection through + directory_get_from_dirserver(). Depending on bootstrap state, + a single directory server is selected and a connection is + eventually made through directory_initiate_command_rend(). + + There appear to be a few options for altering this code to perform + multiple connections. Without refactoring, one approach would be + to make multiple calls to directory_initiate_command_routerstatus() + from directory_get_from_dirserver() if the purpose is + DIR_PURPOSE_FETCH_CONSENSUS and the only directory servers available + are the authorities and the fallback dir mirrors. + + The code in directory_initiate_command_rend() would then + need to be altered to maintain a list of the dircons created for this + purpose as well as avoid immediately queuing the directory_send_command() + request for the DIR_PURPOSE_FETCH_CONSENSUS purpose. A flag would need to + be set on the dircon to be checked in connection_dir_finished_connecting(). + + The function connection_dir_finished_connecting() would need to be altered to + examine the list of pending dircons, determine if this one is the first to + complete, and if so, then call directory_send_command() to download the + consensus and close the other pending dircons. + + An additional timer would need to be installed to re-call + update_consensus_networkstatus_downloads() or a related helper after 5 + seconds. connection_dir_finished_connecting() would cancel this timer. + The helper would check the list of pending connections and ensure it + never exceeds 25.

1 0

[torspec/master] Add a load analysis section.
by nickm＠torproject.org 11 Oct '12

11 Oct '12

commit 42cbe7ea09b87c0cec70413ea40a4436b3eac3db Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Wed Oct 3 19:46:01 2012 -0700 Add a load analysis section. Also tweak some parameters based on the analysis. --- .../xxx-faster-headless-consensus-bootstrap.txt | 51 ++++++++++++++++---- 1 files changed, 41 insertions(+), 10 deletions(-) diff --git a/proposals/xxx-faster-headless-consensus-bootstrap.txt b/proposals/xxx-faster-headless-consensus-bootstrap.txt index 2fd34c1..9aef215 100644 --- a/proposals/xxx-faster-headless-consensus-bootstrap.txt +++ b/proposals/xxx-faster-headless-consensus-bootstrap.txt @@ -17,30 +17,61 @@ Description parallel during the bootstrap process, and download the consensus from the first connection that completes. - Connection attempts will be done in batches of five. Only one connection - will be performed to one of the canonical directory authorities. Four + Connection attempts will be done in batches of three. Only one connection + will be performed to one of the canonical directory authorities. Two connections will be performed to randomly chosen hard coded directory mirrors. - If no connections complete within 5 seconds, another batch of five + If no connections complete within 5 seconds, another batch of three connections will be launched. Otherwise, the first connection to complete - will be used to download the consensus document, after which - bootstrapping will proceed as normal. + will be used to download the consensus document and the others will be + closed, after which bootstrapping will proceed as normal. - If at any time, the total open bootstrap connection attempts exceeds - 25, no new connection attempts are to be launched until existing + If at any time, the total outstanding bootstrap connection attempts + exceeds 15, no new connection attempts are to be launched until existing connection attempts experience full timeout. Fallback Dir Mirror Selection The set of hard coded directory mirrors from #572 shall be chosen using - the 50 Guard nodes with the longest uptime. + the 100 Guard nodes with the longest uptime. The fallback weights will be set using each mirror's fraction of - consensus bandwidth out of the total of all 50 mirrors. + consensus bandwidth out of the total of all 100 mirrors. This list of fallback dir mirrors should be updated with every - major Tor release. + major Tor release. In future releases, the number of dir mirrors + should be set at 20% of the current Guard nodes, rather than fixed at + 100. + +Additional Load with Current Parameter Choices + + This design and the connection count parameters were chosen such that no + additional bandwidth load would be placed on the directory authorities. In + fact, the directory authorities should experience less load, because they will + not need to serve the consensus document for a connection in the event that + one of the directory mirrors complete their connection before the directory + authority does. + + However, the scheme does place additional TLS connection load on the + fallback dir mirrors. Because bootstrapping is rare and all but one of + the TLS connections will be very short-lived and unused, this should not + be a substantial issue. + + The dangerous case is in the event of a prolonged consensus failure + that induces all clients to enter into the bootstrap process. In this case, + the number of TLS connections to the fallback dir mirrors would be between + 2*C/100, or 10,000 for C=500,000 users. If no connections complete before + the five retries, this could reach as high as 50,000 connection attempts, + but this is extremely unlikely to happen in full aggregate. + + However, in the no-consensus scenario today, the directory authorities + would already experience C/9 or 55,555 connection attempts. The + 5-retry scheme increases their total maximum load to about 275,000 + connection attempts, but again this is unlikely to be reached + in aggregate. Additionally, with this scheme, even if the dirauths + are taken down by this load, the dir mirrors should be able to survive + it. Implementation Notes

1 0

[torspec/master] Misc cleanups for 001-process.txt
by nickm＠torproject.org 11 Oct '12

11 Oct '12

commit 24672064abdafeeadee7c69e967f4c5622dd257c Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Wed Oct 3 20:05:09 2012 -0700 Misc cleanups for 001-process.txt --- .../xxx-faster-headless-consensus-bootstrap.txt | 13 +++++++------ 1 files changed, 7 insertions(+), 6 deletions(-) diff --git a/proposals/xxx-faster-headless-consensus-bootstrap.txt b/proposals/xxx-faster-headless-consensus-bootstrap.txt index 9aef215..ce80ae1 100644 --- a/proposals/xxx-faster-headless-consensus-bootstrap.txt +++ b/proposals/xxx-faster-headless-consensus-bootstrap.txt @@ -5,13 +5,14 @@ Status: Open Target: 0.2.4.x+ -Overview +Overview and Motiviation This proposal describes a way for clients to fetch the initial consensus more quickly in situations where some or all of the directory authorities - are unreachable. It is meant to describe a solution for Bug #4483. + are unreachable. This proposal is meant to describe a solution for bug + #4483. -Description +Design: Bootstrap Process Changes The core idea is to attempt to establish bootstrap connections in parallel during the bootstrap process, and download the consensus from the @@ -31,7 +32,7 @@ Description exceeds 15, no new connection attempts are to be launched until existing connection attempts experience full timeout. -Fallback Dir Mirror Selection +Design: Fallback Dir Mirror Selection The set of hard coded directory mirrors from #572 shall be chosen using the 100 Guard nodes with the longest uptime. @@ -44,7 +45,7 @@ Fallback Dir Mirror Selection should be set at 20% of the current Guard nodes, rather than fixed at 100. -Additional Load with Current Parameter Choices +Performance: Additional Load with Current Parameter Choices This design and the connection count parameters were chosen such that no additional bandwidth load would be placed on the directory authorities. In @@ -60,7 +61,7 @@ Additional Load with Current Parameter Choices The dangerous case is in the event of a prolonged consensus failure that induces all clients to enter into the bootstrap process. In this case, - the number of TLS connections to the fallback dir mirrors would be between + the number of initial TLS connections to the fallback dir mirrors would be 2*C/100, or 10,000 for C=500,000 users. If no connections complete before the five retries, this could reach as high as 50,000 connection attempts, but this is extremely unlikely to happen in full aggregate.

1 0

[torspec/master] Add rough rough draft of a mapaddress-based check proposal.
by nickm＠torproject.org 11 Oct '12

11 Oct '12

commit 70194e158836f596b8719272c8c6156b8368ed31 Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Mon Oct 8 18:30:37 2012 -0700 Add rough rough draft of a mapaddress-based check proposal. This thing is going to be tricky to do right... --- proposals/xxx-mapaddress-tor-status.txt | 135 +++++++++++++++++++++++++++++++ 1 files changed, 135 insertions(+), 0 deletions(-) diff --git a/proposals/xxx-mapaddress-tor-status.txt b/proposals/xxx-mapaddress-tor-status.txt new file mode 100644 index 0000000..ddb1a1f --- /dev/null +++ b/proposals/xxx-mapaddress-tor-status.txt @@ -0,0 +1,135 @@ +Title: Internal Mapaddress for Tor Configuration Testing +Author: Mike Perry +Created: 08-10-2012 +Status: Open +Target: 0.2.4.x+ + + +Overview + + This proposal describes a method by which we can replace the + https://check.torproject.org/ testing service with an internal XML + document provided by the Tor client. + +Motivation + + The Tor Check service is a central point of failure in terms of Tor + usability. If it is ever out of sync with the set of exit nodes on the + Tor network or down, user experience is degraded considerably. Moreover, + the check itself is very time-consuming. Users must wait seconds or more + for the result to come back. Worse still, if the user's software *was* + in fact misconfigured, the check.torproject.org DNS resolution and + request leaks out on to the network. + +Design Overview + + The system will have three parts: an internal hard-coded IP address + mapping (127.84.111.114:80), a hard-coded mapaddress to a DNS name + (selftest.torproject.org:80), and a DirPortFrontPage-style simple + HTTP server that serves an XML document for both addresses. + + Upon receipt of a request to the IP address mapping, the system will + create a new 128 bit randomly generated nonce and provide it + in the XML document. + + Requests to http://selftest.torproject.org/ must include a valid, + recent nonce as the GET url path. Upon receipt of a valid nonce, + it is removed from the list of valid nonces. Nonces are only valid + for 60 seconds or until SIGNAL NEWNYM, which ever comes first. + +Design: XML document format for http://127.84.111.114 + + To avoid the need to localize the message in Tor, Tor will only provide + a XML object with connectivity information. Here is an example form: + + <tor-test> + <tor-bootstrap-percent>100</tor-bootstrap-percent> + <tor-version-current>true</tor-version-current> + <dns-nonce>4977eb4842c7c59fa5b830ac4da896d9</dns-nonce> + <tor-test/> + + The tor-bootstrap-percent field represents the results of the Tor client + bootstrap status as integer percentages from bootstrap_status_t. + + The tor-version-current field represents the results of the Tor client + consensus version check. If the bootstrap process has not yet + downloaded a consensus document, this field will have the value + null. + + The dns-nonce field contains a 128-bit secret, encoded in base16. This + field is only present for requests that list the Host: header as + 127.84.111.114. + +Design: XML document format for http://selftest.torproject.org/nonce + + <tor-test> + <tor-bootstrap-percent>100</tor-bootstrap-percent> + <tor-version-current>true</tor-version-current> + <dns-nonce-valid>true</dns-nonce-valid> + <tor-test/> + + The first two fields are the same as for the IP address version. + + The dns-nonce-valid field is only true if the Host header matches + selftest.torproject.org and the nonce is current and valid. Upon + receipt of a valid nonce, that nonce is removed from the list of + valid nonces. + +Design: Request Servicing + + Care must be taken with the dns-nonce generation and usage, to prevent + users from being tracked through leakage of nonce value to application + content. While the usage of XML appears to make this impossible + due to stricter same-origin policy enforcement than JSON, same-origin + enforcement is still fraught with exceptions and loopholes. + + In particular: + + dns-nonce fields MUST be omitted if the HTTP Host: header does not + match the IP address 127.84.111.114. + + Requests to selftest.torproject.org MUST return false for the + dns-nonce-valid field if the HTTP Host: header does not match + selftest.torproject.org, regardless of nonce value. + + Further, requests to selftest.torproject.org MUST validate that + 'selftest.torproject.org' was the actual hostname provided to + SOCKS4A, and not some alternate address mapping (due to DNS rebinding + attacks, for example). + +Design: Application Usage + + Applications will use the system in two steps. First, they will make an + HTTP request to http://127.84.111.114:80/ over Tor's SOCKS port and + parse the resulting XML, if any. + + If the request at this stage fails, the application should inform the + user that either their Tor client is too old, or that it is + misconfigured, depending upon the nature of the failure. + + If the request succeeds and valid XML is returned, the application + will record the value of the dns-nonce field, and then perform a second + request to http://selftest.torproject.org/nonce_value. If the second + request succeeds, and the dns-nonce-valid field is true, the application + may inform the user that their Tor settings are valid. + + If the second request fails, or does not provide the correct dns-nonce, + the application will inform the user that their Tor DNS proxy settings + are incorrect. + + If either tor-bootstrap-percent is not 100, or tor-version-current is + false, applications may choose to inform the user of these facts using + properly localized strings and appropriate UI. + +Security Considerations + + XML was chosen over JSON due to the risks of the identifier leaking + in a way that could enable websites to track the user[1]. + + Because there are many exceptions and circumvention techniques + to the same-origin policy, we have also opted for strict controls + on dns-nonce lifetimes and usage, as well as validation of the Host + header and SOCKS4A request hostnames. + + +1. http://www.hpenterprisesecurity.com/vulncat/en/vulncat/dotnet/javascript_hi…

1 0

[torspec/master] We also need to ignore CORS requests.
by nickm＠torproject.org 11 Oct '12

11 Oct '12

commit cd3296c95e7f2c3639e9b6518a29b80100bece4a Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Tue Oct 9 11:26:05 2012 -0700 We also need to ignore CORS requests. --- proposals/xxx-mapaddress-tor-status.txt | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/proposals/xxx-mapaddress-tor-status.txt b/proposals/xxx-mapaddress-tor-status.txt index ddb1a1f..c226781 100644 --- a/proposals/xxx-mapaddress-tor-status.txt +++ b/proposals/xxx-mapaddress-tor-status.txt @@ -85,6 +85,10 @@ Design: Request Servicing In particular: + Any requests that contain the Origin: header MUST be ignored, + as the Origin: header is only included for third party web content + (CORS). + dns-nonce fields MUST be omitted if the HTTP Host: header does not match the IP address 127.84.111.114.

1 0

[torspec/master] Mention a nonce limit.
by nickm＠torproject.org 11 Oct '12

11 Oct '12

commit ea37bb4d3dbc21e04d5f77e940d6d37da5cb277b Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Wed Oct 10 19:12:47 2012 -0700 Mention a nonce limit. --- proposals/xxx-mapaddress-tor-status.txt | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/proposals/xxx-mapaddress-tor-status.txt b/proposals/xxx-mapaddress-tor-status.txt index c226781..8475bd0 100644 --- a/proposals/xxx-mapaddress-tor-status.txt +++ b/proposals/xxx-mapaddress-tor-status.txt @@ -37,6 +37,11 @@ Design Overview it is removed from the list of valid nonces. Nonces are only valid for 60 seconds or until SIGNAL NEWNYM, which ever comes first. + The list of pending nonces should not be allowed to grow beyond 10 + entries. + + The timeout period and nonce limit should be configurable in torrc. + Design: XML document format for http://127.84.111.114 To avoid the need to localize the message in Tor, Tor will only provide

1 0

[torspec/master] Formatting cleanups.
by nickm＠torproject.org 11 Oct '12

11 Oct '12

commit 94e56740934f39a240a75167f670e1dd34fb9ecf Author: Mike Perry <mikeperry-git(a)fscked.org> Date: Wed Oct 10 19:29:26 2012 -0700 Formatting cleanups. --- .../xxx-faster-headless-consensus-bootstrap.txt | 72 ++++++++++---------- 1 files changed, 37 insertions(+), 35 deletions(-) diff --git a/proposals/xxx-faster-headless-consensus-bootstrap.txt b/proposals/xxx-faster-headless-consensus-bootstrap.txt index ce80ae1..8f5b079 100644 --- a/proposals/xxx-faster-headless-consensus-bootstrap.txt +++ b/proposals/xxx-faster-headless-consensus-bootstrap.txt @@ -7,26 +7,26 @@ Target: 0.2.4.x+ Overview and Motiviation - This proposal describes a way for clients to fetch the initial consensus - more quickly in situations where some or all of the directory authorities - are unreachable. This proposal is meant to describe a solution for bug - #4483. + This proposal describes a way for clients to fetch the initial + consensus more quickly in situations where some or all of the directory + authorities are unreachable. This proposal is meant to describe a + solution for bug #4483. Design: Bootstrap Process Changes The core idea is to attempt to establish bootstrap connections in - parallel during the bootstrap process, and download the consensus from the - first connection that completes. + parallel during the bootstrap process, and download the consensus from + the first connection that completes. - Connection attempts will be done in batches of three. Only one connection - will be performed to one of the canonical directory authorities. Two - connections will be performed to randomly chosen hard coded directory - mirrors. + Connection attempts will be done in batches of three. Only one + connection will be performed to one of the canonical directory + authorities. Two connections will be performed to randomly chosen hard + coded directory mirrors. If no connections complete within 5 seconds, another batch of three - connections will be launched. Otherwise, the first connection to complete - will be used to download the consensus document and the others will be - closed, after which bootstrapping will proceed as normal. + connections will be launched. Otherwise, the first connection to + complete will be used to download the consensus document and the others + will be closed, after which bootstrapping will proceed as normal. If at any time, the total outstanding bootstrap connection attempts exceeds 15, no new connection attempts are to be launched until existing @@ -47,12 +47,12 @@ Design: Fallback Dir Mirror Selection Performance: Additional Load with Current Parameter Choices - This design and the connection count parameters were chosen such that no - additional bandwidth load would be placed on the directory authorities. In - fact, the directory authorities should experience less load, because they will - not need to serve the consensus document for a connection in the event that - one of the directory mirrors complete their connection before the directory - authority does. + This design and the connection count parameters were chosen such that + no additional bandwidth load would be placed on the directory + authorities. In fact, the directory authorities should experience less + load, because they will not need to serve the consensus document for a + connection in the event that one of the directory mirrors complete their + connection before the directory authority does. However, the scheme does place additional TLS connection load on the fallback dir mirrors. Because bootstrapping is rare and all but one of @@ -60,11 +60,12 @@ Performance: Additional Load with Current Parameter Choices be a substantial issue. The dangerous case is in the event of a prolonged consensus failure - that induces all clients to enter into the bootstrap process. In this case, - the number of initial TLS connections to the fallback dir mirrors would be - 2*C/100, or 10,000 for C=500,000 users. If no connections complete before - the five retries, this could reach as high as 50,000 connection attempts, - but this is extremely unlikely to happen in full aggregate. + that induces all clients to enter into the bootstrap process. In this + case, the number of initial TLS connections to the fallback dir mirrors + would be 2*C/100, or 10,000 for C=500,000 users. If no connections + complete before the five retries, this could reach as high as 50,000 + connection attempts, but this is extremely unlikely to happen in full + aggregate. However, in the no-consensus scenario today, the directory authorities would already experience C/9 or 55,555 connection attempts. The @@ -74,7 +75,7 @@ Performance: Additional Load with Current Parameter Choices are taken down by this load, the dir mirrors should be able to survive it. -Implementation Notes +Implementation Notes: Code Modifications The implementation of the bootstrap process is unfortunately mixed in with many types of directory activity. @@ -92,19 +93,20 @@ Implementation Notes DIR_PURPOSE_FETCH_CONSENSUS and the only directory servers available are the authorities and the fallback dir mirrors. - The code in directory_initiate_command_rend() would then - need to be altered to maintain a list of the dircons created for this - purpose as well as avoid immediately queuing the directory_send_command() - request for the DIR_PURPOSE_FETCH_CONSENSUS purpose. A flag would need to - be set on the dircon to be checked in connection_dir_finished_connecting(). + The code in directory_initiate_command_rend() would then need to be + altered to maintain a list of the dircons created for this purpose as + well as avoid immediately queuing the directory_send_command() request + for the DIR_PURPOSE_FETCH_CONSENSUS purpose. A flag would need to be set + on the dircon to be checked in connection_dir_finished_connecting(). - The function connection_dir_finished_connecting() would need to be altered to - examine the list of pending dircons, determine if this one is the first to - complete, and if so, then call directory_send_command() to download the - consensus and close the other pending dircons. + The function connection_dir_finished_connecting() would need to be + altered to examine the list of pending dircons, determine if this one is + the first to complete, and if so, then call directory_send_command() to + download the consensus and close the other pending dircons. An additional timer would need to be installed to re-call update_consensus_networkstatus_downloads() or a related helper after 5 seconds. connection_dir_finished_connecting() would cancel this timer. The helper would check the list of pending connections and ensure it - never exceeds 25. + never exceeds 15. +

1 0