April 2014 - tor-commits - lists.torproject.org

[tor/master] Merge branch 'bug9213_doc_024' into maint-0.2.4
by nickm＠torproject.org 01 Apr '14

01 Apr '14

commit 48578e1512f2d5fa95e4fdde0564d2db8ee7f995 Merge: 24e0b10 a68e6ea Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Apr 1 20:59:13 2014 -0400 Merge branch 'bug9213_doc_024' into maint-0.2.4 changes/bug9213_doc | 5 +++++ doc/tor.1.txt | 5 ++++- 2 files changed, 9 insertions(+), 1 deletion(-)

1 0

[tor/master] Merge remote-tracking branch 'origin/maint-0.2.4'
by nickm＠torproject.org 01 Apr '14

01 Apr '14

commit dfc32177d9ec3b8a639d6831ca4bc92c4f273dcb Merge: b4b9186 48578e1 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Apr 1 21:00:30 2014 -0400 Merge remote-tracking branch 'origin/maint-0.2.4' changes/bug9213_doc | 5 +++++ doc/tor.1.txt | 5 ++++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --cc doc/tor.1.txt index a42d550,eb9ad67..1cc8f84 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@@ -42,13 -37,9 +42,13 @@@ COMMAND-LINE OPTION [[opt-f]] **-f** __FILE__:: Specify a new configuration file to contain further Tor configuration - options. (Default: $HOME/.torrc, or @CONFDIR@/torrc if that file is not + options. (Default: @CONFDIR@/torrc, or $HOME/.torrc if that file is not found) +[[opt-allow-missing-torrc]] **--allow-missing-torrc**:: + Do not require that configuration file specified by **-f** exist if + default torrc can be accessed. + [[opt-defaults-torrc]] **--defaults-torrc** __FILE__:: Specify a file in which to find default values for Tor options. The contents of this file are overridden by those in the regular

1 0

[tor/maint-0.2.4] Merge branch 'bug9213_doc_024' into maint-0.2.4
by nickm＠torproject.org 01 Apr '14

01 Apr '14

commit 48578e1512f2d5fa95e4fdde0564d2db8ee7f995 Merge: 24e0b10 a68e6ea Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Apr 1 20:59:13 2014 -0400 Merge branch 'bug9213_doc_024' into maint-0.2.4 changes/bug9213_doc | 5 +++++ doc/tor.1.txt | 5 ++++- 2 files changed, 9 insertions(+), 1 deletion(-)

1 0

[tor/maint-0.2.4] Fix documentation of torrc search order
by nickm＠torproject.org 01 Apr '14

01 Apr '14

commit a68e6ea9c0288d82ea10076c551f48d63b3813fc Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Mar 27 14:58:46 2014 -0400 Fix documentation of torrc search order We are searching @CONFDIR@ before $HOME, but the documentation implied otherwise. I screwed this up in f5e86bcd6c06d43ff3af5acd8135bd8b577bc3, when I first documented the $HOME/.torrc possibility. Fix for bug 9213; bugfix on 0.2.3.18-rc. --- changes/bug9213_doc | 5 +++++ doc/tor.1.txt | 5 ++++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/changes/bug9213_doc b/changes/bug9213_doc new file mode 100644 index 0000000..2f959dd --- /dev/null +++ b/changes/bug9213_doc @@ -0,0 +1,5 @@ + o Documentation: + - Correctly document that we search for a system torrc file before + looking in ~/.torrc. Fixes documentation side of 9213; bugfix + on 0.2.3.18-rc. + diff --git a/doc/tor.1.txt b/doc/tor.1.txt index 3f8f6da..eb9ad67 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -37,7 +37,7 @@ COMMAND-LINE OPTIONS [[opt-f]] **-f** __FILE__:: Specify a new configuration file to contain further Tor configuration - options. (Default: $HOME/.torrc, or @CONFDIR@/torrc if that file is not + options. (Default: @CONFDIR@/torrc, or $HOME/.torrc if that file is not found) [[opt-defaults-torrc]] **--defaults-torrc** __FILE__:: @@ -2103,6 +2103,9 @@ FILES **@CONFDIR@/torrc**:: The configuration file, which contains "option value" pairs. +**$HOME/.torrc**:: + Fallback location for torrc, if @CONFDIR@/torrc is not found. + **@LOCALSTATEDIR@/lib/tor/**:: The tor process stores keys and other data here.

1 0

[tor/master] Merge remote-tracking branch 'public/bug9870'
by nickm＠torproject.org 01 Apr '14

01 Apr '14

commit b4b91864bb3879587edc7e8c430337f24835b28c Merge: 6bef082 c4bb3c8 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Apr 1 20:48:15 2014 -0400 Merge remote-tracking branch 'public/bug9870' Conflicts: src/or/config.c changes/bug9870 | 5 +++++ src/or/config.c | 38 +++++++++++++++++++++++++------------- 2 files changed, 30 insertions(+), 13 deletions(-) diff --cc src/or/config.c index 909ac14,dcbe88d..c42ceb3 --- a/src/or/config.c +++ b/src/or/config.c @@@ -1342,26 -1360,6 +1361,19 @@@ options_act(const or_options_t *old_opt } #endif + /* If we are a bridge with a pluggable transport proxy but no + Extended ORPort, inform the user that she is missing out. */ + if (server_mode(options) && options->ServerTransportPlugin && + !options->ExtORPort_lines) { + log_notice(LD_CONFIG, "We use pluggable transports but the Extended " + "ORPort is disabled. Tor and your pluggable transports proxy " + "communicate with each other via the Extended ORPort so it " + "is suggested you enable it: it will also allow your Bridge " + "to collect statistics about its clients that use pluggable " + "transports. Please enable it using the ExtORPort torrc option " + "(e.g. set 'ExtORPort auto')."); + } + - if (options->SafeLogging_ != SAFELOG_SCRUB_ALL && - (!old_options || old_options->SafeLogging_ != options->SafeLogging_)) { - log_warn(LD_GENERAL, "Your log may contain sensitive information - you " - "disabled SafeLogging. Please log safely. Don't log unless it " - "serves an important reason. Overwrite the log afterwards."); - } - if (options->Bridges) { mark_bridge_list(); for (cl = options->Bridges; cl; cl = cl->next) {

1 0

[tor/master] Log only one message for dangerous log settings.
by nickm＠torproject.org 01 Apr '14

01 Apr '14

commit c4bb3c8d447400573eb45f3e2076e6b80187f1b9 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Feb 12 15:32:50 2014 -0500 Log only one message for dangerous log settings. We log only one message, containing a complete list of what's wrong. We log the complete list whenever any of the possible things that could have gotten wrong gets worse. Fix for #9870. Bugfix on 10480dff01bece13fab, which we merged in 0.2.5.1-alpha. --- changes/bug9870 | 5 +++++ src/or/config.c | 38 +++++++++++++++++++++++++------------- 2 files changed, 30 insertions(+), 13 deletions(-) diff --git a/changes/bug9870 b/changes/bug9870 new file mode 100644 index 0000000..b4b2c2d --- /dev/null +++ b/changes/bug9870 @@ -0,0 +1,5 @@ + o Minor bugfixes: + + - Log only one message when we start logging in an unsafe + way. Previously, we would log as many messages as we had + problems. Fix for #9870; bugfix on 0.2.5.1-alpha. diff --git a/src/or/config.c b/src/or/config.c index d298177..dcbe88d 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -1148,12 +1148,31 @@ options_act_reversible(const or_options_t *old_options, char **msg) tor_free(severity); tor_log_update_sigsafe_err_fds(); } - if (get_min_log_level() >= LOG_INFO && - get_min_log_level() != old_min_log_level) { - log_warn(LD_GENERAL, "Your log may contain sensitive information: you're " - "logging more than \"notice\". Please log safely. Don't log " - "unless it serves an important reason, and overwrite the log " - "afterwards."); + + { + const char *badness = NULL; + int bad_safelog = 0, bad_severity = 0, new_badness = 0; + if (options->SafeLogging_ != SAFELOG_SCRUB_ALL) { + bad_safelog = 1; + if (!old_options || old_options->SafeLogging_ != options->SafeLogging_) + new_badness = 1; + } + if (get_min_log_level() >= LOG_INFO) { + bad_severity = 1; + if (get_min_log_level() != old_min_log_level) + new_badness = 1; + } + if (bad_safelog && bad_severity) + badness = "you disabled SafeLogging, and " + "you're logging more than \"notice\""; + else if (bad_safelog) + badness = "you disabled SafeLogging"; + else + badness = "you're logging more than \"notice\""; + if (new_badness) + log_warn(LD_GENERAL, "Your log may contain sensitive information - %s. " + "Don't log unless it serves an important reason. " + "Overwrite the log afterwards.", badness); } SMARTLIST_FOREACH(replaced_listeners, connection_t *, conn, @@ -1341,13 +1360,6 @@ options_act(const or_options_t *old_options) } #endif - if (options->SafeLogging_ != SAFELOG_SCRUB_ALL && - (!old_options || old_options->SafeLogging_ != options->SafeLogging_)) { - log_warn(LD_GENERAL, "Your log may contain sensitive information - you " - "disabled SafeLogging. Please log safely. Don't log unless it " - "serves an important reason. Overwrite the log afterwards."); - } - if (options->Bridges) { mark_bridge_list(); for (cl = options->Bridges; cl; cl = cl->next) {

1 0

[bridgedb/develop] More unit tests for Email Server
by isis＠torproject.org 01 Apr '14

01 Apr '14

commit 0556e7daa8504756b34384a0f9b5dfa753556c09 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Fri Mar 21 03:19:23 2014 +0000 More unit tests for Email Server --- lib/bridgedb/test/test_EmailServer.py | 160 +++++++++++++++++++++++++++++++++ 1 file changed, 160 insertions(+) diff --git a/lib/bridgedb/test/test_EmailServer.py b/lib/bridgedb/test/test_EmailServer.py index 4c6bcd7..e8b9360 100644 --- a/lib/bridgedb/test/test_EmailServer.py +++ b/lib/bridgedb/test/test_EmailServer.py @@ -17,8 +17,13 @@ import os import shutil from io import StringIO +import copy from bridgedb import EmailServer +from bridgedb.Dist import BadEmail +from bridgedb.Dist import EmailBasedDistributor +from bridgedb.EmailServer import MailContext +from bridgedb.Time import NoSchedule from bridgedb.persistent import Conf from bridgedb.test.util import fileCheckDecorator from twisted.python import log @@ -29,8 +34,36 @@ TEST_CONFIG_FILE = StringIO(unicode("""\ EMAIL_DIST = True EMAIL_GPG_SIGNING_ENABLED = True EMAIL_GPG_SIGNING_KEY = 'TESTING.subkeys.sec' +EMAIL_DOMAIN_MAP = {} +EMAIL_DOMAIN_RULES = { + 'gmail.com': ["ignore_dots", "dkim"], + 'example.com': [], +} +EMAIL_DOMAINS = ["gmail.com", "example.com"] +EMAIL_USERNAME = "bridges" +EMAIL_SMTP_HOST = "127.0.0.1" +EMAIL_SMTP_PORT = 25 +EMAIL_SMTP_FROM_ADDR = "bridges@localhost" +EMAIL_N_BRIDGES_PER_ANSWER = 3 +EMAIL_FROM_ADDR = "bridges@localhost" +EMAIL_BIND_IP = "127.0.0.1" +EMAIL_PORT = 5225 """)) +class FakeDistributor(EmailBasedDistributor): + def __init__(self, key, domainmap, domainrules, answerParameters=None, + bridges=None): + super(FakeDistributor, self).__init__(key, domainmap, domainrules, + answerParameters) + if bridges: + self.bridges = bridges + else: + self.bridges = [] + + def getBridgesForEmail(self, emailaddr, epoch, N=1, + parameters=None, countryCode=None, bridgeFilterRules=None): + return self.bridges[:N] + class EmailGnuPGTest(unittest.TestCase): """Tests for :func:`bridgedb.EmailServer.getGPGContext`.""" @@ -96,3 +129,130 @@ class EmailGnuPGTest(unittest.TestCase): self.makeBadKey() ctx = EmailServer.getGPGContext(self.config) self.assertTrue(ctx is None) + +class EmailCompositionTests(unittest.TestCase): + """Tests for :func:`bridgedb.EmailServer.getMailResponse`.""" + + def setUp(self): + """Create fake email and associated data""" + configuration = {} + TEST_CONFIG_FILE.seek(0) + compiled = compile(TEST_CONFIG_FILE.read(), '<string>', 'exec') + exec compiled in configuration + self.config = Conf(**configuration) + + # TODO: Add headers if we start validating them + self.lines = ["From: %s(a)%s.com", "To: %s(a)example.net", + "Subject: testing", "\n", "get bridges"] + self.distributor = FakeDistributor('key', {}, {}, []) + self.ctx = MailContext(self.config, self.distributor, NoSchedule()) + + def test_getMailResponseNoFrom(self): + lines = self.lines + lines[0] = "" + lines[1] = self.lines[1] % "bridges" + ret = EmailServer.getMailResponse(lines, self.ctx) + self.assertIsInstance(ret, tuple) + self.assertEqual(len(ret), 2) + self.assertEqual(ret[0], None) + self.assertEqual(ret[1], None) + + def test_getMailResponseBadAddress(self): + lines = copy.copy(self.lines) + lines[0] = self.lines[0] % ("testing?", "example") + lines[1] = self.lines[1] % "bridges" + lines[2] = "" + ret = EmailServer.getMailResponse(lines, self.ctx) + self.assertIsInstance(ret, tuple) + self.assertEqual(len(ret), 2) + self.assertEqual(ret[0], None) + self.assertEqual(ret[1], None) + #lines[0] = self.lines[0] % ("<>>", "example") + lines[0] = "From: %s(a)%s.com" % ("<>>", "example") + ret = EmailServer.getMailResponse(lines, self.ctx) + self.assertIsInstance(ret, tuple) + self.assertEqual(len(ret), 2) + self.assertEqual(ret[0], None) + self.assertEqual(ret[1], None) + + def test_getMailResponseInvalidDomain(self): + lines = copy.copy(self.lines) + lines[0] = self.lines[0] % ("testing", "exa#mple") + ret = EmailServer.getMailResponse(lines, self.ctx) + self.assertIsInstance(ret, tuple) + self.assertEqual(len(ret), 2) + self.assertEqual(ret[0], None) + self.assertEqual(ret[1], None) + lines[0] = self.lines[0] % ("testing", "exam+ple") + ret = EmailServer.getMailResponse(lines, self.ctx) + self.assertIsInstance(ret, tuple) + self.assertEqual(len(ret), 2) + self.assertEqual(ret[0], None) + self.assertEqual(ret[1], None) + + def test_getMailResponseDKIM(self): + lines = copy.copy(self.lines) + lines[0] = self.lines[0] % ("testing", "gmail") + lines.append("X-DKIM-Authentication-Result: ") + ret = EmailServer.getMailResponse(lines, self.ctx) + self.assertIsInstance(ret, tuple) + self.assertEqual(len(ret), 2) + self.assertEqual(ret[0], None) + self.assertEqual(ret[1], None) + lines[0] = self.lines[0] % ("testing", "example") + ret = EmailServer.getMailResponse(lines, self.ctx) + self.assertIsInstance(ret, tuple) + self.assertEqual(len(ret), 2) + self.assertEqual(ret[0], "testing(a)example.com") + self.assertIsInstance(ret[1], StringIO) + mail = ret[1].getvalue() + self.assertNotEqual(mail.find("no bridges currently"), -1) + + def test_getMailResponseMailContent(self): + lines = copy.copy(self.lines) + lines[0] = self.lines[0] % ("testing", "example") + lines.append("transport obfs") + ret = EmailServer.getMailResponse(lines, self.ctx) + self.assertIsInstance(ret, tuple) + self.assertEqual(len(ret), 2) + self.assertEqual(ret[0], "testing(a)example.com") + self.assertIsInstance(ret[1], StringIO) + mail = ret[1].getvalue() + self.assertNotEqual(mail.find("no bridges currently"), -1) + lines.append("transport obfs") + lines.append("unblocked webz") + ret = EmailServer.getMailResponse(lines, self.ctx) + self.assertIsInstance(ret, tuple) + self.assertEqual(len(ret), 2) + self.assertEqual(ret[0], "testing(a)example.com") + self.assertIsInstance(ret[1], StringIO) + mail = ret[1].getvalue() + self.assertNotEqual(mail.find("no bridges currently"), -1) + lines.append("ipv6") + ret = EmailServer.getMailResponse(lines, self.ctx) + self.assertIsInstance(ret, tuple) + self.assertEqual(len(ret), 2) + self.assertEqual(ret[0], "testing(a)example.com") + self.assertIsInstance(ret[1], StringIO) + mail = ret[1].getvalue() + self.assertNotEqual(mail.find("no bridges currently"), -1) + +class EmailServerServiceTests(unittest.TestCase): + def setUp(self): + configuration = {} + TEST_CONFIG_FILE.seek(0) + compiled = compile(TEST_CONFIG_FILE.read(), '<string>', 'exec') + exec compiled in configuration + self.config = Conf(**configuration) + + # TODO: Add headers if we start validating them + self.lines = ["From: %s(a)%s.com", "To: %s(a)example.net", + "Subject: testing", "\n", "get bridges"] + self.distributor = FakeDistributor('key', {}, {}, []) + self.ctx = MailContext(self.config, self.distributor, NoSchedule()) + + def test_receiveMail(self): + self.skip = True + raise unittest.SkipTest("Not finished yet") + from twisted.internet import reactor + EmailServer.addSMTPServer(self.config, self.distributor, NoSchedule)

1 0

[bridgedb/develop] Open and convert the db, if necessary. Store DB filename for all.
by isis＠torproject.org 01 Apr '14

01 Apr '14

commit 1bd2c69fcbb8bf1b59b16e3f6d22066b53ff428c Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Wed Mar 26 05:21:00 2014 +0000 Open and convert the db, if necessary. Store DB filename for all. --- lib/bridgedb/Main.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bridgedb/Main.py b/lib/bridgedb/Main.py index 6c099b8..a28b75f 100644 --- a/lib/bridgedb/Main.py +++ b/lib/bridgedb/Main.py @@ -553,9 +553,10 @@ def startup(options): emailDistributorTmp, ipDistributorTmp) = createBridgeRings(cfg, proxyList, key) - # Initialize our DB file. - db = bridgedb.Storage.Database(cfg.DB_FILE + ".sqlite", cfg.DB_FILE) - bridgedb.Storage.setGlobalDB(db) + # Initialize our DB. + bridgedb.Storage.initializeDBLock() + db = bridgedb.Storage.openOrConvertDatabase(cfg.DB_FILE + ".sqlite", cfg.DB_FILE) + bridgedb.Storage.setDBFilename(cfg.DB_FILE + ".sqlite") load(state, splitter, clear=False) state = persistent.load()

1 0

[bridgedb/develop] Handle previously unhandled exception when replying to mail
by isis＠torproject.org 01 Apr '14

01 Apr '14

commit b6d3091e6a3e76e919f96aa246c99b78617e931d Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Wed Mar 26 22:37:12 2014 +0000 Handle previously unhandled exception when replying to mail --- lib/bridgedb/EmailServer.py | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/lib/bridgedb/EmailServer.py b/lib/bridgedb/EmailServer.py index 79239c7..8b7783f 100644 --- a/lib/bridgedb/EmailServer.py +++ b/lib/bridgedb/EmailServer.py @@ -21,6 +21,7 @@ from twisted.internet import reactor from twisted.internet.defer import Deferred from twisted.internet.task import LoopingCall import twisted.mail.smtp +from twisted.internet.error import ConnectionRefusedError from zope.interface import implements @@ -235,6 +236,17 @@ def buildSpamWarningTemplate(t): + t.gettext(I18n.BRIDGEDB_TEXT[12]) + "\n\n" return msg_template +def _ebReplyToMailFailure(fail): + """Errback for a :api:`twisted.mail.smtp.SMTPSenderFactory`. + + :param fail: A :api:`twisted.python.failure.Failure` which occurred during + the transaction. + """ + logging.debug("EmailServer._ebReplyToMailFailure() called with %r" % fail) + error = fail.getErrorMessage() or "unknown failure." + logging.exception("replyToMail Failure: %s" % error) + return None + def replyToMail(lines, ctx): """Reply to an incoming email. Maybe. @@ -262,7 +274,10 @@ def replyToMail(lines, ctx): d = Deferred() factory = twisted.mail.smtp.SMTPSenderFactory(ctx.smtpFromAddr, sendToUser, - response, d) + response, d, retries=0, + timeout=30) + d.addErrback(_ebReplyToMailFailure) + logging.info("Sending reply to %r", Util.logSafely(sendToUser)) reactor.connectTCP(ctx.smtpServer, ctx.smtpPort, factory) return d

1 0

[bridgedb/develop] Add unit tests for bridgedb.Storage
by isis＠torproject.org 01 Apr '14

01 Apr '14

commit 766cc675c52488e658a0884f0e0b5c8b044731b2 Author: Matthew Finkel <Matthew.Finkel(a)gmail.com> Date: Fri Mar 21 02:48:00 2014 +0000 Add unit tests for bridgedb.Storage --- lib/bridgedb/test/test_Storage.py | 56 +++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/lib/bridgedb/test/test_Storage.py b/lib/bridgedb/test/test_Storage.py new file mode 100644 index 0000000..8b0affd --- /dev/null +++ b/lib/bridgedb/test/test_Storage.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python +"""Unittests for the :mod:`bridgedb.Storage` module.""" + +from twisted.python import log +from twisted.trial import unittest +import bridgedb.Storage as Storage +from twisted.internet import reactor +from twisted.internet.threads import deferToThread +import os +import threading +from time import sleep + +class DatabaseTest(unittest.TestCase): + def setUp(self): + self.dbfname = 'test-bridgedb.sqlite' + Storage.setDBFilename(self.dbfname) + + def tearDown(self): + if os.path.isfile(self.dbfname): + os.unlink(self.dbfname) + Storage.clearGlobalDB() + + def _runAndDie(self, timeout, func): + with func(): + sleep(timeout) + + def _cb_assertTrue(self, result): + self.assertTrue(result) + + def _cb_assertFalse(self, result): + self.assertFalse(result) + + def _eb_Failure(self, failure): + self.fail(failure) + + def test_getDB_FalseWhenLocked(self): + Storage._LOCK = threading.Lock() + Storage._LOCK.acquire() + self.assertFalse(Storage._LOCK.acquire(False)) + + def test_getDB_AcquireLock(self): + Storage.initializeDBLock() + with Storage.getDB() as db: + self.assertIsInstance(db, Storage.Database) + self.assertTrue(Storage.dbIsLocked()) + self.assertEqual(db, Storage._OPENED_DB) + + def test_getDB_ConcurrencyLock(self): + timeout = 1 + d1 = deferToThread(self._runAndDie, timeout, Storage.getDB) + d1.addCallback(self._cb_assertFalse) + d1.addErrback(self._eb_Failure) + d2 = deferToThread(Storage.getDB, False) + d2.addCallback(self._cb_assertFalse) + d2.addErrback(self._eb_Failure) + d2.addCallback(self._cb_assertTrue, Storage.getDB(False))

1 0