June 2014 - tor-commits - lists.torproject.org

[bridgedb/master] Update b.e.server.validateTo() docstring.
by isis＠torproject.org 06 Jun '14

06 Jun '14

commit 5827bc9136d66bc5a97dc80995b6079cc5787890 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed May 28 18:18:13 2014 +0000 Update b.e.server.validateTo() docstring. --- lib/bridgedb/email/server.py | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/lib/bridgedb/email/server.py b/lib/bridgedb/email/server.py index cdc1721..b7e90ef 100644 --- a/lib/bridgedb/email/server.py +++ b/lib/bridgedb/email/server.py @@ -736,9 +736,21 @@ class MailDelivery(object): return origin # This method *cannot* return None, or it'll cause a 503. def validateTo(self, user): - """If the local user that was addressed isn't our configured local user - or doesn't contain a '+' with a prefix matching the local configured - user: Yell. + """Validate the SMTP ``RCPT TO:`` address for the incoming connection. + + The local username and domain name to which this SMTP message is + addressed, after being stripped of any ``'+'`` aliases, **must** be + identical to those in the email address set our + ``EMAIL_SMTP_FROM_ADDR`` configuration file option. + + :type user: :api:`twisted.mail.smtp.User` + :param user: Information about the user this SMTP message was + addressed to. + :raises: A :api:`twisted.mail.smtp.SMTPBadRcpt` if any of the above + conditions weren't met. + :rtype: callable + :returns: A parameterless function which returns an instance of + :class:`SMTPMessage`. """ u = user.dest.local # Hasplus? If yes, strip '+foo'

1 0

[bridgedb/master] Fix CI error on EmailServerServiceTests.test_addServer().
by isis＠torproject.org 06 Jun '14

06 Jun '14

commit 28069af03162e36ee19483f46d9101656de16987 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue May 20 21:04:47 2014 +0000 Fix CI error on EmailServerServiceTests.test_addServer(). The SMTP MAIL FROM email address was unacceptable, because on TravisCI it ended up being `localhost(a)localhost.localdomain`. [0] [0]: https://travis-ci.org/isislovecruft/bridgedb/builds/25636170#L1732 --- lib/bridgedb/test/test_email_server.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/bridgedb/test/test_email_server.py b/lib/bridgedb/test/test_email_server.py index 5f92353..ebb9b85 100644 --- a/lib/bridgedb/test/test_email_server.py +++ b/lib/bridgedb/test/test_email_server.py @@ -683,8 +683,9 @@ class EmailServerServiceTests(SMTPTestCaseMixin, unittest.TestCase): self.transport.protocol = self.proto self.proto.makeConnection(self.transport) - self._test(['HELO localhost', - 'MAIL FROM: testing@localhost', + serverHost = socket.gethostname() + self._test(['HELO %s' % serverHost, + 'MAIL FROM: testing@%s' % serverHost, 'RCPT TO: %s' % self.smtpFromAddr, "DATA", self._buildEmail(body="get transport obfs3")], "250 Delivery in progress",

1 0

[bridgedb/master] Update TravisCI coveralls==0.3 to coveralls==0.4.2.
by isis＠torproject.org 06 Jun '14

06 Jun '14

commit 375b1fbfb50fa5bf5f2c2add20be7b1667553125 Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed May 21 01:06:42 2014 +0000 Update TravisCI coveralls==0.3 to coveralls==0.4.2. --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 55dbbb8..e95fb6c 100644 --- a/.travis.yml +++ b/.travis.yml @@ -30,7 +30,7 @@ python: install: - pip install -r requirements.txt - - pip install --no-use-wheel leekspin==0.1.3 coverage==3.7 coveralls==0.3 sure==1.2.2 + - pip install --no-use-wheel leekspin==0.1.3 coverage==3.7 coveralls==0.4.2 sure==1.2.2 - make install script:

1 0

[bridgedb/master] Merge branch 'hotfix/11215_12147-intervalstart' into develop
by isis＠torproject.org 06 Jun '14

06 Jun '14

commit 625cbe8b6fd2153ae23be1c4a78e80a90b0b2348 Merge: da121b3 f73deea Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed May 28 21:03:19 2014 +0000 Merge branch 'hotfix/11215_12147-intervalstart' into develop lib/bridgedb/HTTPServer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

1 0

[bridgedb/master] Move apt install dependencies to 'install' section in .travis.yml.
by isis＠torproject.org 06 Jun '14

06 Jun '14

commit ba6b18f94dcdf8201f26aa9bff976d45607a24ef Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed May 21 01:11:00 2014 +0000 Move apt install dependencies to 'install' section in .travis.yml. --- .travis.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 7281650..4e3db1e 100644 --- a/.travis.yml +++ b/.travis.yml @@ -22,13 +22,12 @@ addons: before_install: - sudo apt-get update - # Distro development header/library/resource/buildtime dependencies - - sudo apt-get install --no-install-suggests --no-install-recommends build-essential openssl sqlite3 libgpgme11 libgpgme11-dev python-dev python-setuptools python: - "2.7" install: + - sudo apt-get install --no-install-suggests --no-install-recommends build-essential openssl sqlite3 libgpgme11 libgpgme11-dev python-dev python-setuptools - pip install -r requirements.txt - pip install --no-use-wheel leekspin==0.1.3 coverage==3.7.1 coveralls==0.4.2 sure==1.2.2 - make install

1 0

[bridgedb/master] Add utility unittest.TestCase mixin class for testing SMTP.
by isis＠torproject.org 06 Jun '14

06 Jun '14

commit be223814812bad96000da37ab7690704214b2f5a Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue May 20 18:20:42 2014 +0000 Add utility unittest.TestCase mixin class for testing SMTP. * ADD a new class, `test_email_server.SMTPTestsMixin`, which has the following methods useful for testing SMTP within `t.t.unittest.TestCase` subclasses: - SMTPTestsMixin._buildEmail(fromAddr=None, toAddr=None, subject=None, body=None) Creates email text (including headers) for use in an SMTP DATA segment. Includes the SMTP DATA EOM command ('.') at the end. If no keyword arguments are given, the defaults are fairly sane. - SMTPTestsMixin._buildSMTP(commands) Format a list of SMTP protocol commands into a string, using the proper protocol delimiter. - SMTPTestsMixin._test(commands, expected) Send the commands to the TestCase's protocol (this must be the `proto` attribute of your `TestCase`), and then check that the expected output matches what was received from `TestCase.transport` (another attribute you must assign). --- lib/bridgedb/test/test_email_server.py | 113 ++++++++++++++++++++++++++++++++ lib/bridgedb/test/util.py | 10 +++ 2 files changed, 123 insertions(+) diff --git a/lib/bridgedb/test/test_email_server.py b/lib/bridgedb/test/test_email_server.py index 23c65b8..30d27f1 100644 --- a/lib/bridgedb/test/test_email_server.py +++ b/lib/bridgedb/test/test_email_server.py @@ -26,9 +26,12 @@ from bridgedb.persistent import Conf from bridgedb.schedule import Unscheduled from bridgedb.test.test_HTTPServer import DummyBridge from bridgedb.test.util import fileCheckDecorator +from bridgedb.test.util import TestCaseMixin from twisted.python import log from twisted.internet import defer +from twisted.internet import reactor +from twisted.test import proto_helpers from twisted.trial import unittest @@ -416,6 +419,116 @@ class MailDeliveryTest(unittest.TestCase): self.delivery.validateTo, user) +class SMTPTestCaseMixin(TestCaseMixin): + """Utility methods for use within any subclasses of + :api:`twisted.trial.unittest.TestCase` which test SMTP. + + To use me, subclass :api:`twisted.trial.unittest.TestCase` and mix me into + the middle of your class inheritance declarations, like so:: + + class ExampleSMTPTests(SMTPTestCaseMixin, unittest.TestCase): + pass + + and then make certain that your ``TestCase`` subclass has its ``proto`` + and attribute assigned properly:: + + class ExampleSMTPTests(SMTPTestCaseMixin, unittest.TestCase): + def setUp(self): + factory = twisted.mail.smtp.SMTPFactory() + self.proto = self.factory.buildProtocol(('127.0.0.1', 0)) + + + :ivar proto: A :api:`Protocol <twisted.internet.protocol.Protocol>` + associated with a + :api:`ServerFactory <twisted.internet.protocol.ServerFactory>`. + :ivar transport: Anything that implements + :api:`ITransport <twisted.internet.interfaces.ITransport>`. The default + is a :api:`twisted.test.proto_helpers.StringTransportWithDisconection`. + :ivar str smtpFromAddr: The default email address for the server. + """ + + proto = None + transport = proto_helpers.StringTransportWithDisconnection() + smtpFromAddr = None + + def tearDown(self): + """Cleanup method after each ``test_*`` method runs; removes timed out + connections on the reactor and clears the :ivar:`transport`. + """ + self.transport.clear() # Clear bytes from the transport. + + for delay in reactor.getDelayedCalls(): + try: + delay.cancel() + except (AlreadyCalled, AlreadyCancelled): + pass + + def _buildEmail(self, fromAddr=None, toAddr=None, subject=None, body=None): + """Creates email text (including headers) for use in an SMTP DATA + segment. Includes the SMTP DATA EOM command ('.') at the end. If no + keyword arguments are given, the defaults are fairly sane. + + Suitable for testing a :class:`bridgedb.email.server.MailFactory`. + + :param str fromAddr: An email address for the 'From:' header. + :param str toAddr: An email address for the 'To:' header. + :param str subject: The contents of the 'Subject:' header. + :param str body: The contents of the email body. + :rtype: str + :returns: The email text. + """ + fromAddr = fromAddr if fromAddr else 'testing@localhost' + toAddr = toAddr if toAddr else self.smtpFromAddr + subject = subject if subject else 'testing testing one two three' + body = body if body else 'get bridges' + + contents = ['From: %s' % fromAddr, + 'To: %s' % toAddr, + 'Subject: %s' % subject, + '\r\n %s' % body, + '.'] # SMTP DATA EOM command + emailText = self.proto.delimiter.join(contents) + return emailText + + def _buildSMTP(self, commands): + """Format a list of SMTP protocol commands into a string, using the proper + protocol delimiter. + + :param list commands: A list of raw SMTP-protocol command lines. + :rtype: str + :returns: The string for sending those **commands**, suitable for + giving to a :api:`twisted.internet.Protocol.dataReceived` method. + """ + data = self.proto.delimiter.join(commands) + self.proto.delimiter + return data + + def _test(self, commands, expected, noisy=False): + """Send the SMTP **commands** to the ``dataReceived`` method of your + TestCase's protocol (this must be the `proto` attribute of your + `TestCase`, i.e. this uses ``TestCase.proto.dataReceived``). Next, + check that the substring which is **expected** to be within the + server's output matches what was received from :ivar`transport`. + + :param list commands: A sequence of raw SMTP command lines. This will + automatically be passed to :meth:`_buildSMTP`. + :param str expected: A substring which should occur in the "server" + output (taken from the :ivar:`transport`). + :param bool noisy: If ``True``, print the conversation between the + "client" and the "server" in a nicely formatted manner. + """ + data = self._buildSMTP(commands) + self.proto.dataReceived(data) + recv = self.transport.value() + + if noisy: + client = data.replace('\r\n', '\r\n ') + server = recv.replace('\r\n', '\r\n\t\t ') + print('\n CLIENT --------->', '\n %s' % client) + print('\t\t', '<--------- SERVER', '\n\t\t %s' % server) + + self.assertSubstring(expected, recv) + + class EmailServerServiceTests(unittest.TestCase): def setUp(self): self.config = _createConfig() diff --git a/lib/bridgedb/test/util.py b/lib/bridgedb/test/util.py index a012597..ca68d1d 100644 --- a/lib/bridgedb/test/util.py +++ b/lib/bridgedb/test/util.py @@ -14,11 +14,14 @@ from __future__ import print_function from __future__ import unicode_literals +import abc import doctest import os from functools import wraps +from twisted.trial import unittest + def fileCheckDecorator(func): """Method decorator for a t.t.unittest.TestCase test_* method. @@ -62,5 +65,12 @@ def fileCheckDecorator(func): % (str(description), dst, src)) return wrapper + +class TestCaseMixin: + """Subclasses of me can be used as mix-in classes with ``TestCase``s.""" + __metaclass__ = abc.ABCMeta +TestCaseMixin.register(unittest.TestCase) + + if __name__ == "__main__": doctest.run_docstring_examples(fileCheckDecorator, None)

1 0

[bridgedb/master] Add unittests for SMTP protocol responses of b.e.s.MailFactory.
by isis＠torproject.org 06 Jun '14

06 Jun '14

commit 6747778b3f29b298c9f8e6488e2599be21aee987 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue May 20 20:14:40 2014 +0000 Add unittests for SMTP protocol responses of b.e.s.MailFactory. These tests emulate, in Twisted, what the `telnet-an-email` [0] script did by speaking raw SMTP over telnet in an embarrassing pexpect script which was never committed for fear being chained in stocks and having rotten tomatoes thrown at me. Instead of that telnet nonsense, these tests use mocked `ITransport`s from `twisted.test.proto_helpers` to create a transport layer for sending SMTP protocol commands. * FIXES part of #9874 by automating some manual portions of the steps required for testing BridgeDB's email distributor. [0]: https://trac.torproject.org/projects/tor/attachment/ticket/5232/telnet-an-e… --- lib/bridgedb/test/test_email_server.py | 122 +++++++++++++++++++++++++++++++- 1 file changed, 121 insertions(+), 1 deletion(-) diff --git a/lib/bridgedb/test/test_email_server.py b/lib/bridgedb/test/test_email_server.py index 717a006..5f92353 100644 --- a/lib/bridgedb/test/test_email_server.py +++ b/lib/bridgedb/test/test_email_server.py @@ -17,6 +17,7 @@ import io import copy import os import shutil +import socket import types from bridgedb.Dist import EmailBasedDistributor @@ -47,8 +48,9 @@ EMAIL_DOMAIN_MAP = { EMAIL_DOMAIN_RULES = { 'gmail.com': ["ignore_dots", "dkim"], 'example.com': [], + 'localhost': [], } -EMAIL_DOMAINS = ["gmail.com", "example.com"] +EMAIL_DOMAINS = ["gmail.com", "example.com", "localhost"] EMAIL_USERNAME = "bridges" EMAIL_SMTP_HOST = "127.0.0.1" EMAIL_SMTP_PORT = 25 @@ -59,6 +61,7 @@ EMAIL_BIND_IP = "127.0.0.1" EMAIL_PORT = 5225 """)) + def _createConfig(configFile=TEST_CONFIG_FILE): configuration = {} TEST_CONFIG_FILE.seek(0) @@ -529,6 +532,123 @@ class SMTPTestCaseMixin(TestCaseMixin): self.assertSubstring(expected, recv) +class MailFactoryTests(SMTPTestCaseMixin, unittest.TestCase): + """Unittests for :class:`bridgedb.email.server.MailFactory`.""" + + def setUp(self): + """Set up a localhost MailFactory handler incoming SMTP connections.""" + config = _createConfig() + context = _createMailContext(config) + factory = server.MailFactory(context) + factory.protocol.timeout = None # Otherwise the reactor gets dirty + + self.smtpFromAddr = context.smtpFromAddr # 'bridges@localhost' + self.proto = factory.buildProtocol(('127.0.0.1', 0)) + self.transport = proto_helpers.StringTransportWithDisconnection() + self.proto.setTimeout(None) + # Set the protocol; StringTransportWithDisconnection is a bit janky: + self.transport.protocol = self.proto + self.proto.makeConnection(self.transport) + + def test_MailFactory_HELO_localhost(self): + """Send 'HELO localhost' to the server's transport.""" + ip = self.transport.getPeer().host + self._test(['HELO localhost'], + "Hello %s, nice to meet you" % ip) + + def test_MailFactory_MAIL_FROM_testing_at_localhost(self): + """Send 'MAIL FROM: human@localhost'.""" + self._test(['HELO localhost', + 'MAIL FROM: testing@localhost'], + "250 Sender address accepted") + + def test_MailFactory_MAIL_FROM_testing_at_gethostname(self): + """Send 'MAIL FROM: human@hostname' for the local hostname.""" + hostname = socket.gethostname() or "computer" + self._test(['HELO localhost', + 'MAIL FROM: testing@%s' % hostname], + "250 Sender address accepted") + + def test_MailFactory_MAIL_FROM_testing_at_ipaddress(self): + """Send 'MAIL FROM: human@ipaddr' for the loopback IP address.""" + hostname = socket.gethostbyname(socket.gethostname()) or "127.0.0.1" + self._test(['HELO localhost', + 'MAIL FROM: testing@%s' % hostname], + "250 Sender address accepted") + + def test_MailFactory_RCPT_TO_config_EMAIL_SMTP_FROM_ADDR(self): + """Send 'RCPT TO:' with the context.smtpFromAddr.""" + self._test(['HELO localhost', + 'MAIL FROM: testing@localhost', + 'RCPT TO: %s' % self.smtpFromAddr], + "250 Recipient address accepted") + + def test_MailFactory_DATA_blank(self): + """A DATA command with nothing after it should receive:: + '354 Continue' + in response. + """ + self._test(['HELO localhost', + 'MAIL FROM: testing@localhost', + 'RCPT TO: %s' % self.smtpFromAddr, + "DATA"], + "354 Continue") + + def test_MailFactory_DATA_get_help(self): + """A DATA command with ``'get help'`` in the email body should + receive:: + '250 Delivery in progress' + in response. + """ + emailText = self._buildEmail(body="get help") + self._test(['HELO localhost', + 'MAIL FROM: testing@localhost', + 'RCPT TO: %s' % self.smtpFromAddr, + "DATA", emailText], + "250 Delivery in progress", + noisy=True) + + def test_MailFactory_DATA_get_transport_obfs3(self): + """A DATA command with ``'get transport obfs3'`` in the email body + should receive:: + '250 Delivery in progress' + in response. + """ + emailText = self._buildEmail(body="get transport obfs3") + self._test(['HELO localhost', + 'MAIL FROM: testing@localhost', + 'RCPT TO: %s' % self.smtpFromAddr, + "DATA", emailText], + "250 Delivery in progress", + noisy=True) + + def test_MailFactory_DATA_To_bridges_plus_zh_CN(self): + """Test sending to 'bridges+zh_CN' address for Chinese translations.""" + # TODO: Add tests which use '+' syntax in mailTo in order to test + # email translations. Do this when some strings have been translated. + emailTo = list(self.smtpFromAddr.partition('@')) + emailTo.insert(1, '+zh_CN') + emailTo = ''.join(emailTo) + emailText = self._buildEmail(toAddr=emailTo) + self._test(['HELO localhost', + 'MAIL FROM: testing@localhost', + 'RCPT TO: %s' % emailTo, + "DATA", emailText], + "250 Delivery in progress", + noisy=True) + + def test_MailFactory_DATA_get_bridges_QUIT(self): + """Test sending 'DATA' with 'get bridges', then sending 'QUIT'.""" + emailText = self._buildEmail() + self._test(['HELO localhost', + 'MAIL FROM: testing@localhost', + 'RCPT TO: %s' % self.smtpFromAddr, + "DATA", emailText, + "QUIT"], + "221 See you later", + noisy=True) + + class EmailServerServiceTests(SMTPTestCaseMixin, unittest.TestCase): """Unittests for :func:`bridgedb.email.server.addServer`."""

1 0

[bridgedb/master] Tell TravisCI to put bridges.torproject.org into /etc/hosts.
by isis＠torproject.org 06 Jun '14

06 Jun '14

commit 0a1f2cbeaedd17d1ecbcba0f9e8943bff2fb59b6 Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue May 20 21:21:06 2014 +0000 Tell TravisCI to put bridges.torproject.org into /etc/hosts. See http://docs.travis-ci.com/user/hosts/ --- .travis.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.travis.yml b/.travis.yml index 09bfe59..55dbbb8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -16,6 +16,10 @@ notifications: on_success: never on_failure: change +addons: + hosts: + - bridges.torproject.org + before_install: - sudo apt-get update # Distro development header/library/resource/buildtime dependencies

1 0

[bridgedb/master] Merge branch 'hotfix/0.2.1-docstrings' into develop
by isis＠torproject.org 06 Jun '14

06 Jun '14

commit e2de6afb4332452664b8ff7dbc707c3384ae7bf6 Merge: 625cbe8 c053bff Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed May 28 21:18:43 2014 +0000 Merge branch 'hotfix/0.2.1-docstrings' into develop bridgedb.conf | 2 +- lib/bridgedb/email/server.py | 86 ++++++++++++++++++++++++++---------------- 2 files changed, 55 insertions(+), 33 deletions(-)

1 0

[bridgedb/master] Update coverage==3.7 to coverage==3.7.1 for TravisCI and setup.py.
by isis＠torproject.org 06 Jun '14

06 Jun '14

commit 12e42c89a41189d01874de84939360c7df4ac0bf Author: Isis Lovecruft <isis(a)torproject.org> Date: Wed May 21 01:08:16 2014 +0000 Update coverage==3.7 to coverage==3.7.1 for TravisCI and setup.py. --- .travis.yml | 2 +- setup.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index e95fb6c..7281650 100644 --- a/.travis.yml +++ b/.travis.yml @@ -30,7 +30,7 @@ python: install: - pip install -r requirements.txt - - pip install --no-use-wheel leekspin==0.1.3 coverage==3.7 coveralls==0.4.2 sure==1.2.2 + - pip install --no-use-wheel leekspin==0.1.3 coverage==3.7.1 coveralls==0.4.2 sure==1.2.2 - make install script: diff --git a/setup.py b/setup.py index fbb6602..4654415 100644 --- a/setup.py +++ b/setup.py @@ -284,7 +284,7 @@ setuptools.setup( 'bridgedb.test'], scripts=['scripts/bridgedb'], extras_require={'test': ["sure==1.2.2", - "coverage==3.7", + "coverage==3.7.1", "leekspin==0.1.3"]}, zip_safe=False, cmdclass=get_cmdclass(),

1 0