February 2020 - tor-commits - lists.torproject.org

[bridgedb/master] Fix next item iteration
by phw＠torproject.org 19 Feb '20

19 Feb '20

commit bd2ee8d0fca1ab311f9644ff826a430b30234453 Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Jan 11 16:26:19 2020 -0800 Fix next item iteration Python 3.x replaced the generator's next() method with a next() builtin... https://stackoverflow.com/questions/21622193/python-3-2-coroutine-attribute… No significant difference in test outcome... before: FAILED (skips=106, failures=16, errors=273, successes=409) after: FAILED (… [View More]

1 0

[bridgedb/master] Drop unicode argument from gettext's install
by phw＠torproject.org 19 Feb '20

19 Feb '20

commit 61e82530f027463b164f9181ab3de1db08bd6668 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Jan 13 16:10:54 2020 -0800 Drop unicode argument from gettext's install Python 3.x is unicode by default and gettext doesn't have this argument any longer so simply dropping it... https://docs.python.org/3/library/gettext.html#gettext.NullTranslations.ins… This fixes... Traceback (most recent call last): File "/home/atagar/… [View More]

1 0

[bridgedb/master] Fix translation tests
by phw＠torproject.org 19 Feb '20

19 Feb '20

commit 2e9a582e7124462d297bceb352dc179a64ca0639 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Jan 13 17:49:50 2020 -0800 Fix translation tests Adjustments so test_translations.py passes. There were a few issues, most notably... * Python's map() builtin returns a list in 2.x and generator in 3.x. Generators are more efficient to work with, but when indexing into the object we must first convert it into a list. … [View More]Traceback (most recent call last): File "/home/atagar/Desktop/tor/bridgedb/bridgedb/test/test_translations.py", line 48, in test_getLocaleFromHTTPRequest_withLangParam self.assertEqual(parsed[0], 'ar') builtins.TypeError: 'map' object is not subscriptable * BridgeDB's getFirstSupportedLang returns 'en-US' by default when no supported language is present. That isn't a language due to its '-US' suffix, causing the following... Traceback (most recent call last): File "/home/atagar/Desktop/tor/bridgedb/bridgedb/test/test_translations.py", line 85, in test_usingRTLLang self.assertTrue(translations.usingRTLLang(['fa'])) File "/home/atagar/Desktop/tor/bridgedb/bridgedb/translations.py", line 150, in usingRTLLang print("text_direction: %s" % babel.core.Locale.parse(lang).text_direction) File "/usr/local/lib/python3.5/dist-packages/babel/core.py", line 268, in parse parts = parse_locale(identifier, sep=sep) File "/usr/local/lib/python3.5/dist-packages/babel/core.py", line 1094, in parse_locale raise ValueError('expected only letters, got %r' % lang) builtins.ValueError: expected only letters, got 'en-us' * Test expects 'fa' to be a supported language. BridgeDB's "_langs.get_langs()" returns an empty list for me. Possibly a setup issue, but even if it was this should better explain what the test needs. Skipping this assertion if setup didn't enable farsi. Test results changed as follows... before: FAILED (skips=106, failures=17, errors=258, successes=423) after: FAILED (skips=106, failures=17, errors=256, successes=425) --- bridgedb/test/test_translations.py | 12 ++++++++---- bridgedb/translations.py | 19 +++++++++++++------ 2 files changed, 21 insertions(+), 10 deletions(-) diff --git a/bridgedb/test/test_translations.py b/bridgedb/test/test_translations.py index 7d14cc0..939511d 100644 --- a/bridgedb/test/test_translations.py +++ b/bridgedb/test/test_translations.py @@ -10,7 +10,7 @@ from twisted.trial import unittest -from bridgedb import translations +from bridgedb import _langs, translations from bridgedb.test.https_helpers import DummyRequest @@ -45,10 +45,11 @@ class TranslationsMiscTests(unittest.TestCase): }) parsed = translations.getLocaleFromHTTPRequest(request) + + self.assertEqual(len(parsed), 3) self.assertEqual(parsed[0], 'ar') self.assertEqual(parsed[1], 'en') self.assertEqual(parsed[2], 'en_US') - self.assertEqual(len(parsed), 3) def test_getLocaleFromHTTPRequest_withLangParam_AcceptLanguage(self): """This request uses a '?lang=ar' param, with an 'Accept-Language' @@ -61,11 +62,12 @@ class TranslationsMiscTests(unittest.TestCase): request.args.update({b'lang': [b'fa']}) parsed = translations.getLocaleFromHTTPRequest(request) + + self.assertEqual(len(parsed), 3) self.assertEqual(parsed[0], 'fa') self.assertEqual(parsed[1], 'en') self.assertEqual(parsed[2], 'en_US') #self.assertEqual(parsed[3], 'en-gb') - self.assertEqual(len(parsed), 3) def test_getLocaleFromPlusAddr(self): emailAddr = 'bridges(a)torproject.org' @@ -80,4 +82,6 @@ class TranslationsMiscTests(unittest.TestCase): def test_usingRTLLang(self): self.assertFalse(translations.usingRTLLang(['foo_BAR'])) self.assertFalse(translations.usingRTLLang(['en'])) - self.assertTrue(translations.usingRTLLang(['fa'])) + + if 'fa' in _langs.get_langs(): + self.assertTrue(translations.usingRTLLang(['fa'])) diff --git a/bridgedb/translations.py b/bridgedb/translations.py index 10c27fc..194fa8a 100644 --- a/bridgedb/translations.py +++ b/bridgedb/translations.py @@ -58,7 +58,10 @@ def getFirstSupportedLang(langs): if l in supported: lang = l break - return lang + + # crop locales (like 'en-US') to just the language + + return lang.split('-')[0] def getLocaleFromHTTPRequest(request): """Retrieve the languages from an HTTP ``Accept-Language:`` header. @@ -77,16 +80,20 @@ def getLocaleFromHTTPRequest(request): logging.debug("Client sent no 'Accept-Language' header. Using fallback.") header = 'en,en-US' - langs = headers.parseAcceptLanguage(header) + langs = list(headers.parseAcceptLanguage(header)) if not safelog.safe_logging: # pragma: no cover logging.debug("Client Accept-Language (top 5): %s" % langs[:5]) # Check if we got a ?lang=foo argument, and if we did, insert it first - chosenLang = request.args.get("lang", [None,])[0] + chosenLang = request.args.get(b"lang", [None,])[0] if chosenLang: logging.debug("Client requested language: %r" % chosenLang) langs.insert(0, chosenLang) + # normalize languages to be unicode + + langs = list(map(lambda l: l if isinstance(l, str) else l.decode('utf-8'), langs)) + installTranslations(langs) return langs @@ -140,11 +147,11 @@ def usingRTLLang(langs): :returns: ``True`` if the preferred language is right-to-left; ``False`` otherwise. """ + lang = getFirstSupportedLang(langs) - rtl = False try: - rtl = babel.core.Locale.parse(lang).text_direction == "rtl" + return babel.core.Locale.parse(lang).text_direction == "rtl" except ValueError as err: logging.warning("Couldn't parse locale %s: %s" % (lang, err)) - return rtl + return False [View Less]

1 0

[bridgedb/master] Drop unicode argument from gettext.install()
by phw＠torproject.org 19 Feb '20

19 Feb '20

commit 0ac0e46a2afaa67cbf6719770e4fff592f148496 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Jan 13 19:02:30 2020 -0800 Drop unicode argument from gettext.install() Oops, I removed this deprecated argument from gettext.translation's install method but I missed that there were similar calls to this function as well. This changes the test results as follows... before: FAILED (skips=109, failures=18, errors=383, successes=470) after:… [View More]

1 0

[bridgedb/master] Error concatenating range with a list
by phw＠torproject.org 19 Feb '20

19 Feb '20

commit f76c06102fe83417dc7c391c9c614f7fd54ff8c0 Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Jan 15 17:19:08 2020 -0800 Error concatenating range with a list BridgeDB relies on catch-alls to address its underlying buginess, which is making this harder. Removed one to uncover our http_server's actual exception... Traceback (most recent call last): File "/home/atagar/Desktop/tor/bridgedb/bridgedb/test/test_https_server.py", line 740, … [View More]

1 0

[bridgedb/master] Normalize getHMAC inputs
by phw＠torproject.org 19 Feb '20

19 Feb '20

commit 7ac4b8ea8605470371d78ba6e3c58731c9549c57 Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Jan 14 14:29:42 2020 -0800 Normalize getHMAC inputs Calculating the hmac requires bytes. This fixes... Traceback (most recent call last): File "/home/atagar/Desktop/tor/bridgedb/bridgedb/test/test_https_distributor.py", line 152, in test_HTTPSDistributor_prepopulateRings_without_proxies dist = distributor.HTTPSDistributor(3, self.key) … [View More] File "/home/atagar/Desktop/tor/bridgedb/bridgedb/distributors/https/distributor.py", line 93, in __init__ key2 = getHMAC(key, "Assign-Bridges-To-Rings") File "/home/atagar/Desktop/tor/bridgedb/bridgedb/crypto.py", line 193, in getHMAC h = hmac.new(key, value, digestmod=DIGESTMOD) File "/usr/lib/python3.5/hmac.py", line 144, in new return HMAC(key, msg, digestmod) File "/usr/lib/python3.5/hmac.py", line 42, in __init__ raise TypeError("key: expected bytes or bytearray, but got %r" % type(key).__name__) builtins.TypeError: key: expected bytes or bytearray, but got 'str' Test results changed as follows... before: FAILED (skips=109, failures=22, errors=318, successes=531) after: FAILED (skips=109, failures=21, errors=283, successes=567) --- bridgedb/crypto.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/bridgedb/crypto.py b/bridgedb/crypto.py index b39c659..bd3d783 100644 --- a/bridgedb/crypto.py +++ b/bridgedb/crypto.py @@ -190,6 +190,12 @@ def getKey(filename): def getHMAC(key, value): """Return the HMAC of **value** using the **key**.""" + + # normalize inputs to be bytes + + key = key.encode('utf-8') if isinstance(key, str) else key + value = value.encode('utf-8') if isinstance(value, str) else value + h = hmac.new(key, value, digestmod=DIGESTMOD) return h.digest() @@ -200,14 +206,19 @@ def getHMACFunc(key, hex=True): :rtype: callable :returns: A function which can be uses to generate HMACs. """ + + key = key.encode('utf-8') if isinstance(key, str) else key h = hmac.new(key, digestmod=DIGESTMOD) + def hmac_fn(value): + value = value.encode('utf-8') if isinstance(value, str) else value h_tmp = h.copy() h_tmp.update(value) if hex: return h_tmp.hexdigest() else: return h_tmp.digest() + return hmac_fn def removePKCS1Padding(message): [View Less]

1 0

[bridgedb/master] Replace string.letters with string.ascii_letters
by phw＠torproject.org 19 Feb '20

19 Feb '20

commit 6d9b2a675d515c370a60730d6d9c91d32decd038 Author: Damian Johnson <atagar(a)torproject.org> Date: Fri Jan 17 12:06:25 2020 -0800 Replace string.letters with string.ascii_letters Python 3 renamed that constant. This fixes... Traceback (most recent call last): File "/home/atagar/Desktop/tor/bridgedb/bridgedb/test/test_parse_nickname.py", line 37, in test_parse_nickname_isValidRouterNickname_invalid_symbols self.assertFalse(… [View More]

1 0

[bridgedb/master] Replace rfc822 with email.message
by phw＠torproject.org 19 Feb '20

19 Feb '20

commit 1138daa8e1202d1649a70c7b238ac7ebcade23a8 Author: Damian Johnson <atagar(a)torproject.org> Date: Fri Jan 10 16:13:01 2020 -0800 Replace rfc822 with email.message Hardest part of this migration yet. Python 2's deprecated rfc8222 module was removed in python 3. Unfortunately Isis used a catch-all here, covering the fact that this code has always been a bit broken (due to overriding our 'addr' import with a local variable)... Traceback (most … [View More]recent call last): File "/home/atagar/Desktop/tor/bridgedb/bridgedb/test/test_email_autoresponder.py", line 353, in test_SMTPAutoresponder_getMailFrom_givemebridges_at_seriously recipient = str(self.responder.getMailFrom()) File "/home/atagar/Desktop/tor/bridgedb/bridgedb/distributors/email/autoresponder.py", line 535, in getMailFrom raise addr.BadEmail(allRecipients) builtins.AttributeError: 'str' object has no attribute 'BadEmail' Finally the single test that exercises this code path works... before: FAILED (skips=109, failures=24, errors=225, successes=623) after: FAILED (skips=109, failures=23, errors=225, successes=624) --- bridgedb/distributors/email/autoresponder.py | 19 ++++++++++--------- bridgedb/distributors/email/server.py | 12 ++++-------- bridgedb/test/test_email_dkim.py | 4 ++-- bridgedb/test/test_email_server.py | 6 +++--- 4 files changed, 19 insertions(+), 22 deletions(-) diff --git a/bridgedb/distributors/email/autoresponder.py b/bridgedb/distributors/email/autoresponder.py index 1d2b618..3711eae 100644 --- a/bridgedb/distributors/email/autoresponder.py +++ b/bridgedb/distributors/email/autoresponder.py @@ -39,6 +39,7 @@ Functionality for autoresponding to incoming emails. from __future__ import unicode_literals from __future__ import print_function +import email import io import logging import time @@ -439,8 +440,8 @@ class SMTPAutoresponder(smtp.SMTPClient): if not body: return # The client was already warned. - messageID = self.incoming.message.getheader("Message-ID", None) - subject = self.incoming.message.getheader("Subject", None) + messageID = self.incoming.message.get("Message-ID", None) + subject = self.incoming.message.get("Subject", None) response = generateResponse(recipient, client, body, subject, messageID, self.incoming.context.gpgSignFunc) @@ -461,13 +462,13 @@ class SMTPAutoresponder(smtp.SMTPClient): """ clients = [] addrHeader = None - try: fromAddr = self.incoming.message.getaddr("From")[1] + try: fromAddr = email.utils.parseaddr(self.incoming.message.get("From"))[1] except (IndexError, TypeError, AttributeError): pass else: addrHeader = fromAddr if not addrHeader: logging.warn("No From header on incoming mail.") - try: senderHeader = self.incoming.message.getaddr("Sender")[1] + try: senderHeader = email.utils.parseaddr(self.incoming.message.get("Sender"))[1] except (IndexError, TypeError, AttributeError): pass else: addrHeader = senderHeader if not addrHeader: @@ -509,10 +510,10 @@ class SMTPAutoresponder(smtp.SMTPClient): try: ourAddress = smtp.Address(self.incoming.context.fromAddr) - allRecipients = self.incoming.message.getaddrlist("To") + allRecipients = self.incoming.message.get_all("To") - for _, addr in allRecipients: - recipient = smtp.Address(addr) + for address in allRecipients: + recipient = smtp.Address(address) if not ourAddress.domain in recipient.domain: logging.debug(("Not our domain (%s) or subdomain, skipping" " email address: %s") @@ -525,11 +526,11 @@ class SMTPAutoresponder(smtp.SMTPClient): " email address: %s") % str(recipient)) continue # Only check the username before the first '+': - beforePlus = recipient.local.split('+', 1)[0] + beforePlus = recipient.local.split(b'+', 1)[0] if beforePlus == ourAddress.local: ours = str(recipient) if not ours: - raise addr.BadEmail(allRecipients) + raise addr.BadEmail('No email address accepted, please see log', allRecipients) except Exception as error: logging.error(("Couldn't find our email address in incoming email " diff --git a/bridgedb/distributors/email/server.py b/bridgedb/distributors/email/server.py index 0f96812..8a2fbea 100644 --- a/bridgedb/distributors/email/server.py +++ b/bridgedb/distributors/email/server.py @@ -49,10 +49,10 @@ Servers which interface with clients and distribute bridges over SMTP. from __future__ import unicode_literals +import email.message import logging import io import socket -import rfc822 from twisted.internet import defer from twisted.internet import reactor @@ -222,7 +222,7 @@ class SMTPMessage(object): if self.nBytes > self.context.maximumSize: self.ignoring = True else: - self.lines.append(line) + self.lines.append(line.decode('utf-8') if isinstance(line, bytes) else line) if not safelog.safe_logging: try: ln = line.rstrip("\r\n").encode('utf-8', 'replace') @@ -251,12 +251,8 @@ class SMTPMessage(object): :rtype: :api:`twisted.mail.smtp.rfc822.Message` :returns: A ``Message`` comprised of all lines received thus far. """ - rawMessage = io.StringIO() - for line in self.lines: - line = line.decode('utf-8') if isinstance(line, bytes) else line - rawMessage.writelines(line + '\n') - rawMessage.seek(0) - return rfc822.Message(rawMessage) + + return email.message_from_string('\n'.join(self.lines)) @implementer(smtp.IMessageDelivery) diff --git a/bridgedb/test/test_email_dkim.py b/bridgedb/test/test_email_dkim.py index e8fe3f3..2330a71 100644 --- a/bridgedb/test/test_email_dkim.py +++ b/bridgedb/test/test_email_dkim.py @@ -11,8 +11,8 @@ """Unittests for the :mod:`bridgedb.distributors.email.dkim` module.""" +import email.message import io -import rfc822 from twisted.trial import unittest @@ -47,7 +47,7 @@ get bridges def _createMessage(self, messageString): """Create an ``email.message.Message`` from a string.""" - messageIO = io.StringIO(messageString if isinstance(messageString, str) else messageString.decode('utf-8')) + messageIO = io.StringIO(unicode(messageString)) return rfc822.Message(messageIO) def test_checkDKIM_good(self): diff --git a/bridgedb/test/test_email_server.py b/bridgedb/test/test_email_server.py index bc331f4..081a0e4 100644 --- a/bridgedb/test/test_email_server.py +++ b/bridgedb/test/test_email_server.py @@ -13,10 +13,10 @@ from __future__ import print_function +import email.message import socket import string import types -import rfc822 from twisted.python import log from twisted.internet import defer @@ -95,10 +95,10 @@ class SMTPMessageTests(unittest.TestCase): defer.Deferred) def test_SMTPMessage_getIncomingMessage(self): - """``getIncomingMessage`` should return a ``rfc822.Message``.""" + """``getIncomingMessage`` should return a ``email.message.Message``.""" self.message.lineReceived(self.line) self.assertIsInstance(self.message.getIncomingMessage(), - rfc822.Message) + email.message.Message) class SMTPIncomingDeliveryTests(unittest.TestCase): [View Less]

1 0

[bridgedb/master] Syntax error due to long integer syntax
by phw＠torproject.org 19 Feb '20

19 Feb '20

commit 61472aee2f50cd9b2b02e4045f04ceee44ea4970 Author: Damian Johnson <atagar(a)torproject.org> Date: Fri Jan 17 12:24:12 2020 -0800 Syntax error due to long integer syntax Python 3 no longer has a separate type for long integers (all numbers are covered by ints). This fixes... Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/twisted/trial/runner.py", line 823, in loadByName return self.suiteFactory([self.… [View More]

1 0

[bridgedb/master] Fix hex conversion
by phw＠torproject.org 19 Feb '20

19 Feb '20

commit fd4b1bbaeea1791cf69bbbfdd89ce7bc6e111088 Author: Damian Johnson <atagar(a)torproject.org> Date: Fri Jan 17 12:47:39 2020 -0800 Fix hex conversion Replacing hex encoding with a python 3 equivalent... Python 2: >>> 'hello'.encode('hex') '68656c6c6f' Python 3: >>> import binascii >>> binascii.hexlify('hello'.encode('utf-8')) b'68656c6c6f' This fixes... … [View More]

1 0