tor-commits March 2017

tor-commits@lists.torproject.org

20 participants
1183 discussions

[stem/master] Note ed25519 certificate support on our site
by atagar＠torproject.org 30 Mar '17

30 Mar '17

commit db601c3a701acb99daded402ba0f3df55b097728 Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Mar 29 21:20:09 2017 -0700 Note ed25519 certificate support on our site --- docs/api.rst | 1 + docs/api/descriptor/certificate.rst | 5 +++++ docs/change_log.rst | 1 + docs/contents.rst | 1 + 4 files changed, 8 insertions(+) diff --git a/docs/api.rst b/docs/api.rst index 208450b..0644d0b 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -37,6 +37,7 @@ remotely like Tor does. * `stem.descriptor.router_status_entry <api/descriptor/router_status_entry.html>`_ - Relay entries within a network status document. * `stem.descriptor.hidden_service_descriptor <api/descriptor/hidden_service_descriptor.html>`_ - Descriptors generated for hidden services. * `stem.descriptor.tordnsel <api/descriptor/tordnsel.html>`_ - `TorDNSEL <https://www.torproject.org/projects/tordnsel.html.en>`_ exit lists. + * `stem.descriptor.tordnsel <api/descriptor/certificate.html>`_ - `Ed25519 certificates <https://gitweb.torproject.org/torspec.git/tree/cert-spec.txt>`_. * `stem.descriptor.reader <api/descriptor/reader.html>`_ - Reads and parses descriptor files from disk. * `stem.descriptor.remote <api/descriptor/remote.html>`_ - Downloads descriptors from directory mirrors and authorities. diff --git a/docs/api/descriptor/certificate.rst b/docs/api/descriptor/certificate.rst new file mode 100644 index 0000000..f4c2aa3 --- /dev/null +++ b/docs/api/descriptor/certificate.rst @@ -0,0 +1,5 @@ +Certificate +=========== + +.. automodule:: stem.descriptor.certificate + diff --git a/docs/change_log.rst b/docs/change_log.rst index e7993d1..bad162b 100644 --- a/docs/change_log.rst +++ b/docs/change_log.rst @@ -51,6 +51,7 @@ The following are only available within Stem's `git repository * **Descriptors** + * Support and validation for `ed25519 certificates <api/descriptor/certificate.html>`_ (`spec <https://gitweb.torproject.org/torspec.git/tree/cert-spec.txt>`_, :trac:`21558`) * Moved from the deprecated `pycrypto <https://www.dlitz.net/software/pycrypto/>`_ module to `cryptography <https://pypi.python.org/pypi/cryptography>`_ for validating signatures (:trac:`21086`) * Sped descriptor reading by ~25% by deferring defaulting when validating * Support for protocol descriptor fields (:spec:`eb4fb3c`) diff --git a/docs/contents.rst b/docs/contents.rst index 930d4cf..5cbd5bf 100644 --- a/docs/contents.rst +++ b/docs/contents.rst @@ -37,6 +37,7 @@ Contents api/manual api/version + api/descriptor/certificate api/descriptor/descriptor api/descriptor/server_descriptor api/descriptor/extrainfo_descriptor

1 0

[stem/master] Add Ed25519 certificate extension support
by atagar＠torproject.org 30 Mar '17

30 Mar '17

commit 3ce6631670cf3a0ffb5d44736bd426c3eaa82b5d Author: Patrick O'Doherty <p(a)trickod.com> Date: Sun Sep 18 17:24:02 2016 -0700 Add Ed25519 certificate extension support Parse the identity-ed25519 certificate block and validate the extension contents in accordance with prop #220. Validates the router-sig-ed25519 signature against the provided certified key. Start of work to verify onion-key-crosscert blocks --- requirements.txt | 1 + stem/descriptor/__init__.py | 12 +- stem/descriptor/certificate.py | 198 ++++++++++++++++++++++++++++++ stem/descriptor/server_descriptor.py | 29 +++++ stem/prereq.py | 21 ++++ test/unit/descriptor/server_descriptor.py | 12 ++ 6 files changed, 272 insertions(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 6dc054c..3cd7160 100644 --- a/requirements.txt +++ b/requirements.txt @@ -3,3 +3,4 @@ pyflakes pycodestyle tox cryptography +pynacl diff --git a/stem/descriptor/__init__.py b/stem/descriptor/__init__.py index 6e8c5a4..4e90889 100644 --- a/stem/descriptor/__init__.py +++ b/stem/descriptor/__init__.py @@ -636,7 +636,17 @@ class Descriptor(object): raise ValueError("Digest is for the range ending with '%s' but that isn't in our descriptor" % end) digest_content = raw_descriptor[start_index:end_index + len(end)] - digest_hash = hashlib.sha1(stem.util.str_tools._to_bytes(digest_content)) + return self._digest_for_bytes(digest_content) + + def _digest_for_bytes(self, bytes_to_sign): + """ + Provides a digest of the provided bytes + + :param bytes bytes_to_sign: the bytes for which we should generate a digest + + :returns: the digest string encoded in uppercase hex + """ + digest_hash = hashlib.sha1(bytes_to_sign) return stem.util.str_tools._to_unicode(digest_hash.hexdigest().upper()) def __getattr__(self, name): diff --git a/stem/descriptor/certificate.py b/stem/descriptor/certificate.py new file mode 100644 index 0000000..e26992f --- /dev/null +++ b/stem/descriptor/certificate.py @@ -0,0 +1,198 @@ +# Copyright 2016, Patrick O'Doherty and The Tor Project +# See LICENSE for licensing information + +""" +Parsing for the Tor server descriptor Ed25519 Certificates, which is used to +validate the Ed25519 key used to sign the relay descriptor. + +Certificates can optionally contain CertificateExtension objects depending on their type and purpose. Currently Ed25519KeyCertificate certificates will contain one SignedWithEd25519KeyCertificateExtensio + + +**Module Overview:** + +:: + + Certificate - Tor Certificate + |- Ed25519KeyCertificate - Certificate for Ed25519 signing key + +- +- verify_descriptor_signature - verify a relay descriptor against a signature + + + CertificateExtension - Certificate extension + +- - SignedWithEd25519KeyCertificateExtension - Ed25519 signing key extension +""" + +import base64 +import hashlib +import time +from collections import OrderedDict + +import stem.util.str_tools + +import nacl.signing +from nacl.exceptions import BadSignatureError + + +SIGNATURE_LENGTH = 64 +STANDARD_ATTRIBUTES_LENGTH = 40 +CERTIFICATE_FLAGS_LENGTH = 4 +ED25519_ROUTER_SIGNATURE_PREFIX = 'Tor router descriptor signature v1' + + +def _bytes_to_long(b): + return long(b.encode('hex'), 16) + + +def _parse_long_offset(offset, length): + def _parse(raw_contents): + return _bytes_to_long(raw_contents[offset:(offset + length)]) + + return _parse + + +def _parse_offset(offset, length): + def _parse(raw_contents): + return raw_contents[offset:(offset + length)] + + return _parse + + +def _parse_certificate(raw_contents, master_key_bytes, validate = False): + version, cert_type = raw_contents[0:2] + + if version == '\x01': + if cert_type == '\x04': + return Ed25519KeyCertificate(raw_contents, master_key_bytes, validate = validate) + elif cert_type == '\x05': + # TLS link certificated signed with ed25519 signing key + pass + elif cert_type == '\x06': + # Ed25519 authentication signed with ed25519 signing key + pass + else: + raise ValueError("Unknown Certificate type %s" % cert_type.encode('hex')) + else: + raise ValueError("Unknown Certificate version %s" % version.encode('hex')) + + +def _parse_extensions(raw_contents): + n_extensions = _bytes_to_long(raw_contents[39:40]) + if n_extensions == 0: + return [] + + extensions = [] + extension_bytes = raw_contents[STANDARD_ATTRIBUTES_LENGTH:-SIGNATURE_LENGTH] + while len(extension_bytes) > 0: + ext_length = _bytes_to_long(extension_bytes[0:2]) + ext_type, ext_flags = extension_bytes[2:CERTIFICATE_FLAGS_LENGTH] + try: + ext_data = extension_bytes[CERTIFICATE_FLAGS_LENGTH:(CERTIFICATE_FLAGS_LENGTH + ext_length)] + except: + raise ValueError('Certificate contained truncated extension') + + if ext_type == SignedWithEd25519KeyCertificateExtension.TYPE: + extension = SignedWithEd25519KeyCertificateExtension(ext_type, ext_flags, ext_data) + else: + raise ValueError('Invalid certificate extension type: %s' % ext_type.encode('hex')) + + extensions.append(extension) + extension_bytes = extension_bytes[CERTIFICATE_FLAGS_LENGTH + ext_length:] + + if len(extensions) != n_extensions: + raise ValueError('n_extensions was %d but parsed %d' % (n_extensions, len(extensions))) + + return extensions + + +def _parse_signature(cert): + return cert[-SIGNATURE_LENGTH:] + + +class Certificate(object): + """ + See proposal #220 <https://gitweb.torproject.org/torspec.git/tree/proposals/220-ecc-id-keys.txt> + """ + + ATTRIBUTES = { + 'version': _parse_offset(0, 1), + 'cert_type': _parse_offset(1, 1), + 'expiration_date': _parse_long_offset(2, 4), + 'cert_key_type': _parse_offset(6, 1), + 'certified_key': _parse_offset(7, 32), + 'n_extensions': _parse_long_offset(39, 1), + 'extensions': _parse_extensions, + 'signature': _parse_signature + } + + def __init__(self, raw_contents, identity_key, validate = False): + self.certificate_bytes = raw_contents + self.identity_key = identity_key + + self.__set_certificate_entries(raw_contents) + + def __set_certificate_entries(self, raw_contents): + entries = OrderedDict() + for key, func in Certificate.ATTRIBUTES.iteritems(): + try: + entries[key] = func(raw_contents) + except IndexError: + raise ValueError('Unable to get bytes for %s from certificate' % key) + + for key, value in entries.iteritems(): + setattr(self, key, value) + + +class Ed25519KeyCertificate(Certificate): + def __init__(self, raw_contents, identity_key, validate = False): + super(Ed25519KeyCertificate, self).__init__(raw_contents, identity_key, validate = False) + + if validate: + if len(self.extensions) == 0: + raise ValueError('Ed25519KeyCertificate missing SignedWithEd25519KeyCertificateExtension extension') + + self._verify_signature() + + if (self.expiration_date * 3600) < int(time.time()): + raise ValueError('Expired Ed25519KeyCertificate') + + def verify_descriptor_signature(self, descriptor, signature): + missing_padding = len(signature) % 4 + signature_bytes = base64.b64decode(stem.util.str_tools._to_bytes(signature) + b'=' * missing_padding) + verify_key = nacl.signing.VerifyKey(self.certified_key) + + signed_part = descriptor[:descriptor.index('router-sig-ed25519 ') + len('router-sig-ed25519 ')] + descriptor_with_prefix = ED25519_ROUTER_SIGNATURE_PREFIX + signed_part + descriptor_sha256_digest = hashlib.sha256(descriptor_with_prefix).digest() + verify_key.verify(descriptor_sha256_digest, signature_bytes) + + def _verify_signature(self): + if self.identity_key: + verify_key = nacl.signing.VerifyKey(base64.b64decode(self.identity_key + '=')) + else: + verify_key = nacl.singing.VerifyKey(self.extensions[0].ext_data) + + try: + verify_key.verify(self.certificate_bytes[:-SIGNATURE_LENGTH], self.signature) + except BadSignatureError: + raise ValueError('Ed25519KeyCertificate signature invalid') + + +class CertificateExtension(object): + KNOWN_TYPES = ['\x04'] + + def __init__(self, ext_type, ext_flags, ext_data): + self.ext_type = ext_type + self.ext_flags = ext_flags + self.ext_data = ext_data + + def is_known_type(self): + return self.ext_type in CertificateExtension.KNOWN_TYPES + + def affects_validation(self): + return self.ext_flags == '\x01' + + +class SignedWithEd25519KeyCertificateExtension(CertificateExtension): + TYPE = '\x04' + + def __init__(self, ext_type, ext_flags, ext_data): + super(SignedWithEd25519KeyCertificateExtension, self).__init__(ext_type, ext_flags, ext_data) diff --git a/stem/descriptor/server_descriptor.py b/stem/descriptor/server_descriptor.py index 3aa5fb0..c5bcaad 100644 --- a/stem/descriptor/server_descriptor.py +++ b/stem/descriptor/server_descriptor.py @@ -62,6 +62,8 @@ from stem.descriptor import ( _parse_key_block, ) +from stem.descriptor.certificate import _parse_certificate + try: # added in python 3.2 from functools import lru_cache @@ -662,6 +664,14 @@ class ServerDescriptor(Descriptor): if expected_last_keyword and expected_last_keyword != list(entries.keys())[-1]: raise ValueError("Descriptor must end with a '%s' entry" % expected_last_keyword) + if 'identity-ed25519' in entries.keys(): + if not 'router-sig-ed25519' in entries.keys(): + raise ValueError("Descriptor must have router-sig-ed25519 entry to accompany identity-ed25519") + + if 'router-sig-ed25519' != list(entries.keys())[-2]: + if 'router-sig-ed25519' != list(entries.keys())[-1]: + raise ValueError("Descriptor must end with a 'router-sig-ed25519' entry") + if not self.exit_policy: raise ValueError("Descriptor must have at least one 'accept' or 'reject' entry") @@ -750,6 +760,25 @@ class RelayDescriptor(ServerDescriptor): if signed_digest != self.digest(): raise ValueError('Decrypted digest does not match local digest (calculated: %s, local: %s)' % (signed_digest, self.digest())) + if stem.prereq.is_nacl_available() and self.ed25519_certificate: + self.certificate = _parse_certificate(_bytes_for_block(self.ed25519_certificate), + self.ed25519_master_key, + validate) + + if self.ed25519_master_key is not None: + if self.certificate.identity_key != self.ed25519_master_key: + raise ValueError("master-key-ed25519 does not match ed25519 certificate identity key") + + self.certificate.verify_descriptor_signature(stem.util.str_tools._to_unicode(raw_contents), + self.ed25519_signature) + + onion_key_bytes = _bytes_for_block(self.onion_key) + from Crypto.Util import asn1 + seq = asn1.DerSequence() + seq.decode(onion_key_bytes) + self._digest_for_signature(self.onion_key, self.onion_key_crosscert) + + @lru_cache() def digest(self): """ diff --git a/stem/prereq.py b/stem/prereq.py index 585b619..e8769a3 100644 --- a/stem/prereq.py +++ b/stem/prereq.py @@ -15,6 +15,7 @@ Checks for stem dependencies. We require python 2.6 or greater (including the check_requirements - checks for minimum requirements for running stem is_python_3 - checks if python 3.0 or later is available is_crypto_available - checks if the cryptography module is available + is_nacl_available - checks if the pynacl module is available """ import inspect @@ -27,6 +28,7 @@ except ImportError: from stem.util.lru_cache import lru_cache CRYPTO_UNAVAILABLE = "Unable to import the cryptography module. Because of this we'll be unable to verify descriptor signature integrity. You can get cryptography from: https://pypi.python.org/pypi/cryptography" +NACL_UNAVAILABLE = "Unable to import the pynacl module. Because of this we'll be unable to verify descriptor ed25519 certificate integrity. You can get pynacl from https://github.com/pyca/pynacl/" def check_requirements(): @@ -146,3 +148,22 @@ def is_mock_available(): return True except ImportError: return False + +@lru_cache() +def is_nacl_available(): + """ + Checks if the pynacl functions we use are available. This is used for + verifying ed25519 certificates in relay descriptor signatures. + + :returns: **True** if we can use pynacl and **False** otherwise + """ + + from stem.util import log + + try: + from nacl import encoding + from nacl import signing + return True + except ImportError: + log.log_once('stem.prereq.is_nacl_available', log.INFO, NACL_UNAVAILABLE) + return False diff --git a/test/unit/descriptor/server_descriptor.py b/test/unit/descriptor/server_descriptor.py index ba8271d..c4e9b67 100644 --- a/test/unit/descriptor/server_descriptor.py +++ b/test/unit/descriptor/server_descriptor.py @@ -3,6 +3,7 @@ Unit tests for stem.descriptor.server_descriptor. """ import datetime +import time import io import pickle import tarfile @@ -245,6 +246,7 @@ Qlx9HNCqCY877ztFRC624ja2ql6A2hBcuoYMbkHjcQ4= self.assertTrue(isinstance(str(desc), str)) + @patch('time.time', Mock(return_value = time.mktime(datetime.date(2010, 1, 1).timetuple()))) def test_with_ed25519(self): """ Parses a descriptor with a ed25519 identity key, as added by proposal 228 @@ -299,6 +301,16 @@ Qlx9HNCqCY877ztFRC624ja2ql6A2hBcuoYMbkHjcQ4= self.assertEqual('B5E441051D139CCD84BC765D130B01E44DAC29AD', desc.digest()) self.assertEqual([], desc.get_unrecognized_lines()) + @patch('time.time', Mock(return_value = time.mktime(datetime.date(2020, 1, 1).timetuple()))) + def test_with_ed25519_expired_cert(self): + """ + Parses a server descriptor with an expired ed25519 certificate + """ + desc_text = open(get_resource('bridge_descriptor_with_ed25519'), 'rb').read() + desc_iter = stem.descriptor.server_descriptor._parse_file(io.BytesIO(desc_text), validate = True) + self.assertRaises(ValueError, list, desc_iter) + + def test_bridge_with_ed25519(self): """ Parses a bridge descriptor with ed25519.

1 0

[stem/master] stem.descriptor.certificate unit tests
by atagar＠torproject.org 30 Mar '17

30 Mar '17

commit 311c4d7bab40c3f3bb80c9e52a217a390e635aab Author: Patrick O'Doherty <p(a)trickod.com> Date: Tue Feb 7 20:23:49 2017 -0800 stem.descriptor.certificate unit tests Unit tests to ensure that invalid certificate data raises the appropriately descriptive ValueError when using the _parse_certificate and _parse_extensions functions provided by stem.descriptor.certificate --- stem/descriptor/certificate.py | 6 +- test/settings.cfg | 1 + test/unit/descriptor/certificate.py | 108 ++++++++++++++++++++++++++++++++++++ 3 files changed, 112 insertions(+), 3 deletions(-) diff --git a/stem/descriptor/certificate.py b/stem/descriptor/certificate.py index 63ebace..ff78538 100644 --- a/stem/descriptor/certificate.py +++ b/stem/descriptor/certificate.py @@ -5,7 +5,7 @@ Parsing for the Tor server descriptor Ed25519 Certificates, which is used to validate the Ed25519 key used to sign the relay descriptor. -Certificates can optionally contain CertificateExtension objects depending on their type and purpose. Currently Ed25519KeyCertificate certificates will contain one SignedWithEd25519KeyCertificateExtensio +Certificates can optionally contain CertificateExtension objects depending on their type and purpose. Currently Ed25519KeyCertificate certificates will contain one SignedWithEd25519KeyCertificateExtension **Module Overview:** @@ -82,9 +82,9 @@ def _parse_extensions(raw_contents): extensions = [] extension_bytes = raw_contents[STANDARD_ATTRIBUTES_LENGTH:-SIGNATURE_LENGTH] while len(extension_bytes) > 0: - ext_length = _bytes_to_long(extension_bytes[0:2]) - ext_type, ext_flags = extension_bytes[2:CERTIFICATE_FLAGS_LENGTH] try: + ext_length = _bytes_to_long(extension_bytes[0:2]) + ext_type, ext_flags = extension_bytes[2:CERTIFICATE_FLAGS_LENGTH] ext_data = extension_bytes[CERTIFICATE_FLAGS_LENGTH:(CERTIFICATE_FLAGS_LENGTH + ext_length)] except: raise ValueError('Certificate contained truncated extension') diff --git a/test/settings.cfg b/test/settings.cfg index c8172f6..558487d 100644 --- a/test/settings.cfg +++ b/test/settings.cfg @@ -190,6 +190,7 @@ test.unit_tests |test.unit.descriptor.networkstatus.document_v3.TestNetworkStatusDocument |test.unit.descriptor.networkstatus.bridge_document.TestBridgeNetworkStatusDocument |test.unit.descriptor.hidden_service_descriptor.TestHiddenServiceDescriptor +|test.unit.descriptor.certificate.TestCertificate |test.unit.exit_policy.rule.TestExitPolicyRule |test.unit.exit_policy.policy.TestExitPolicy |test.unit.version.TestVersion diff --git a/test/unit/descriptor/certificate.py b/test/unit/descriptor/certificate.py new file mode 100644 index 0000000..450685f --- /dev/null +++ b/test/unit/descriptor/certificate.py @@ -0,0 +1,108 @@ +""" +Unit tests for stem.descriptor.certificate. +""" + +import unittest + +import stem.descriptor.certificate + +import nacl.signing +import nacl.encoding + + +class TestCertificate(unittest.TestCase): + + def test_with_invalid_version(self): + cert_bytes = '\x02\x04' + self.assertRaisesRegexp( + ValueError, + 'Unknown Certificate version', + stem.descriptor.certificate._parse_certificate, + cert_bytes, + None + ) + + def test_with_invalid_type(self): + cert_bytes = '\x01\x07' + self.assertRaisesRegexp( + ValueError, + 'Unknown Certificate type', + stem.descriptor.certificate._parse_certificate, + cert_bytes, + None + ) + + def test_parse_extensions_truncated_extension(self): + cert_bytes = '\x00' * 39 # First 40 bytes are standard fields + cert_bytes += '\x01' # n_extensions = 1 + cert_bytes += '\x00\x08' # extension length = 8 bytes + cert_bytes += stem.descriptor.certificate.SIGNATURE_LENGTH * '\x00' # pad empty signature block + + self.assertRaisesRegexp( + ValueError, + 'Certificate contained truncated extension', + stem.descriptor.certificate._parse_extensions, + cert_bytes + ) + + def test_parse_extensions_invalid_certificate_extension_type(self): + cert_bytes = '\x00' * 39 # First 40 bytes are standard fields + cert_bytes += '\x01' # n_extensions = 1 + cert_bytes += '\x00\x08' # extension length = 8 bytes + cert_bytes += '\x00' * 6 # pad out to 8 bytes + cert_bytes += stem.descriptor.certificate.SIGNATURE_LENGTH * '\x00' # pad empty signature block + + self.assertRaisesRegexp( + ValueError, + 'Invalid certificate extension type:', + stem.descriptor.certificate._parse_extensions, + cert_bytes + ) + + def test_parse_extensions_invalid_n_extensions_count(self): + cert_bytes = '\x00' * 39 # First 40 bytes are standard fields + cert_bytes += '\x02' # n_extensions = 2 + cert_bytes += '\x00\x08' # extension length = 8 bytes + cert_bytes += '\x04' # certificate type + cert_bytes += '\x00' * 5 # pad out to 8 bytes + cert_bytes += stem.descriptor.certificate.SIGNATURE_LENGTH * '\x00' # pad empty signature block + + self.assertRaisesRegexp( + ValueError, + 'n_extensions was 2 but parsed 1', + stem.descriptor.certificate._parse_extensions, + cert_bytes + ) + + def test_ed25519_key_certificate_without_extensions(self): + cert_bytes = '\x01\x04' + '\x00' * 37 # First 40 bytes are standard fields + cert_bytes += '\x00' # n_extensions = 0 + cert_bytes += stem.descriptor.certificate.SIGNATURE_LENGTH * '\x00' # pad empty signature block + + self.assertRaisesRegexp( + ValueError, + 'Ed25519KeyCertificate missing SignedWithEd25519KeyCertificateExtension extension', + stem.descriptor.certificate._parse_certificate, + cert_bytes, + None, + validate = True + ) + + def test_certificate_with_invalid_signature(self): + master_key = nacl.signing.SigningKey.generate() + master_key_base64 = master_key.encode(nacl.encoding.Base64Encoder) + + cert_bytes = '\x01\x04' + '\x00' * 37 # 40 byte preamble of standard fields + cert_bytes += '\x01' # n_extensions = 1 + cert_bytes += '\x00\x08' # extentsion length = 8 bytes + cert_bytes += '\x04' + '\x00' * 5 # certificate type + padding out to 8 bytes + cert_bytes += stem.descriptor.certificate.SIGNATURE_LENGTH * '\x00' # empty signature block + + self.assertRaisesRegexp( + ValueError, + 'Ed25519KeyCertificate signature invalid', + stem.descriptor.certificate._parse_certificate, + cert_bytes, + master_key_base64, + validate = True + )

1 0

[stem/master] Note soft pynacl dependency in faq
by atagar＠torproject.org 30 Mar '17

30 Mar '17

commit bdab91ad27f15e2fbf800590db0b6af92ba54bf2 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Feb 27 16:14:53 2017 -0800 Note soft pynacl dependency in faq --- docs/faq.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/faq.rst b/docs/faq.rst index 69033d4..50718be 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -53,7 +53,8 @@ Does Stem have any dependencies? **No.** All you need in order to use Stem is Python. When it is available Stem will use `cryptography -<https://pypi.python.org/pypi/cryptography>`_ to validate descriptor signatures. +<https://pypi.python.org/pypi/cryptography>`_ and `PyNaCl +<https://pypi.python.org/pypi/PyNaCl/>`_ to validate descriptor signatures. However, there is no need to install cryptography unless you need this functionality. @@ -72,6 +73,7 @@ cryptography with... % sudo apt-get install python-dev % sudo pip install cryptography + % sudo pip install pynacl .. _what_python_versions_is_stem_compatible_with:

1 0

[stem/master] Note pynacl version when running tests
by atagar＠torproject.org 30 Mar '17

30 Mar '17

commit 9071c587f82df3158dc33059685cf0dec3551df0 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Feb 27 16:29:57 2017 -0800 Note pynacl version when running tests Including our pynacl version in our testing header. This greatly helps when troubleshooting testing failures from others. --- run_tests.py | 1 + test/util.py | 9 +++++++++ 2 files changed, 10 insertions(+) diff --git a/run_tests.py b/run_tests.py index bceade3..fc6d906 100755 --- a/run_tests.py +++ b/run_tests.py @@ -161,6 +161,7 @@ def main(): tor_version_check, Task('checking python version', test.util.check_python_version), Task('checking cryptography version', test.util.check_cryptography_version), + Task('checking pynacl version', test.util.check_pynacl_version), Task('checking mock version', test.util.check_mock_version), Task('checking pyflakes version', test.util.check_pyflakes_version), Task('checking pycodestyle version', test.util.check_pycodestyle_version), diff --git a/test/util.py b/test/util.py index ce1f8e8..18765ef 100644 --- a/test/util.py +++ b/test/util.py @@ -23,6 +23,7 @@ Tasks are... |- check_tor_version - checks our version of tor |- check_python_version - checks our version of python |- check_cryptography_version - checks our version of cryptography + |- check_pynacl_version - checks our version of pynacl |- check_pyflakes_version - checks our version of pyflakes |- check_pycodestyle_version - checks our version of pycodestyle |- clean_orphaned_pyc - removes any *.pyc without a corresponding *.py @@ -222,6 +223,14 @@ def check_cryptography_version(): return 'missing' +def check_pynacl_version(): + if stem.prereq._is_pynacl_available(): + import nacl + return nacl.__version__ + else: + return 'missing' + + def check_mock_version(): if stem.prereq.is_mock_available(): try:

1 0

[stem/master] Correct stylistic issues
by atagar＠torproject.org 30 Mar '17

30 Mar '17

commit 54ea347242f0966240fc0d072983e636b0f7caab Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Feb 27 15:49:22 2017 -0800 Correct stylistic issues Handful of things spotted by pycodestyle. --- stem/descriptor/server_descriptor.py | 15 ++++++--------- stem/prereq.py | 1 + test/settings.cfg | 4 ++++ test/unit/descriptor/server_descriptor.py | 1 - 4 files changed, 11 insertions(+), 10 deletions(-) diff --git a/stem/descriptor/server_descriptor.py b/stem/descriptor/server_descriptor.py index 7d134dd..a8a1ba2 100644 --- a/stem/descriptor/server_descriptor.py +++ b/stem/descriptor/server_descriptor.py @@ -666,8 +666,8 @@ class ServerDescriptor(Descriptor): raise ValueError("Descriptor must end with a '%s' entry" % expected_last_keyword) if 'identity-ed25519' in entries.keys(): - if not 'router-sig-ed25519' in entries.keys(): - raise ValueError("Descriptor must have router-sig-ed25519 entry to accompany identity-ed25519") + if 'router-sig-ed25519' not in entries.keys(): + raise ValueError('Descriptor must have router-sig-ed25519 entry to accompany identity-ed25519') if 'router-sig-ed25519' != list(entries.keys())[-2]: if 'router-sig-ed25519' != list(entries.keys())[-1]: @@ -772,13 +772,11 @@ class RelayDescriptor(ServerDescriptor): validate) if self.certificate.identity_key != self.ed25519_master_key: - raise ValueError("master-key-ed25519 does not match ed25519 certificate identity key") + raise ValueError('master-key-ed25519 does not match ed25519 certificate identity key') self.certificate.verify_descriptor_signature(raw_contents, self.ed25519_signature) - - @lru_cache() def digest(self): """ @@ -791,7 +789,6 @@ class RelayDescriptor(ServerDescriptor): return self._digest_for_content(b'router ', b'\nrouter-signature\n') - @lru_cache() def onion_key_crosscert_digest(self): """ @@ -804,10 +801,10 @@ class RelayDescriptor(ServerDescriptor): :raises: ValueError if the digest cannot be calculated """ - signing_key_digest = hashlib.sha1(_bytes_for_block(self.signing_key)).digest() - data = signing_key_digest + base64.b64decode(self.ed25519_master_key + b'=') - return data.encode("hex").upper() + signing_key_digest = hashlib.sha1(_bytes_for_block(self.signing_key)).digest() + data = signing_key_digest + base64.b64decode(self.ed25519_master_key + b'=') + return data.encode('hex').upper() def _compare(self, other, method): if not isinstance(other, RelayDescriptor): diff --git a/stem/prereq.py b/stem/prereq.py index e8769a3..5d6e80a 100644 --- a/stem/prereq.py +++ b/stem/prereq.py @@ -149,6 +149,7 @@ def is_mock_available(): except ImportError: return False + @lru_cache() def is_nacl_available(): """ diff --git a/test/settings.cfg b/test/settings.cfg index 558487d..c953773 100644 --- a/test/settings.cfg +++ b/test/settings.cfg @@ -152,6 +152,10 @@ pyflakes.ignore stem/prereq.py => 'modes' imported but unused pyflakes.ignore stem/prereq.py => 'Cipher' imported but unused pyflakes.ignore stem/prereq.py => 'algorithms' imported but unused pyflakes.ignore stem/prereq.py => 'unittest' imported but unused +pyflakes.ignore stem/prereq.py => 'unittest.mock' imported but unused +pyflakes.ignore stem/prereq.py => 'long_to_bytes' imported but unused +pyflakes.ignore stem/prereq.py => 'encoding' imported but unused +pyflakes.ignore stem/prereq.py => 'signing' imported but unused pyflakes.ignore stem/interpreter/__init__.py => undefined name 'raw_input' pyflakes.ignore stem/util/conf.py => undefined name 'unicode' pyflakes.ignore stem/util/test_tools.py => 'pyflakes' imported but unused diff --git a/test/unit/descriptor/server_descriptor.py b/test/unit/descriptor/server_descriptor.py index c4e9b67..8170532 100644 --- a/test/unit/descriptor/server_descriptor.py +++ b/test/unit/descriptor/server_descriptor.py @@ -310,7 +310,6 @@ Qlx9HNCqCY877ztFRC624ja2ql6A2hBcuoYMbkHjcQ4= desc_iter = stem.descriptor.server_descriptor._parse_file(io.BytesIO(desc_text), validate = True) self.assertRaises(ValueError, list, desc_iter) - def test_bridge_with_ed25519(self): """ Parses a bridge descriptor with ed25519.

1 0

[stem/master] Fix Python3 bytes type issue using Cryptography.
by atagar＠torproject.org 30 Mar '17

30 Mar '17

commit 23ab3005664b951af6135f18472cbccb178fc988 Author: Paras Chetal <paras.chetal(a)gmail.com> Date: Sat Mar 4 14:33:50 2017 -0800 Fix Python3 bytes type issue using Cryptography. Signed-off-by: Patrick O'Doherty <p(a)trickod.com> --- stem/descriptor/hidden_service_descriptor.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stem/descriptor/hidden_service_descriptor.py b/stem/descriptor/hidden_service_descriptor.py index ce3a134..dc42286 100644 --- a/stem/descriptor/hidden_service_descriptor.py +++ b/stem/descriptor/hidden_service_descriptor.py @@ -326,7 +326,7 @@ class HiddenServiceDescriptor(Descriptor): # try decrypting the session key - cipher = Cipher(algorithms.AES(authentication_cookie), modes.CTR('\x00' * len(iv)), default_backend()) + cipher = Cipher(algorithms.AES(authentication_cookie), modes.CTR(stem.util.str_tools._to_bytes('\x00' * len(iv))), default_backend()) decryptor = cipher.decryptor() session_key = decryptor.update(encrypted_session_key) + decryptor.finalize()

1 0

[stem/master] Fix Python 3 compatibility issues & broken unit test.
by atagar＠torproject.org 30 Mar '17

30 Mar '17

commit e0eab2e3bce1dd404c7435b920485f2781146c90 Author: Patrick O'Doherty <p(a)trickod.com> Date: Sat Mar 11 19:02:45 2017 -0800 Fix Python 3 compatibility issues & broken unit test. Fix a number of exceptions caused by mixing str and bytes types, which while OK in Python 2 cause runtime exceptions in Python 3. Fix broken truncated extension parsing test. Not quite sure how this test ever failed, as python's slices (in both 2 and 3) always truncate instead of raising exceptions for out of bounds indices. --- stem/descriptor/certificate.py | 57 +++++++++++++++++----------- stem/descriptor/hidden_service_descriptor.py | 2 +- stem/descriptor/server_descriptor.py | 5 ++- test/unit/descriptor/certificate.py | 15 ++++---- test/unit/manual.py | 3 ++ 5 files changed, 49 insertions(+), 33 deletions(-) diff --git a/stem/descriptor/certificate.py b/stem/descriptor/certificate.py index 8904b58..09cf1cf 100644 --- a/stem/descriptor/certificate.py +++ b/stem/descriptor/certificate.py @@ -22,6 +22,7 @@ contain one SignedWithEd25519KeyCertificateExtension. """ import base64 +import binascii import hashlib import time @@ -37,11 +38,14 @@ except ImportError: SIGNATURE_LENGTH = 64 STANDARD_ATTRIBUTES_LENGTH = 40 CERTIFICATE_FLAGS_LENGTH = 4 -ED25519_ROUTER_SIGNATURE_PREFIX = 'Tor router descriptor signature v1' +ED25519_ROUTER_SIGNATURE_PREFIX = b'Tor router descriptor signature v1' def _bytes_to_long(b): - return long(b.encode('hex'), 16) + if stem.prereq.is_python_3(): + return int(binascii.hexlify(stem.util.str_tools._to_bytes(b)), 16) + else: + return long(binascii.hexlify(b), 16) def _parse_long_offset(offset, length): @@ -59,21 +63,22 @@ def _parse_offset(offset, length): def _parse_certificate(raw_contents, master_key_bytes, validate = False): - version, cert_type = raw_contents[0:2] + version = raw_contents[0:1] + cert_type = raw_contents[1:2] - if version == '\x01': - if cert_type == '\x04': + if version == b'\x01': + if cert_type == b'\x04': return Ed25519KeyCertificate(raw_contents, master_key_bytes, validate = validate) - elif cert_type == '\x05': + elif cert_type == b'\x05': # TLS link certificated signed with ed25519 signing key pass - elif cert_type == '\x06': + elif cert_type == b'\x06': # Ed25519 authentication signed with ed25519 signing key pass else: - raise ValueError("Unknown Certificate type %s" % cert_type.encode('hex')) + raise ValueError("Unknown Certificate type %s" % binascii.hexlify(cert_type)) else: - raise ValueError("Unknown Certificate version %s" % version.encode('hex')) + raise ValueError("Unknown Certificate version %s" % binascii.hexlify(version)) def _parse_extensions(raw_contents): @@ -83,18 +88,19 @@ def _parse_extensions(raw_contents): extensions = [] extension_bytes = raw_contents[STANDARD_ATTRIBUTES_LENGTH:-SIGNATURE_LENGTH] + while len(extension_bytes) > 0: - try: - ext_length = _bytes_to_long(extension_bytes[0:2]) - ext_type, ext_flags = extension_bytes[2:CERTIFICATE_FLAGS_LENGTH] - ext_data = extension_bytes[CERTIFICATE_FLAGS_LENGTH:(CERTIFICATE_FLAGS_LENGTH + ext_length)] - except: + ext_length = _bytes_to_long(extension_bytes[0:2]) + ext_type = extension_bytes[2:3] + ext_flags = extension_bytes[3:CERTIFICATE_FLAGS_LENGTH] + ext_data = extension_bytes[CERTIFICATE_FLAGS_LENGTH:(CERTIFICATE_FLAGS_LENGTH + ext_length)] + if len(ext_type) == 0 or len(ext_flags) == 0 or len(ext_data) == 0: raise ValueError('Certificate contained truncated extension') if ext_type == SignedWithEd25519KeyCertificateExtension.TYPE: extension = SignedWithEd25519KeyCertificateExtension(ext_type, ext_flags, ext_data) else: - raise ValueError('Invalid certificate extension type: %s' % ext_type.encode('hex')) + raise ValueError('Invalid certificate extension type: %s' % binascii.hexlify(ext_type)) extensions.append(extension) extension_bytes = extension_bytes[CERTIFICATE_FLAGS_LENGTH + ext_length:] @@ -127,19 +133,23 @@ class Certificate(object): def __init__(self, raw_contents, identity_key, validate = False): self.certificate_bytes = raw_contents - self.identity_key = identity_key + + if type(identity_key) == bytes: + self.identity_key = stem.util.str_tools._to_unicode(identity_key) + else: + self.identity_key = identity_key self.__set_certificate_entries(raw_contents) def __set_certificate_entries(self, raw_contents): entries = OrderedDict() - for key, func in Certificate.ATTRIBUTES.iteritems(): + for key, func in Certificate.ATTRIBUTES.items(): try: entries[key] = func(raw_contents) except IndexError: raise ValueError('Unable to get bytes for %s from certificate' % key) - for key, value in entries.iteritems(): + for key, value in entries.items(): setattr(self, key, value) @@ -167,7 +177,7 @@ class Ed25519KeyCertificate(Certificate): signature_bytes = base64.b64decode(stem.util.str_tools._to_bytes(signature) + b'=' * missing_padding) verify_key = nacl.signing.VerifyKey(self.certified_key) - signed_part = descriptor[:descriptor.index('router-sig-ed25519 ') + len('router-sig-ed25519 ')] + signed_part = descriptor[:descriptor.index(b'router-sig-ed25519 ') + len('router-sig-ed25519 ')] descriptor_with_prefix = ED25519_ROUTER_SIGNATURE_PREFIX + signed_part descriptor_sha256_digest = hashlib.sha256(descriptor_with_prefix).digest() @@ -181,10 +191,11 @@ class Ed25519KeyCertificate(Certificate): raise ValueError('Certificate validation requires the pynacl module') import nacl.signing + import nacl.encoding from nacl.exceptions import BadSignatureError if self.identity_key: - verify_key = nacl.signing.VerifyKey(base64.b64decode(self.identity_key + '=')) + verify_key = nacl.signing.VerifyKey(self.identity_key + '=', encoder=nacl.encoding.Base64Encoder) else: verify_key = nacl.singing.VerifyKey(self.extensions[0].ext_data) @@ -195,7 +206,7 @@ class Ed25519KeyCertificate(Certificate): class CertificateExtension(object): - KNOWN_TYPES = ['\x04'] + KNOWN_TYPES = [b'\x04'] def __init__(self, ext_type, ext_flags, ext_data): self.ext_type = ext_type @@ -206,11 +217,11 @@ class CertificateExtension(object): return self.ext_type in CertificateExtension.KNOWN_TYPES def affects_validation(self): - return self.ext_flags == '\x01' + return self.ext_flags == b'\x01' class SignedWithEd25519KeyCertificateExtension(CertificateExtension): - TYPE = '\x04' + TYPE = b'\x04' def __init__(self, ext_type, ext_flags, ext_data): super(SignedWithEd25519KeyCertificateExtension, self).__init__(ext_type, ext_flags, ext_data) diff --git a/stem/descriptor/hidden_service_descriptor.py b/stem/descriptor/hidden_service_descriptor.py index dc42286..9c87500 100644 --- a/stem/descriptor/hidden_service_descriptor.py +++ b/stem/descriptor/hidden_service_descriptor.py @@ -326,7 +326,7 @@ class HiddenServiceDescriptor(Descriptor): # try decrypting the session key - cipher = Cipher(algorithms.AES(authentication_cookie), modes.CTR(stem.util.str_tools._to_bytes('\x00' * len(iv))), default_backend()) + cipher = Cipher(algorithms.AES(authentication_cookie), modes.CTR(b'\x00' * len(iv)), default_backend()) decryptor = cipher.decryptor() session_key = decryptor.update(encrypted_session_key) + decryptor.finalize() diff --git a/stem/descriptor/server_descriptor.py b/stem/descriptor/server_descriptor.py index a695966..9cd709b 100644 --- a/stem/descriptor/server_descriptor.py +++ b/stem/descriptor/server_descriptor.py @@ -35,6 +35,7 @@ import functools import hashlib import re import base64 +import binascii import stem.descriptor.extrainfo_descriptor import stem.exit_policy @@ -800,8 +801,8 @@ class RelayDescriptor(ServerDescriptor): """ signing_key_digest = hashlib.sha1(_bytes_for_block(self.signing_key)).digest() - data = signing_key_digest + base64.b64decode(self.ed25519_master_key + b'=') - return data.encode('hex').upper() + data = signing_key_digest + base64.b64decode(self.ed25519_master_key + '=') + return stem.util.str_tools._to_unicode(binascii.hexlify(data).upper()) def _compare(self, other, method): if not isinstance(other, RelayDescriptor): diff --git a/test/unit/descriptor/certificate.py b/test/unit/descriptor/certificate.py index 61f6068..4e71e84 100644 --- a/test/unit/descriptor/certificate.py +++ b/test/unit/descriptor/certificate.py @@ -11,7 +11,7 @@ import test.runner class TestCertificate(unittest.TestCase): def test_with_invalid_version(self): - cert_bytes = '\x02\x04' + cert_bytes = b'\x02\x04' self.assertRaisesRegexp( ValueError, 'Unknown Certificate version', @@ -21,7 +21,7 @@ class TestCertificate(unittest.TestCase): ) def test_with_invalid_type(self): - cert_bytes = '\x01\x07' + cert_bytes = b'\x01\x07' self.assertRaisesRegexp( ValueError, 'Unknown Certificate type', @@ -34,13 +34,14 @@ class TestCertificate(unittest.TestCase): cert_bytes = '\x00' * 39 # First 40 bytes are standard fields cert_bytes += '\x01' # n_extensions = 1 cert_bytes += '\x00\x08' # extension length = 8 bytes + cert_bytes += '\x04' # ext_type = 0x04 cert_bytes += stem.descriptor.certificate.SIGNATURE_LENGTH * '\x00' # pad empty signature block self.assertRaisesRegexp( ValueError, 'Certificate contained truncated extension', stem.descriptor.certificate._parse_extensions, - cert_bytes + stem.util.str_tools._to_bytes(cert_bytes) ) def test_parse_extensions_invalid_certificate_extension_type(self): @@ -54,7 +55,7 @@ class TestCertificate(unittest.TestCase): ValueError, 'Invalid certificate extension type:', stem.descriptor.certificate._parse_extensions, - cert_bytes + stem.util.str_tools._to_bytes(cert_bytes) ) def test_parse_extensions_invalid_n_extensions_count(self): @@ -69,7 +70,7 @@ class TestCertificate(unittest.TestCase): ValueError, 'n_extensions was 2 but parsed 1', stem.descriptor.certificate._parse_extensions, - cert_bytes + stem.util.str_tools._to_bytes(cert_bytes) ) def test_ed25519_key_certificate_without_extensions(self): @@ -81,7 +82,7 @@ class TestCertificate(unittest.TestCase): ValueError, 'Ed25519KeyCertificate missing SignedWithEd25519KeyCertificateExtension extension', stem.descriptor.certificate._parse_certificate, - cert_bytes, + stem.util.str_tools._to_bytes(cert_bytes), None, validate = True ) @@ -107,7 +108,7 @@ class TestCertificate(unittest.TestCase): ValueError, 'Ed25519KeyCertificate signature invalid', stem.descriptor.certificate._parse_certificate, - cert_bytes, + stem.util.str_tools._to_bytes(cert_bytes), master_key_base64, validate = True ) diff --git a/test/unit/manual.py b/test/unit/manual.py index 5f939c3..671ed93 100644 --- a/test/unit/manual.py +++ b/test/unit/manual.py @@ -173,6 +173,9 @@ class TestManual(unittest.TestCase): self.assertEqual(EXPECTED_CLI_OPTIONS, manual.commandline_options) self.assertEqual(EXPECTED_SIGNALS, manual.signals) self.assertEqual(EXPECTED_FILES, manual.files) + if EXPECTED_CONFIG_OPTIONS != manual.config_options: + print(EXPECTED_CONFIG_OPTIONS) + print(manual.config_options) self.assertEqual(EXPECTED_CONFIG_OPTIONS, manual.config_options) def test_parsing_with_unknown_options(self):

1 0

[stem/master] Update copyright header for certificate.py
by atagar＠torproject.org 30 Mar '17

30 Mar '17

commit 9cd41ae6991e975b6432e9d29b2597a336a65dbb Author: Patrick O'Doherty <p(a)trickod.com> Date: Sun Feb 26 16:24:43 2017 -0800 Update copyright header for certificate.py --- stem/descriptor/certificate.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stem/descriptor/certificate.py b/stem/descriptor/certificate.py index ff78538..0c3b2b6 100644 --- a/stem/descriptor/certificate.py +++ b/stem/descriptor/certificate.py @@ -1,4 +1,4 @@ -# Copyright 2016, Patrick O'Doherty and The Tor Project +# Copyright 2017, Damian Johnson and The Tor Project # See LICENSE for licensing information """

1 0

[stem/master] Drop final reference to pycrypto in test/util.py
by atagar＠torproject.org 30 Mar '17

30 Mar '17

commit 25e9e22d39d21627cf09003476370dd5115bca4e Author: Patrick O'Doherty <p(a)trickod.com> Date: Tue Feb 28 23:28:34 2017 -0800 Drop final reference to pycrypto in test/util.py --- test/util.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/util.py b/test/util.py index 18765ef..092d8d6 100644 --- a/test/util.py +++ b/test/util.py @@ -217,8 +217,8 @@ def check_python_version(): def check_cryptography_version(): if stem.prereq.is_crypto_available(): - import Crypto - return Crypto.__version__ + import cryptography + return cryptography.__version__ else: return 'missing'

1 0

← Newer
1
...
7
8
9
10
11
12
13
...
119
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits March 2017