February 2018 - tor-commits - lists.torproject.org

[stem/master] Move Relay class into stem.client
by atagar＠torproject.org 07 Feb '18

07 Feb '18

commit 9b3a868a80e91527d9cdf276b3e27eef1bf34b02 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Feb 4 13:09:28 2018 -0800 Move Relay class into stem.client On reflection I'd really prefer to keep all these modules colocated in this module, much as stem.descriptor covers all things descriptor related. --- stem/__init__.py | 1 - stem/client/__init__.py | 594 ++++++---------------------------------- stem/client/cell.py | 36 ++- stem/client/datatype.py | 533 +++++++++++++++++++++++++++++++++++ stem/relay.py | 137 --------- test/integ/client/__init__.py | 2 +- test/integ/client/connection.py | 2 +- test/settings.cfg | 2 +- test/unit/client/address.py | 2 +- test/unit/client/cell.py | 2 +- test/unit/client/certificate.py | 2 +- test/unit/client/kdf.py | 6 +- test/unit/client/size.py | 2 +- 13 files changed, 660 insertions(+), 661 deletions(-) diff --git a/stem/__init__.py b/stem/__init__.py index 2d2e1c6d..83a48903 100644 --- a/stem/__init__.py +++ b/stem/__init__.py @@ -494,7 +494,6 @@ __all__ = [ 'exit_policy', 'prereq', 'process', - 'relay', 'socket', 'version', 'ControllerError', diff --git a/stem/client/__init__.py b/stem/client/__init__.py index 9f4217c5..85473685 100644 --- a/stem/client/__init__.py +++ b/stem/client/__init__.py @@ -2,556 +2,142 @@ # See LICENSE for licensing information """ -Support for `Tor's ORPort protocol -<https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt>`_. - -**This module only consists of low level components, and is not intended for -users.** See our :class:`~stem.relay.Relay` the API you probably want. +Interaction with a Tor relay's ORPort. :class:`~stem.client.Relay` is +a wrapper for :class:`~stem.socket.RelaySocket`, much the same way as +:class:`~stem.control.Controller` provides higher level functions for +:class:`~stem.socket.ControlSocket`. .. versionadded:: 1.7.0 :: - split - splits bytes into substrings - - Field - Packable and unpackable datatype. - |- Size - Field of a static size. - |- Address - Relay address. - |- Certificate - Relay certificate. + Relay - Connection with a tor relay's ORPort. + | +- connect - Establishes a connection with a relay. | - |- pack - encodes content - |- unpack - decodes content - +- pop - decodes content with remainder - - KDF - KDF-TOR derivatived attributes - +- from_value - parses key material - -.. data:: AddrType (enum) - - Form an address takes. - - ===================== =========== - AddressType Description - ===================== =========== - **HOSTNAME** relay hostname - **IPv4** IPv4 address - **IPv6** IPv6 address - **ERROR_TRANSIENT** temporarily error retrieving address - **ERROR_PERMANENT** permanent error retrieving address - **UNKNOWN** unrecognized address type - ===================== =========== - -.. data:: RelayCommand (enum) - - Command concerning streams and circuits we've established with a relay. - Commands have two characteristics... - - * **forward/backward**: **forward** commands are issued from the orgin, - whereas **backward** come from the relay - - * **stream/circuit**: **steam** commands concern an individual steam, whereas - **circuit** concern the entire circuit we've established with a relay - - ===================== =========== - RelayCommand Description - ===================== =========== - **BEGIN** begin a stream (**forward**, **stream**) - **DATA** transmit data (**forward/backward**, **stream**) - **END** end a stream (**forward/backward**, **stream**) - **CONNECTED** BEGIN reply (**backward**, **stream**) - **SENDME** ready to accept more cells (**forward/backward**, **stream/circuit**) - **EXTEND** extend the circuit through another relay (**forward**, **circuit**) - **EXTENDED** EXTEND reply (**backward**, **circuit**) - **TRUNCATE** remove last circuit hop (**forward**, **circuit**) - **TRUNCATED** TRUNCATE reply (**backward**, **circuit**) - **DROP** ignorable no-op (**forward/backward**, **circuit**) - **RESOLVE** request DNS resolution (**forward**, **stream**) - **RESOLVED** RESOLVE reply (**backward**, **stream**) - **BEGIN_DIR** request descriptor (**forward**, **steam**) - **EXTEND2** ntor EXTEND request (**forward**, **circuit**) - **EXTENDED2** EXTEND2 reply (**backward**, **circuit**) - **UNKNOWN** unrecognized command - ===================== =========== - -.. data:: CertType (enum) - - Relay certificate type. - - ===================== =========== - CertType Description - ===================== =========== - **LINK** link key certificate certified by RSA1024 identity - **IDENTITY** RSA1024 Identity certificate - **AUTHENTICATE** RSA1024 AUTHENTICATE cell link certificate - **UNKNOWN** unrecognized certificate type - ===================== =========== - -.. data:: CloseReason (enum) - - Reason a relay is closed. - - ===================== =========== - CloseReason Description - ===================== =========== - **NONE** no reason given - **PROTOCOL** tor protocol violation - **INTERNAL** internal error - **REQUESTED** client sent a TRUNCATE command - **HIBERNATING** relay suspended, trying to save bandwidth - **RESOURCELIMIT** out of memory, sockets, or circuit IDs - **CONNECTFAILED** unable to reach relay - **OR_IDENTITY** connected, but its OR identity was not as expected - **OR_CONN_CLOSED** connection that was carrying this circuit died - **FINISHED** circuit has expired for being dirty or old - **TIMEOUT** circuit construction took too long - **DESTROYED** circuit was destroyed without a client TRUNCATE - **NOSUCHSERVICE** request was for an unknown hidden service - **UNKNOWN** unrecognized reason - ===================== =========== + |- is_alive - reports if our connection is open or closed + |- connection_time - time when we last connected or disconnected + +- close - shuts down our connection """ -import collections -import hashlib -import io -import struct - -import stem.prereq +import stem +import stem.client.cell +import stem.socket import stem.util.connection -import stem.util.enum - -from stem.util import _hash_attr -ZERO = '\x00' -HASH_LEN = 20 -KEY_LEN = 16 +from stem.client.datatype import AddrType, Address __all__ = [ 'cell', + 'datatype', ] +DEFAULT_LINK_PROTOCOLS = (3, 4, 5) -class _IntegerEnum(stem.util.enum.Enum): - """ - Integer backed enumeration. Enumerations of this type always have an implicit - **UNKNOWN** value for integer values that lack a mapping. - """ - - def __init__(self, *args): - self._enum_to_int = {} - self._int_to_enum = {} - parent_args = [] - - for entry in args: - if len(entry) == 2: - enum, int_val = entry - str_val = enum - elif len(entry) == 3: - enum, str_val, int_val = entry - else: - raise ValueError('IntegerEnums can only be constructed with two or three value tuples: %s' % repr(entry)) - - self._enum_to_int[str_val] = int_val - self._int_to_enum[int_val] = str_val - parent_args.append((enum, str_val)) - - parent_args.append(('UNKNOWN', 'UNKNOWN')) - super(_IntegerEnum, self).__init__(*parent_args) - - def get(self, val): - """ - Privides the (enum, int_value) tuple for a given value. - """ - if isinstance(val, int): - return self._int_to_enum.get(val, self.UNKNOWN), val - elif val in self: - return val, self._enum_to_int.get(val, val) - else: - raise ValueError("Invalid enumeration '%s', options are %s" % (val, ', '.join(self))) - - -AddrType = _IntegerEnum( - ('HOSTNAME', 0), - ('IPv4', 4), - ('IPv6', 6), - ('ERROR_TRANSIENT', 16), - ('ERROR_PERMANENT', 17), -) - -RelayCommand = _IntegerEnum( - ('BEGIN', 'RELAY_BEGIN', 1), - ('DATA', 'RELAY_DATA', 2), - ('END', 'RELAY_END', 3), - ('CONNECTED', 'RELAY_CONNECTED', 4), - ('SENDME', 'RELAY_SENDME', 5), - ('EXTEND', 'RELAY_EXTEND', 6), - ('EXTENDED', 'RELAY_EXTENDED', 7), - ('TRUNCATE', 'RELAY_TRUNCATE', 8), - ('TRUNCATED', 'RELAY_TRUNCATED', 9), - ('DROP', 'RELAY_DROP', 10), - ('RESOLVE', 'RELAY_RESOLVE', 11), - ('RESOLVED', 'RELAY_RESOLVED', 12), - ('BEGIN_DIR', 'RELAY_BEGIN_DIR', 13), - ('EXTEND2', 'RELAY_EXTEND2', 14), - ('EXTENDED2', 'RELAY_EXTENDED2', 15), -) - -CertType = _IntegerEnum( - ('LINK', 1), - ('IDENTITY', 2), - ('AUTHENTICATE', 3), -) - -CloseReason = _IntegerEnum( - ('NONE', 0), - ('PROTOCOL', 1), - ('INTERNAL', 2), - ('REQUESTED', 3), - ('HIBERNATING', 4), - ('RESOURCELIMIT', 5), - ('CONNECTFAILED', 6), - ('OR_IDENTITY', 7), - ('OR_CONN_CLOSED', 8), - ('FINISHED', 9), - ('TIMEOUT', 10), - ('DESTROYED', 11), - ('NOSUCHSERVICE', 12), -) - -STREAM_ID_REQUIRED = ( - RelayCommand.BEGIN, - RelayCommand.DATA, - RelayCommand.END, - RelayCommand.CONNECTED, - RelayCommand.RESOLVE, - RelayCommand.RESOLVED, - RelayCommand.BEGIN_DIR, -) - -STREAM_ID_DISALLOWED = ( - RelayCommand.EXTEND, - RelayCommand.EXTENDED, - RelayCommand.TRUNCATE, - RelayCommand.TRUNCATED, - RelayCommand.DROP, - RelayCommand.EXTEND2, - RelayCommand.EXTENDED2, -) - - -def split(content, size): - """ - Simple split of bytes into two substrings. - - :param bytes content: string to split - :param int size: index to split the string on - - :returns: two value tuple with the split bytes +class Relay(object): """ + Connection with a Tor relay's ORPort. - return content[:size], content[size:] - - -class Field(object): + :var int link_protocol: link protocol version we established """ - Packable and unpackable datatype. - """ - - def pack(self): - """ - Encodes field into bytes. - - :returns: **bytes** that can be communicated over Tor's ORPort - - :raises: **ValueError** if incorrect type or size - """ - raise NotImplementedError('Not yet available') - - @classmethod - def unpack(cls, packed): - """ - Decodes bytes into a field of this type. - - :param bytes packed: content to decode - - :returns: instance of this class - - :raises: **ValueError** if packed data is malformed - """ - - unpacked, remainder = cls.pop(packed) - - if remainder: - raise ValueError('%s is the wrong size for a %s field' % (repr(packed), cls.__name__)) - - return unpacked + def __init__(self, orport, link_protocol): + self.link_protocol = link_protocol + self._orport = orport @staticmethod - def pop(packed): + def connect(address, port, link_protocols = DEFAULT_LINK_PROTOCOLS): """ - Decodes bytes as this field type, providing it and the remainder. + Establishes a connection with the given ORPort. - :param bytes packed: content to decode + :param str address: ip address of the relay + :param int port: ORPort of the relay + :param tuple link_protocols: acceptable link protocol versions - :returns: tuple of the form (unpacked, remainder) - - :raises: **ValueError** if packed data is malformed + :raises: + * **ValueError** if address or port are invalid + * :class:`stem.SocketError` if we're unable to establish a connection """ - raise NotImplementedError('Not yet available') - - def __eq__(self, other): - return hash(self) == hash(other) if isinstance(other, Field) else False - - def __ne__(self, other): - return not self == other - - -class Size(Field): - """ - Unsigned `struct.pack format - <https://docs.python.org/2/library/struct.html#format-characters>` for - network-order fields. - - ==================== =========== - Pack Description - ==================== =========== - CHAR Unsigned char (1 byte) - SHORT Unsigned short (2 bytes) - LONG Unsigned long (4 bytes) - LONG_LONG Unsigned long long (8 bytes) - ==================== =========== - """ - - def __init__(self, name, size, pack_format): - self.name = name - self.size = size - self.format = pack_format - - @staticmethod - def pop(packed): - raise NotImplementedError("Use our constant's unpack() and pop() instead") - - def pack(self, content): - if not isinstance(content, int): - raise ValueError('Size.pack encodes an integer, but was a %s' % type(content).__name__) - - packed = struct.pack(self.format, content) - - if self.size != len(packed): - raise ValueError('%s is the wrong size for a %s field' % (repr(packed), self.name)) - - return packed - - def unpack(self, packed): - if self.size != len(packed): - raise ValueError('%s is the wrong size for a %s field' % (repr(packed), self.name)) - - return struct.unpack(self.format, packed)[0] - - def pop(self, packed): - return self.unpack(packed[:self.size]), packed[self.size:] - - -class Address(Field): - """ - Relay address. - - :var stem.client.AddrType type: address type - :var int type_int: integer value of the address type - :var unicode value: address value - :var bytes value_bin: encoded address value - """ - - def __init__(self, value, addr_type = None): - if addr_type is None: - if stem.util.connection.is_valid_ipv4_address(value): - addr_type = AddrType.IPv4 - elif stem.util.connection.is_valid_ipv6_address(value): - addr_type = AddrType.IPv6 - else: - raise ValueError('Address type is required unless an IPv4 or IPv6 address') - - self.type, self.type_int = AddrType.get(addr_type) - - if self.type == AddrType.IPv4: - if stem.util.connection.is_valid_ipv4_address(value): - self.value = value - self.value_bin = ''.join([Size.CHAR.pack(int(v)) for v in value.split('.')]) - else: - if len(value) != 4: - raise ValueError('Packed IPv4 addresses should be four bytes, but was: %s' % repr(value)) - - self.value = '.'.join([str(Size.CHAR.unpack(value[i])) for i in range(4)]) - self.value_bin = value - elif self.type == AddrType.IPv6: - if stem.util.connection.is_valid_ipv6_address(value): - self.value = stem.util.connection.expand_ipv6_address(value).lower() - self.value_bin = ''.join([Size.SHORT.pack(int(v, 16)) for v in self.value.split(':')]) - else: - if len(value) != 16: - raise ValueError('Packed IPv6 addresses should be sixteen bytes, but was: %s' % repr(value)) - - self.value = ':'.join(['%04x' % Size.SHORT.unpack(value[i * 2:(i + 1) * 2]) for i in range(8)]) - self.value_bin = value + if stem.util.connection.is_valid_ipv4_address(address): + addr_type = AddrType.IPv4 + elif stem.util.connection.is_valid_ipv6_address(address): + addr_type = AddrType.IPv6 else: - # The spec doesn't really tell us what form to expect errors to be. For - # now just leaving the value unset so we can fill it in later when we - # know what would be most useful. - - self.value = None - self.value_bin = value - - def pack(self): - cell = io.BytesIO() - cell.write(Size.CHAR.pack(self.type_int)) - cell.write(Size.CHAR.pack(len(self.value_bin))) - cell.write(self.value_bin) - return cell.getvalue() - - @staticmethod - def pop(content): - if not content: - raise ValueError('Payload empty where an address was expected') - elif len(content) < 2: - raise ValueError('Insuffient data for address headers') - - addr_type, content = Size.CHAR.pop(content) - addr_length, content = Size.CHAR.pop(content) - - if len(content) < addr_length: - raise ValueError('Address specified a payload of %i bytes, but only had %i' % (addr_length, len(content))) - - addr_value, content = split(content, addr_length) - - return Address(addr_value, addr_type), content - - def __hash__(self): - return _hash_attr(self, 'type_int', 'value_bin') - - -class Certificate(Field): - """ - Relay certificate as defined in tor-spec section 4.2. - - :var stem.client.CertType type: certificate type - :var int type_int: integer value of the certificate type - :var bytes value: certificate value - """ - - def __init__(self, cert_type, value): - self.type, self.type_int = CertType.get(cert_type) - self.value = value + raise ValueError("'%s' isn't an IPv4 or IPv6 address" % address) + + if not stem.util.connection.is_valid_port(port): + raise ValueError("'%s' isn't a valid port" % port) + elif not link_protocols: + raise ValueError("Connection can't be established without a link protocol.") + + try: + conn = stem.socket.RelaySocket(address, port) + except stem.SocketError as exc: + if 'Connection refused' in str(exc): + raise stem.SocketError("Failed to connect to %s:%i. Maybe it isn't an ORPort?" % (address, port)) + elif 'SSL: UNKNOWN_PROTOCOL' in str(exc): + raise stem.SocketError("Failed to SSL authenticate to %s:%i. Maybe it isn't an ORPort?" % (address, port)) + else: + raise - def pack(self): - cell = io.BytesIO() - cell.write(Size.CHAR.pack(self.type_int)) - cell.write(Size.SHORT.pack(len(self.value))) - cell.write(self.value) - return cell.getvalue() + conn.send(stem.client.cell.VersionsCell(link_protocols).pack()) + response = conn.recv() - @staticmethod - def pop(content): - cert_type, content = Size.CHAR.pop(content) - cert_size, content = Size.SHORT.pop(content) + # Link negotiation ends right away if we lack a common protocol + # version. (#25139) - if cert_size > len(content): - raise ValueError('CERTS cell should have a certificate with %i bytes, but only had %i remaining' % (cert_size, len(content))) + if not response: + conn.close() + raise stem.SocketError('Unable to establish a common link protocol with %s:%i' % (address, port)) - cert_bytes, content = split(content, cert_size) - return Certificate(cert_type, cert_bytes), content + versions_reply = stem.client.cell.Cell.pop(response, 2)[0] + common_protocols = set(link_protocols).intersection(versions_reply.versions) - def __hash__(self): - return _hash_attr(self, 'type_int', 'value') + if not common_protocols: + conn.close() + raise stem.SocketError('Unable to find a common link protocol. We support %s but %s:%i supports %s.' % (', '.join(link_protocols), address, port, ', '.join(versions_reply.versions))) + # TODO: we should fill in our address, right? + # TODO: what happens if we skip the NETINFO? -class KDF(collections.namedtuple('KDF', ['key_hash', 'forward_digest', 'backward_digest', 'forward_key', 'backward_key'])): - """ - Computed KDF-TOR derived values for TAP, CREATE_FAST handshakes, and hidden - service protocols as defined tor-spec section 5.2.1. - - :var bytes key_hash: hash that proves knowledge of our shared key - :var bytes forward_digest: forward digest hash seed - :var bytes backward_digest: backward digest hash seed - :var bytes forward_key: forward encryption key - :var bytes backward_key: backward encryption key - """ + link_protocol = max(common_protocols) + conn.send(stem.client.cell.NetinfoCell(Address(address, addr_type), []).pack(link_protocol)) - @staticmethod - def from_value(key_material): - # Derived key material, as per... - # - # K = H(K0 | [00]) | H(K0 | [01]) | H(K0 | [02]) | ... + return Relay(conn, link_protocol) - derived_key = '' - counter = 0 + def is_alive(self): + """ + Checks if our socket is currently connected. This is a pass-through for our + socket's :func:`~stem.socket.BaseSocket.is_alive` method. - while len(derived_key) < KEY_LEN * 2 + HASH_LEN * 3: - derived_key += hashlib.sha1(key_material + Size.CHAR.pack(counter)).digest() - counter += 1 + :returns: **bool** that's **True** if our socket is connected and **False** otherwise + """ - key_hash, derived_key = split(derived_key, HASH_LEN) - forward_digest, derived_key = split(derived_key, HASH_LEN) - backward_digest, derived_key = split(derived_key, HASH_LEN) - forward_key, derived_key = split(derived_key, KEY_LEN) - backward_key, derived_key = split(derived_key, KEY_LEN) + return self._orport.is_alive() - return KDF(key_hash, forward_digest, backward_digest, forward_key, backward_key) + def connection_time(self): + """ + Provides the unix timestamp for when our socket was either connected or + disconnected. That is to say, the time we connected if we're currently + connected and the time we disconnected if we're not connected. + :returns: **float** for when we last connected or disconnected, zero if + we've never connected + """ -class Circuit(collections.namedtuple('Circuit', ['socket', 'id', 'forward_digest', 'backward_digest', 'forward_key', 'backward_key'])): - """ - Circuit through which requests can be made of a `Tor relay's ORPort - <https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt>`_. - - :var stem.socket.RelaySocket socket: socket through which this circuit has been established - :var int id: circuit id - :var hashlib.sha1 forward_digest: digest for forward integrity check - :var hashlib.sha1 backward_digest: digest for backward integrity check - :var bytes forward_key: forward encryption key - :var bytes backward_key: backward encryption key - """ + return self._orport.connection_time() - @staticmethod - def create(relay_socket, circ_id, link_version): + def close(self): """ - Constructs a new circuit over the given ORPort. + Closes our socket connection. This is a pass-through for our socket's + :func:`~stem.socket.BaseSocket.close` method. """ - if not stem.prereq.is_crypto_available(): - raise ImportError('Circuit construction requires the cryptography module') - - from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes - from cryptography.hazmat.backends import default_backend - - create_fast_cell = stem.client.cell.CreateFastCell(circ_id) - relay_socket.send(create_fast_cell.pack(link_version)) - - response = stem.client.cell.Cell.unpack(relay_socket.recv(), link_version) - created_fast_cells = filter(lambda cell: isinstance(cell, stem.client.cell.CreatedFastCell), response) - - if not created_fast_cells: - raise ValueError('We should get a CREATED_FAST response from a CREATE_FAST request') - - created_fast_cell = created_fast_cells[0] - kdf = KDF.from_value(create_fast_cell.key_material + created_fast_cell.key_material) - ctr = modes.CTR(ZERO * (algorithms.AES.block_size / 8)) - - if created_fast_cell.derivative_key != kdf.key_hash: - raise ValueError('Remote failed to prove that it knows our shared key') - - return Circuit( - relay_socket, - circ_id, - hashlib.sha1(kdf.forward_digest), - hashlib.sha1(kdf.backward_digest), - Cipher(algorithms.AES(kdf.forward_key), ctr, default_backend()).encryptor(), - Cipher(algorithms.AES(kdf.backward_key), ctr, default_backend()).decryptor(), - ) + return self._orport.close() + def __enter__(self): + return self -setattr(Size, 'CHAR', Size('CHAR', 1, '!B')) -setattr(Size, 'SHORT', Size('SHORT', 2, '!H')) -setattr(Size, 'LONG', Size('LONG', 4, '!L')) -setattr(Size, 'LONG_LONG', Size('LONG_LONG', 8, '!Q')) + def __exit__(self, exit_type, value, traceback): + self.close() diff --git a/stem/client/cell.py b/stem/client/cell.py index 2bae4fc5..a018b2a3 100644 --- a/stem/client/cell.py +++ b/stem/client/cell.py @@ -44,15 +44,33 @@ import os import random import sys -import stem.client - from stem import UNDEFINED -from stem.client import HASH_LEN, ZERO, Address, Size, split +from stem.client.datatype import HASH_LEN, ZERO, Address, Certificate, CloseReason, RelayCommand, Size, split from stem.util import _hash_attr, datetime_to_unix FIXED_PAYLOAD_LEN = 509 AUTH_CHALLENGE_SIZE = 32 +STREAM_ID_REQUIRED = ( + RelayCommand.BEGIN, + RelayCommand.DATA, + RelayCommand.END, + RelayCommand.CONNECTED, + RelayCommand.RESOLVE, + RelayCommand.RESOLVED, + RelayCommand.BEGIN_DIR, +) + +STREAM_ID_DISALLOWED = ( + RelayCommand.EXTEND, + RelayCommand.EXTENDED, + RelayCommand.TRUNCATE, + RelayCommand.TRUNCATED, + RelayCommand.DROP, + RelayCommand.EXTEND2, + RelayCommand.EXTENDED2, +) + class Cell(object): """ @@ -297,14 +315,14 @@ class RelayCell(CircuitCell): def __init__(self, circ_id, command, data, digest = 0, stream_id = 0): super(RelayCell, self).__init__(circ_id) - self.command, self.command_int = stem.client.RelayCommand.get(command) + self.command, self.command_int = RelayCommand.get(command) self.data = data self.digest = digest self.stream_id = stream_id - if not stream_id and self.command in stem.client.STREAM_ID_REQUIRED: + if not stream_id and self.command in STREAM_ID_REQUIRED: raise ValueError('%s relay cells require a stream id' % self.command) - elif stream_id and self.command in stem.client.STREAM_ID_DISALLOWED: + elif stream_id and self.command in STREAM_ID_DISALLOWED: raise ValueError('%s relay cells concern the circuit itself and cannot have a stream id' % self.command) def pack(self, link_version): @@ -345,9 +363,9 @@ class DestroyCell(CircuitCell): VALUE = 4 IS_FIXED_SIZE = True - def __init__(self, circ_id, reason = stem.client.CloseReason.NONE): + def __init__(self, circ_id, reason = CloseReason.NONE): super(DestroyCell, self).__init__(circ_id) - self.reason, self.reason_int = stem.client.CloseReason.get(reason) + self.reason, self.reason_int = CloseReason.get(reason) def pack(self, link_version): return DestroyCell._pack(link_version, Size.CHAR.pack(self.reason_int), self.circ_id) @@ -610,7 +628,7 @@ class CertsCell(Cell): if not content: raise ValueError('CERTS cell indicates it should have %i certificates, but only contained %i' % (cert_count, len(certs))) - cert, content = stem.client.Certificate.pop(content) + cert, content = Certificate.pop(content) certs.append(cert) return CertsCell(certs) diff --git a/stem/client/datatype.py b/stem/client/datatype.py new file mode 100644 index 00000000..f5805a2a --- /dev/null +++ b/stem/client/datatype.py @@ -0,0 +1,533 @@ +# Copyright 2018, Damian Johnson and The Tor Project +# See LICENSE for licensing information + +""" +Support for `Tor's ORPort protocol +<https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt>`_. + +**This module only consists of low level components, and is not intended for +users.** See our :class:`~stem.client.Relay` the API you probably want. + +.. versionadded:: 1.7.0 + +:: + + split - splits bytes into substrings + + Field - Packable and unpackable datatype. + |- Size - Field of a static size. + |- Address - Relay address. + |- Certificate - Relay certificate. + | + |- pack - encodes content + |- unpack - decodes content + +- pop - decodes content with remainder + + KDF - KDF-TOR derivatived attributes + +- from_value - parses key material + +.. data:: AddrType (enum) + + Form an address takes. + + ===================== =========== + AddressType Description + ===================== =========== + **HOSTNAME** relay hostname + **IPv4** IPv4 address + **IPv6** IPv6 address + **ERROR_TRANSIENT** temporarily error retrieving address + **ERROR_PERMANENT** permanent error retrieving address + **UNKNOWN** unrecognized address type + ===================== =========== + +.. data:: RelayCommand (enum) + + Command concerning streams and circuits we've established with a relay. + Commands have two characteristics... + + * **forward/backward**: **forward** commands are issued from the orgin, + whereas **backward** come from the relay + + * **stream/circuit**: **steam** commands concern an individual steam, whereas + **circuit** concern the entire circuit we've established with a relay + + ===================== =========== + RelayCommand Description + ===================== =========== + **BEGIN** begin a stream (**forward**, **stream**) + **DATA** transmit data (**forward/backward**, **stream**) + **END** end a stream (**forward/backward**, **stream**) + **CONNECTED** BEGIN reply (**backward**, **stream**) + **SENDME** ready to accept more cells (**forward/backward**, **stream/circuit**) + **EXTEND** extend the circuit through another relay (**forward**, **circuit**) + **EXTENDED** EXTEND reply (**backward**, **circuit**) + **TRUNCATE** remove last circuit hop (**forward**, **circuit**) + **TRUNCATED** TRUNCATE reply (**backward**, **circuit**) + **DROP** ignorable no-op (**forward/backward**, **circuit**) + **RESOLVE** request DNS resolution (**forward**, **stream**) + **RESOLVED** RESOLVE reply (**backward**, **stream**) + **BEGIN_DIR** request descriptor (**forward**, **steam**) + **EXTEND2** ntor EXTEND request (**forward**, **circuit**) + **EXTENDED2** EXTEND2 reply (**backward**, **circuit**) + **UNKNOWN** unrecognized command + ===================== =========== + +.. data:: CertType (enum) + + Relay certificate type. + + ===================== =========== + CertType Description + ===================== =========== + **LINK** link key certificate certified by RSA1024 identity + **IDENTITY** RSA1024 Identity certificate + **AUTHENTICATE** RSA1024 AUTHENTICATE cell link certificate + **UNKNOWN** unrecognized certificate type + ===================== =========== + +.. data:: CloseReason (enum) + + Reason a relay is closed. + + ===================== =========== + CloseReason Description + ===================== =========== + **NONE** no reason given + **PROTOCOL** tor protocol violation + **INTERNAL** internal error + **REQUESTED** client sent a TRUNCATE command + **HIBERNATING** relay suspended, trying to save bandwidth + **RESOURCELIMIT** out of memory, sockets, or circuit IDs + **CONNECTFAILED** unable to reach relay + **OR_IDENTITY** connected, but its OR identity was not as expected + **OR_CONN_CLOSED** connection that was carrying this circuit died + **FINISHED** circuit has expired for being dirty or old + **TIMEOUT** circuit construction took too long + **DESTROYED** circuit was destroyed without a client TRUNCATE + **NOSUCHSERVICE** request was for an unknown hidden service + **UNKNOWN** unrecognized reason + ===================== =========== +""" + +import collections +import hashlib +import io +import struct + +import stem.prereq +import stem.util.connection +import stem.util.enum + +from stem.util import _hash_attr + +ZERO = '\x00' +HASH_LEN = 20 +KEY_LEN = 16 + + +class _IntegerEnum(stem.util.enum.Enum): + """ + Integer backed enumeration. Enumerations of this type always have an implicit + **UNKNOWN** value for integer values that lack a mapping. + """ + + def __init__(self, *args): + self._enum_to_int = {} + self._int_to_enum = {} + parent_args = [] + + for entry in args: + if len(entry) == 2: + enum, int_val = entry + str_val = enum + elif len(entry) == 3: + enum, str_val, int_val = entry + else: + raise ValueError('IntegerEnums can only be constructed with two or three value tuples: %s' % repr(entry)) + + self._enum_to_int[str_val] = int_val + self._int_to_enum[int_val] = str_val + parent_args.append((enum, str_val)) + + parent_args.append(('UNKNOWN', 'UNKNOWN')) + super(_IntegerEnum, self).__init__(*parent_args) + + def get(self, val): + """ + Privides the (enum, int_value) tuple for a given value. + """ + + if isinstance(val, int): + return self._int_to_enum.get(val, self.UNKNOWN), val + elif val in self: + return val, self._enum_to_int.get(val, val) + else: + raise ValueError("Invalid enumeration '%s', options are %s" % (val, ', '.join(self))) + + +AddrType = _IntegerEnum( + ('HOSTNAME', 0), + ('IPv4', 4), + ('IPv6', 6), + ('ERROR_TRANSIENT', 16), + ('ERROR_PERMANENT', 17), +) + +RelayCommand = _IntegerEnum( + ('BEGIN', 'RELAY_BEGIN', 1), + ('DATA', 'RELAY_DATA', 2), + ('END', 'RELAY_END', 3), + ('CONNECTED', 'RELAY_CONNECTED', 4), + ('SENDME', 'RELAY_SENDME', 5), + ('EXTEND', 'RELAY_EXTEND', 6), + ('EXTENDED', 'RELAY_EXTENDED', 7), + ('TRUNCATE', 'RELAY_TRUNCATE', 8), + ('TRUNCATED', 'RELAY_TRUNCATED', 9), + ('DROP', 'RELAY_DROP', 10), + ('RESOLVE', 'RELAY_RESOLVE', 11), + ('RESOLVED', 'RELAY_RESOLVED', 12), + ('BEGIN_DIR', 'RELAY_BEGIN_DIR', 13), + ('EXTEND2', 'RELAY_EXTEND2', 14), + ('EXTENDED2', 'RELAY_EXTENDED2', 15), +) + +CertType = _IntegerEnum( + ('LINK', 1), + ('IDENTITY', 2), + ('AUTHENTICATE', 3), +) + +CloseReason = _IntegerEnum( + ('NONE', 0), + ('PROTOCOL', 1), + ('INTERNAL', 2), + ('REQUESTED', 3), + ('HIBERNATING', 4), + ('RESOURCELIMIT', 5), + ('CONNECTFAILED', 6), + ('OR_IDENTITY', 7), + ('OR_CONN_CLOSED', 8), + ('FINISHED', 9), + ('TIMEOUT', 10), + ('DESTROYED', 11), + ('NOSUCHSERVICE', 12), +) + + +def split(content, size): + """ + Simple split of bytes into two substrings. + + :param bytes content: string to split + :param int size: index to split the string on + + :returns: two value tuple with the split bytes + """ + + return content[:size], content[size:] + + +class Field(object): + """ + Packable and unpackable datatype. + """ + + def pack(self): + """ + Encodes field into bytes. + + :returns: **bytes** that can be communicated over Tor's ORPort + + :raises: **ValueError** if incorrect type or size + """ + + raise NotImplementedError('Not yet available') + + @classmethod + def unpack(cls, packed): + """ + Decodes bytes into a field of this type. + + :param bytes packed: content to decode + + :returns: instance of this class + + :raises: **ValueError** if packed data is malformed + """ + + unpacked, remainder = cls.pop(packed) + + if remainder: + raise ValueError('%s is the wrong size for a %s field' % (repr(packed), cls.__name__)) + + return unpacked + + @staticmethod + def pop(packed): + """ + Decodes bytes as this field type, providing it and the remainder. + + :param bytes packed: content to decode + + :returns: tuple of the form (unpacked, remainder) + + :raises: **ValueError** if packed data is malformed + """ + + raise NotImplementedError('Not yet available') + + def __eq__(self, other): + return hash(self) == hash(other) if isinstance(other, Field) else False + + def __ne__(self, other): + return not self == other + + +class Size(Field): + """ + Unsigned `struct.pack format + <https://docs.python.org/2/library/struct.html#format-characters>` for + network-order fields. + + ==================== =========== + Pack Description + ==================== =========== + CHAR Unsigned char (1 byte) + SHORT Unsigned short (2 bytes) + LONG Unsigned long (4 bytes) + LONG_LONG Unsigned long long (8 bytes) + ==================== =========== + """ + + def __init__(self, name, size, pack_format): + self.name = name + self.size = size + self.format = pack_format + + @staticmethod + def pop(packed): + raise NotImplementedError("Use our constant's unpack() and pop() instead") + + def pack(self, content): + if not isinstance(content, int): + raise ValueError('Size.pack encodes an integer, but was a %s' % type(content).__name__) + + packed = struct.pack(self.format, content) + + if self.size != len(packed): + raise ValueError('%s is the wrong size for a %s field' % (repr(packed), self.name)) + + return packed + + def unpack(self, packed): + if self.size != len(packed): + raise ValueError('%s is the wrong size for a %s field' % (repr(packed), self.name)) + + return struct.unpack(self.format, packed)[0] + + def pop(self, packed): + return self.unpack(packed[:self.size]), packed[self.size:] + + +class Address(Field): + """ + Relay address. + + :var stem.client.AddrType type: address type + :var int type_int: integer value of the address type + :var unicode value: address value + :var bytes value_bin: encoded address value + """ + + def __init__(self, value, addr_type = None): + if addr_type is None: + if stem.util.connection.is_valid_ipv4_address(value): + addr_type = AddrType.IPv4 + elif stem.util.connection.is_valid_ipv6_address(value): + addr_type = AddrType.IPv6 + else: + raise ValueError('Address type is required unless an IPv4 or IPv6 address') + + self.type, self.type_int = AddrType.get(addr_type) + + if self.type == AddrType.IPv4: + if stem.util.connection.is_valid_ipv4_address(value): + self.value = value + self.value_bin = ''.join([Size.CHAR.pack(int(v)) for v in value.split('.')]) + else: + if len(value) != 4: + raise ValueError('Packed IPv4 addresses should be four bytes, but was: %s' % repr(value)) + + self.value = '.'.join([str(Size.CHAR.unpack(value[i])) for i in range(4)]) + self.value_bin = value + elif self.type == AddrType.IPv6: + if stem.util.connection.is_valid_ipv6_address(value): + self.value = stem.util.connection.expand_ipv6_address(value).lower() + self.value_bin = ''.join([Size.SHORT.pack(int(v, 16)) for v in self.value.split(':')]) + else: + if len(value) != 16: + raise ValueError('Packed IPv6 addresses should be sixteen bytes, but was: %s' % repr(value)) + + self.value = ':'.join(['%04x' % Size.SHORT.unpack(value[i * 2:(i + 1) * 2]) for i in range(8)]) + self.value_bin = value + else: + # The spec doesn't really tell us what form to expect errors to be. For + # now just leaving the value unset so we can fill it in later when we + # know what would be most useful. + + self.value = None + self.value_bin = value + + def pack(self): + cell = io.BytesIO() + cell.write(Size.CHAR.pack(self.type_int)) + cell.write(Size.CHAR.pack(len(self.value_bin))) + cell.write(self.value_bin) + return cell.getvalue() + + @staticmethod + def pop(content): + if not content: + raise ValueError('Payload empty where an address was expected') + elif len(content) < 2: + raise ValueError('Insuffient data for address headers') + + addr_type, content = Size.CHAR.pop(content) + addr_length, content = Size.CHAR.pop(content) + + if len(content) < addr_length: + raise ValueError('Address specified a payload of %i bytes, but only had %i' % (addr_length, len(content))) + + addr_value, content = split(content, addr_length) + + return Address(addr_value, addr_type), content + + def __hash__(self): + return _hash_attr(self, 'type_int', 'value_bin') + + +class Certificate(Field): + """ + Relay certificate as defined in tor-spec section 4.2. + + :var stem.client.CertType type: certificate type + :var int type_int: integer value of the certificate type + :var bytes value: certificate value + """ + + def __init__(self, cert_type, value): + self.type, self.type_int = CertType.get(cert_type) + self.value = value + + def pack(self): + cell = io.BytesIO() + cell.write(Size.CHAR.pack(self.type_int)) + cell.write(Size.SHORT.pack(len(self.value))) + cell.write(self.value) + return cell.getvalue() + + @staticmethod + def pop(content): + cert_type, content = Size.CHAR.pop(content) + cert_size, content = Size.SHORT.pop(content) + + if cert_size > len(content): + raise ValueError('CERTS cell should have a certificate with %i bytes, but only had %i remaining' % (cert_size, len(content))) + + cert_bytes, content = split(content, cert_size) + return Certificate(cert_type, cert_bytes), content + + def __hash__(self): + return _hash_attr(self, 'type_int', 'value') + + +class KDF(collections.namedtuple('KDF', ['key_hash', 'forward_digest', 'backward_digest', 'forward_key', 'backward_key'])): + """ + Computed KDF-TOR derived values for TAP, CREATE_FAST handshakes, and hidden + service protocols as defined tor-spec section 5.2.1. + + :var bytes key_hash: hash that proves knowledge of our shared key + :var bytes forward_digest: forward digest hash seed + :var bytes backward_digest: backward digest hash seed + :var bytes forward_key: forward encryption key + :var bytes backward_key: backward encryption key + """ + + @staticmethod + def from_value(key_material): + # Derived key material, as per... + # + # K = H(K0 | [00]) | H(K0 | [01]) | H(K0 | [02]) | ... + + derived_key = '' + counter = 0 + + while len(derived_key) < KEY_LEN * 2 + HASH_LEN * 3: + derived_key += hashlib.sha1(key_material + Size.CHAR.pack(counter)).digest() + counter += 1 + + key_hash, derived_key = split(derived_key, HASH_LEN) + forward_digest, derived_key = split(derived_key, HASH_LEN) + backward_digest, derived_key = split(derived_key, HASH_LEN) + forward_key, derived_key = split(derived_key, KEY_LEN) + backward_key, derived_key = split(derived_key, KEY_LEN) + + return KDF(key_hash, forward_digest, backward_digest, forward_key, backward_key) + + +class Circuit(collections.namedtuple('Circuit', ['socket', 'id', 'forward_digest', 'backward_digest', 'forward_key', 'backward_key'])): + """ + Circuit through which requests can be made of a `Tor relay's ORPort + <https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt>`_. + + :var stem.socket.RelaySocket socket: socket through which this circuit has been established + :var int id: circuit id + :var hashlib.sha1 forward_digest: digest for forward integrity check + :var hashlib.sha1 backward_digest: digest for backward integrity check + :var bytes forward_key: forward encryption key + :var bytes backward_key: backward encryption key + """ + + @staticmethod + def create(relay_socket, circ_id, link_version): + """ + Constructs a new circuit over the given ORPort. + """ + + if not stem.prereq.is_crypto_available(): + raise ImportError('Circuit construction requires the cryptography module') + + from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes + from cryptography.hazmat.backends import default_backend + + create_fast_cell = stem.client.cell.CreateFastCell(circ_id) + relay_socket.send(create_fast_cell.pack(link_version)) + + response = stem.client.cell.Cell.unpack(relay_socket.recv(), link_version) + created_fast_cells = filter(lambda cell: isinstance(cell, stem.client.cell.CreatedFastCell), response) + + if not created_fast_cells: + raise ValueError('We should get a CREATED_FAST response from a CREATE_FAST request') + + created_fast_cell = created_fast_cells[0] + kdf = KDF.from_value(create_fast_cell.key_material + created_fast_cell.key_material) + ctr = modes.CTR(ZERO * (algorithms.AES.block_size / 8)) + + if created_fast_cell.derivative_key != kdf.key_hash: + raise ValueError('Remote failed to prove that it knows our shared key') + + return Circuit( + relay_socket, + circ_id, + hashlib.sha1(kdf.forward_digest), + hashlib.sha1(kdf.backward_digest), + Cipher(algorithms.AES(kdf.forward_key), ctr, default_backend()).encryptor(), + Cipher(algorithms.AES(kdf.backward_key), ctr, default_backend()).decryptor(), + ) + + +setattr(Size, 'CHAR', Size('CHAR', 1, '!B')) +setattr(Size, 'SHORT', Size('SHORT', 2, '!H')) +setattr(Size, 'LONG', Size('LONG', 4, '!L')) +setattr(Size, 'LONG_LONG', Size('LONG_LONG', 8, '!Q')) diff --git a/stem/relay.py b/stem/relay.py deleted file mode 100644 index 14b7833f..00000000 --- a/stem/relay.py +++ /dev/null @@ -1,137 +0,0 @@ -# Copyright 2018, Damian Johnson and The Tor Project -# See LICENSE for licensing information - -""" -Interaction with a Tor relay's ORPort. :class:`~stem.relay.Relay` is -a wrapper for :class:`~stem.socket.RelaySocket`, much the same way as -:class:`~stem.control.Controller` provides higher level functions for -:class:`~stem.socket.ControlSocket`. - -.. versionadded:: 1.7.0 - -:: - - Relay - Connection with a tor relay's ORPort. - | +- connect - Establishes a connection with a relay. - | - |- is_alive - reports if our connection is open or closed - |- connection_time - time when we last connected or disconnected - +- close - shuts down our connection -""" - -import stem -import stem.client -import stem.client.cell -import stem.socket -import stem.util.connection - -DEFAULT_LINK_PROTOCOLS = (3, 4, 5) - - -class Relay(object): - """ - Connection with a Tor relay's ORPort. - - :var int link_protocol: link protocol version we established - """ - - def __init__(self, orport, link_protocol): - self.link_protocol = link_protocol - self._orport = orport - - @staticmethod - def connect(address, port, link_protocols = DEFAULT_LINK_PROTOCOLS): - """ - Establishes a connection with the given ORPort. - - :param str address: ip address of the relay - :param int port: ORPort of the relay - :param tuple link_protocols: acceptable link protocol versions - - :raises: - * **ValueError** if address or port are invalid - * :class:`stem.SocketError` if we're unable to establish a connection - """ - - if stem.util.connection.is_valid_ipv4_address(address): - addr_type = stem.client.AddrType.IPv4 - elif stem.util.connection.is_valid_ipv6_address(address): - addr_type = stem.client.AddrType.IPv6 - else: - raise ValueError("'%s' isn't an IPv4 or IPv6 address" % address) - - if not stem.util.connection.is_valid_port(port): - raise ValueError("'%s' isn't a valid port" % port) - elif not link_protocols: - raise ValueError("Connection can't be established without a link protocol.") - - try: - conn = stem.socket.RelaySocket(address, port) - except stem.SocketError as exc: - if 'Connection refused' in str(exc): - raise stem.SocketError("Failed to connect to %s:%i. Maybe it isn't an ORPort?" % (address, port)) - elif 'SSL: UNKNOWN_PROTOCOL' in str(exc): - raise stem.SocketError("Failed to SSL authenticate to %s:%i. Maybe it isn't an ORPort?" % (address, port)) - else: - raise - - conn.send(stem.client.cell.VersionsCell(link_protocols).pack()) - response = conn.recv() - - # Link negotiation ends right away if we lack a common protocol - # version. (#25139) - - if not response: - conn.close() - raise stem.SocketError('Unable to establish a common link protocol with %s:%i' % (address, port)) - - versions_reply = stem.client.cell.Cell.pop(response, 2)[0] - common_protocols = set(link_protocols).intersection(versions_reply.versions) - - if not common_protocols: - conn.close() - raise stem.SocketError('Unable to find a common link protocol. We support %s but %s:%i supports %s.' % (', '.join(link_protocols), address, port, ', '.join(versions_reply.versions))) - - # TODO: we should fill in our address, right? - # TODO: what happens if we skip the NETINFO? - - link_protocol = max(common_protocols) - conn.send(stem.client.cell.NetinfoCell(stem.client.Address(address, addr_type), []).pack(link_protocol)) - - return Relay(conn, link_protocol) - - def is_alive(self): - """ - Checks if our socket is currently connected. This is a pass-through for our - socket's :func:`~stem.socket.BaseSocket.is_alive` method. - - :returns: **bool** that's **True** if our socket is connected and **False** otherwise - """ - - return self._orport.is_alive() - - def connection_time(self): - """ - Provides the unix timestamp for when our socket was either connected or - disconnected. That is to say, the time we connected if we're currently - connected and the time we disconnected if we're not connected. - - :returns: **float** for when we last connected or disconnected, zero if - we've never connected - """ - - return self._orport.connection_time() - - def close(self): - """ - Closes our socket connection. This is a pass-through for our socket's - :func:`~stem.socket.BaseSocket.close` method. - """ - - return self._orport.close() - - def __enter__(self): - return self - - def __exit__(self, exit_type, value, traceback): - self.close() diff --git a/test/integ/client/__init__.py b/test/integ/client/__init__.py index 8d77a653..345e69fc 100644 --- a/test/integ/client/__init__.py +++ b/test/integ/client/__init__.py @@ -1,5 +1,5 @@ """ -Integration tests for tor's ORPort (stem.relay and stem.client). +Integration tests for stem.client. """ __all__ = [ diff --git a/test/integ/client/connection.py b/test/integ/client/connection.py index f5399a71..86a9cbcc 100644 --- a/test/integ/client/connection.py +++ b/test/integ/client/connection.py @@ -8,7 +8,7 @@ import unittest import stem import test.runner -from stem.relay import Relay +from stem.client import Relay class TestConnection(unittest.TestCase): diff --git a/test/settings.cfg b/test/settings.cfg index 6d543bac..aaab0790 100644 --- a/test/settings.cfg +++ b/test/settings.cfg @@ -146,7 +146,7 @@ pycodestyle.ignore test/unit/util/connection.py => W291: _tor tor 158 # issue. pyflakes.ignore run_tests.py => 'unittest' imported but unused -pyflakes.ignore stem/client/__init__.py => redefinition of unused 'pop' from * +pyflakes.ignore stem/client/datatype.py => redefinition of unused 'pop' from * pyflakes.ignore stem/util/__init__.py => undefined name 'long' pyflakes.ignore stem/util/__init__.py => undefined name 'unicode' pyflakes.ignore stem/control.py => undefined name 'controller' diff --git a/test/unit/client/address.py b/test/unit/client/address.py index c92f7722..f3da4971 100644 --- a/test/unit/client/address.py +++ b/test/unit/client/address.py @@ -6,7 +6,7 @@ import collections import re import unittest -from stem.client import AddrType, Address +from stem.client.datatype import AddrType, Address ExpectedAddress = collections.namedtuple('ExpectedAddress', ['type', 'type_int', 'value', 'value_bin']) diff --git a/test/unit/client/cell.py b/test/unit/client/cell.py index 1fde76cf..f53355dd 100644 --- a/test/unit/client/cell.py +++ b/test/unit/client/cell.py @@ -6,7 +6,7 @@ import datetime import os import unittest -from stem.client import ZERO, CertType, CloseReason, Address, Certificate +from stem.client.datatype import ZERO, CertType, CloseReason, Address, Certificate from test.unit.client import test_data from stem.client.cell import ( diff --git a/test/unit/client/certificate.py b/test/unit/client/certificate.py index 873de51d..b6782acc 100644 --- a/test/unit/client/certificate.py +++ b/test/unit/client/certificate.py @@ -4,7 +4,7 @@ Unit tests for stem.client.Certificate. import unittest -from stem.client import CertType, Certificate +from stem.client.datatype import CertType, Certificate class TestCertificate(unittest.TestCase): diff --git a/test/unit/client/kdf.py b/test/unit/client/kdf.py index 56665d31..f3922538 100644 --- a/test/unit/client/kdf.py +++ b/test/unit/client/kdf.py @@ -4,7 +4,7 @@ Unit tests for stem.client.KDF. import unittest -import stem.client +from stem.client.datatype import KDF KEY_1 = '\xec\xec.\xeb7R\xf2\n\xcb\xce\x97\xf4\x86\x82\x19#\x10\x0f\x08\xf0\xa2Z\xdeJ\x8f2\x8cc\xf6\xfa\x0e\t\x83f\xc5\xe2\xb3\x94\xa8\x13' KEY_2 = '\xe0v\xe4\xfaTB\x91\x1c\x81Gz\xa0\tI\xcb{\xc56\xcfV\xc2\xa0\x19\x9c\x98\x9a\x06\x0e\xc5\xfa\xb0z\x83\xa6\x10\xf6r"<b' @@ -12,14 +12,14 @@ KEY_2 = '\xe0v\xe4\xfaTB\x91\x1c\x81Gz\xa0\tI\xcb{\xc56\xcfV\xc2\xa0\x19\x9c\x98 class TestKDF(unittest.TestCase): def test_parsing(self): - k1 = stem.client.KDF.from_value(KEY_1) + k1 = KDF.from_value(KEY_1) self.assertEqual('\xca+\x81\x05\x14\x9d)o\xa6\x82\xe9B\xa8?\xf2\xaf\x85\x1b]6', k1.key_hash) self.assertEqual('\xac\xcc\xbc\x91\xb1\xaf\xd7\xe0\xe9\x9dF#\xd8\xdbz\xe8\xe6\xca\x83,', k1.forward_digest) self.assertEqual('*\xe5scX\xbb+\xca \xcb\xa4\xbc\xad\x0f\x95\x0cO\xcc\xac\xf1', k1.backward_digest) self.assertEqual('\xc3\xbe\xc9\xe1\xf4\x90f\xdai\xf3\xf3\xf5\x14\xb5\xb9\x03', k1.forward_key) self.assertEqual('U\xaf\x1e\x1b\xb1q||\x86A<_\xf7\xa0%\x86', k1.backward_key) - k2 = stem.client.KDF.from_value(KEY_1) + k2 = KDF.from_value(KEY_1) self.assertEqual('\xca+\x81\x05\x14\x9d)o\xa6\x82\xe9B\xa8?\xf2\xaf\x85\x1b]6', k2.key_hash) self.assertEqual('\xac\xcc\xbc\x91\xb1\xaf\xd7\xe0\xe9\x9dF#\xd8\xdbz\xe8\xe6\xca\x83,', k2.forward_digest) self.assertEqual('*\xe5scX\xbb+\xca \xcb\xa4\xbc\xad\x0f\x95\x0cO\xcc\xac\xf1', k2.backward_digest) diff --git a/test/unit/client/size.py b/test/unit/client/size.py index 6078139c..59416662 100644 --- a/test/unit/client/size.py +++ b/test/unit/client/size.py @@ -5,7 +5,7 @@ Unit tests for stem.client.Size. import re import unittest -from stem.client import Size +from stem.client.datatype import Size class TestSize(unittest.TestCase):

1 0

[stem/master] Move circuit construction into the Relay class
by atagar＠torproject.org 07 Feb '18

07 Feb '18

commit 2ca71ddb1d91154b7dd4a9c8e0c9e3bac06be204 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Feb 4 14:49:30 2018 -0800 Move circuit construction into the Relay class Preferably I'd like to keep all socket activity within the Relay class. We might need to bend this in practice but lets first give it a try. --- stem/client/__init__.py | 68 +++++++++++++++++++++++++++++++- stem/client/cell.py | 100 +++++++++++++++++++++++------------------------- stem/client/datatype.py | 51 ------------------------ 3 files changed, 114 insertions(+), 105 deletions(-) diff --git a/stem/client/__init__.py b/stem/client/__init__.py index 85473685..fdb436b6 100644 --- a/stem/client/__init__.py +++ b/stem/client/__init__.py @@ -19,12 +19,14 @@ a wrapper for :class:`~stem.socket.RelaySocket`, much the same way as +- close - shuts down our connection """ +import hashlib + import stem import stem.client.cell import stem.socket import stem.util.connection -from stem.client.datatype import AddrType, Address +from stem.client.datatype import ZERO, AddrType, Address, KDF __all__ = [ 'cell', @@ -44,6 +46,7 @@ class Relay(object): def __init__(self, orport, link_protocol): self.link_protocol = link_protocol self._orport = orport + self._circuits = {} @staticmethod def connect(address, port, link_protocols = DEFAULT_LINK_PROTOCOLS): @@ -136,8 +139,71 @@ class Relay(object): return self._orport.close() + def create_circuit(self): + """ + Establishes a new circuit. + """ + + # Find an unused circuit id. Since we're initiating the circuit we pick any + # value from a range that's determined by our link protocol. + + circ_id = 0x80000000 if self.link_protocol > 3 else 0x01 + + while circ_id in self._circuits: + circ_id += 1 + + create_fast_cell = stem.client.cell.CreateFastCell(circ_id) + self._orport.send(create_fast_cell.pack(self.link_protocol)) + + response = stem.client.cell.Cell.unpack(self._orport.recv(), self.link_protocol) + created_fast_cells = filter(lambda cell: isinstance(cell, stem.client.cell.CreatedFastCell), response) + + if not created_fast_cells: + raise ValueError('We should get a CREATED_FAST response from a CREATE_FAST request') + + created_fast_cell = created_fast_cells[0] + kdf = KDF.from_value(create_fast_cell.key_material + created_fast_cell.key_material) + + if created_fast_cell.derivative_key != kdf.key_hash: + raise ValueError('Remote failed to prove that it knows our shared key') + + circ = Circuit(self, circ_id, kdf) + self._circuits[circ.id] = circ + + return circ + def __enter__(self): return self def __exit__(self, exit_type, value, traceback): self.close() + + +class Circuit(object): + """ + Circuit through which requests can be made of a `Tor relay's ORPort + <https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt>`_. + + :var stem.client.Relay relay: relay through which this circuit has been established + :var int id: circuit id + :var hashlib.sha1 forward_digest: digest for forward integrity check + :var hashlib.sha1 backward_digest: digest for backward integrity check + :var bytes forward_key: forward encryption key + :var bytes backward_key: backward encryption key + """ + + def __init__(self, relay, circ_id, kdf): + if not stem.prereq.is_crypto_available(): + raise ImportError('Circuit construction requires the cryptography module') + + from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes + from cryptography.hazmat.backends import default_backend + + ctr = modes.CTR(ZERO * (algorithms.AES.block_size / 8)) + + self.relay = relay + self.id = circ_id + self.forward_digest = hashlib.sha1(kdf.forward_digest) + self.backward_digest = hashlib.sha1(kdf.backward_digest) + self.forward_key = Cipher(algorithms.AES(kdf.forward_key), ctr, default_backend()).encryptor() + self.backward_key = Cipher(algorithms.AES(kdf.backward_key), ctr, default_backend()).decryptor() diff --git a/stem/client/cell.py b/stem/client/cell.py index a018b2a3..ed8932c4 100644 --- a/stem/client/cell.py +++ b/stem/client/cell.py @@ -113,16 +113,16 @@ class Cell(object): raise ValueError("'%s' isn't a valid cell value" % value) - def pack(self, link_version): + def pack(self, link_protocol): raise NotImplementedError('Unpacking not yet implemented for %s cells' % type(self).NAME) @staticmethod - def unpack(content, link_version): + def unpack(content, link_protocol): """ Unpacks all cells from a response. :param bytes content: payload to decode - :param int link_version: link protocol version + :param int link_protocol: link protocol version :returns: :class:`~stem.client.cell.Cell` generator @@ -132,16 +132,16 @@ class Cell(object): """ while content: - cell, content = Cell.pop(content, link_version) + cell, content = Cell.pop(content, link_protocol) yield cell @staticmethod - def pop(content, link_version): + def pop(content, link_protocol): """ Unpacks the first cell. :param bytes content: payload to decode - :param int link_version: link protocol version + :param int link_protocol: link protocol version :returns: (:class:`~stem.client.cell.Cell`, remainder) tuple @@ -150,7 +150,7 @@ class Cell(object): * NotImplementedError if unable to unpack this cell type """ - circ_id, content = Size.SHORT.pop(content) if link_version < 4 else Size.LONG.pop(content) + circ_id, content = Size.SHORT.pop(content) if link_protocol < 4 else Size.LONG.pop(content) command, content = Size.CHAR.pop(content) cls = Cell.by_value(command) @@ -163,10 +163,10 @@ class Cell(object): raise ValueError('%s cell should have a payload of %i bytes, but only had %i' % (cls.NAME, payload_len, len(content))) payload, content = split(content, payload_len) - return cls._unpack(payload, circ_id, link_version), content + return cls._unpack(payload, circ_id, link_protocol), content @classmethod - def _pack(cls, link_version, payload, circ_id = 0): + def _pack(cls, link_protocol, payload, circ_id = 0): """ Provides bytes that can be used on the wire for these cell attributes. Format of a properly packed cell depends on if it's fixed or variable @@ -178,7 +178,7 @@ class Cell(object): Variable: [ CircuitID ][ Command ][ Size ][ Payload ] :param str name: cell command - :param int link_version: link protocol version + :param int link_protocol: link protocol version :param bytes payload: cell payload :param int circ_id: circuit id, if a CircuitCell @@ -188,16 +188,10 @@ class Cell(object): """ if isinstance(cls, CircuitCell) and circ_id is None: - if cls.NAME.startswith('CREATE'): - # Since we're initiating the circuit we pick any value from a range - # that's determined by our link version. - - circ_id = 0x80000000 if link_version > 3 else 0x01 - else: - raise ValueError('%s cells require a circ_id' % cls.NAME) + raise ValueError('%s cells require a circ_id' % cls.NAME) cell = io.BytesIO() - cell.write(Size.LONG.pack(circ_id) if link_version > 3 else Size.SHORT.pack(circ_id)) + cell.write(Size.LONG.pack(circ_id) if link_protocol > 3 else Size.SHORT.pack(circ_id)) cell.write(Size.CHAR.pack(cls.VALUE)) cell.write(b'' if cls.IS_FIXED_SIZE else Size.SHORT.pack(len(payload))) cell.write(payload) @@ -206,7 +200,7 @@ class Cell(object): if cls.IS_FIXED_SIZE: cell_size = cell.seek(0, io.SEEK_END) - fixed_cell_len = 514 if link_version > 3 else 512 + fixed_cell_len = 514 if link_protocol > 3 else 512 if cell_size > fixed_cell_len: raise ValueError('Payload of %s is too large (%i bytes), must be less than %i' % (cls.NAME, cell_size, fixed_cell_len)) @@ -216,12 +210,12 @@ class Cell(object): return cell.getvalue() @classmethod - def _unpack(cls, content, circ_id, link_version): + def _unpack(cls, content, circ_id, link_protocol): """ Subclass implementation for unpacking cell content. :param bytes content: payload to decode - :param int link_version: link protocol version + :param int link_protocol: link protocol version :param int circ_id: circuit id cell is for :returns: instance of this cell type @@ -268,11 +262,11 @@ class PaddingCell(Cell): self.payload = payload - def pack(self, link_version): - return PaddingCell._pack(link_version, self.payload) + def pack(self, link_protocol): + return PaddingCell._pack(link_protocol, self.payload) @classmethod - def _unpack(cls, content, circ_id, link_version): + def _unpack(cls, content, circ_id, link_protocol): return PaddingCell(content) def __hash__(self): @@ -325,7 +319,7 @@ class RelayCell(CircuitCell): elif stream_id and self.command in STREAM_ID_DISALLOWED: raise ValueError('%s relay cells concern the circuit itself and cannot have a stream id' % self.command) - def pack(self, link_version): + def pack(self, link_protocol): payload = io.BytesIO() payload.write(Size.CHAR.pack(self.command_int)) payload.write(Size.SHORT.pack(0)) # 'recognized' field @@ -334,10 +328,10 @@ class RelayCell(CircuitCell): payload.write(Size.SHORT.pack(len(self.data))) payload.write(self.data) - return RelayCell._pack(link_version, payload.getvalue(), self.circ_id) + return RelayCell._pack(link_protocol, payload.getvalue(), self.circ_id) @classmethod - def _unpack(cls, content, circ_id, link_version): + def _unpack(cls, content, circ_id, link_protocol): command, content = Size.CHAR.pop(content) _, content = Size.SHORT.pop(content) # 'recognized' field stream_id, content = Size.SHORT.pop(content) @@ -367,11 +361,11 @@ class DestroyCell(CircuitCell): super(DestroyCell, self).__init__(circ_id) self.reason, self.reason_int = CloseReason.get(reason) - def pack(self, link_version): - return DestroyCell._pack(link_version, Size.CHAR.pack(self.reason_int), self.circ_id) + def pack(self, link_protocol): + return DestroyCell._pack(link_protocol, Size.CHAR.pack(self.reason_int), self.circ_id) @classmethod - def _unpack(cls, content, circ_id, link_version): + def _unpack(cls, content, circ_id, link_protocol): content = content.rstrip(ZERO) if not content: @@ -406,11 +400,11 @@ class CreateFastCell(CircuitCell): super(CreateFastCell, self).__init__(circ_id) self.key_material = key_material - def pack(self, link_version): - return CreateFastCell._pack(link_version, self.key_material, self.circ_id) + def pack(self, link_protocol): + return CreateFastCell._pack(link_protocol, self.key_material, self.circ_id) @classmethod - def _unpack(cls, content, circ_id, link_version): + def _unpack(cls, content, circ_id, link_protocol): content = content.rstrip(ZERO) if len(content) != HASH_LEN: @@ -447,11 +441,11 @@ class CreatedFastCell(CircuitCell): self.key_material = key_material self.derivative_key = derivative_key - def pack(self, link_version): - return CreatedFastCell._pack(link_version, self.key_material + self.derivative_key, self.circ_id) + def pack(self, link_protocol): + return CreatedFastCell._pack(link_protocol, self.key_material + self.derivative_key, self.circ_id) @classmethod - def _unpack(cls, content, circ_id, link_version): + def _unpack(cls, content, circ_id, link_protocol): content = content.rstrip(ZERO) if len(content) != HASH_LEN * 2: @@ -477,7 +471,7 @@ class VersionsCell(Cell): def __init__(self, versions): self.versions = versions - def pack(self, link_version = None): + def pack(self, link_protocol = None): # Used for link version negotiation so we don't have that yet. This is fine # since VERSION cells avoid most version dependent attributes. @@ -485,14 +479,14 @@ class VersionsCell(Cell): return VersionsCell._pack(2, payload) @classmethod - def _unpack(cls, content, circ_id, link_version): - link_versions = [] + def _unpack(cls, content, circ_id, link_protocol): + link_protocols = [] while content: version, content = Size.SHORT.pop(content) - link_versions.append(version) + link_protocols.append(version) - return VersionsCell(link_versions) + return VersionsCell(link_protocols) def __hash__(self): return _hash_attr(self, 'versions') @@ -516,7 +510,7 @@ class NetinfoCell(Cell): self.receiver_address = receiver_address self.sender_addresses = sender_addresses - def pack(self, link_version): + def pack(self, link_protocol): payload = io.BytesIO() payload.write(Size.LONG.pack(int(datetime_to_unix(self.timestamp)))) payload.write(self.receiver_address.pack()) @@ -525,10 +519,10 @@ class NetinfoCell(Cell): for addr in self.sender_addresses: payload.write(addr.pack()) - return NetinfoCell._pack(link_version, payload.getvalue()) + return NetinfoCell._pack(link_protocol, payload.getvalue()) @classmethod - def _unpack(cls, content, circ_id, link_version): + def _unpack(cls, content, circ_id, link_protocol): if len(content) < 4: raise ValueError('NETINFO cell expected to start with a timestamp') @@ -591,11 +585,11 @@ class VPaddingCell(Cell): self.payload = payload - def pack(self, link_version): - return VPaddingCell._pack(link_version, self.payload) + def pack(self, link_protocol): + return VPaddingCell._pack(link_protocol, self.payload) @classmethod - def _unpack(cls, content, circ_id, link_version): + def _unpack(cls, content, circ_id, link_protocol): return VPaddingCell(payload = content) def __hash__(self): @@ -616,11 +610,11 @@ class CertsCell(Cell): def __init__(self, certs): self.certificates = certs - def pack(self, link_version): - return CertsCell._pack(link_version, Size.CHAR.pack(len(self.certificates)) + ''.join([cert.pack() for cert in self.certificates])) + def pack(self, link_protocol): + return CertsCell._pack(link_protocol, Size.CHAR.pack(len(self.certificates)) + ''.join([cert.pack() for cert in self.certificates])) @classmethod - def _unpack(cls, content, circ_id, link_version): + def _unpack(cls, content, circ_id, link_protocol): cert_count, content = Size.CHAR.pop(content) certs = [] @@ -659,7 +653,7 @@ class AuthChallengeCell(Cell): self.challenge = challenge self.methods = methods - def pack(self, link_version): + def pack(self, link_protocol): payload = io.BytesIO() payload.write(self.challenge) payload.write(Size.SHORT.pack(len(self.methods))) @@ -667,10 +661,10 @@ class AuthChallengeCell(Cell): for method in self.methods: payload.write(Size.SHORT.pack(method)) - return AuthChallengeCell._pack(link_version, payload.getvalue()) + return AuthChallengeCell._pack(link_protocol, payload.getvalue()) @classmethod - def _unpack(cls, content, circ_id, link_version): + def _unpack(cls, content, circ_id, link_protocol): if len(content) < AUTH_CHALLENGE_SIZE + 2: raise ValueError('AUTH_CHALLENGE payload should be at least 34 bytes, but was %i' % len(content)) diff --git a/stem/client/datatype.py b/stem/client/datatype.py index f5805a2a..8b4c8e64 100644 --- a/stem/client/datatype.py +++ b/stem/client/datatype.py @@ -476,57 +476,6 @@ class KDF(collections.namedtuple('KDF', ['key_hash', 'forward_digest', 'backward return KDF(key_hash, forward_digest, backward_digest, forward_key, backward_key) -class Circuit(collections.namedtuple('Circuit', ['socket', 'id', 'forward_digest', 'backward_digest', 'forward_key', 'backward_key'])): - """ - Circuit through which requests can be made of a `Tor relay's ORPort - <https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt>`_. - - :var stem.socket.RelaySocket socket: socket through which this circuit has been established - :var int id: circuit id - :var hashlib.sha1 forward_digest: digest for forward integrity check - :var hashlib.sha1 backward_digest: digest for backward integrity check - :var bytes forward_key: forward encryption key - :var bytes backward_key: backward encryption key - """ - - @staticmethod - def create(relay_socket, circ_id, link_version): - """ - Constructs a new circuit over the given ORPort. - """ - - if not stem.prereq.is_crypto_available(): - raise ImportError('Circuit construction requires the cryptography module') - - from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes - from cryptography.hazmat.backends import default_backend - - create_fast_cell = stem.client.cell.CreateFastCell(circ_id) - relay_socket.send(create_fast_cell.pack(link_version)) - - response = stem.client.cell.Cell.unpack(relay_socket.recv(), link_version) - created_fast_cells = filter(lambda cell: isinstance(cell, stem.client.cell.CreatedFastCell), response) - - if not created_fast_cells: - raise ValueError('We should get a CREATED_FAST response from a CREATE_FAST request') - - created_fast_cell = created_fast_cells[0] - kdf = KDF.from_value(create_fast_cell.key_material + created_fast_cell.key_material) - ctr = modes.CTR(ZERO * (algorithms.AES.block_size / 8)) - - if created_fast_cell.derivative_key != kdf.key_hash: - raise ValueError('Remote failed to prove that it knows our shared key') - - return Circuit( - relay_socket, - circ_id, - hashlib.sha1(kdf.forward_digest), - hashlib.sha1(kdf.backward_digest), - Cipher(algorithms.AES(kdf.forward_key), ctr, default_backend()).encryptor(), - Cipher(algorithms.AES(kdf.backward_key), ctr, default_backend()).decryptor(), - ) - - setattr(Size, 'CHAR', Size('CHAR', 1, '!B')) setattr(Size, 'SHORT', Size('SHORT', 2, '!H')) setattr(Size, 'LONG', Size('LONG', 4, '!L'))

1 0

[stem/master] Accept string and hash digests
by atagar＠torproject.org 07 Feb '18

07 Feb '18

commit bdc96f01bf91c3f7758e12a58e5e388b88734ad9 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Feb 5 10:14:14 2018 -0800 Accept string and hash digests The RELAY cells actually work from a truncated digest. Only the two ends know the full sha1 digest for the message series that has been sent. Adjusting the RelayCell so it takes all three forms that the digest comes in. --- stem/client/cell.py | 12 ++++++++++++ test/unit/client/cell.py | 8 ++++++++ 2 files changed, 20 insertions(+) diff --git a/stem/client/cell.py b/stem/client/cell.py index ed8932c4..17f27202 100644 --- a/stem/client/cell.py +++ b/stem/client/cell.py @@ -308,6 +308,18 @@ class RelayCell(CircuitCell): IS_FIXED_SIZE = True def __init__(self, circ_id, command, data, digest = 0, stream_id = 0): + if 'hashlib.HASH' in str(type(digest)): + # Unfortunately hashlib generates from a dynamic private class so + # isinstance() isn't such a great option. + + digest = Size.LONG.unpack(digest.digest()[:4]) + elif isinstance(digest, str): + digest = Size.LONG.unpack(digest[:4]) + elif isinstance(digest, int): + pass + else: + raise ValueError('RELAY cell digest must be a hash, string, or int but was a %s' % type(digest).__name__) + super(RelayCell, self).__init__(circ_id) self.command, self.command_int = RelayCommand.get(command) self.data = data diff --git a/test/unit/client/cell.py b/test/unit/client/cell.py index f53355dd..19744c6d 100644 --- a/test/unit/client/cell.py +++ b/test/unit/client/cell.py @@ -3,6 +3,7 @@ Unit tests for the stem.client.cell. """ import datetime +import hashlib import os import unittest @@ -153,6 +154,13 @@ class TestCell(unittest.TestCase): self.assertEqual(digest, cell.digest) self.assertEqual(stream_id, cell.stream_id) + digest = hashlib.sha1('hi') + self.assertEqual(3257622417, RelayCell(5, 'RELAY_BEGIN_DIR', '', digest, 564346860).digest) + self.assertEqual(3257622417, RelayCell(5, 'RELAY_BEGIN_DIR', '', digest.digest(), 564346860).digest) + self.assertEqual(3257622417, RelayCell(5, 'RELAY_BEGIN_DIR', '', 3257622417, 564346860).digest) + self.assertRaisesRegexp(ValueError, 'RELAY cell digest must be a hash, string, or int but was a list', RelayCell, 5, 'RELAY_BEGIN_DIR', '', [], 564346860) + self.assertRaisesRegexp(ValueError, "Invalid enumeration 'NO_SUCH_COMMAND', options are RELAY_BEGIN, RELAY_DATA", RelayCell, 5, 'NO_SUCH_COMMAND', '', 5, 564346860) + def test_destroy_cell(self): for cell_bytes, (circ_id, reason, reason_int) in DESTROY_CELLS.items(): self.assertEqual(cell_bytes, DestroyCell(circ_id, reason).pack(5))

1 0

[stem/master] Initial Circuit.send() method
by atagar＠torproject.org 07 Feb '18

07 Feb '18

commit b6ec30d450e929976acf3fce9378f0c3a777d938 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Feb 5 11:24:34 2018 -0800 Initial Circuit.send() method Quite a few things I dislike about this, but it works! Next up: testing and refinement... --- stem/client/__init__.py | 37 ++++++++++++++++++++++++++++++++++++- stem/client/cell.py | 17 +++++++++++++++-- 2 files changed, 51 insertions(+), 3 deletions(-) diff --git a/stem/client/__init__.py b/stem/client/__init__.py index fdb436b6..4f9ed17c 100644 --- a/stem/client/__init__.py +++ b/stem/client/__init__.py @@ -19,6 +19,7 @@ a wrapper for :class:`~stem.socket.RelaySocket`, much the same way as +- close - shuts down our connection """ +import copy import hashlib import stem @@ -26,7 +27,7 @@ import stem.client.cell import stem.socket import stem.util.connection -from stem.client.datatype import ZERO, AddrType, Address, KDF +from stem.client.datatype import ZERO, AddrType, Address, KDF, split __all__ = [ 'cell', @@ -207,3 +208,37 @@ class Circuit(object): self.backward_digest = hashlib.sha1(kdf.backward_digest) self.forward_key = Cipher(algorithms.AES(kdf.forward_key), ctr, default_backend()).encryptor() self.backward_key = Cipher(algorithms.AES(kdf.backward_key), ctr, default_backend()).decryptor() + + def send(self, command, data, stream_id = 0): + """ + Sends a message over the circuit. + + :param stem.client.RelayCommand command: command to be issued + :param bytes data: message payload + :param int stream_id: specific stream this concerns + """ + + # TODO: move RelayCommand to this base module? + # TODO: add lock + + orig_digest = self.forward_digest.copy() + orig_key = copy.copy(self.forward_key) + + try: + cell = stem.client.cell.RelayCell(self.id, command, data, 0, stream_id) + payload_without_digest = cell.pack(self.relay.link_protocol)[3:] + self.forward_digest.update(payload_without_digest) + + cell = stem.client.cell.RelayCell(self.id, command, data, self.forward_digest, stream_id) + header, payload = split(cell.pack(self.relay.link_protocol), 3) + encrypted_payload = header + self.forward_key.update(payload) + + self.relay._orport.send(encrypted_payload) + except: + self.forward_digest = orig_digest + self.forward_key = orig_key + raise + + def close(self): + self.relay._orport.send(stem.client.cell.DestroyCell(self.id).pack(self.relay.link_protocol)) + del self.relay._circuits[self.id] diff --git a/stem/client/cell.py b/stem/client/cell.py index 17f27202..c4a19940 100644 --- a/stem/client/cell.py +++ b/stem/client/cell.py @@ -307,7 +307,7 @@ class RelayCell(CircuitCell): VALUE = 3 IS_FIXED_SIZE = True - def __init__(self, circ_id, command, data, digest = 0, stream_id = 0): + def __init__(self, circ_id, command, data, digest = 0, stream_id = 0, raw_content = None): if 'hashlib.HASH' in str(type(digest)): # Unfortunately hashlib generates from a dynamic private class so # isinstance() isn't such a great option. @@ -325,6 +325,7 @@ class RelayCell(CircuitCell): self.data = data self.digest = digest self.stream_id = stream_id + self._raw_content = raw_content if not stream_id and self.command in STREAM_ID_REQUIRED: raise ValueError('%s relay cells require a stream id' % self.command) @@ -342,8 +343,20 @@ class RelayCell(CircuitCell): return RelayCell._pack(link_protocol, payload.getvalue(), self.circ_id) + def decrypt(self, circ): + # TODO: clearly funky, just a spot to start... + + if not self._raw_content: + raise ValueError('Only received cells can be decrypted') + + decrypted = circ.backward_key.update(self._raw_content) + return RelayCell._unpack(decrypted, self.circ_id, 3) + + @classmethod def _unpack(cls, content, circ_id, link_protocol): + orig_content = content + command, content = Size.CHAR.pop(content) _, content = Size.SHORT.pop(content) # 'recognized' field stream_id, content = Size.SHORT.pop(content) @@ -351,7 +364,7 @@ class RelayCell(CircuitCell): data_len, content = Size.SHORT.pop(content) data, content = split(content, data_len) - return RelayCell(circ_id, command, data, digest, stream_id) + return RelayCell(circ_id, command, data, digest, stream_id, orig_content) def __hash__(self): return _hash_attr(self, 'command_int', 'stream_id', 'digest', 'data')

1 0

[stem/master] Make Relay thread safe
by atagar＠torproject.org 07 Feb '18

07 Feb '18

commit 331127483838c416b279d30cf041deb678984ab2 Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Feb 6 09:53:08 2018 -0800 Make Relay thread safe --- stem/client/__init__.py | 91 +++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 40 deletions(-) diff --git a/stem/client/__init__.py b/stem/client/__init__.py index 8d34b626..5a9d09e6 100644 --- a/stem/client/__init__.py +++ b/stem/client/__init__.py @@ -16,11 +16,18 @@ a wrapper for :class:`~stem.socket.RelaySocket`, much the same way as | |- is_alive - reports if our connection is open or closed |- connection_time - time when we last connected or disconnected - +- close - shuts down our connection + |- close - shuts down our connection + | + +- create_circuit - establishes a new circuit + + Circuit - Circuit we've established through a relay. + |- send - sends a message through this circuit + +- close - closes this circuit """ import copy import hashlib +import threading import stem import stem.client.cell @@ -47,6 +54,7 @@ class Relay(object): def __init__(self, orport, link_protocol): self.link_protocol = link_protocol self._orport = orport + self._orport_lock = threading.RLock() self._circuits = {} @staticmethod @@ -138,40 +146,42 @@ class Relay(object): :func:`~stem.socket.BaseSocket.close` method. """ - return self._orport.close() + with self._orport_lock: + return self._orport.close() def create_circuit(self): """ Establishes a new circuit. """ - # Find an unused circuit id. Since we're initiating the circuit we pick any - # value from a range that's determined by our link protocol. + with self._orport_lock: + # Find an unused circuit id. Since we're initiating the circuit we pick any + # value from a range that's determined by our link protocol. - circ_id = 0x80000000 if self.link_protocol > 3 else 0x01 + circ_id = 0x80000000 if self.link_protocol > 3 else 0x01 - while circ_id in self._circuits: - circ_id += 1 + while circ_id in self._circuits: + circ_id += 1 - create_fast_cell = stem.client.cell.CreateFastCell(circ_id) - self._orport.send(create_fast_cell.pack(self.link_protocol)) + create_fast_cell = stem.client.cell.CreateFastCell(circ_id) + self._orport.send(create_fast_cell.pack(self.link_protocol)) - response = stem.client.cell.Cell.unpack(self._orport.recv(), self.link_protocol) - created_fast_cells = filter(lambda cell: isinstance(cell, stem.client.cell.CreatedFastCell), response) + response = stem.client.cell.Cell.unpack(self._orport.recv(), self.link_protocol) + created_fast_cells = filter(lambda cell: isinstance(cell, stem.client.cell.CreatedFastCell), response) - if not created_fast_cells: - raise ValueError('We should get a CREATED_FAST response from a CREATE_FAST request') + if not created_fast_cells: + raise ValueError('We should get a CREATED_FAST response from a CREATE_FAST request') - created_fast_cell = created_fast_cells[0] - kdf = KDF.from_value(create_fast_cell.key_material + created_fast_cell.key_material) + created_fast_cell = created_fast_cells[0] + kdf = KDF.from_value(create_fast_cell.key_material + created_fast_cell.key_material) - if created_fast_cell.derivative_key != kdf.key_hash: - raise ValueError('Remote failed to prove that it knows our shared key') + if created_fast_cell.derivative_key != kdf.key_hash: + raise ValueError('Remote failed to prove that it knows our shared key') - circ = Circuit(self, circ_id, kdf) - self._circuits[circ.id] = circ + circ = Circuit(self, circ_id, kdf) + self._circuits[circ.id] = circ - return circ + return circ def __enter__(self): return self @@ -219,30 +229,31 @@ class Circuit(object): """ # TODO: move RelayCommand to this base module? - # TODO: add lock - orig_digest = self.forward_digest.copy() - orig_key = copy.copy(self.forward_key) + with self.relay._orport_lock: + orig_digest = self.forward_digest.copy() + orig_key = copy.copy(self.forward_key) - try: - cell = stem.client.cell.RelayCell(self.id, command, data, 0, stream_id) - payload_without_digest = cell.pack(self.relay.link_protocol)[3:] - self.forward_digest.update(payload_without_digest) + try: + cell = stem.client.cell.RelayCell(self.id, command, data, 0, stream_id) + payload_without_digest = cell.pack(self.relay.link_protocol)[3:] + self.forward_digest.update(payload_without_digest) - cell = stem.client.cell.RelayCell(self.id, command, data, self.forward_digest, stream_id) - header, payload = split(cell.pack(self.relay.link_protocol), 3) - encrypted_payload = header + self.forward_key.update(payload) + cell = stem.client.cell.RelayCell(self.id, command, data, self.forward_digest, stream_id) + header, payload = split(cell.pack(self.relay.link_protocol), 3) + encrypted_payload = header + self.forward_key.update(payload) - self.relay._orport.send(encrypted_payload) - reply = next(stem.client.cell.Cell.unpack(self.relay._orport.recv(), self.relay.link_protocol)) + self.relay._orport.send(encrypted_payload) + reply = next(stem.client.cell.Cell.unpack(self.relay._orport.recv(), self.relay.link_protocol)) - decrypted = self.backward_key.update(reply.pack(3)[3:]) - return stem.client.cell.RelayCell._unpack(decrypted, self.id, 3) - except: - self.forward_digest = orig_digest - self.forward_key = orig_key - raise + decrypted = self.backward_key.update(reply.pack(3)[3:]) + return stem.client.cell.RelayCell._unpack(decrypted, self.id, 3) + except: + self.forward_digest = orig_digest + self.forward_key = orig_key + raise def close(self): - self.relay._orport.send(stem.client.cell.DestroyCell(self.id).pack(self.relay.link_protocol)) - del self.relay._circuits[self.id] + with self.relay._orport_lock: + self.relay._orport.send(stem.client.cell.DestroyCell(self.id).pack(self.relay.link_protocol)) + del self.relay._circuits[self.id]

1 0

[stem/master] Only validate RELAY cells when unencrypted
by atagar＠torproject.org 07 Feb '18

07 Feb '18

commit 0d0a018d721db77015453d284bb348e7db72cb84 Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Feb 7 10:35:59 2018 -0800 Only validate RELAY cells when unencrypted When the cell is encrypted these fields are obviously essentially noise. In practice this seems to be causing inconsistent failures roughly 1/10 of the time... Traceback (most recent call last): File "client-or-stream-raw.py", line 30, in <module> reply = circ.send('RELAY_BEGIN_DIR', stream_id = 1) File "/home/atagar/Desktop/tor/endosome/stem/client/__init__.py", line 246, in send reply = next(stem.client.cell.Cell.unpack(self.relay._orport.recv(), self.relay.link_protocol)) File "/home/atagar/Desktop/tor/endosome/stem/client/cell.py", line 135, in unpack cell, content = Cell.pop(content, link_protocol) File "/home/atagar/Desktop/tor/endosome/stem/client/cell.py", line 166, in pop return cls._unpack(payload, circ_id, link_protocol), content File "/home/atagar/Desktop/tor/endosome/stem/client/cell.py", line 349, in _unpack return RelayCell(circ_id, command, data, digest, stream_id, recognized) File "/home/atagar/Desktop/tor/endosome/stem/client/cell.py", line 327, in __init__ raise ValueError('%s relay cells concern the circuit itself and cannot have a stream id' % self.command) ValueError: RELAY_TRUNCATE relay cells concern the circuit itself and cannot have a stream id --- stem/client/__init__.py | 4 +--- stem/client/cell.py | 9 +++++---- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/stem/client/__init__.py b/stem/client/__init__.py index f4fc718b..2b853baa 100644 --- a/stem/client/__init__.py +++ b/stem/client/__init__.py @@ -218,7 +218,7 @@ class Circuit(object): self.forward_key = Cipher(algorithms.AES(kdf.forward_key), ctr, default_backend()).encryptor() self.backward_key = Cipher(algorithms.AES(kdf.backward_key), ctr, default_backend()).decryptor() - def send(self, command, data, stream_id = 0): + def send(self, command, data = '', stream_id = 0): """ Sends a message over the circuit. @@ -227,8 +227,6 @@ class Circuit(object): :param int stream_id: specific stream this concerns """ - # TODO: move RelayCommand to this base module? - with self.relay._orport_lock: orig_digest = self.forward_digest.copy() orig_key = copy.copy(self.forward_key) diff --git a/stem/client/cell.py b/stem/client/cell.py index 13f14f3c..c5f8f20b 100644 --- a/stem/client/cell.py +++ b/stem/client/cell.py @@ -321,10 +321,11 @@ class RelayCell(CircuitCell): self.digest = digest self.stream_id = stream_id - if not stream_id and self.command in STREAM_ID_REQUIRED: - raise ValueError('%s relay cells require a stream id' % self.command) - elif stream_id and self.command in STREAM_ID_DISALLOWED: - raise ValueError('%s relay cells concern the circuit itself and cannot have a stream id' % self.command) + if digest == 0: + if not stream_id and self.command in STREAM_ID_REQUIRED: + raise ValueError('%s relay cells require a stream id' % self.command) + elif stream_id and self.command in STREAM_ID_DISALLOWED: + raise ValueError('%s relay cells concern the circuit itself and cannot have a stream id' % self.command) def pack(self, link_protocol): payload = io.BytesIO()

1 0

[stem/master] Make Relay iterable to get circuits
by atagar＠torproject.org 07 Feb '18

07 Feb '18

commit 94cd4204b86a21f21f73c28f40073841269f6503 Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Feb 6 10:26:30 2018 -0800 Make Relay iterable to get circuits Thought is: iterate over relays to get circuits, and iterate over circuits to get streams. --- stem/client/__init__.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/stem/client/__init__.py b/stem/client/__init__.py index 5a9d09e6..3f8507e5 100644 --- a/stem/client/__init__.py +++ b/stem/client/__init__.py @@ -183,6 +183,11 @@ class Relay(object): return circ + def __iter__(self): + with self._orport_lock: + for circ in self._circuits.values(): + yield circ + def __enter__(self): return self

1 0

[stem/master] Omit our address from NETINFO
by atagar＠torproject.org 07 Feb '18

07 Feb '18

commit cfadcde5a08af28e2e39eb467c0ba81aace662ef Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Feb 6 11:41:02 2018 -0800 Omit our address from NETINFO We already weren't providing it so this just adjusts the comment talking about why. There's a few issues... 1. Getting our address with python is clunky. Simplest approach is... socket.gethostbyname(socket.gethostname()) ... but I suspect what we really want is our external address, which I'm unsure how best to get. 2. We don't have a usecase at present where it helps. Tor is quite happy to not receive one, and seems if I provide junk data it's happy to accept that too. For now just opting to exclude it until we have a concrete use case, which in turn will inform if the internal address is acceptable or we must resolve something else. --- stem/client/__init__.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/stem/client/__init__.py b/stem/client/__init__.py index b743a0eb..f4fc718b 100644 --- a/stem/client/__init__.py +++ b/stem/client/__init__.py @@ -103,7 +103,9 @@ class Relay(object): conn.close() raise stem.SocketError('Unable to find a common link protocol. We support %s but %s:%i supports %s.' % (', '.join(link_protocols), address, port, ', '.join(versions_reply.versions))) - # TODO: we should fill in our address, right? + # Establishing connections requires sending a NETINFO, but including our + # address is optional. We can revisit including it when we have a usecase + # where it would help. link_protocol = max(common_protocols) conn.send(stem.client.cell.NetinfoCell(Address(address), []).pack(link_protocol))

1 0

[stem/master] Drop todo note about recognized field
by atagar＠torproject.org 07 Feb '18

07 Feb '18

commit a50a07da4bccad91e4883c174cd4bfa00a6f49a8 Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Feb 7 10:21:25 2018 -0800 Drop todo note about recognized field Sent a little patch to clarify this field... https://trac.torproject.org/projects/tor/ticket/25171 --- stem/client/cell.py | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/stem/client/cell.py b/stem/client/cell.py index 4ac8d40e..13f14f3c 100644 --- a/stem/client/cell.py +++ b/stem/client/cell.py @@ -292,18 +292,11 @@ class RelayCell(CircuitCell): :var stem.client.RelayCommand command: command to be issued :var int command_int: integer value of our command :var bytes data: payload of the cell - :var int recognized: zero if endpoint is this hop, non-zero otherwise + :var int recognized: zero if cell is decrypted, non-zero otherwise :var int digest: running digest held with the relay :var int stream_id: specific stream this concerns """ - # TODO: Relay cells also have a 'recognized' field but from the spec I really - # haven't a clue what the heck it is. The spec makes multiple mentions to - # "when the 'recognized' field of a RELAY cell is zero" but no mention to if - # it's non-zero or what the field actually is. :/ - # - # For now just leaving it out. I'll file a ticket to ask about it later. - NAME = 'RELAY' VALUE = 3 IS_FIXED_SIZE = True

1 0

[stem/master] Building blocks for basic circuit requests
by atagar＠torproject.org 07 Feb '18

07 Feb '18

commit 79520a8290578d5fd96eaf06c8080f91ace0bddf Merge: 76e8eb5f 0d0a018d Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Feb 7 11:21:03 2018 -0800 Building blocks for basic circuit requests Low-level building blocks for circuit requests. This is *not* the API we'll vend to end users (it's much too raw), but we now have everything from Endosome [1] we need to download descriptors... import stem.client with stem.client.Relay.connect('127.0.0.1', 12345, [3]) as relay: circ = relay.create_circuit() circ.send('RELAY_BEGIN_DIR', stream_id = 1) desc = circ.send('RELAY_DATA', 'GET /tor/server/authority HTTP/1.0\r\n\r\n', stream_id = 1).data circ.close() print(desc) When run this prints our own descriptor... % python demo.py HTTP/1.0 200 OK Date: Wed, 07 Feb 2018 18:42:41 GMT Content-Type: text/plain Content-Encoding: identity Expires: Fri, 09 Feb 2018 18:42:41 GMT router Unnamed 97.113.177.53 12345 0 23456 identity-ed25519 -----BEGIN ED25519 CERT----- AQQABm/qAazUltT1iUUbIMw8VNNhGb50FDHKJz6S94FLQNxL0LObAQAgBAAapbO9 iLFD0l9SEiEMFQWIT2VnbLyCZKvbrxTs5ULC1l1hQPoui6Y/lEd3yjrQhIs/vl6R 1S6FbwSFDmiXOzq47mFrse4C71ht3TpLOD0F3wiyjWtsqU1k7iPmmpejUgs= -----END ED25519 CERT----- master-key-ed25519 GqWzvYixQ9JfUhIhDBUFiE Next steps are to... a. Support ORPort downloads in 'stem.descriptor.remote'. b. Add more integration tests. c. Give more thought to the API we'd like to vend. d. Brainstorm a GSoC project idea that expands these capabilities. However, first there's some other projects (in particular v3 hidden services) that need my time. [1] https://github.com/teor2345/endosome stem/__init__.py | 1 + stem/client/__init__.py | 440 +++++++++++++++-------------------- stem/client/cell.py | 419 ++++++++++++++++++++++------------ stem/client/datatype.py | 482 +++++++++++++++++++++++++++++++++++++++ stem/control.py | 9 +- stem/socket.py | 1 + test/integ/__init__.py | 1 + test/integ/client/__init__.py | 7 + test/integ/client/connection.py | 66 ++++++ test/integ/control/controller.py | 4 +- test/runner.py | 4 +- test/settings.cfg | 4 +- test/unit/client/__init__.py | 1 + test/unit/client/address.py | 27 ++- test/unit/client/cell.py | 146 +++++++++--- test/unit/client/certificate.py | 2 +- test/unit/client/kdf.py | 27 +++ test/unit/client/size.py | 2 +- 18 files changed, 1178 insertions(+), 465 deletions(-)

1 0