April 2018 - tor-commits - lists.torproject.org

[stem/master] Resync manual data
by atagar＠torproject.org 24 Apr '18

24 Apr '18

commit 530b3bc144a7d323ce30aa604eff63560c716400 Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Apr 21 16:53:56 2018 -0700 Resync manual data Tests we're adding are for the ONLINE test target, so getting us to a spot first where everything passes. --- stem/cached_tor_manual.sqlite | Bin 239616 -> 238592 bytes stem/manual.py | 15 +++++++++++++-- stem/settings.cfg | 9 ++------- test/integ/manual.py | 38 ++++---------------------------------- 4 files changed, 19 insertions(+), 43 deletions(-) diff --git a/stem/cached_tor_manual.sqlite b/stem/cached_tor_manual.sqlite index 749bf86b..6b17f4da 100644 Binary files a/stem/cached_tor_manual.sqlite and b/stem/cached_tor_manual.sqlite differ diff --git a/stem/manual.py b/stem/manual.py index 0bff9b68..3d734026 100644 --- a/stem/manual.py +++ b/stem/manual.py @@ -758,8 +758,19 @@ def _add_config_options(config_options, category, lines): last_option, usage, description = None, None, [] - if lines and lines[0].startswith('The following options'): - lines = lines[lines.index(''):] # drop the initial description + # Drop the section description. Each ends with a paragraph saying 'The + # following options...'. + + desc_paragraph_index = None + + for i, line in enumerate(lines): + if 'The following options' in line: + desc_paragraph_index = i + break + + if desc_paragraph_index is not None: + lines = lines[desc_paragraph_index:] # trim to the description paragrah + lines = lines[lines.index(''):] # drop the paragraph for line in lines: if line and not line.startswith(' '): diff --git a/stem/settings.cfg b/stem/settings.cfg index 5ce1cfa0..7e84210c 100644 --- a/stem/settings.cfg +++ b/stem/settings.cfg @@ -32,7 +32,6 @@ manual.important ExitPolicy manual.important MyFamily manual.important Nickname manual.important ORPort -manual.important PortForwarding manual.important AccountingMax manual.important AccountingStart @@ -190,8 +189,8 @@ manual.summary.RejectPlaintextPorts Prevents connections on risky ports manual.summary.OptimisticData Use exits without confirmation that prior connections succeeded manual.summary.Tor2webMode Establish non-anonymous hidden service connections manual.summary.Tor2webRendezvousPoints Rendezvous points to use for hidden services when in Tor2webMode -manual.summary._HSLayer2Nodes # TODO: https://trac.torproject.org/projects/tor/ticket/25581 -manual.summary._HSLayer3Nodes # TODO: https://trac.torproject.org/projects/tor/ticket/25581 +manual.summary.HSLayer2Nodes permissible relays for the second hop of HS circuits +manual.summary.HSLayer3Nodes permissible relays for the third hop of HS circuits manual.summary.UseMicrodescriptors Retrieve microdescriptors rather than server descriptors manual.summary.PathBiasCircThreshold Number of circuits through a guard before applying bias checks manual.summary.PathBiasNoticeRate Fraction of circuits that must succeed before logging a notice @@ -222,7 +221,6 @@ manual.summary.BridgeDistribution Distribution method BrideDB should provide our manual.summary.ContactInfo Contact information for this relay manual.summary.ExitRelay Allow relaying of exit traffic manual.summary.ExitPolicy Traffic destinations that can exit from this relay -manual.summary.ExitPolicyDefault # TODO: https://trac.torproject.org/projects/tor/ticket/25582 manual.summary.ExitPolicyRejectPrivate Prevent exiting on the local network manual.summary.ExitPolicyRejectLocalInterfaces More extensive prevention of exiting on the local network manual.summary.ReducedExitPolicy Customized reduced exit policy @@ -232,8 +230,6 @@ manual.summary.MyFamily Other relays this operator administers manual.summary.Nickname Identifier for this relay manual.summary.NumCPUs Number of processes spawned for decryption manual.summary.ORPort Port used to accept relay traffic -manual.summary.PortForwarding Use UPnP or NAT-PMP if needed to relay -manual.summary.PortForwardingHelper Executable for configuring port forwarding manual.summary.PublishServerDescriptor Types of descriptors published manual.summary.ShutdownWaitLength Delay before quitting after receiving a SIGINT signal manual.summary.SSLKeyLifetime Lifetime for our link certificate @@ -369,7 +365,6 @@ manual.summary.TestingDirAuthVoteHSDir Relays to give the HSDir flag to manual.summary.TestingDirAuthVoteHSDirIsStrict Only grant the HSDir flag to relays listed by TestingDirAuthVoteHSDir manual.summary.TestingEnableConnBwEvent Allow controllers to request CONN_BW events manual.summary.TestingEnableCellStatsEvent Allow controllers to request CELL_STATS events -manual.summary.TestingEnableTbEmptyEvent Allow controllers to request TB_EMPTY events manual.summary.TestingMinExitFlagThreshold Lower bound for assigning the Exit flag manual.summary.TestingLinkCertLifetime Duration of our ed25519 certificate manual.summary.TestingAuthKeyLifetime Duration for our ed25519 signing key diff --git a/test/integ/manual.py b/test/integ/manual.py index b08179db..fa7e0ca5 100644 --- a/test/integ/manual.py +++ b/test/integ/manual.py @@ -52,39 +52,8 @@ EXPECTED_FILE_DESCRIPTION = 'Specify a new configuration file to contain further EXPECTED_BANDWIDTH_RATE_DESCRIPTION = 'A token bucket limits the average incoming bandwidth usage on this node to the specified number of bytes per second, and the average outgoing bandwidth usage to that same value. If you want to run a relay in the public network, this needs to be at the very least 75 KBytes for a relay (that is, 600 kbits) or 50 KBytes for a bridge (400 kbits) -- but of course, more is better; we recommend at least 250 KBytes (2 mbits) if possible. (Default: 1 GByte)\n\nNote that this option, and other bandwidth-limiting options, apply to TCP data only: They do not count TCP headers or DNS traffic.\n\nWith this option, and in other options that take arguments in bytes, KBytes, and so on, other formats are also supported. Notably, "KBytes" can also be written as "kilobytes" or "kb"; "MBytes" can be written as "megabytes" or "MB"; "kbits" can be written as "kilobits"; and so forth. Tor also accepts "byte" and "bit" in the singular. The prefixes "tera" and "T" are a lso recognized. If no units are given, we default to bytes. To avoid confusion, we recommend writing "bytes" or "bits" explicitly, since it\'s easy to forget that "B" means bytes, not bits.' -EXPECTED_EXIT_POLICY_DESCRIPTION = """ -Set an exit policy for this server. Each policy is of the form "accept[6]|reject[6] ADDR[/MASK][:PORT]". If /MASK is omitted then this policy just applies to the host given. Instead of giving a host or network you can also use "*" to denote the universe (0.0.0.0/0 and ::/128), or *4 to denote all IPv4 addresses, and *6 to denote all IPv6 addresses. PORT can be a single port number, an interval of ports "FROM_PORT-TO_PORT", or "*". If PORT is omitted, that means "*". - -For example, "accept 18.7.22.69:*,reject 18.0.0.0/8:*,accept *:*" would reject any IPv4 traffic destined for MIT except for web.mit.edu, and accept any other IPv4 or IPv6 traffic. - -Tor also allows IPv6 exit policy entries. For instance, "reject6 [FC00::]/7:*" rejects all destinations that share 7 most significant bit prefix with address FC00::. Respectively, "accept6 [C000::]/3:*" accepts all destinations that share 3 most significant bit prefix with address C000::. - -accept6 and reject6 only produce IPv6 exit policy entries. Using an IPv4 address with accept6 or reject6 is ignored and generates a warning. accept/reject allows either IPv4 or IPv6 addresses. Use *4 as an IPv4 wildcard address, and *6 as an IPv6 wildcard address. accept/reject * expands to matching IPv4 and IPv6 wildcard address rules. - -To specify all IPv4 and IPv6 internal and link-local networks (including 0.0.0.0/8, 169.254.0.0/16, 127.0.0.0/8, 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12, [::]/8, [FC00::]/7, [FE80::]/10, [FEC0::]/10, [FF00::]/8, and [::]/127), you can use the "private" alias instead of an address. ("private" always produces rules for IPv4 and IPv6 addresses, even when used with accept6/reject6.) - -Private addresses are rejected by default (at the beginning of your exit policy), along with any configured primary public IPv4 and IPv6 addresses. These private addresses are rejected unless you set the ExitPolicyRejectPrivate config option to 0. For example, once you've done that, you could allow HTTP to 127.0.0.1 and block all other connections to internal networks with "accept 127.0.0.1:80,reject private:*", though that may also allow connections to your own computer that are addressed to its public (external) IP address. See RFC 1918 and RFC 3330 for more details about internal and reserved IP address space. See ExitPolicyRejectLocalInterfaces if you want to block every address on the relay, even those that aren't advertised in the descriptor. - -This directive can be specified multiple times so you don't have to put it all on one line. - -Policies are considered first to last, and the first match wins. If you want to allow the same ports on IPv4 and IPv6, write your rules using accept/reject *. If you want to allow different ports on IPv4 and IPv6, write your IPv6 rules using accept6/reject6 *6, and your IPv4 rules using accept/reject *4. If you want to _replace_ the default exit policy, end your exit policy with either a reject *:* or an accept *:*. Otherwise, you're _augmenting_ (prepending to) the default exit policy. - -If you want to use a reduced exit policy rather than the default exit policy, set "ReducedExitPolicy 1". If you want to replace the default exit policy with your custom exit policy, end your exit policy with either a reject : or an accept :. Otherwise, you're augmenting (prepending to) the default or reduced exit policy. - -The default exit policy is: - - reject *:25 - reject *:119 - reject *:135-139 - reject *:445 - reject *:563 - reject *:1214 - reject *:4661-4666 - reject *:6346-6429 - reject *:6699 - reject *:6881-6999 - accept *:* -""".strip() +EXPECTED_EXIT_POLICY_DESCRIPTION_START = 'Set an exit policy for this server. Each policy' +EXPECTED_EXIT_POLICY_DESCRIPTION_END = 'it applies to both IPv4 and IPv6 addresses.' class TestManual(unittest.TestCase): @@ -164,7 +133,8 @@ class TestManual(unittest.TestCase): return manual = stem.manual.Manual.from_man(self.man_path) - self.assertEqual(EXPECTED_EXIT_POLICY_DESCRIPTION, manual.config_options['ExitPolicy'].description) + self.assertTrue(manual.config_options['ExitPolicy'].description.startswith(EXPECTED_EXIT_POLICY_DESCRIPTION_START)) + self.assertTrue(manual.config_options['ExitPolicy'].description.endswith(EXPECTED_EXIT_POLICY_DESCRIPTION_END)) def test_that_cache_is_up_to_date(self): """

1 0

[stem/master] Parse descriptor fetch headers
by atagar＠torproject.org 24 Apr '18

24 Apr '18

commit 9f1a2824b5a135a2b19c8426cb85c2deef11afde Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Apr 23 19:36:40 2018 -0700 Parse descriptor fetch headers Oh interesting! I didn't realize that the ORPort response was an HTTP response, headers and all. --- stem/descriptor/remote.py | 45 +++++++++++++++++++++++++++++++++++++++------ 1 file changed, 39 insertions(+), 6 deletions(-) diff --git a/stem/descriptor/remote.py b/stem/descriptor/remote.py index ebf9e786..5655c84c 100644 --- a/stem/descriptor/remote.py +++ b/stem/descriptor/remote.py @@ -256,14 +256,29 @@ def get_consensus(authority_v3ident = None, microdescriptor = False, **query_arg def _download_from_orport(endpoint, resource): """ - Downloads descriptors from the given orport. + Downloads descriptors from the given orport. Payload is just like an http + response (headers and all)... + + :: + + HTTP/1.0 200 OK + Date: Mon, 23 Apr 2018 18:43:47 GMT + Content-Type: text/plain + X-Your-Address-Is: 216.161.254.25 + Content-Encoding: identity + Expires: Wed, 25 Apr 2018 18:43:47 GMT + + router dannenberg 193.23.244.244 443 0 80 + identity-ed25519 + ... rest of the descriptor content... :param stem.ORPort endpoint: endpoint to download from :param str resource: descriptor resource to download - :returns: **str** with the descirptor data + :returns: two value tuple of the form (data, reply_headers) :raises: + * :class:`stem.ProtocolError` if not a valid descriptor response * :class:`stem.SocketError` if unable to establish a connection """ @@ -272,7 +287,25 @@ def _download_from_orport(endpoint, resource): with stem.client.Relay.connect(endpoint.address, endpoint.port, link_protocol) as relay: with relay.create_circuit() as circ: circ.send('RELAY_BEGIN_DIR', stream_id = 1) - return circ.send('RELAY_DATA', resource, stream_id = 1).data + lines = circ.send('RELAY_DATA', resource, stream_id = 1).data.splitlines() + first_line = lines.pop(0) + + if first_line != 'HTTP/1.0 200 OK': + raise stem.ProtocolError("Response should begin with HTTP success, but was '%s'" % first_line) + + headers = {} + next_line = lines.pop(0) + + while next_line: + if ': ' not in next_line: + raise stem.ProtocolError("'%s' is not a HTTP header:\n\n%s" % next_line) + + key, value = next_line.split(': ', 1) + headers[key] = value + + next_line = lines.pop(0) + + return '\n'.join(lines), headers def _download_from_dirport(url, compression, timeout): @@ -445,6 +478,8 @@ class Query(object): :var bool is_done: flag that indicates if our request has finished :var float start_time: unix timestamp when we first started running + :var http.client.HTTPMessage reply_headers: headers provided in the response, + **None** if we haven't yet made our request :var float runtime: time our query took, this is **None** if it's not yet finished @@ -464,8 +499,6 @@ class Query(object): :var str download_url: last url used to download the descriptor, this is unset until we've actually made a download attempt - :var http.client.HTTPMessage reply_headers: headers provided in the response, - **None** if we haven't yet made our request :param bool start: start making the request when constructed (default is **True**) :param bool block: only return after the request has been completed, this is @@ -642,7 +675,7 @@ class Query(object): endpoint = self._pick_endpoint(use_authority = retries == 0 and self.fall_back_to_authority) if isinstance(endpoint, stem.ORPort): - self.content = _download_from_orport(endpoint, self.resource) + self.content, self.reply_headers = _download_from_orport(endpoint, self.resource) elif isinstance(endpoint, stem.DirPort): self.download_url = 'http://%s:%i/%s' % (endpoint.address, endpoint.port, self.resource.lstrip('/')) self.content, self.reply_headers = _download_from_dirport(self.download_url, self.compression, timeout)

1 0

[stem/master] Unit test downloading descriptors from orport
by atagar＠torproject.org 24 Apr '18

24 Apr '18

commit 04186deb315d7b2d3ec1074757604d5f98656116 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Apr 23 19:37:57 2018 -0700 Unit test downloading descriptors from orport --- test/unit/descriptor/remote.py | 88 +++++++++++++++++++++++++++++++----------- 1 file changed, 65 insertions(+), 23 deletions(-) diff --git a/test/unit/descriptor/remote.py b/test/unit/descriptor/remote.py index b2cfa7c1..7d92c695 100644 --- a/test/unit/descriptor/remote.py +++ b/test/unit/descriptor/remote.py @@ -29,9 +29,9 @@ except ImportError: try: # added in python 3.3 - from unittest.mock import patch, Mock + from unittest.mock import patch, Mock, MagicMock except ImportError: - from mock import patch, Mock + from mock import patch, Mock, MagicMock # The urlopen() method is in a different location depending on if we're using # python 2.x or 3.x. The 2to3 converter accounts for this in imports, but not @@ -126,10 +126,20 @@ HEADER = '\r\n'.join([ 'Content-Encoding: %s', ]) +ORPORT_DESCRIPTOR = 'HTTP/1.0 200 OK\n' + HEADER + '\n\n' + TEST_DESCRIPTOR -def _urlopen_mock(data, encoding = 'identity'): - urlopen_mock = Mock() - urlopen_mock().read.return_value = data + +def _orport_mock(data): + connect_mock = MagicMock() + relay_mock = connect_mock().__enter__() + circ_mock = relay_mock.create_circuit().__enter__() + circ_mock.send().data = data + return connect_mock + + +def _dirport_mock(data, encoding = 'identity'): + dirport_mock = Mock() + dirport_mock().read.return_value = data if stem.prereq.is_python_3(): headers = HTTPMessage() @@ -138,11 +148,11 @@ def _urlopen_mock(data, encoding = 'identity'): key, value = line.split(': ', 1) headers.add_header(key, encoding if key == 'Content-Encoding' else value) - urlopen_mock().headers = headers + dirport_mock().headers = headers else: - urlopen_mock().headers = HTTPMessage(io.BytesIO(HEADER % encoding)) + dirport_mock().headers = HTTPMessage(io.BytesIO(HEADER % encoding)) - return urlopen_mock + return dirport_mock class TestDescriptorDownloader(unittest.TestCase): @@ -150,6 +160,38 @@ class TestDescriptorDownloader(unittest.TestCase): # prevent our mocks from impacting other tests stem.descriptor.remote.SINGLETON_DOWNLOADER = None + @patch('stem.client.Relay.connect', _orport_mock(ORPORT_DESCRIPTOR)) + def test_using_orport(self): + """ + Download a descriptor through the ORPort. + """ + + reply = stem.descriptor.remote.their_server_descriptor( + endpoints = [stem.ORPort('12.34.56.78', 1100)], + fall_back_to_authority = False, + validate = True, + ) + + self.assertEqual(1, len(list(reply))) + self.assertEqual('moria1', list(reply)[0].nickname) + self.assertEqual(5, len(reply.reply_headers)) + + @patch(URL_OPEN, _dirport_mock(TEST_DESCRIPTOR)) + def test_using_dirport(self): + """ + Download a descriptor through the DirPort. + """ + + reply = stem.descriptor.remote.their_server_descriptor( + endpoints = [stem.DirPort('12.34.56.78', 1100)], + fall_back_to_authority = False, + validate = True, + ) + + self.assertEqual(1, len(list(reply))) + self.assertEqual('moria1', list(reply)[0].nickname) + self.assertEqual(5, len(reply.reply_headers)) + def test_gzip_url_override(self): query = stem.descriptor.remote.Query(TEST_RESOURCE, start = False) self.assertEqual([Compression.PLAINTEXT], query.compression) @@ -177,7 +219,7 @@ class TestDescriptorDownloader(unittest.TestCase): query = stem.descriptor.remote.Query(TEST_RESOURCE, compression = Compression.LZMA, start = False) self.assertEqual([Compression.PLAINTEXT], query.compression) - @patch(URL_OPEN, _urlopen_mock(read_resource('compressed_identity'), encoding = 'identity')) + @patch(URL_OPEN, _dirport_mock(read_resource('compressed_identity'), encoding = 'identity')) def test_compression_plaintext(self): """ Download a plaintext descriptor. @@ -192,7 +234,7 @@ class TestDescriptorDownloader(unittest.TestCase): self.assertEqual(1, len(descriptors)) self.assertEqual('moria1', descriptors[0].nickname) - @patch(URL_OPEN, _urlopen_mock(read_resource('compressed_gzip'), encoding = 'gzip')) + @patch(URL_OPEN, _dirport_mock(read_resource('compressed_gzip'), encoding = 'gzip')) def test_compression_gzip(self): """ Download a gip compressed descriptor. @@ -207,7 +249,7 @@ class TestDescriptorDownloader(unittest.TestCase): self.assertEqual(1, len(descriptors)) self.assertEqual('moria1', descriptors[0].nickname) - @patch(URL_OPEN, _urlopen_mock(read_resource('compressed_zstd'), encoding = 'x-zstd')) + @patch(URL_OPEN, _dirport_mock(read_resource('compressed_zstd'), encoding = 'x-zstd')) def test_compression_zstd(self): """ Download a zstd compressed descriptor. @@ -226,7 +268,7 @@ class TestDescriptorDownloader(unittest.TestCase): self.assertEqual(1, len(descriptors)) self.assertEqual('moria1', descriptors[0].nickname) - @patch(URL_OPEN, _urlopen_mock(read_resource('compressed_lzma'), encoding = 'x-tor-lzma')) + @patch(URL_OPEN, _dirport_mock(read_resource('compressed_lzma'), encoding = 'x-tor-lzma')) def test_compression_lzma(self): """ Download a lzma compressed descriptor. @@ -245,7 +287,7 @@ class TestDescriptorDownloader(unittest.TestCase): self.assertEqual(1, len(descriptors)) self.assertEqual('moria1', descriptors[0].nickname) - @patch(URL_OPEN, _urlopen_mock(TEST_DESCRIPTOR)) + @patch(URL_OPEN, _dirport_mock(TEST_DESCRIPTOR)) def test_reply_headers(self): query = stem.descriptor.remote.get_server_descriptors('9695DFC35FFEB861329B9F1AB04C46397020CE31', start = False) self.assertEqual(None, query.reply_headers) # initially we don't have a reply @@ -268,7 +310,7 @@ class TestDescriptorDownloader(unittest.TestCase): self.assertEqual(1, len(descriptors)) self.assertEqual('moria1', descriptors[0].nickname) - @patch(URL_OPEN, _urlopen_mock(TEST_DESCRIPTOR)) + @patch(URL_OPEN, _dirport_mock(TEST_DESCRIPTOR)) def test_query_download(self): """ Check Query functionality when we successfully download a descriptor. @@ -293,7 +335,7 @@ class TestDescriptorDownloader(unittest.TestCase): self.assertEqual('9695DFC35FFEB861329B9F1AB04C46397020CE31', desc.fingerprint) self.assertEqual(TEST_DESCRIPTOR.strip(), desc.get_bytes()) - @patch(URL_OPEN, _urlopen_mock(b'some malformed stuff')) + @patch(URL_OPEN, _dirport_mock(b'some malformed stuff')) def test_query_with_malformed_content(self): """ Query with malformed descriptor content. @@ -321,12 +363,12 @@ class TestDescriptorDownloader(unittest.TestCase): self.assertRaises(ValueError, query.run) @patch(URL_OPEN) - def test_query_with_timeout(self, urlopen_mock): + def test_query_with_timeout(self, dirport_mock): def urlopen_call(*args, **kwargs): time.sleep(0.06) raise socket.timeout('connection timed out') - urlopen_mock.side_effect = urlopen_call + dirport_mock.side_effect = urlopen_call query = stem.descriptor.remote.Query( TEST_RESOURCE, @@ -341,7 +383,7 @@ class TestDescriptorDownloader(unittest.TestCase): # Check that we don't make a third. self.assertRaises(socket.timeout, query.run) - self.assertEqual(2, urlopen_mock.call_count) + self.assertEqual(2, dirport_mock.call_count) def test_query_with_invalid_endpoints(self): invalid_endpoints = { @@ -355,7 +397,7 @@ class TestDescriptorDownloader(unittest.TestCase): expected_error = re.escape('Endpoints must be an stem.ORPort, stem.DirPort, or two value tuple. ' + error_suffix) self.assertRaisesRegexp(ValueError, expected_error, stem.descriptor.remote.Query, TEST_RESOURCE, 'server-descriptor 1.0', endpoints = endpoints) - @patch(URL_OPEN, _urlopen_mock(TEST_DESCRIPTOR)) + @patch(URL_OPEN, _dirport_mock(TEST_DESCRIPTOR)) def test_can_iterate_multiple_times(self): query = stem.descriptor.remote.Query( TEST_RESOURCE, @@ -381,7 +423,7 @@ class TestDescriptorDownloader(unittest.TestCase): self.assertTrue(len(fallback_directories) > 10) self.assertEqual('5.39.92.199', fallback_directories['0BEA4A88D069753218EAAAD6D22EA87B9A1319D6'].address) - @patch(URL_OPEN, _urlopen_mock(FALLBACK_DIR_CONTENT)) + @patch(URL_OPEN, _dirport_mock(FALLBACK_DIR_CONTENT)) def test_fallback_directories_from_remote(self): fallback_directories = stem.descriptor.remote.FallbackDirectory.from_remote() header = OrderedDict((('type', 'fallback'), ('version', '2.0.0'), ('timestamp', '20170526090242'))) @@ -465,15 +507,15 @@ class TestDescriptorDownloader(unittest.TestCase): self.assertEqual(expected, stem.descriptor.remote.FallbackDirectory.from_cache(tmp.name)) - @patch(URL_OPEN, _urlopen_mock(b'')) + @patch(URL_OPEN, _dirport_mock(b'')) def test_fallback_directories_from_remote_empty(self): self.assertRaisesRegexp(IOError, 'did not have any content', stem.descriptor.remote.FallbackDirectory.from_remote) - @patch(URL_OPEN, _urlopen_mock(b'\n'.join(FALLBACK_DIR_CONTENT.splitlines()[1:]))) + @patch(URL_OPEN, _dirport_mock(b'\n'.join(FALLBACK_DIR_CONTENT.splitlines()[1:]))) def test_fallback_directories_from_remote_no_header(self): self.assertRaisesRegexp(IOError, 'does not have a type field indicating it is fallback directory metadata', stem.descriptor.remote.FallbackDirectory.from_remote) - @patch(URL_OPEN, _urlopen_mock(FALLBACK_DIR_CONTENT.replace(b'version=2.0.0', b'version'))) + @patch(URL_OPEN, _dirport_mock(FALLBACK_DIR_CONTENT.replace(b'version=2.0.0', b'version'))) def test_fallback_directories_from_remote_malformed_header(self): self.assertRaisesRegexp(IOError, 'Malformed fallback directory header line: /\* version \*/', stem.descriptor.remote.FallbackDirectory.from_remote)

1 0

[stem/master] Method to get descriptor of relay we're downloading from
by atagar＠torproject.org 24 Apr '18

24 Apr '18

commit 7b05dec592452ff57dbf79a2c20261f18ca51c30 Author: Damian Johnson <atagar(a)torproject.org> Date: Mon Apr 23 10:11:58 2018 -0700 Method to get descriptor of relay we're downloading from Besides using this module to download descriptors I'd like to use it for a relay health check. For that I'd like to get the descriptor of the relay we're downloading from. The DirPort protocol surfaced this but Stem didn't, so adding a little function for it. --- docs/change_log.rst | 1 + stem/descriptor/remote.py | 36 +++++++++++++++++++++++++++++++++++- 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/docs/change_log.rst b/docs/change_log.rst index b6121982..18005e1e 100644 --- a/docs/change_log.rst +++ b/docs/change_log.rst @@ -59,6 +59,7 @@ The following are only available within Stem's `git repository * Added zstd and lzma compression support (:spec:`1cb56af`) * Added server descriptor's new is_hidden_service_dir attribute * Don't retry downloading descriptors when we've timed out + * Added :func:`~stem.descriptor.remote.their_server_descriptor` * Added the reply_headers attribute to :class:`~stem.descriptor.remote.Query` * Supplying a User-Agent when downloading descriptors. * Reduced maximum descriptors fetched by the remote module to match tor's new limit (:trac:`24743`) diff --git a/stem/descriptor/remote.py b/stem/descriptor/remote.py index eb9c7b62..c191b02a 100644 --- a/stem/descriptor/remote.py +++ b/stem/descriptor/remote.py @@ -45,6 +45,7 @@ content. For example... :: get_instance - Provides a singleton DescriptorDownloader used for... + |- their_server_descriptor - provides the server descriptor of the relay we download from |- get_server_descriptors - provides present server descriptors |- get_extrainfo_descriptors - provides present extrainfo descriptors +- get_consensus - provides the present consensus or router status entries @@ -63,6 +64,7 @@ content. For example... DescriptorDownloader - Configurable class for issuing queries |- use_directory_mirrors - use directory mirrors to download future descriptors + |- their_server_descriptor - provides the server descriptor of the relay we download from |- get_server_descriptors - provides present server descriptors |- get_extrainfo_descriptors - provides present extrainfo descriptors |- get_consensus - provides the present consensus or router status entries @@ -201,6 +203,21 @@ def get_instance(): return SINGLETON_DOWNLOADER +def their_server_descriptor(**query_args): + """ + Provides the server descriptor of the relay we're downloading from. + + .. versionadded:: 1.7.0 + + :param query_args: additional arguments for the + :class:`~stem.descriptor.remote.Query` constructor + + :returns: :class:`~stem.descriptor.remote.Query` for the server descriptors + """ + + return get_instance().their_server_descriptor(**query_args) + + def get_server_descriptors(fingerprints = None, **query_args): """ Shorthand for @@ -247,9 +264,12 @@ def _download_from_orport(endpoint, resource): :returns: **str** with the descirptor data :raises: + * :class:`stem.SocketError` if unable to establish a connection """ - with stem.client.Relay.connect(endpoint.address, endpoint.port, endpoint.link_protocols) as relay: + link_protocol = endpoint.link_protocols if endpoint.link_protocols else [3] + + with stem.client.Relay.connect(endpoint.address, endpoint.port, link_protocol) as relay: with relay.create_circuit() as circ: circ.send('RELAY_BEGIN_DIR', stream_id = 1) return circ.send('RELAY_DATA', resource, stream_id = 1).data @@ -694,6 +714,20 @@ class DescriptorDownloader(object): return consensus + def their_server_descriptor(self, **query_args): + """ + Provides the server descriptor of the relay we're downloading from. + + .. versionadded:: 1.7.0 + + :param query_args: additional arguments for the + :class:`~stem.descriptor.remote.Query` constructor + + :returns: :class:`~stem.descriptor.remote.Query` for the server descriptors + """ + + return self.query('/tor/server/authority', **query_args) + def get_server_descriptors(self, fingerprints = None, **query_args): """ Provides the server descriptors with the given fingerprints. If no

1 0

[stem/master] ORPort descriptor downloading support
by atagar＠torproject.org 24 Apr '18

24 Apr '18

commit 820881a6dd9a5ff0e6b4f0b78b733f1bcd8e096d Merge: 27b2cbd7 b0433da3 Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Apr 24 12:30:17 2018 -0700 ORPort descriptor downloading support DescriptorDownloader support for downloading from relay ORPorts. To do this simply provide an ORPort as the endpoint... reply = stem.descriptor.remote.get_server_descriptors( endpoint = stem.ORPort('12.34.56.78', 90), ) Like the stem.client module on which this is based this is still very much alpha functionality. No doubt as we use this in the wild we'll discover rough edges. That said, in practice seems to work. Originally we planned for this to be part of Dave's SoP project, but I need this capability for something of my own (health check for my tor relay). No matter - just means Dave has more time for his other project tasks. :P docs/change_log.rst | 4 +- stem/__init__.py | 55 +++++++++ stem/cached_tor_manual.sqlite | Bin 239616 -> 238592 bytes stem/client/__init__.py | 11 +- stem/descriptor/remote.py | 247 ++++++++++++++++++++++++++++++---------- stem/manual.py | 15 ++- stem/settings.cfg | 9 +- stem/util/connection.py | 2 + test/integ/descriptor/remote.py | 27 +++++ test/integ/manual.py | 38 +------ test/settings.cfg | 3 +- test/unit/descriptor/remote.py | 111 ++++++++++++++---- test/unit/endpoint.py | 46 ++++++++ 13 files changed, 438 insertions(+), 130 deletions(-)

1 0

[stem/master] Avoid downloading from tor26
by atagar＠torproject.org 24 Apr '18

24 Apr '18

commit b0433da3dc236070b385f69446dc0855b6085434 Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Apr 24 12:28:07 2018 -0700 Avoid downloading from tor26 Occasionally our integ tests with the ONLINE target take five minutes to run. Reason is that sometimes we attempt to download descriptors from tor26 which does not vend a usable DirPort. Good thing our tests caught this - DescriptorDownloader users would be nailed by this too. --- stem/descriptor/remote.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/stem/descriptor/remote.py b/stem/descriptor/remote.py index fda73327..7908becf 100644 --- a/stem/descriptor/remote.py +++ b/stem/descriptor/remote.py @@ -464,6 +464,11 @@ class Query(object): :class:`~stem.ORPort` instances. Usage of tuples for this argument is deprecated and will be removed in the future. + .. versionchanged:: 1.7.0 + Avoid downloading from tor26. This directory authority throttles its + DirPort to such an extent that requests either time out or take on the + order of minutes. + :var str resource: resource being fetched, such as '/tor/server/all' :var str descriptor_type: type of descriptors being fetched (for options see :func:`~stem.descriptor.__init__.parse_file`), this is guessed from the @@ -667,7 +672,7 @@ class Query(object): """ if use_authority or not self.endpoints: - picked = random.choice(list(get_authorities().values())) + picked = random.choice([auth for auth in get_authorities().values() if auth.nickname != 'tor26']) return stem.DirPort(picked.address, picked.dir_port) else: return random.choice(self.endpoints)

1 0

[stem/master] Link protocol hardcoded in circuit
by atagar＠torproject.org 24 Apr '18

24 Apr '18

commit 92296cd438497846128693f605ef30208a79d739 Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Apr 24 10:28:50 2018 -0700 Link protocol hardcoded in circuit Oops. Iirc things indeed only work with a v3 link protocol right now, but none the less we shouldn't hardcode it in the circuit class. --- stem/client/__init__.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stem/client/__init__.py b/stem/client/__init__.py index be7e16ad..11a97171 100644 --- a/stem/client/__init__.py +++ b/stem/client/__init__.py @@ -245,8 +245,8 @@ class Circuit(object): self.relay._orport.send(encrypted_payload) reply = next(stem.client.cell.Cell.unpack(self.relay._orport.recv(), self.relay.link_protocol)) - decrypted = self.backward_key.update(reply.pack(3)[3:]) - return stem.client.cell.RelayCell._unpack(decrypted, self.id, 3) + decrypted = self.backward_key.update(reply.pack(self.relay.link_protocol)[3:]) + return stem.client.cell.RelayCell._unpack(decrypted, self.id, self.relay.link_protocol) except: self.forward_digest = orig_digest self.forward_key = orig_key

1 0

[stem/master] Integ test downloading descriptors from ORPort
by atagar＠torproject.org 24 Apr '18

24 Apr '18

commit d0374a7b70e836abe70b15a827abf55314ce5825 Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Apr 24 12:10:33 2018 -0700 Integ test downloading descriptors from ORPort Sweet, finally it's working for realz! --- docs/change_log.rst | 3 ++- stem/descriptor/remote.py | 9 ++++++--- test/integ/descriptor/remote.py | 27 +++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 4 deletions(-) diff --git a/docs/change_log.rst b/docs/change_log.rst index 18005e1e..b237f25f 100644 --- a/docs/change_log.rst +++ b/docs/change_log.rst @@ -55,8 +55,9 @@ The following are only available within Stem's `git repository * **Descriptors** + * `stem.descriptor.remote <api/descriptor/remote.html>`_ can now download from relay ORPorts + * Zstd and lzma compression support (:spec:`1cb56af`) * `Fallback directory v2 support <https://lists.torproject.org/pipermail/tor-dev/2017-December/012721.html>`_, which adds *nickname* and *extrainfo* - * Added zstd and lzma compression support (:spec:`1cb56af`) * Added server descriptor's new is_hidden_service_dir attribute * Don't retry downloading descriptors when we've timed out * Added :func:`~stem.descriptor.remote.their_server_descriptor` diff --git a/stem/descriptor/remote.py b/stem/descriptor/remote.py index 5bd186c3..fda73327 100644 --- a/stem/descriptor/remote.py +++ b/stem/descriptor/remote.py @@ -282,12 +282,12 @@ def _download_from_orport(endpoint, resource): * :class:`stem.SocketError` if unable to establish a connection """ - link_protocol = endpoint.link_protocols if endpoint.link_protocols else [3] + link_protocols = endpoint.link_protocols if endpoint.link_protocols else [3] - with stem.client.Relay.connect(endpoint.address, endpoint.port, link_protocol) as relay: + with stem.client.Relay.connect(endpoint.address, endpoint.port, link_protocols) as relay: with relay.create_circuit() as circ: circ.send('RELAY_BEGIN_DIR', stream_id = 1) - lines = b''.join([cell.data for cell in circ.send('RELAY_DATA', resource, stream_id = 1)]).splitlines() + lines = b''.join([cell.data for cell in circ.send('RELAY_DATA', 'GET %s HTTP/1.0\r\n\r\n' % resource, stream_id = 1)]).splitlines() first_line = lines.pop(0) if first_line != 'HTTP/1.0 200 OK': @@ -447,6 +447,9 @@ class Query(object): argument is overwritten with Compression.GZIP. .. versionchanged:: 1.7.0 + Added support for downloading from ORPorts. + + .. versionchanged:: 1.7.0 Added the compression argument. .. versionchanged:: 1.7.0 diff --git a/test/integ/descriptor/remote.py b/test/integ/descriptor/remote.py index 3123ef06..f56e1b02 100644 --- a/test/integ/descriptor/remote.py +++ b/test/integ/descriptor/remote.py @@ -4,6 +4,7 @@ Integration tests for stem.descriptor.remote. import unittest +import stem import stem.descriptor import stem.descriptor.extrainfo_descriptor import stem.descriptor.networkstatus @@ -16,6 +17,32 @@ import test.require class TestDescriptorDownloader(unittest.TestCase): @test.require.only_run_once @test.require.online + def test_downloading_via_orport(self): + moria1 = stem.descriptor.remote.get_authorities()['moria1'] + + desc = list(stem.descriptor.remote.their_server_descriptor( + endpoints = [stem.ORPort(moria1.address, moria1.or_port)], + fall_back_to_authority = False, + ).run())[0] + + self.assertEqual('moria1', desc.nickname) + self.assertTrue(isinstance(desc, stem.descriptor.stem.descriptor.server_descriptor.ServerDescriptor)) + + @test.require.only_run_once + @test.require.online + def test_downloading_via_dirport(self): + moria1 = stem.descriptor.remote.get_authorities()['moria1'] + + desc = list(stem.descriptor.remote.their_server_descriptor( + endpoints = [stem.DirPort(moria1.address, moria1.dir_port)], + fall_back_to_authority = False, + ).run())[0] + + self.assertEqual('moria1', desc.nickname) + self.assertTrue(isinstance(desc, stem.descriptor.stem.descriptor.server_descriptor.ServerDescriptor)) + + @test.require.only_run_once + @test.require.online def test_shorthand_aliases(self): """ Quick sanity test that we can call our shorthand aliases for getting

1 0

[stem/master] Provide all cells circuits reply with
by atagar＠torproject.org 24 Apr '18

24 Apr '18

commit 6b5ff7cd17759fcf628237f9277503d36969d850 Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Apr 24 11:44:38 2018 -0700 Provide all cells circuits reply with Oops, callers of send() on our circuit only got the first cell sent in repsonse. These are fixed sized cells, so when the response is larger than 509 bytes (such as in the case of descriptors) multiple will be received. It's important that this returns a list rather than yielding a generator because we need to ensure we issue recv() calls to drain messages off the socket. Otherwise the next call will hang. --- stem/client/__init__.py | 11 ++++++++--- stem/descriptor/remote.py | 2 +- test/unit/descriptor/remote.py | 9 ++++++++- 3 files changed, 17 insertions(+), 5 deletions(-) diff --git a/stem/client/__init__.py b/stem/client/__init__.py index 11a97171..aa4aa274 100644 --- a/stem/client/__init__.py +++ b/stem/client/__init__.py @@ -227,6 +227,8 @@ class Circuit(object): :param stem.client.RelayCommand command: command to be issued :param bytes data: message payload :param int stream_id: specific stream this concerns + + :returns: **list** of :class:`~stem.client.cell.RelayCell` responses """ with self.relay._orport_lock: @@ -242,11 +244,14 @@ class Circuit(object): header, payload = split(cell.pack(self.relay.link_protocol), 3) encrypted_payload = header + self.forward_key.update(payload) + reply = [] self.relay._orport.send(encrypted_payload) - reply = next(stem.client.cell.Cell.unpack(self.relay._orport.recv(), self.relay.link_protocol)) - decrypted = self.backward_key.update(reply.pack(self.relay.link_protocol)[3:]) - return stem.client.cell.RelayCell._unpack(decrypted, self.id, self.relay.link_protocol) + for cell in stem.client.cell.Cell.unpack(self.relay._orport.recv(), self.relay.link_protocol): + decrypted = self.backward_key.update(cell.pack(self.relay.link_protocol)[3:]) + reply.append(stem.client.cell.RelayCell._unpack(decrypted, self.id, self.relay.link_protocol)) + + return reply except: self.forward_digest = orig_digest self.forward_key = orig_key diff --git a/stem/descriptor/remote.py b/stem/descriptor/remote.py index 5655c84c..5bd186c3 100644 --- a/stem/descriptor/remote.py +++ b/stem/descriptor/remote.py @@ -287,7 +287,7 @@ def _download_from_orport(endpoint, resource): with stem.client.Relay.connect(endpoint.address, endpoint.port, link_protocol) as relay: with relay.create_circuit() as circ: circ.send('RELAY_BEGIN_DIR', stream_id = 1) - lines = circ.send('RELAY_DATA', resource, stream_id = 1).data.splitlines() + lines = b''.join([cell.data for cell in circ.send('RELAY_DATA', resource, stream_id = 1)]).splitlines() first_line = lines.pop(0) if first_line != 'HTTP/1.0 200 OK': diff --git a/test/unit/descriptor/remote.py b/test/unit/descriptor/remote.py index 7d92c695..040d4c6c 100644 --- a/test/unit/descriptor/remote.py +++ b/test/unit/descriptor/remote.py @@ -130,10 +130,17 @@ ORPORT_DESCRIPTOR = 'HTTP/1.0 200 OK\n' + HEADER + '\n\n' + TEST_DESCRIPTOR def _orport_mock(data): + cells = [] + + for hunk in [data[i:i + 50] for i in range(0, len(data), 50)]: + cell = Mock() + cell.data = hunk + cells.append(cell) + connect_mock = MagicMock() relay_mock = connect_mock().__enter__() circ_mock = relay_mock.create_circuit().__enter__() - circ_mock.send().data = data + circ_mock.send.return_value = cells return connect_mock

1 0

[tor/release-0.3.3] Add distcheck support to travis configuration.
by nickm＠torproject.org 24 Apr '18

24 Apr '18

commit 61251337153448afa1cc2baae252c3f0d6353162 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Apr 18 14:24:54 2018 -0400 Add distcheck support to travis configuration. Implements 25814. --- .travis.yml | 5 ++++- changes/travis_distcheck | 4 ++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index dc0a8ac67..6da3027b8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -90,6 +90,8 @@ matrix: - compiler: gcc - compiler: gcc env: COVERAGE_OPTIONS="--enable-coverage" + - compiler: gcc + env: DISTCHECK="yes" ## The "sudo: required" forces non-containerized builds, working ## around a Travis CI environment issue: clang LeakAnalyzer fails ## because it requires ptrace and the containerized environment no @@ -118,7 +120,8 @@ script: - ./autogen.sh - ./configure $RUST_OPTIONS $COVERAGE_OPTIONS --disable-asciidoc --enable-fatal-warnings --disable-silent-rules --enable-fragile-hardening ## We run `make check` because that's what https://jenkins.torproject.org does. - - make check + - if [[ "$DISTCHECK" == "" ]]; then make check; fi + - if [[ "$DISTCHECK" != "" ]]; then make distcheck DISTCHECK_CONFIGURE_FLAGS="$RUST_OPTIONS $COVERAGE_OPTIONS --disable-asciidoc --enable-fatal-warnings --disable-silent-rules --enable-fragile-hardening"; fi after_failure: ## `make check` will leave a log file with more details of test failures. diff --git a/changes/travis_distcheck b/changes/travis_distcheck new file mode 100644 index 000000000..0f278fe7e --- /dev/null +++ b/changes/travis_distcheck @@ -0,0 +1,4 @@ + o Minor features (continuous integration): + - Our .travis.yml configuration now includes support for testing + the results of "make distcheck". (It's not uncommon for "make check" to + pass but "make distcheck" to fail.) Closes ticket 25814.

1 0