March 2012 - tor-commits - lists.torproject.org

[tor/master] explain that bug 5090 allows a post-auth heap overflow
by arma＠torproject.org 26 Mar '12

26 Mar '12

commit bca8bf62c6683374321cd2a306b6455e441661b9 Author: Roger Dingledine <arma(a)torproject.org> Date: Sun Mar 25 23:09:23 2012 -0400 explain that bug 5090 allows a post-auth heap overflow resolves bug 5402. --- ChangeLog | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/ChangeLog b/ChangeLog index 5de4d63..a6dc608 100644 --- a/ChangeLog +++ b/ChangeLog @@ -43,8 +43,11 @@ Changes in version 0.2.3.13-alpha - 2012-03-1? - Detect and reject certain misformed escape sequences in configuration values. Previously, these values would cause us to crash if received in a torrc file or over an (authenticated) - control port. Bug found by Esteban Manchado Velázquez. Patch by - "flupzor". Fixes bug 5090; bugfix on 0.2.0.16-alpha. + control port. Bug found by Esteban Manchado Velázquez, and + independently by Robert Connolly from Matta Consulting who further + noted that it allows a post-authentication heap overflow. Patch + by "flupzor". Fixes bugs 5090 and 5402 (CVE 2012-1668); bugfix + on 0.2.0.16-alpha. - Ensure that variables set in Tor's environment cannot override environment variables which Tor tries to pass to a managed pluggable-transport proxy. Previously, Tor would pass every

1 0

[stem/master] Skipping auth message check when cookie missing
by atagar＠torproject.org 26 Mar '12

26 Mar '12

commit 896bb442fe9cc3039a98188fd68ddfcba4f0c1e5 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 25 12:39:35 2012 -0700 Skipping auth message check when cookie missing When we first run our tests we don't have an authentication cookie unless we've run with the RUN_COOKIE target. We have special handling for this, but it did not account for running with an open connection (the default) in which case we'd encounter a second exception while trying to check the failure message. --- test/integ/connection/authentication.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/test/integ/connection/authentication.py b/test/integ/connection/authentication.py index 0762b23..aafde5d 100644 --- a/test/integ/connection/authentication.py +++ b/test/integ/connection/authentication.py @@ -211,7 +211,7 @@ class TestAuthenticate(unittest.TestCase): # auth but the function will short circuit with failure due to the # missing file. - self.assertRaises(stem.connection.UnreadableCookieFile, self._check_auth, auth_type, auth_value) + self.assertRaises(stem.connection.UnreadableCookieFile, self._check_auth, auth_type, auth_value, False) elif _can_authenticate(auth_type): self._check_auth(auth_type, auth_value) else:

1 0

[stem/master] Unit tests for server descriptor parsing
by atagar＠torproject.org 26 Mar '12

26 Mar '12

commit 4502701f0ec8462c0848128f000ec18c14a94167 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 25 14:52:43 2012 -0700 Unit tests for server descriptor parsing Borrowing heavily from Karsten's metrics-lib unit tests [1], plus a few of our own. This doesn't include everything, some are covered by util tests and others are effectively covered by parsing metrics and cached-descriptor files. That said, I wouldn't be against including more - my fingers are just getting numb... While looking through those tests I discovered these deltas... * When parsing the descriptor he validated that we had the correct first and last entry but we didn't. This was a bug, now fixed. * He checks that the 'protocols' line only contains numberic protocols but we accept any space separated content. The spec doesn't specifify what the protocols can be so I'm leaving this alone. * He had checks that a 'published' line with a year of '3012' or '1912' would fail. I can understand having a sanity check but the spec does not say that those dates are invalid so again leaving it as-is. * Metrics lib has validation for the exit policy but we don't. This is because we haven't yet implemented an ExitPolicy class - gsathya is currently working on that in... https://trac.torproject.org/projects/tor/ticket/5454 * Karsten is parsing and validating read/write-history lines. We, on the other hand, log an INFO level warning when these appear and assert in the integ tests that they (along with 'eventdns') do not exist in our cached-descriptor file. My understanding is that these are deprecated entries and should not appear outside of the extra-info descriptors. Maybe I'm misunderstanding something here... On everything else we either match or are maybe slightly more strict about only allowing content conforming to the spec. [1] https://gitweb.torproject.org/metrics-lib.git/blob/HEAD:/test/org/torprojec… --- run_tests.py | 2 + stem/descriptor/server_descriptor.py | 8 +- test/unit/descriptor/server_descriptor.py | 276 +++++++++++++++++++++++++++++ 3 files changed, 285 insertions(+), 1 deletions(-) diff --git a/run_tests.py b/run_tests.py index 3cb259d..2c36a04 100755 --- a/run_tests.py +++ b/run_tests.py @@ -18,6 +18,7 @@ import test.unit.connection.protocolinfo import test.unit.socket.control_line import test.unit.socket.control_message import test.unit.descriptor.reader +import test.unit.descriptor.server_descriptor import test.unit.util.conf import test.unit.util.connection import test.unit.util.enum @@ -88,6 +89,7 @@ UNIT_TESTS = ( test.unit.util.system.TestSystem, test.unit.util.tor_tools.TestTorTools, test.unit.descriptor.reader.TestDescriptorReader, + test.unit.descriptor.server_descriptor.TestServerDescriptor, test.unit.version.TestVersion, test.unit.socket.control_message.TestControlMessage, test.unit.socket.control_line.TestControlLine, diff --git a/stem/descriptor/server_descriptor.py b/stem/descriptor/server_descriptor.py index 899cfcc..a82a657 100644 --- a/stem/descriptor/server_descriptor.py +++ b/stem/descriptor/server_descriptor.py @@ -312,11 +312,13 @@ class ServerDescriptorV3(stem.descriptor.Descriptor): entries = {} remaining_contents = contents.split("\n") + first_entry, last_entry = remaining_contents[0], remaining_contents[0] while remaining_contents: line = remaining_contents.pop(0) # last line can be empty if not line and not remaining_contents: continue + last_entry = line # Some lines have an 'opt ' for backward compatability. They should be # ignored. This prefix is being removed in... @@ -355,7 +357,11 @@ class ServerDescriptorV3(stem.descriptor.Descriptor): if keyword in entries and len(entries[keyword]) > 1: raise ValueError("The '%s' entry can only appear once in a descriptor" % keyword) - if not self.exit_policy: + if not first_entry.startswith(ENTRY_START): + raise ValueError("Descriptor must start with a '%s' entry" % ENTRY_START) + elif not last_entry.startswith(ENTRY_END): + raise ValueError("Descriptor must end with a '%s' entry" % ENTRY_END) + elif not self.exit_policy: raise ValueError("Descriptor must have at least one 'accept' or 'reject' entry") # parse all the entries into our attributes diff --git a/test/unit/descriptor/server_descriptor.py b/test/unit/descriptor/server_descriptor.py new file mode 100644 index 0000000..cde6714 --- /dev/null +++ b/test/unit/descriptor/server_descriptor.py @@ -0,0 +1,276 @@ +""" +Unit tests for stem.descriptor.server_descriptor. +""" + +import datetime +import StringIO +import unittest + +import stem.descriptor.server_descriptor +from stem.descriptor.server_descriptor import ServerDescriptorV3 + +CRYPTO_BLOB = """ +MIGJAoGBAJv5IIWQ+WDWYUdyA/0L8qbIkEVH/cwryZWoIaPAzINfrw1WfNZGtBmg +skFtXhOHHqTRN4GPPrZsAIUOQGzQtGb66IQgT4tO/pj+P6QmSCCdTfhvGfgTCsC+ +WPi4Fl2qryzTb3QO5r5x7T8OsG2IBUET1bLQzmtbC560SYR49IvVAgMBAAE= +""" + +SAMPLE_DESCRIPTOR_ATTR = ( + ("router", "caerSidi 71.35.133.197 9001 0 0"), + ("published", "2012-03-01 17:15:27"), + ("bandwidth", "153600 256000 104590"), + ("reject", "*:*"), + ("onion-key", "\n-----BEGIN RSA PUBLIC KEY-----%s-----END RSA PUBLIC KEY-----" % CRYPTO_BLOB), + ("signing-key", "\n-----BEGIN RSA PUBLIC KEY-----%s-----END RSA PUBLIC KEY-----" % CRYPTO_BLOB), + ("router-signature", "\n-----BEGIN SIGNATURE-----%s-----END SIGNATURE-----" % CRYPTO_BLOB), +) + +def _make_descriptor(attr = None, exclude = None): + """ + Constructs a minimal server descriptor with the given attributes. + + Arguments: + attr (dict) - keyword/value mappings to be included in the descriptor + exclude (list) - mandatory keywords to exclude from the descriptor + + Returns: + str with customized descriptor content + """ + + descriptor_lines = [] + if attr == None: attr = {} + if exclude == None: exclude = [] + + for keyword, value in SAMPLE_DESCRIPTOR_ATTR: + if keyword in exclude: continue + elif keyword in attr: + value = attr[keyword] + del attr[keyword] + + # if this is the last entry then we should dump in any unused attributes + if keyword == "router-signature": + for attr_keyword, attr_value in attr.items(): + descriptor_lines.append("%s %s" % (attr_keyword, attr_value)) + + descriptor_lines.append("%s %s" % (keyword, value)) + + return "\n".join(descriptor_lines) + +class TestServerDescriptor(unittest.TestCase): + def test_minimal_descriptor(self): + """ + Basic sanity check that we can parse a descriptor with minimal attributes. + """ + + desc_text = _make_descriptor() + desc = ServerDescriptorV3(desc_text) + + self.assertEquals("caerSidi", desc.nickname) + self.assertEquals("71.35.133.197", desc.address) + self.assertEquals(None, desc.fingerprint) + self.assertTrue(CRYPTO_BLOB in desc.onion_key) + self.assertTrue(CRYPTO_BLOB in desc.signing_key) + self.assertTrue(CRYPTO_BLOB in desc.signature) + + def test_with_opt(self): + """ + Includes an 'opt <keyword> <value>' entry. + """ + + desc_text = _make_descriptor({"opt": "contact www.atagar.com/contact/"}) + desc = ServerDescriptorV3(desc_text) + self.assertEquals("www.atagar.com/contact/", desc.contact) + + def test_unrecognized_line(self): + """ + Includes unrecognized content in the descriptor. + """ + + desc_text = _make_descriptor({"pepperjack": "is oh so tasty!"}) + desc = ServerDescriptorV3(desc_text) + self.assertEquals(["pepperjack is oh so tasty!"], desc.get_unrecognized_lines()) + + def test_proceeding_line(self): + """ + Includes a line prior to the 'router' entry. + """ + + desc_text = "hibernate 1\n" + _make_descriptor() + self._expect_invalid_attr(desc_text) + + def test_trailing_line(self): + """ + Includes a line after the 'router-signature' entry. + """ + + desc_text = _make_descriptor() + "\nhibernate 1" + self._expect_invalid_attr(desc_text) + + def test_nickname_missing(self): + """ + Constructs with a malformed router entry. + """ + + desc_text = _make_descriptor({"router": " 71.35.133.197 9001 0 0"}) + self._expect_invalid_attr(desc_text, "nickname") + + def test_nickname_too_long(self): + """ + Constructs with a nickname that is an invalid length. + """ + + desc_text = _make_descriptor({"router": "saberrider2008ReallyLongNickname 71.35.133.197 9001 0 0"}) + self._expect_invalid_attr(desc_text, "nickname", "saberrider2008ReallyLongNickname") + + def test_nickname_invalid_char(self): + """ + Constructs with an invalid relay nickname. + """ + + desc_text = _make_descriptor({"router": "$aberrider2008 71.35.133.197 9001 0 0"}) + self._expect_invalid_attr(desc_text, "nickname", "$aberrider2008") + + def test_address_malformed(self): + """ + Constructs with an invalid ip address. + """ + + desc_text = _make_descriptor({"router": "caerSidi 371.35.133.197 9001 0 0"}) + self._expect_invalid_attr(desc_text, "address", "371.35.133.197") + + def test_port_too_high(self): + """ + Constructs with an ORPort that is too large. + """ + + desc_text = _make_descriptor({"router": "caerSidi 71.35.133.197 900001 0 0"}) + self._expect_invalid_attr(desc_text, "or_port", 900001) + + def test_port_malformed(self): + """ + Constructs with an ORPort that isn't numeric. + """ + + desc_text = _make_descriptor({"router": "caerSidi 71.35.133.197 900a1 0 0"}) + self._expect_invalid_attr(desc_text, "or_port") + + def test_port_newline(self): + """ + Constructs with a newline replacing the ORPort. + """ + + desc_text = _make_descriptor({"router": "caerSidi 71.35.133.197 \n 0 0"}) + self._expect_invalid_attr(desc_text, "or_port") + + def test_platform_empty(self): + """ + Constructs with an empty platform entry. + """ + + desc_text = _make_descriptor({"platform": ""}) + desc = ServerDescriptorV3(desc_text, validate = False) + self.assertEquals("", desc.platform) + + # does the same but with 'platform ' replaced with 'platform' + desc_text = desc_text.replace("platform ", "platform") + desc = ServerDescriptorV3(desc_text, validate = False) + self.assertEquals("", desc.platform) + + def test_protocols_no_circuit_versions(self): + """ + Constructs with a protocols line without circuit versions. + """ + + desc_text = _make_descriptor({"opt": "protocols Link 1 2"}) + self._expect_invalid_attr(desc_text, "circuit_protocols") + + def test_published_leap_year(self): + """ + Constructs with a published entry for a leap year, and when the date is + invalid. + """ + + desc_text = _make_descriptor({"published": "2011-02-29 04:03:19"}) + self._expect_invalid_attr(desc_text, "published") + + desc_text = _make_descriptor({"published": "2012-02-29 04:03:19"}) + expected_published = datetime.datetime(2012, 2, 29, 4, 3, 19) + self.assertEquals(expected_published, ServerDescriptorV3(desc_text).published) + + def test_published_no_time(self): + """ + Constructs with a published entry without a time component. + """ + + desc_text = _make_descriptor({"published": "2012-01-01"}) + self._expect_invalid_attr(desc_text, "published") + + def test_annotations(self): + """ + Checks that content before a descriptor are parsed as annotations. + """ + + desc_text = "@pepperjack very tasty\n@mushrooms not so much\n" + desc_text += _make_descriptor() + desc_text += "\ntrailing text that should be ignored, ho hum" + + # running parse_file_v3 should provide an iterator with a single descriptor + desc_iter = stem.descriptor.server_descriptor.parse_file_v3(StringIO.StringIO(desc_text)) + desc_entries = list(desc_iter) + self.assertEquals(1, len(desc_entries)) + desc = desc_entries[0] + + self.assertEquals("caerSidi", desc.nickname) + self.assertEquals("@pepperjack very tasty", desc.get_annotation_lines()[0]) + self.assertEquals("@mushrooms not so much", desc.get_annotation_lines()[1]) + self.assertEquals({"@pepperjack": "very tasty", "@mushrooms": "not so much"}, desc.get_annotations()) + self.assertEquals([], desc.get_unrecognized_lines()) + + def test_duplicate_field(self): + """ + Constructs with a field appearing twice. + """ + + desc_text = _make_descriptor({"<replace>": ""}) + desc_text = desc_text.replace("<replace>", "contact foo\ncontact bar") + self._expect_invalid_attr(desc_text, "contact", "foo") + + def test_missing_required_attr(self): + """ + Test making a descriptor with a missing required attribute. + """ + + for attr in stem.descriptor.server_descriptor.REQUIRED_FIELDS: + desc_text = _make_descriptor(exclude = [attr]) + self.assertRaises(ValueError, ServerDescriptorV3, desc_text) + + # check that we can still construct it without validation + desc = ServerDescriptorV3(desc_text, validate = False) + + # for one of them checks that the corresponding values are None + if attr == "router": + self.assertEquals(None, desc.nickname) + self.assertEquals(None, desc.address) + self.assertEquals(None, desc.or_port) + self.assertEquals(None, desc.socks_port) + self.assertEquals(None, desc.dir_port) + + def _expect_invalid_attr(self, desc_text, attr = None, expected_value = None): + """ + Asserts that construction will fail due to desc_text having a malformed + attribute. If an attr is provided then we check that it matches an expected + value when we're constructed without validation. + """ + + self.assertRaises(ValueError, ServerDescriptorV3, desc_text) + desc = ServerDescriptorV3(desc_text, validate = False) + + if attr: + # check that the invalid attribute matches the expected value when + # constructed without validation + + self.assertEquals(expected_value, getattr(desc, attr)) + else: + # check a default attribute + self.assertEquals("caerSidi", desc.nickname) +

1 0

[stem/master] Fixing reader's pydoc example
by atagar＠torproject.org 26 Mar '12

26 Mar '12

commit cf0e790c809d4057eceb37550b33f6f5bf67b065 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 25 16:02:23 2012 -0700 Fixing reader's pydoc example --- stem/descriptor/reader.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/stem/descriptor/reader.py b/stem/descriptor/reader.py index de39d54..61bee5c 100644 --- a/stem/descriptor/reader.py +++ b/stem/descriptor/reader.py @@ -41,7 +41,7 @@ again... time.sleep(1) - save_processed_files(reader.get_processed_files(), "/tmp/used_descriptors") + save_processed_files("/tmp/used_descriptors", reader.get_processed_files()) load_processed_files - Loads a listing of processed files.

1 0

[stem/master] Integer truncation could break reader's last modified check
by atagar＠torproject.org 26 Mar '12

26 Mar '12

commit 975d98d7337d4dd566aa5e9967e62afe2fdb0bc0 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 25 16:12:32 2012 -0700 Integer truncation could break reader's last modified check When saving the last modified timestamps they're saved as ints. However, the timestamp itself is a float, so the following check for an unchanging timestamp would fail. --- stem/descriptor/reader.py | 2 +- test/integ/descriptor/reader.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/stem/descriptor/reader.py b/stem/descriptor/reader.py index 61bee5c..4683744 100644 --- a/stem/descriptor/reader.py +++ b/stem/descriptor/reader.py @@ -326,7 +326,7 @@ class DescriptorReader: # This is a file. Register it's last modified timestamp and check if # it's a file that we should skip. - last_modified = os.stat(target).st_mtime + last_modified = int(os.stat(target).st_mtime) last_used = self._processed_files.get(target) if last_used and last_used >= last_modified: diff --git a/test/integ/descriptor/reader.py b/test/integ/descriptor/reader.py index 717a55b..1444dbb 100644 --- a/test/integ/descriptor/reader.py +++ b/test/integ/descriptor/reader.py @@ -279,7 +279,7 @@ class TestDescriptorReader(unittest.TestCase): for root, _, files in os.walk(DESCRIPTOR_TEST_DATA): for filename in files: path = os.path.join(root, filename) - last_modified = os.stat(path).st_mtime + last_modified = int(os.stat(path).st_mtime) expected_results[path] = last_modified reader = stem.descriptor.reader.DescriptorReader([DESCRIPTOR_TEST_DATA])

1 0

[stem/master] Merge branch 'metrics-lib'
by atagar＠torproject.org 26 Mar '12

26 Mar '12

commit bc0e578615d9a856ab9052acc5a116774c49d133 Merge: 989fd95 975d98d Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 25 17:09:21 2012 -0700 Merge branch 'metrics-lib' run_tests.py | 12 + stem/__init__.py | 2 +- stem/descriptor/__init__.py | 87 +++ stem/descriptor/reader.py | 418 ++++++++++++++ stem/descriptor/server_descriptor.py | 574 ++++++++++++++++++++ stem/socket.py | 2 +- stem/util/__init__.py | 2 +- stem/util/connection.py | 56 ++ stem/util/tor_tools.py | 51 ++ test/integ/connection/authentication.py | 2 +- test/integ/descriptor/__init__.py | 6 + test/integ/descriptor/data/descriptor_archive.tar | Bin 0 -> 20480 bytes .../descriptor/data/descriptor_archive.tar.bz2 | Bin 0 -> 3322 bytes .../descriptor/data/descriptor_archive.tar.gz | Bin 0 -> 2844 bytes test/integ/descriptor/data/example_descriptor | 30 + test/integ/descriptor/data/riddle | 20 + test/integ/descriptor/data/tiny.png | Bin 0 -> 188 bytes test/integ/descriptor/reader.py | 410 ++++++++++++++ test/integ/descriptor/server_descriptor.py | 133 +++++ test/mocking.py | 41 ++- test/runner.py | 10 +- test/unit/descriptor/__init__.py | 6 + test/unit/descriptor/reader.py | 96 ++++ test/unit/descriptor/server_descriptor.py | 276 ++++++++++ test/unit/util/__init__.py | 2 +- test/unit/util/connection.py | 51 ++ test/unit/util/tor_tools.py | 55 ++ 27 files changed, 2331 insertions(+), 11 deletions(-)

1 0

[stem/master] Revising implementation details for processed files
by atagar＠torproject.org 26 Mar '12

26 Mar '12

commit d93d894a3339216b7dc1791ea8c009cd280db7ff Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 4 11:24:24 2012 -0800 Revising implementation details for processed files Like metrics-lib, the DescriptorReader will track the last modified timestamp for the descriptor files we have processed. Revising the pydocs for it and adding untested functions to save and load. Next step is to add integ tests. --- stem/descriptor/reader.py | 125 ++++++++++++++++++++++++++++++++++++-------- 1 files changed, 102 insertions(+), 23 deletions(-) diff --git a/stem/descriptor/reader.py b/stem/descriptor/reader.py index b6404f3..6bb3112 100644 --- a/stem/descriptor/reader.py +++ b/stem/descriptor/reader.py @@ -8,14 +8,11 @@ Example: ] reader = DescriptorReader(my_descriptors) - reader.start() - # prints all of the descriptor contents - for descriptor in reader: - print descriptor - - reader.stop() - reader.join() + with reader: + # prints all of the descriptor contents + for descriptor in reader: + print descriptor """ import os @@ -23,11 +20,96 @@ import theading import mimetypes import Queue +def load_processed_files(path): + """ + Loads a dictionary of 'path => last modified timestamp' mappings, as + persisted by save_processed_files(), from a file. + + Arguments: + path (str) - location to load the processed files dictionary from + + Returns: + dict of 'path (str) => last modified timestamp (int)' mappings + + Raises: + IOError if unable to read the file + TypeError if unable to parse the file's contents + """ + + processed_files = {} + + with open(path) as input_file: + for line in input_file.readlines(): + line = line.strip() + + if not line: continue # skip blank lines + + if not " " in line: + raise TypeError("Malformed line: %s" % line) + + path, timestamp = line.rsplit(" ", 1) + + if not os.path.isabs(path): + raise TypeError("'%s' is not an absolute path" % path) + elif not timestamp.isdigit(): + raise TypeError("'%s' is not an integer timestamp" % timestamp) + + processed_files[path] = int(timestamp) + + return processed_files + +def save_processed_files(processed_files, path): + """ + Persists a dictionary of 'path => last modified timestamp' mappings (as + provided by the DescriptorReader's get_processed_files() method) so that they + can be loaded later and applied to another DescriptorReader. + + Arguments: + processed_files (dict) - 'path => last modified' mappings + path (str) - location to save the processed files dictionary to + + Raises: + IOError if unable to write to the file + TypeError if processed_files is of the wrong type + """ + + # makes the parent directory if it doesn't already exist + try: + path_dir = os.path.dirname(path) + if not os.path.exists(path_dir): os.makedirs(path_dir) + except OSError, exc: raise IOError(exc) + + with open(path, "w") as output_file: + for path, timestamp in processed_files.items(): + output_file.write("%s %i" % (path, timestamp)) + class DescriptorReader(threading.Thread): """ Iterator for the descriptor data on the local file system. This can process text files, tarball archives (gzip or bzip2), or recurse directories. + This keeps track the last modified timestamps for descriptor files we have + used, and if you call restart() then this will only provide descriptors from + new files or files that have changed since them. + + You can also save this listing of processed files and later apply it another + DescriptorReader. For instance, to only print the descriptors that have + changed since the last ran... + + reader = DescriptorReader(["/tmp/descriptor_data"]) + + try: + processed_files = load_processed_files("/tmp/used_descriptors") + reader.set_processed_files(processed_files) + except: pass # could not load, mabye this is the first run + + # only prints descriptors that have changed since we last ran + with reader: + for descriptor in reader: + print descriptor + + save_processed_files(reader.get_processed_files(), "/tmp/used_descriptors") + This ignores files that cannot be processed (either due to read errors or because they don't contain descriptor data). The caller can be notified of files that are skipped by restering a listener with register_skip_listener(). @@ -37,13 +119,14 @@ class DescriptorReader(threading.Thread): self.targets = targets self.skip_listeners = [] self.processed_files = {} + self._stop_event = threading.Event() def stop(self): """ Stops further reading of descriptors. """ - pass # TODO: implement + self._stop_event.set() def get_processed_files(self): """ @@ -59,7 +142,7 @@ class DescriptorReader(threading.Thread): files we have processed """ - return self.processed_files + return dict(self.processed_files) def set_processed_files(self, processed_files): """ @@ -67,19 +150,6 @@ class DescriptorReader(threading.Thread): listing of processed files. With the get_processed_files() method this can be used to skip descriptors that we have already read. For instance... - # gets the initial descriptors - reader = DescriptorReader(["/tmp/descriptor_data"]) - - with reader: - initial_descriptors = list(reader) - processed_files = reader.get_processed_files() - - # only gets the descriptors that have changed since we last checked - reader = DescriptorReader(["/tmp/descriptor_data"]) - reader.set_processed_files(processed_files) - - with reader: - new_descriptors = list(reader) Arguments: processed_files (dict) - mapping of absolute paths (str) to unix @@ -104,7 +174,16 @@ class DescriptorReader(threading.Thread): self.skip_listeners.append(listener) def run(self): - pass # TODO: implement + # os.walk(path, followlinks = True) + # + # >>> mimetypes.guess_type("/home/atagar/Desktop/control-spec.txt") + # ('text/plain', None) + # + # >>> mimetypes.guess_type("/home/atagar/Desktop/server-descriptors-2012-03.tar.bz2") + # ('application/x-tar', 'bzip2') + + while not self._stop_event.isSet(): + pass # TODO: implement def _notify_skip_listener(self, path, exception): for listener in self.skip_listeners:

1 0

[stem/master] Skeletion for descriptor reader
by atagar＠torproject.org 26 Mar '12

26 Mar '12

commit 5a7bb2584ec976b64a91986e5aa8ca60d8ab60a0 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 4 01:22:36 2012 -0800 Skeletion for descriptor reader Initial skeleton for a class to read descriptor data, not including most of the functional bits. This will be similar to the metrics-lib DescriptorReader class, but with a slightly different API and the ability to handle tarball archives. --- stem/descriptor/__init__.py | 6 ++ stem/descriptor/reader.py | 119 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+), 0 deletions(-) diff --git a/stem/descriptor/__init__.py b/stem/descriptor/__init__.py new file mode 100644 index 0000000..fef6750 --- /dev/null +++ b/stem/descriptor/__init__.py @@ -0,0 +1,6 @@ +""" +Utilities for parsing and processing descriptor data. +""" + +__all__ = ["reader"] + diff --git a/stem/descriptor/reader.py b/stem/descriptor/reader.py new file mode 100644 index 0000000..b6404f3 --- /dev/null +++ b/stem/descriptor/reader.py @@ -0,0 +1,119 @@ +""" +Reads descriptors from local directories and archives. + +Example: + my_descriptors = [ + "/tmp/server-descriptors-2012-03.tar.bz2", + "/tmp/archived_descriptors/", + ] + + reader = DescriptorReader(my_descriptors) + reader.start() + + # prints all of the descriptor contents + for descriptor in reader: + print descriptor + + reader.stop() + reader.join() +""" + +import os +import theading +import mimetypes +import Queue + +class DescriptorReader(threading.Thread): + """ + Iterator for the descriptor data on the local file system. This can process + text files, tarball archives (gzip or bzip2), or recurse directories. + + This ignores files that cannot be processed (either due to read errors or + because they don't contain descriptor data). The caller can be notified of + files that are skipped by restering a listener with register_skip_listener(). + """ + + def __init__(self, targets): + self.targets = targets + self.skip_listeners = [] + self.processed_files = {} + + def stop(self): + """ + Stops further reading of descriptors. + """ + + pass # TODO: implement + + def get_processed_files(self): + """ + For each file we have provided descriptor data for this provides a mapping + of the form... + + absolute_path (str) => modified_time (int) + + This includes entries set through the set_processed_files() method. + + Returns: + dict with the paths and unix timestamp for the last modified times of the + files we have processed + """ + + return self.processed_files + + def set_processed_files(self, processed_files): + """ + Appends a dictionary of 'path => modified timestamp' mappings to our + listing of processed files. With the get_processed_files() method this can + be used to skip descriptors that we have already read. For instance... + + # gets the initial descriptors + reader = DescriptorReader(["/tmp/descriptor_data"]) + + with reader: + initial_descriptors = list(reader) + processed_files = reader.get_processed_files() + + # only gets the descriptors that have changed since we last checked + reader = DescriptorReader(["/tmp/descriptor_data"]) + reader.set_processed_files(processed_files) + + with reader: + new_descriptors = list(reader) + + Arguments: + processed_files (dict) - mapping of absolute paths (str) to unix + timestamps for the last modified time (int) + """ + + self.processed_files.update(processed_files) + + def register_skip_listener(self, listener): + """ + Registers a listener for files that are skipped. This listener is expected + to be a functor of the form... + + my_listener(path, exception) + + Arguments: + listener (functor) - functor to be notified of files that are skipped to + read errors or because they couldn't be parsed as + valid descriptor data + """ + + self.skip_listeners.append(listener) + + def run(self): + pass # TODO: implement + + def _notify_skip_listener(self, path, exception): + for listener in self.skip_listeners: + listener(path, exception) + + def __enter__(self): + self.start() + + def __exit__(self, type, value, traceback): + self.stop() + self.join() +

1 0

[stem/master] Adding mocking support for builtin functions
by atagar＠torproject.org 26 Mar '12

26 Mar '12

commit cbc0cb1a509911f89e8b14428efa65f0aa5b22b6 Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Mar 7 20:17:29 2012 -0800 Adding mocking support for builtin functions Builtin functions, like open, lack the __dict__ attribute among other things which prevents the current mocking scheme from working. Adding workarounds so we can accomidate them. This also adds a function for supporting the 'with' keyword on mock objects. --- test/mocking.py | 41 ++++++++++++++++++++++++++++++++++++++++- 1 files changed, 40 insertions(+), 1 deletions(-) diff --git a/test/mocking.py b/test/mocking.py index e5f46be..62e04c9 100644 --- a/test/mocking.py +++ b/test/mocking.py @@ -24,6 +24,7 @@ Instance Constructors import inspect import itertools import StringIO +import __builtin__ import stem.connection import stem.socket @@ -38,6 +39,8 @@ MOCK_ID = itertools.count(0) MOCK_STATE = {} +BUILTIN_TYPE = type(open) + def no_op(): def _no_op(*args): pass return _no_op @@ -54,6 +57,19 @@ def raise_exception(exception): def _raise(*args): raise exception return _raise +def support_with(obj): + """ + Provides no-op support for the 'with' keyword, adding __enter__ and __exit__ + methods to the object. The __enter__ provides the object itself and __exit__ + does nothing. + + Arguments: + obj (object) - object to support the 'with' keyword + """ + + obj.__dict__["__enter__"] = return_value(obj) + obj.__dict__["__exit__"] = no_op() + def mock(target, mock_call): """ Mocks the given function, saving the initial implementation so it can be @@ -64,6 +80,24 @@ def mock(target, mock_call): mock_call (functor) - mocking to replace the function with """ + # Builtin functions need special care because the builtin_function_or_method + # type lacks the normal '__dict__'. + + if isinstance(target, BUILTIN_TYPE): + # check if we have already mocked this function + target_function = target.__name__ + is_mocked = False + + for module, function_name, _ in MOCK_STATE.values(): + if module == __builtin__ and function_name == target_function: + is_mocked = True + + if not is_mocked: + MOCK_STATE[MOCK_ID.next()] = (__builtin__, target_function, target) + + setattr(__builtin__, target.__name__, mock_call) + return + if "mock_id" in target.__dict__: # we're overriding an already mocked function mocking_id = target.__dict__["mock_id"] @@ -130,7 +164,12 @@ def revert_mocking(): for mock_id in mock_ids: module, function, impl = MOCK_STATE[mock_id] - module.__dict__[function] = impl + + if module == __builtin__: + setattr(__builtin__, function, impl) + else: + module.__dict__[function] = impl + del MOCK_STATE[mock_id] MOCK_STATE.clear()

1 0

[stem/master] Typo in import statement
by atagar＠torproject.org 26 Mar '12

26 Mar '12

commit 5f8064b3ba8f47162dc781a5804839c029599d14 Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Mar 7 20:20:09 2012 -0800 Typo in import statement Basic syntax issue (import typo) --- stem/descriptor/reader.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/stem/descriptor/reader.py b/stem/descriptor/reader.py index 6bb3112..ec6a2b2 100644 --- a/stem/descriptor/reader.py +++ b/stem/descriptor/reader.py @@ -16,7 +16,7 @@ Example: """ import os -import theading +import threading import mimetypes import Queue

1 0