February 2013 - tor-commits - lists.torproject.org

[stem/master] Accounting for different exception type from base64.b64decode()
by atagar＠torproject.org 02 Feb '13

02 Feb '13

commit 264c9e673f71dde707a03ec6b24931f1bd0f3fa4 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Jan 27 16:02:06 2013 -0800 Accounting for different exception type from base64.b64decode() When the base64.b64decode() function encounters a value with malformed padding it raises a TypeError in python 2.x and a binascii.Error in python 3.x. ====================================================================== ERROR: test_malformed_fingerprint ---… [View More]------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/router_status_entry.py", line 231, in test_malformed_fingerprint self._expect_invalid_attr(content, "fingerprint") File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/router_status_entry.py", line 525, in _expect_invalid_attr entry = RouterStatusEntryV3(content, False) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 330, in __init__ super(RouterStatusEntryV3, self).__init__(content, validate, document) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 150, in __init__ self._parse(entries, validate) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 337, in _parse _parse_r_line(self, value, validate, True) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 496, in _parse_r_line desc.fingerprint = _decode_fingerprint(r_comp[1], validate) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 703, in _decode_fingerprint identity_decoded = base64.b64decode(stem.util.str_tools.to_bytes(identity)) File "/usr/lib/python3.2/base64.py", line 92, in b64decode return binascii.a2b_base64(s) binascii.Error: Incorrect padding --- stem/descriptor/router_status_entry.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/stem/descriptor/router_status_entry.py b/stem/descriptor/router_status_entry.py index 7f9875b..ce5a7db 100644 --- a/stem/descriptor/router_status_entry.py +++ b/stem/descriptor/router_status_entry.py @@ -17,6 +17,7 @@ sources... """ import base64 +import binascii import datetime import stem.descriptor @@ -701,7 +702,7 @@ def _decode_fingerprint(identity, validate): try: identity_decoded = base64.b64decode(stem.util.str_tools.to_bytes(identity)) - except TypeError: + except (TypeError, binascii.Error): if not validate: return None [View Less]

1 0

[stem/master] Converted _required_fields() encounter TypeErrors
by atagar＠torproject.org 02 Feb '13

02 Feb '13

commit 8d5d2f14edf92c7240aae0b7894fea6e27b41699 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Jan 27 16:17:34 2013 -0800 Converted _required_fields() encounter TypeErrors The server and extrainfo descriptors' _required_fields() functions get converted by 2to3 into a line that tries to concatenate a list with a tuple (raising a TypeError). On reflection the converted version which uses list comprehension is a little nicer than filter(), so using a … [View More]

1 0

[stem/master] Mocking for open() broken in python 3
by atagar＠torproject.org 02 Feb '13

02 Feb '13

commit 8b95332960e5343dd9b716ef25682548f0de9d7e Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Jan 27 16:32:06 2013 -0800 Mocking for open() broken in python 3 In python 2.x there's both __builtin__.open() and io.open(), but in python 3.x there's just io.open() (which is aliased by builtin). Accounting for this when we mock it for the descriptor reader unit tests. I'm a little puzzled why the mocking wasn't failing more spectacularly before … [View More](the __builtin__ import should always fail on python 3), but oh well - fixed now. With this the unit tests finally pass with python 3!!! ====================================================================== ERROR: test_load_processed_files ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/reader.py", line 52, in test_load_processed_files self.assertEquals(expected_value, stem.descriptor.reader.load_processed_files("")) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/reader.py", line 179, in load_processed_files with open(path) as input_file: IOError: [Errno 2] No such file or directory: '' ====================================================================== FAIL: test_get_connections ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/util/proc.py", line 171, in test_get_connections self.assertEquals(expected_results, proc.get_connections(pid)) AssertionError: Lists differ: [('17.17.17.17', 4369, '34.34.... != [] First list contains 2 additional elements. First extra element 0: ('17.17.17.17', 4369, '34.34.34.34', 8738) + [] - [('17.17.17.17', 4369, '34.34.34.34', 8738), - ] --- test/mocking.py | 5 +++-- test/unit/descriptor/reader.py | 36 ++++++++++++++++++------------------ test/unit/util/proc.py | 18 ++++++++++++++---- 3 files changed, 35 insertions(+), 24 deletions(-) diff --git a/test/mocking.py b/test/mocking.py index ea88992..6472659 100644 --- a/test/mocking.py +++ b/test/mocking.py @@ -52,7 +52,6 @@ import hashlib import inspect import itertools import StringIO -import __builtin__ import stem.descriptor.extrainfo_descriptor import stem.descriptor.networkstatus @@ -409,12 +408,14 @@ def revert_mocking(): # makes the following check fail. Haven't a clue why. if stem.prereq.is_python_3(): + import builtins builtin_module = builtins else: + import __builtin__ builtin_module = __builtin__ if module == builtin_module: - setattr(__builtin__, function, impl) + setattr(builtin_module, function, impl) else: setattr(module, function, impl) diff --git a/test/unit/descriptor/reader.py b/test/unit/descriptor/reader.py index acd77b5..bc24805 100644 --- a/test/unit/descriptor/reader.py +++ b/test/unit/descriptor/reader.py @@ -6,9 +6,21 @@ import StringIO import unittest import stem.descriptor.reader +import stem.prereq import test.mocking as mocking +def _mock_open(content): + test_content = StringIO.StringIO(content) + mocking.support_with(test_content) + + if stem.prereq.is_python_3(): + import builtins + mocking.mock(builtins.open, mocking.return_value(test_content), builtins) + else: + mocking.mock(open, mocking.return_value(test_content)) + + class TestDescriptorReader(unittest.TestCase): def tearDown(self): mocking.revert_mocking() @@ -36,9 +48,7 @@ class TestDescriptorReader(unittest.TestCase): "/dir/after empty line": 12345, } - test_content = StringIO.StringIO("\n".join(test_lines)) - mocking.support_with(test_content) - mocking.mock(open, mocking.return_value(test_content)) + _mock_open("\n".join(test_lines)) self.assertEquals(expected_value, stem.descriptor.reader.load_processed_files("")) def test_load_processed_files_empty(self): @@ -46,9 +56,7 @@ class TestDescriptorReader(unittest.TestCase): Tests the load_processed_files() function with an empty file. """ - test_content = StringIO.StringIO("") - mocking.support_with(test_content) - mocking.mock(open, mocking.return_value(test_content)) + _mock_open("") self.assertEquals({}, stem.descriptor.reader.load_processed_files("")) def test_load_processed_files_no_file(self): @@ -57,9 +65,7 @@ class TestDescriptorReader(unittest.TestCase): it is missing the file path. """ - test_content = StringIO.StringIO(" 12345") - mocking.support_with(test_content) - mocking.mock(open, mocking.return_value(test_content)) + _mock_open(" 12345") self.assertRaises(TypeError, stem.descriptor.reader.load_processed_files, "") def test_load_processed_files_no_timestamp(self): @@ -68,9 +74,7 @@ class TestDescriptorReader(unittest.TestCase): it is missing the timestamp. """ - test_content = StringIO.StringIO("/dir/file ") - mocking.support_with(test_content) - mocking.mock(open, mocking.return_value(test_content)) + _mock_open("/dir/file ") self.assertRaises(TypeError, stem.descriptor.reader.load_processed_files, "") def test_load_processed_files_malformed_file(self): @@ -79,9 +83,7 @@ class TestDescriptorReader(unittest.TestCase): it has an invalid file path. """ - test_content = StringIO.StringIO("not_an_absolute_file 12345") - mocking.support_with(test_content) - mocking.mock(open, mocking.return_value(test_content)) + _mock_open("not_an_absolute_file 12345") self.assertRaises(TypeError, stem.descriptor.reader.load_processed_files, "") def test_load_processed_files_malformed_timestamp(self): @@ -90,7 +92,5 @@ class TestDescriptorReader(unittest.TestCase): it has a non-numeric timestamp. """ - test_content = StringIO.StringIO("/dir/file 123a") - mocking.support_with(test_content) - mocking.mock(open, mocking.return_value(test_content)) + _mock_open("/dir/file 123a") self.assertRaises(TypeError, stem.descriptor.reader.load_processed_files, "") diff --git a/test/unit/util/proc.py b/test/unit/util/proc.py index 1ba2c60..0a6c0a0 100644 --- a/test/unit/util/proc.py +++ b/test/unit/util/proc.py @@ -6,6 +6,8 @@ import os import StringIO import unittest +import stem.prereq + from stem.util import proc from test import mocking @@ -155,10 +157,18 @@ class TestProc(unittest.TestCase): tcp = '\n 0: 11111111:1111 22222222:2222 01 44444444:44444444 55:55555555 66666666 1111 8 99999999' udp = '\n A: BBBBBBBB:BBBB CCCCCCCC:CCCC DD EEEEEEEE:EEEEEEEE FF:FFFFFFFF GGGGGGGG 1111 H IIIIIIII' - mocking.mock(open, mocking.return_for_args({ - ('/proc/net/tcp',): StringIO.StringIO(tcp), - ('/proc/net/udp',): StringIO.StringIO(udp) - })) + if stem.prereq.is_python_3(): + import builtins + + mocking.mock(builtins.open, mocking.return_for_args({ + ('/proc/net/tcp',): StringIO.StringIO(tcp), + ('/proc/net/udp',): StringIO.StringIO(udp) + }), builtins) + else: + mocking.mock(open, mocking.return_for_args({ + ('/proc/net/tcp',): StringIO.StringIO(tcp), + ('/proc/net/udp',): StringIO.StringIO(udp) + })) # tests the edge case of pid = 0 self.assertEquals([], proc.get_connections(0)) [View Less]

1 0

[stem/master] Socket closures raise a ValueError in python 3
by atagar＠torproject.org 02 Feb '13

02 Feb '13

commit 1e2e6204eb8ba618e79c7bb1459e53e5154147fa Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Jan 27 17:43:22 2013 -0800 Socket closures raise a ValueError in python 3 Evidently reading from a closed socket file raises a ValueError rather than socket.error in python 3... Exception in thread Tor Listener: Traceback (most recent call last): File "/usr/lib/python3.2/threading.py", line 736, in _bootstrap_inner self.run() File "… [View More]

1 0

[stem/master] Converting tor initialization lines to unicode
by atagar＠torproject.org 02 Feb '13

02 Feb '13

commit 7507edec828603a860d9ebf0cbb049060d32fc3e Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Jan 27 17:41:08 2013 -0800 Converting tor initialization lines to unicode The stdout of the tor process is an ASCII byte stream, causing our following regex search() call to error. --- stem/process.py | 9 ++++++++- 1 files changed, 8 insertions(+), 1 deletions(-) diff --git a/stem/process.py b/stem/process.py index 19d9028..52a931e 100644 --- a/stem/process.… [View More]

1 0

[stem/master] Making sure that the socket's file is writeable
by atagar＠torproject.org 02 Feb '13

02 Feb '13

commit 6219ed6e6c517e0ce7d25b5d54355ed85329615e Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Jan 27 18:02:49 2013 -0800 Making sure that the socket's file is writeable In python 2.x the socket.makefile() method *should* only be readable (according to the default argument) but I've been using it for write operations too all this time. In python 3.x doing this raises "io.UnsupportedOperation: not writable" so guess we gotta fix it now. :) ==… [View More]==================================================================== ERROR: test_get_connections ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/util/proc.py", line 92, in test_get_connections elif not test.runner.get_runner().is_ptraceable(): File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 413, in is_ptraceable tor_version = self.get_tor_version() File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 567, in get_tor_version control_socket = self.get_tor_socket() File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 534, in get_tor_socket stem.connection.authenticate(control_socket, CONTROL_PASSWORD, self.get_chroot()) File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 321, in authenticate protocolinfo_response = get_protocolinfo(controller) File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 800, in get_protocolinfo protocolinfo_response = _msg(controller, "PROTOCOLINFO 1") File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 836, in _msg controller.send(message) File "/home/atagar/Desktop/stem/test/data/python3/stem/socket.py", line 82, in send send_message(self._socket_file, message, raw) File "/home/atagar/Desktop/stem/test/data/python3/stem/socket.py", line 422, in send_message control_file.write(message) io.UnsupportedOperation: not writable --- stem/socket.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/stem/socket.py b/stem/socket.py index 9d48bd5..cad4572 100644 --- a/stem/socket.py +++ b/stem/socket.py @@ -181,7 +181,7 @@ class ControlSocket(object): with self._recv_lock: self._socket = self._make_socket() - self._socket_file = self._socket.makefile() + self._socket_file = self._socket.makefile(mode = "rw") self._is_alive = True # It's possible for this to have a transient failure... [View Less]

1 0

[stem/master] Converting str stats for proc tests
by atagar＠torproject.org 02 Feb '13

02 Feb '13

commit 6a85a34c753f34c7cc79f4ad4e9f67d92bce1046 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Jan 27 19:01:32 2013 -0800 Converting str stats for proc tests The proc tests were comparing strings with floats. It coincidentally passed, but wasn't really testing what we wanted. In python 3.x the comparisons fail due to the type mismatch. ====================================================================== ERROR: test_get_stats ----------… [View More]

1 0

[stem/master] Disabling newline translation in our socket
by atagar＠torproject.org 02 Feb '13

02 Feb '13

commit c2b92450016ccfbe46edec151136c8a8a9f694cb Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Jan 27 18:18:45 2013 -0800 Disabling newline translation in our socket In python 3 the socket file object tries to be helpful by translating newline characters. By default '\n', '\r', and '\r\n' are all translated to '\n'. Tor uses '\r\n' newlines and we check for this so disabling universal newline translation. ====================================… [View More]================================== ERROR: test_get_pid_by_port_netstat ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 321, in authenticate protocolinfo_response = get_protocolinfo(controller) File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 800, in get_protocolinfo protocolinfo_response = _msg(controller, "PROTOCOLINFO 1") File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 837, in _msg return controller.recv() File "/home/atagar/Desktop/stem/test/data/python3/stem/socket.py", line 114, in recv return recv_message(socket_file) File "/home/atagar/Desktop/stem/test/data/python3/stem/socket.py", line 511, in recv_message raise stem.ProtocolError("All lines should end with CRLF") stem.ProtocolError: All lines should end with CRLF During handling of the above exception, another exception occurred: Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/util/system.py", line 245, in test_get_pid_by_port_netstat elif not runner.is_ptraceable(): File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 413, in is_ptraceable tor_version = self.get_tor_version() File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 567, in get_tor_version control_socket = self.get_tor_socket() File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 534, in get_tor_socket stem.connection.authenticate(control_socket, CONTROL_PASSWORD, self.get_chroot()) File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 323, in authenticate raise IncorrectSocketType("unable to use the control socket") stem.connection.IncorrectSocketType: unable to use the control socket --- stem/socket.py | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/stem/socket.py b/stem/socket.py index cad4572..f0786ab 100644 --- a/stem/socket.py +++ b/stem/socket.py @@ -35,6 +35,7 @@ import re import socket import threading +import stem.prereq import stem.response from stem.util import log @@ -181,7 +182,12 @@ class ControlSocket(object): with self._recv_lock: self._socket = self._make_socket() - self._socket_file = self._socket.makefile(mode = "rw") + + if stem.prereq.is_python_3(): + self._socket_file = self._socket.makefile(mode = "rw", newline = "") + else: + self._socket_file = self._socket.makefile(mode = "rw") + self._is_alive = True # It's possible for this to have a transient failure... [View Less]

1 0

[stem/master] Skipping orphaned file check if testing with python 3
by atagar＠torproject.org 02 Feb '13

02 Feb '13

commit 313e5251da4a390fa283c440ffbe0f0ef048ed38 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Jan 27 19:31:42 2013 -0800 Skipping orphaned file check if testing with python 3 The orphaned *.pyc check is both broken with python 3 and pointless since it's an exported copy of the python 2 codebase (we're not acually making changes to it). --- run_tests.py | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diff --git a/run_tests.py b/… [View More]

1 0

[stem/master] Returning unicode from stem.util.system.call()
by atagar＠torproject.org 02 Feb '13

02 Feb '13

commit 7d1e8a4ffc08ca3b5e7fb34d3b025ec3c8755497 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Jan 27 19:08:40 2013 -0800 Returning unicode from stem.util.system.call() The call() method is the basic call for shelling out and, like tor's stdout earlier, its output was ASCII bytes. This is fine in python 2.x, but causes problems for python 3.x. Both are fine with this being unicode output so going with that. ==================================… [View More]

1 0