- tor-commits - lists.torproject.org

[tor/master] further changelog cleanups
by arma＠torproject.org 23 Apr '12

23 Apr '12

commit 54d0cdeb939fb19f6e430a62c255acac87b68f6a Author: Roger Dingledine <arma(a)torproject.org> Date: Mon Apr 23 01:22:59 2012 -0400 further changelog cleanups --- ChangeLog | 86 ++++++++++++++++++++++++++++-------------------------------- 1 files changed, 40 insertions(+), 46 deletions(-) diff --git a/ChangeLog b/ChangeLog index 43db8aa..9936e91 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,63 +1,57 @@ -Changes in version 0.2.3.14-alpha - 2012-04-?? +Changes in version 0.2.3.14-alpha - 2012-04-2? o Directory authority changes: - Change IP address for ides (v3 directory authority), and rename it to turtles. - o Security fixes: - - When using the debuging BridgePassword field, a bridge authority - now compares alleged passwords by hashing them, then comparing - the result to a digest of the expected authenticator. This avoids - a potential side-channel attack in the previous code, which - had foolishly used strcmp(). Fortunately, the BridgePassword field - *is not in use*, but if it had been, the timing - behavior of strcmp() might have allowed an adversary to guess the - BridgePassword value, and enumerate the bridges. Bugfix on - 0.2.0.14-alpha. Fixes bug 5543. - o Major bugfixes: - - Do not allow the presence of one consensus flavor to keep us from - downloading another. Previously, we had one "time to download a - consensus" timer, which didn't understand the idea of having one - consensus but wanting to download another. Fixes bug 4011; fix on - 0.2.3.1-alpha. - - If authorities are unable to get a set of v2 consensus documents - from other directory authorities, they no longer fail-back and - try to fetch them from regular directory caches. Fixes bug 5635; - bugfix on 0.2.2.26-beta, where routers stopped downloading v2 - consensus documents entirely. - - Prevent a client-side assertion failure when receiving an INTRODUCE2 + - When we start a Tor client with a normal consensus already cached, + be willing to download a microdescriptor consensus. Fixes bug 4011; + fix on 0.2.3.1-alpha. + - If authorities are unable to get a v2 consensus document from other + directory authorities, they no longer fall back to fetching + them from regular directory caches. Fixes bug 5635; bugfix on + 0.2.2.26-beta, where routers stopped downloading v2 consensus + documents entirely. + - Avoid a client-side assertion failure when receiving an INTRODUCE2 cell on a general purpose circuit. Fixes bug 5644; bugfix on 0.2.1.6-alpha. - Avoid logging uninitialized data when unable to decode a hidden - service descriptor cookie. Fixes bug 5647; bugfix on 0.2.1.5-alpha. + service descriptor cookie. Fixes bug 5647; bugfix on 0.2.1.5-alpha. o Major features (performance): - - When built to use the newly OpenSSL 1.0.1, and built for an x86 or - x86_64 instruction set, take advantage of OpenSSL's AESNI, - bitsliced, or vectorized AES implementations as appropriate. These - can be much, much faster than other AES implementations. - - o Minor bugfixes: + - When built to use OpenSSL 1.0.1, and built for an x86 or x86_64 + instruction set, take advantage of OpenSSL's AESNI, bitsliced, or + vectorized AES implementations as appropriate. These can be much, + much faster than other AES implementations. + + o Minor bugfixes (0.2.2.x and earlier): + - Change the BridgePassword feature (part of the "bridge community" + design, which is not yet implemented) to use a time-independent + comparison. The old behavior might have allowed an adversary + to use timing to guess the BridgePassword value. Fixes bug 5543; + bugfix on 0.2.0.14-alpha. + - Enforce correct return behavior of tor_vsscanf() when the '%%' + pattern is used. Fixes bug 5558. Bugfix on 0.2.1.13. + - When sending an HTTP/1.1 proxy request, include a Host header. + Fixes bug 5593; bugfix on 0.2.2.1-alpha. - Don't log that we have "decided to publish new relay descriptor" unless we are actually publishing a descriptor. Fixes bug 3942; - bugfix on 0.2.3.2-alpha. - - Fix bug stomping on ORPort option NoListen and ignoring option - NoAdvertise. Fixes bug 5151; bugfix on 0.2.3.9-alpha. - - In the testsuite, provide a large enough buffer in the tor_sscanf - unit test. We'd otherwise overrun that buffer and crash during the - unit tests. Fixes bug 5449; bugfix on 0.2.3.12-alpha. Thanks weasel - for spotting the bug. + bugfix on 0.2.2.28-beta. + + o Minor bugfixes (0.2.3.x): - Fix a bug where a bridge authority crashes (on a failed assert) if it has seen no directory requests when it's time to write statistics to disk. Fixes bug 5508. Bugfix on 0.2.3.6-alpha. - - Enforce correct return behavior of tor_vsscanf(), when the '%%' - pattern is used. Fixes bug 5558. Bugfix on 0.2.1.13. + - Fix bug stomping on ORPort option NoListen and ignoring option + NoAdvertise. Fixes bug 5151; bugfix on 0.2.3.9-alpha. + - In the testsuite, provide a large enough buffer in the tor_sscanf + unit test. Otherwise we'd overrun that buffer and crash during + the unit tests. Found by weasel. Fixes bug 5449; bugfix on + 0.2.3.12-alpha. - Make sure we create the keys directory if it doesn't exist and we're - about to store the dynamic diffie hellman parameters. Fixes bug 5572; - bugfix on 0.2.3.13-alpha. - - When sending an HTTP/1.1 proxy request, include a Host header. - Fixes bug 5593; bugfix on 0.2.2.1-alpha. + about to store the dynamic Diffie-Hellman parameters. Fixes bug + 5572; bugfix on 0.2.3.13-alpha. - Fix a small memory leak when trying to decode incorrect base16 authenticator during SAFECOOKIE authentication. Found by Coverity Scan. Fixes CID 507. Bugfix on 0.2.3.13-alpha. @@ -75,14 +69,14 @@ Changes in version 0.2.3.14-alpha - 2012-04-?? - Update to the April 3 2012 Maxmind GeoLite Country database. o Documentation: - - Begin a state-contents.txt file in doc to explain the contents - of the Tor state file. Fixes bug 2987. + - Begin a doc/state-contents.txt file to explain the contents of + the Tor state file. Fixes bug 2987. - Document unit of bandwidth related options in sample torrc. Fixes bug 5621. o Removed features: - - The "torify" script no longer supports the "tsocks" sockifier - tool, since it doesn't support DNS and UDP right for Tor. + - The "torify" script no longer supports the "tsocks" socksifier + tool, since tsocks doesn't support DNS and UDP right for Tor. Everyone should be using torsocks instead. Fixes bugs 3530 and 5180. Based on a patch by "ugh".

1 0

r25622: {website} finish the process of not recommending a particular web serv (website/trunk/docs/en)
by Roger Dingledine 23 Apr '12

23 Apr '12

Author: arma Date: 2012-04-23 04:27:55 +0000 (Mon, 23 Apr 2012) New Revision: 25622 Modified: website/trunk/docs/en/tor-hidden-service.wml Log: finish the process of not recommending a particular web server for hidden service operators Modified: website/trunk/docs/en/tor-hidden-service.wml =================================================================== --- website/trunk/docs/en/tor-hidden-service.wml 2012-04-23 04:12:22 UTC (rev 25621) +++ website/trunk/docs/en/tor-hidden-service.wml 2012-04-23 04:27:55 UTC (rev 25622) @@ -74,16 +74,22 @@ + You need to configure your web server so it doesn't give away any + information about you, your computer, or your location. Be sure to + bind the web server only to localhost (if people could get to it + directly, they could confirm that your computer is the one offering + the hidden service). Be sure that its error messages don't list + your hostname or other hints. Consider putting the web server in a + sandbox or VM to limit the damage from code vulnerabilities. + + + Once your web server is set up, make sure it works: open your browser and go to <a href="http://localhost:8080/">http://localhost:8080/</a>, where 8080 is the webserver port you chose during setup (you can choose any port, 8080 is just an example). Then try putting a file in the main html directory, and make sure it shows up when you access the site. - The reason we bind the web server only to localhost is to make sure - it isn't publically accessible. If people could get to it directly, - they could confirm that your computer is the one offering the - hidden service. <hr> @@ -193,16 +199,6 @@ want to make a backup copy of the <var>private_key</var> file somewhere. - We avoided recommending Apache above, a) because many people might - already be running it for a public web server on their computer, and b) - because it's big - and has lots of places where it might reveal your IP address or other - identifying information, for example in 404 pages. For people who need - more functionality, though, Apache may be the right answer. Can - somebody make us a checklist of ways to lock down your Apache when you're - using it as a hidden service? Savant probably has these problems too. - - If you want to forward multiple virtual ports for a single hidden service, just add more <var>HiddenServicePort</var> lines. If you want to run multiple hidden services from the same Tor

1 0

[stem/master] Concurrency is still hard
by atagar＠torproject.org 23 Apr '12

23 Apr '12

commit 4f8be72088e40bb4a6b62e9e39a4c3d9121ec472 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Apr 22 20:40:53 2012 -0700 Concurrency is still hard It felt like the handle_close flag was hacky and now I know why. If recv() tried to call close() while a send() call was in flight and would also eventually call close() then we'd encounter a similer type of deadlock as commit d5162f4 tried to fix. This did not come up for control port connections, but it did frequently occure for control sockets (as seen by deadlock when running the RUN_SOCKET target). This did not always occure because the recv() sometimes acquired the send_lock before the next send() call, making everything proceed correctly. Fixing this by having recv() make a non-blocking attempt to get the send_lock. If it works then we know that we can safely close, and if we can't acquire the lock then we know that a send() or close() call is already in progress and can leave the closing to them. This does introduce a possible issue where the send() call succeeds, the recv() call fails with a SocketClosed, and we're left in a state of being alive. However, this is both a weird use case (how can the send() work if the socket is closed?) and also not strictly wrong (raising a SocketClosed does not mean that we've finished shutting ourselves down). It's a little tempting to add a dedicated close_lock to account for this, but after looking into that I realized that this would make the concurrency far more convoluted. --- stem/socket.py | 27 +++++++++++++++++---------- 1 files changed, 17 insertions(+), 10 deletions(-) diff --git a/stem/socket.py b/stem/socket.py index 55129f3..6f0bfda 100644 --- a/stem/socket.py +++ b/stem/socket.py @@ -89,9 +89,6 @@ class ControlSocket: self._socket, self._socket_file = None, None self._is_alive = False - # indicates that we're in the midst of calling close() - self._handling_close = False - # Tracks sending and receiving separately. This should be safe, and doing # so prevents deadlock where we block writes because we're waiting to read # a message that isn't coming. @@ -149,10 +146,24 @@ class ControlSocket: return recv_message(socket_file) except SocketClosed, exc: # If recv_message raises a SocketClosed then we should properly shut - # everything down. However, if this was caused *from* a close call - # and it's joining on our thread then this would risk deadlock. + # everything down. However, there's a couple cases where this will + # cause deadlock... + # + # * this socketClosed was *caused by* a close() call, which is joining + # on our thread + # + # * a send() call that's currently in flight is about to call close(), + # also attempting to join on us + # + # To resolve this we make a non-blocking call to acquire the send lock. + # If we get it then great, we can close safely. If not then one of the + # above are in progress and we leave the close to them. + + if self.is_alive(): + if self._send_lock.acquire(False): + self.close() + self._send_lock.release() - if self.is_alive() and not self._handling_close: self.close() raise exc def is_alive(self): @@ -202,8 +213,6 @@ class ControlSocket: Shuts down the socket. If it's already closed then this is a no-op. """ - self._handling_close = True - with self._send_lock: # Function is idempotent with one exception: we notify _close() if this # is causing our is_alive() state to change. @@ -235,8 +244,6 @@ class ControlSocket: if is_change: self._close() - - self._handling_close = False def _get_send_lock(self): """

1 0

[stem/master] Bug with socket tests when we had an emulated chroot
by atagar＠torproject.org 23 Apr '12

23 Apr '12

commit f921c569d3565a554f90a6ae72b52032eca86cb9 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Apr 22 20:57:31 2012 -0700 Bug with socket tests when we had an emulated chroot Forgot to make one of the fixes for handling chroot tests in a socket use case. --- test/integ/connection/connect.py | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/test/integ/connection/connect.py b/test/integ/connection/connect.py index d61ddf7..29cf2dc 100644 --- a/test/integ/connection/connect.py +++ b/test/integ/connection/connect.py @@ -44,12 +44,15 @@ class TestConnect(unittest.TestCase): Basic sanity checks for the connect_socket_file function. """ + runner = test.runner.get_runner() + control_socket = stem.connection.connect_socket_file( socket_path = test.runner.CONTROL_SOCKET_PATH, password = test.runner.CONTROL_PASSWORD, + chroot_path = runner.get_chroot(), controller = stem.connection.Controller.NONE) - if test.runner.Torrc.SOCKET in test.runner.get_runner().get_options(): + if test.runner.Torrc.SOCKET in runner.get_options(): test.runner.exercise_socket(self, control_socket) control_socket.close() else:

1 0

[stem/master] Ensuring that we're closed when msg() raises SocketClosed
by atagar＠torproject.org 23 Apr '12

23 Apr '12

commit fcbbc58d700bb6a22a4d1e7bd7d5fcb22999c9f2 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Apr 22 21:14:33 2012 -0700 Ensuring that we're closed when msg() raises SocketClosed The ControlSocket's recv() method cannot assure that we're closed when it raises a SocketClosed exception (as explained in 4f8be72), but the BaseController's msg() method can. Integ tests were inconsistently failing because I expected to have this assurance. --- stem/control.py | 29 +++++++++++++++++++---------- 1 files changed, 19 insertions(+), 10 deletions(-) diff --git a/stem/control.py b/stem/control.py index 0ed4032..b3bd29c 100644 --- a/stem/control.py +++ b/stem/control.py @@ -167,16 +167,25 @@ class BaseController: break - self._socket.send(message) - response = self._reply_queue.get() - - # If the message we received back had an exception then re-raise it to the - # caller. Otherwise return the response. - - if isinstance(response, stem.socket.ControllerError): - raise response - else: - return response + try: + self._socket.send(message) + response = self._reply_queue.get() + + # If the message we received back had an exception then re-raise it to the + # caller. Otherwise return the response. + + if isinstance(response, stem.socket.ControllerError): + raise response + else: + return response + except stem.socket.SocketClosed, exc: + # If the recv() thread caused the SocketClosed then we could still be + # in the process of closing. Calling close() here so that we can + # provide an assurance to the caller that when we raise a SocketClosed + # exception we are shut down afterward for realz. + + self.close() + raise exc def is_alive(self): """

1 0

r25621: {website} Delete the troubleshooting paragraph that I wrote back when (website/trunk/docs/en)
by Roger Dingledine 23 Apr '12

23 Apr '12

Author: arma Date: 2012-04-23 04:12:22 +0000 (Mon, 23 Apr 2012) New Revision: 25621 Modified: website/trunk/docs/en/tor-hidden-service.wml Log: Delete the troubleshooting paragraph that I wrote back when people were commonly not configuring their browsers to use a proxy. Now that everybody is using Torbutton, "you didn't configure your browser correctly" shouldn't be the issue. And also the faq entry no longer helps you configure your browser correctly. Modified: website/trunk/docs/en/tor-hidden-service.wml =================================================================== --- website/trunk/docs/en/tor-hidden-service.wml 2012-04-22 00:47:42 UTC (rev 25620) +++ website/trunk/docs/en/tor-hidden-service.wml 2012-04-23 04:12:22 UTC (rev 25621) @@ -32,14 +32,6 @@ Go</a> search engine and someone hosting a <a href="http://duskgytldkxiuqc6.onion/">sample site</a>. - - It will typically take 10-60 seconds to load (or to decide that the - service is currently unreachable). If it fails immediately and your - browser pops up an alert saying that "www.duskgytldkxiuqc6.onion could - not be found, please check the name and try again" then you haven't - configured Tor correctly; see <a href="<page docs/faq>#DoesntWork">the - it-doesn't-work FAQ entry</a> for some help. - This page describes the steps for setting up your own hidden service

1 0

[stem/master] Fixing testing thread leak and adding check for it
by atagar＠torproject.org 23 Apr '12

23 Apr '12

commit 7271f04a9cdaf6bb6eb633424891c17654b2807f Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Apr 22 18:38:16 2012 -0700 Fixing testing thread leak and adding check for it Python does not gracefully handle lingering threads at shutdown, even if they're daemons (... which is dumb since that's the only point of the daemon thread). Having threads causes occasional nonsensical stacktraces with things like "'NoneType' object has no attribute 'socket'" for module references. Adding a check after running our integ tests that only the main thread remains, and erroring if that's not the case. This also fixes the thread leak that was probably the issue in... https://trac.torproject.org/projects/tor/ticket/5512 --- run_tests.py | 24 +++++++++++++++++++++--- stem/control.py | 9 ++++++++- test/integ/control/base_controller.py | 8 ++++---- 3 files changed, 33 insertions(+), 8 deletions(-) diff --git a/run_tests.py b/run_tests.py index dcfcdef..9ee6042 100755 --- a/run_tests.py +++ b/run_tests.py @@ -9,6 +9,7 @@ import sys import time import getopt import unittest +import threading import StringIO import test.output @@ -204,6 +205,9 @@ def load_user_configuration(test_config): if __name__ == '__main__': start_time = time.time() + # override flag to indicate at the end that testing failed somewhere + testing_failed = False + # loads and validates our various configurations test_config = stem.util.conf.get_config("test") @@ -320,6 +324,20 @@ if __name__ == '__main__': print test.output.print_logging(logging_buffer) + + # We should have joined on all threads. If not then that indicates a + # leak that could both likely be a bug and disrupt further targets. + + active_threads = threading.enumerate() + + if len(active_threads) > 1: + test.output.print_line("Threads lingering after test run:", *ERROR_ATTR) + + for lingering_thread in active_threads: + test.output.print_line(" %s" % lingering_thread, *ERROR_ATTR) + + testing_failed = True + break except KeyboardInterrupt: test.output.print_line(" aborted starting tor: keyboard interrupt\n", *ERROR_ATTR) break @@ -357,11 +375,11 @@ if __name__ == '__main__': if runtime < 1: runtime_label = "(%0.1f seconds)" % runtime else: runtime_label = "(%i seconds)" % runtime - if error_tracker.has_error_occured(): - test.output.print_line("TESTING FAILED %s" % runtime_label, term.Color.RED, term.Attr.BOLD) + if testing_failed or error_tracker.has_error_occured(): + test.output.print_line("TESTING FAILED %s" % runtime_label, *ERROR_ATTR) for line in error_tracker: - test.output.print_line(" %s" % line, term.Color.RED, term.Attr.BOLD) + test.output.print_line(" %s" % line, *ERROR_ATTR) else: test.output.print_line("TESTING PASSED %s" % runtime_label, term.Color.GREEN, term.Attr.BOLD) print diff --git a/stem/control.py b/stem/control.py index 55c6e7a..0ed4032 100644 --- a/stem/control.py +++ b/stem/control.py @@ -15,7 +15,8 @@ BaseController - Base controller class asynchronous message handling. |- close - shuts down our connection to the tor process |- get_socket - provides the socket used for control communication |- add_status_listener - notifies a callback of changes in our status - +- remove_status_listener - prevents further notification of status changes + |- remove_status_listener - prevents further notification of status changes + +- __enter__ / __exit__ - manages socket connection """ import time @@ -269,6 +270,12 @@ class BaseController: self._status_listeners = new_listeners return is_changed + def __enter__(self): + return self + + def __exit__(self, exit_type, value, traceback): + self.close() + def _handle_event(self, event_message): """ Callback to be overwritten by subclasses for event listening. This is diff --git a/test/integ/control/base_controller.py b/test/integ/control/base_controller.py index 1abab93..940ef99 100644 --- a/test/integ/control/base_controller.py +++ b/test/integ/control/base_controller.py @@ -40,8 +40,8 @@ class TestBaseController(unittest.TestCase): """ if test.runner.Torrc.PORT in test.runner.get_runner().get_options(): - controller = stem.control.BaseController.from_port(control_port = test.runner.CONTROL_PORT) - self.assertTrue(isinstance(controller, stem.control.BaseController)) + with stem.control.BaseController.from_port(control_port = test.runner.CONTROL_PORT) as controller: + self.assertTrue(isinstance(controller, stem.control.BaseController)) else: self.assertRaises(stem.socket.SocketError, stem.control.BaseController.from_port, "127.0.0.1", test.runner.CONTROL_PORT) @@ -51,8 +51,8 @@ class TestBaseController(unittest.TestCase): """ if test.runner.Torrc.SOCKET in test.runner.get_runner().get_options(): - controller = stem.control.BaseController.from_socket_file(test.runner.CONTROL_SOCKET_PATH) - self.assertTrue(isinstance(controller, stem.control.BaseController)) + with stem.control.BaseController.from_socket_file(socket_path = test.runner.CONTROL_SOCKET_PATH) as controller: + self.assertTrue(isinstance(controller, stem.control.BaseController)) else: self.assertRaises(stem.socket.SocketError, stem.control.BaseController.from_socket_file, test.runner.CONTROL_SOCKET_PATH)

1 0

[stem/master] Catching unexpected exception from descriptor reader
by atagar＠torproject.org 22 Apr '12

22 Apr '12

commit 6bb517d876243aafd5cf3ae0eef7ecbe45958389 Author: Damian Johnson <atagar(a)torproject.org> Date: Fri Apr 20 12:25:18 2012 -0700 Catching unexpected exception from descriptor reader The os.stat() call can raise an OSError, so catching that and notifying the skip listener. --- stem/descriptor/reader.py | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-) diff --git a/stem/descriptor/reader.py b/stem/descriptor/reader.py index 83b29c6..0bddd53 100644 --- a/stem/descriptor/reader.py +++ b/stem/descriptor/reader.py @@ -357,9 +357,13 @@ class DescriptorReader: # This is a file. Register its last modified timestamp and check if # it's a file that we should skip. - last_modified = int(os.stat(target).st_mtime) - last_used = self._processed_files.get(target) - new_processed_files[target] = last_modified + try: + last_modified = int(os.stat(target).st_mtime) + last_used = self._processed_files.get(target) + new_processed_files[target] = last_modified + except OSError, exc: + self._notify_skip_listeners(target, ReadFailed(exc)) + return if last_used and last_used >= last_modified: self._notify_skip_listeners(target, AlreadyRead(last_modified, last_used))

1 0

[stem/master] Testing option for chroot setups
by atagar＠torproject.org 22 Apr '12

22 Apr '12

commit 13f7ce34e735b2547f3ab669bed0ceb571624c0f Author: Damian Johnson <atagar(a)torproject.org> Date: Thu Apr 19 19:14:42 2012 -0700 Testing option for chroot setups Adding a new target to simulate a chroot setup by stripping the test directory from any control socket responses. As expected this is causing integ tests that validate this path to fail. Sathyanarayanan took the first several stabs at this, and this is just another potential option for... https://trac.torproject.org/projects/tor/ticket/4896 --- run_tests.py | 1 + test/runner.py | 38 ++++++++++++++++++++++++++++++++++++++ test/settings.cfg | 2 ++ test/testrc.sample | 11 ++--------- 4 files changed, 43 insertions(+), 9 deletions(-) diff --git a/run_tests.py b/run_tests.py index 1fee4da..dcfcdef 100755 --- a/run_tests.py +++ b/run_tests.py @@ -63,6 +63,7 @@ CONFIG = stem.util.conf.config_dict("test", { Target = stem.util.enum.UppercaseEnum( "ONLINE", "RELATIVE", + "CHROOT", "RUN_NONE", "RUN_OPEN", "RUN_PASSWORD", diff --git a/test/runner.py b/test/runner.py index 32c3872..3636b44 100644 --- a/test/runner.py +++ b/test/runner.py @@ -49,6 +49,7 @@ CONFIG = stem.util.conf.config_dict("test", { "integ.log": "./test/data/log", "integ.target.online": False, "integ.target.relative_data_dir": False, + "integ.target.chroot": False, }) STATUS_ATTR = (term.Color.BLUE, term.Attr.BOLD) @@ -135,6 +136,20 @@ def get_runner(): if not INTEG_RUNNER: INTEG_RUNNER = Runner() return INTEG_RUNNER +class MockChrootFile: + """ + Wrapper around a file object that strips given content from readline() + responses. This is used to simulate a chroot setup by removing the restign + directory from the paths we report. + """ + + def __init__(self, wrapped_file, strip_text): + self.wrapped_file = wrapped_file + self.strip_text = strip_text + + def readline(self): + return self.wrapped_file.readline().replace(self.strip_text, "") + class Runner: def __init__(self): self._runner_lock = threading.RLock() @@ -146,6 +161,10 @@ class Runner: self._torrc_contents = "" self._custom_opts = None self._tor_process = None + + # set if we monkey patch stem.socket.recv_message() + + self._original_recv_message = None def start(self, tor_cmd, extra_torrc_opts): """ @@ -198,6 +217,20 @@ class Runner: self._run_setup() self._start_tor(tor_cmd) + # strip the testing directory from recv_message responses if we're + # simulating a chroot setup + + if CONFIG["integ.target.chroot"] and not self._original_recv_message: + # TODO: when we have a function for telling stem the chroot we'll + # need to set that too + + self._original_recv_message = stem.socket.recv_message + + def _chroot_recv_message(control_file): + return self._original_recv_message(MockChrootFile(control_file, data_dir_path)) + + stem.socket.recv_message = _chroot_recv_message + # revert our cwd back to normal if CONFIG["integ.target.relative_data_dir"]: os.chdir(original_cwd) @@ -226,6 +259,11 @@ class Runner: if self._test_dir and CONFIG["integ.test_directory"] == "": shutil.rmtree(self._test_dir, ignore_errors = True) + # reverts any mockin of stem.socket.recv_message + if self._original_recv_message: + stem.socket.recv_message = self._original_recv_message + self._original_recv_message = None + self._test_dir = "" self._tor_cmd = None self._tor_cwd = "" diff --git a/test/settings.cfg b/test/settings.cfg index 74bb933..77156c5 100644 --- a/test/settings.cfg +++ b/test/settings.cfg @@ -30,6 +30,7 @@ msg.help target.config ONLINE => integ.target.online target.config RELATIVE => integ.target.relative_data_dir +target.config CHROOT => integ.target.chroot target.config RUN_NONE => integ.target.run.none target.config RUN_OPEN => integ.target.run.open target.config RUN_PASSWORD => integ.target.run.password @@ -44,6 +45,7 @@ target.config RUN_ALL => integ.target.run.all target.description ONLINE => Includes tests that require network activity. target.description RELATIVE => Uses a relative path for tor's data directory. +target.description CHROOT => Simulates a chroot setup. target.description RUN_NONE => Configuration without a way for controllers to connect. target.description RUN_OPEN => Configuration with an open control port (default). target.description RUN_PASSWORD => Configuration with password authentication. diff --git a/test/testrc.sample b/test/testrc.sample index a1cd5a6..165a8d8 100644 --- a/test/testrc.sample +++ b/test/testrc.sample @@ -26,15 +26,7 @@ # integ.target.relative_data_dir # Uses a relative path for the tor data directory if set. # -# integ.target.run.none -# integ.target.run.open -# integ.target.run.password -# integ.target.run.cookie -# integ.target.run.multiple -# integ.target.run.socket -# integ.target.run.scookie -# integ.target.run.ptrace -# integ.target.run.all +# integ.target.run.* # Runs the integration test suite for all of the given connection and # authentication configurations. If the 'all' option is set then the other # flags are ignored. @@ -50,6 +42,7 @@ integ.log ./test/data/log integ.target.online false integ.target.relative_data_dir false +integ.target.chroot false integ.target.run.none false integ.target.run.open true integ.target.run.password false

1 0

[stem/master] Handling chroot in stem.connection and integ tests
by atagar＠torproject.org 22 Apr '12

22 Apr '12

commit 636fd3704d6ddcba6a2a780119570fc2e25dd77f Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Apr 22 15:05:17 2012 -0700 Handling chroot in stem.connection and integ tests On ticket 4896 [1] I proposed a couple ideas for how we could handle chroot setups, but on reflection neither of them were very good. At a low level we can't reliably expand paths, nor should we try. If the user makes a raw 'GETINFO config-file' query then we should simply return what tor gives us, not try to 'fix' it by expanding the path. Rather, we should correct for chroot prefixes at a higher level like the controller. My current plan is... * The Controller class will have an optional chroot_path constructor argument, and a get_chroot() method. All of the Controller's methods and those of subclasses should take it into account for tor resource paths. * The stem.connection functions now accept a chroot_path argument. We need this since they will construct Controller instances, and also do the initial authentication (we need to know about chroots for cookie authentication). [1] https://trac.torproject.org/projects/tor/ticket/4896 --- stem/connection.py | 25 +++++++++++----- test/integ/connection/authentication.py | 16 ++++++----- test/integ/connection/connect.py | 5 +++- test/integ/connection/protocolinfo.py | 9 +++++- test/integ/control/base_controller.py | 6 ++-- test/integ/socket/control_message.py | 4 +++ test/runner.py | 45 +++++++++++++++++++++++------- test/unit/connection/authentication.py | 4 +- 8 files changed, 80 insertions(+), 34 deletions(-) diff --git a/stem/connection.py b/stem/connection.py index 419dfa3..d950049 100644 --- a/stem/connection.py +++ b/stem/connection.py @@ -194,7 +194,7 @@ AUTHENTICATE_EXCEPTIONS = ( AuthenticationFailure, ) -def connect_port(control_addr = "127.0.0.1", control_port = 9051, password = None, controller = Controller.NONE): +def connect_port(control_addr = "127.0.0.1", control_port = 9051, password = None, chroot_path = None, controller = Controller.NONE): """ Convenience function for quickly getting a control connection. This is very handy for debugging or CLI setup, handling setup and prompting for a password @@ -205,6 +205,7 @@ def connect_port(control_addr = "127.0.0.1", control_port = 9051, password = Non control_addr (str) - ip address of the controller control_port (int) - port number of the controller password (str) - passphrase to authenticate to the socket + chroot_path (str) - path prefix if in a chroot environment controller (Controller) - controller type to be returned Returns: @@ -220,9 +221,9 @@ def connect_port(control_addr = "127.0.0.1", control_port = 9051, password = Non print exc return None - return _connect(control_port, password, controller) + return _connect(control_port, password, chroot_path, controller) -def connect_socket_file(socket_path = "/var/run/tor/control", password = None, controller = Controller.NONE): +def connect_socket_file(socket_path = "/var/run/tor/control", password = None, chroot_path = None, controller = Controller.NONE): """ Convenience function for quickly getting a control connection. For more information see the connect_port function. @@ -230,6 +231,7 @@ def connect_socket_file(socket_path = "/var/run/tor/control", password = None, c Arguments: socket_path (str) - path where the control socket is located password (str) - passphrase to authenticate to the socket + chroot_path (str) - path prefix if in a chroot environment controller (Controller) - controller type to be returned Returns: @@ -242,15 +244,16 @@ def connect_socket_file(socket_path = "/var/run/tor/control", password = None, c print exc return None - return _connect(control_socket, password, controller) + return _connect(control_socket, password, chroot_path, controller) -def _connect(control_socket, password, controller): +def _connect(control_socket, password, chroot_path, controller): """ Common implementation for the connect_* functions. Arguments: control_socket (stem.socket.ControlSocket) - socket being authenticated to password (str) - passphrase to authenticate to the socket + chroot_path (str) - path prefix if in a chroot environment controller (Controller) - controller type to be returned Returns: @@ -258,7 +261,7 @@ def _connect(control_socket, password, controller): """ try: - authenticate(control_socket, password) + authenticate(control_socket, password, chroot_path) if controller == Controller.NONE: return control_socket @@ -274,7 +277,7 @@ def _connect(control_socket, password, controller): print "Unable to authenticate: %s" % exc return None -def authenticate(control_socket, password = None, protocolinfo_response = None): +def authenticate(control_socket, password = None, chroot_path = None, protocolinfo_response = None): """ Authenticates to a control socket using the information provided by a PROTOCOLINFO response. In practice this will often be all we need to @@ -288,6 +291,7 @@ def authenticate(control_socket, password = None, protocolinfo_response = None): control_socket (stem.socket.ControlSocket) - socket to be authenticated password (str) - passphrase to present to the socket if it uses password authentication (skips password auth if None) + chroot_path (str) - path prefix if in a chroot environment protocolinfo_response (stem.connection.ProtocolInfoResponse) - tor protocolinfo response, this is retrieved on our own if None @@ -397,7 +401,12 @@ def authenticate(control_socket, password = None, protocolinfo_response = None): elif auth_type == AuthMethod.PASSWORD: authenticate_password(control_socket, password, False) elif auth_type == AuthMethod.COOKIE: - authenticate_cookie(control_socket, protocolinfo_response.cookie_path, False) + cookie_path = protocolinfo_response.cookie_path + + if chroot_path: + cookie_path = os.path.join(chroot_path, cookie_path.lstrip(os.path.sep)) + + authenticate_cookie(control_socket, cookie_path, False) return # success! except OpenAuthRejected, exc: diff --git a/test/integ/connection/authentication.py b/test/integ/connection/authentication.py index aafde5d..f80b848 100644 --- a/test/integ/connection/authentication.py +++ b/test/integ/connection/authentication.py @@ -85,8 +85,9 @@ class TestAuthenticate(unittest.TestCase): Tests that the authenticate function can authenticate to our socket. """ - with test.runner.get_runner().get_tor_socket(False) as control_socket: - stem.connection.authenticate(control_socket, test.runner.CONTROL_PASSWORD) + runner = test.runner.get_runner() + with runner.get_tor_socket(False) as control_socket: + stem.connection.authenticate(control_socket, test.runner.CONTROL_PASSWORD, runner.get_chroot()) test.runner.exercise_socket(self, control_socket) def test_authenticate_general_example(self): @@ -94,7 +95,8 @@ class TestAuthenticate(unittest.TestCase): Tests the authenticate function with something like its pydoc example. """ - tor_options = test.runner.get_runner().get_options() + runner = test.runner.get_runner() + tor_options = runner.get_options() try: control_socket = stem.socket.ControlPort(control_port = test.runner.CONTROL_PORT) @@ -105,7 +107,7 @@ class TestAuthenticate(unittest.TestCase): try: # this authenticate call should work for everything but password-only auth - stem.connection.authenticate(control_socket) + stem.connection.authenticate(control_socket, chroot_path = runner.get_chroot()) test.runner.exercise_socket(self, control_socket) except stem.connection.IncorrectSocketType: self.fail() @@ -141,7 +143,7 @@ class TestAuthenticate(unittest.TestCase): if is_password_only: self.assertRaises(stem.connection.MissingPassword, stem.connection.authenticate, control_socket) else: - stem.connection.authenticate(control_socket) + stem.connection.authenticate(control_socket, chroot_path = runner.get_chroot()) test.runner.exercise_socket(self, control_socket) # tests with the incorrect password @@ -149,12 +151,12 @@ class TestAuthenticate(unittest.TestCase): if is_password_only: self.assertRaises(stem.connection.IncorrectPassword, stem.connection.authenticate, control_socket, "blarg") else: - stem.connection.authenticate(control_socket, "blarg") + stem.connection.authenticate(control_socket, "blarg", runner.get_chroot()) test.runner.exercise_socket(self, control_socket) # tests with the right password with runner.get_tor_socket(False) as control_socket: - stem.connection.authenticate(control_socket, test.runner.CONTROL_PASSWORD) + stem.connection.authenticate(control_socket, test.runner.CONTROL_PASSWORD, runner.get_chroot()) test.runner.exercise_socket(self, control_socket) def test_authenticate_none(self): diff --git a/test/integ/connection/connect.py b/test/integ/connection/connect.py index 9b6337a..d61ddf7 100644 --- a/test/integ/connection/connect.py +++ b/test/integ/connection/connect.py @@ -25,12 +25,15 @@ class TestConnect(unittest.TestCase): Basic sanity checks for the connect_port function. """ + runner = test.runner.get_runner() + control_socket = stem.connection.connect_port( control_port = test.runner.CONTROL_PORT, password = test.runner.CONTROL_PASSWORD, + chroot_path = runner.get_chroot(), controller = stem.connection.Controller.NONE) - if test.runner.Torrc.PORT in test.runner.get_runner().get_options(): + if test.runner.Torrc.PORT in runner.get_options(): test.runner.exercise_socket(self, control_socket) control_socket.close() else: diff --git a/test/integ/connection/protocolinfo.py b/test/integ/connection/protocolinfo.py index a0214a2..728c746 100644 --- a/test/integ/connection/protocolinfo.py +++ b/test/integ/connection/protocolinfo.py @@ -114,12 +114,17 @@ class TestProtocolInfo(unittest.TestCase): the test configuration. """ - tor_options = test.runner.get_runner().get_options() + runner = test.runner.get_runner() + tor_options = runner.get_options() auth_methods, auth_cookie_path = [], None if test.runner.Torrc.COOKIE in tor_options: auth_methods.append(stem.connection.AuthMethod.COOKIE) - auth_cookie_path = test.runner.get_runner().get_auth_cookie_path() + chroot_path = runner.get_chroot() + auth_cookie_path = runner.get_auth_cookie_path() + + if chroot_path and auth_cookie_path.startswith(chroot_path): + auth_cookie_path = auth_cookie_path[len(chroot_path):] if test.runner.Torrc.PASSWORD in tor_options: auth_methods.append(stem.connection.AuthMethod.PASSWORD) diff --git a/test/integ/control/base_controller.py b/test/integ/control/base_controller.py index da4cc08..1abab93 100644 --- a/test/integ/control/base_controller.py +++ b/test/integ/control/base_controller.py @@ -77,10 +77,10 @@ class TestBaseController(unittest.TestCase): runner = test.runner.get_runner() with runner.get_tor_socket() as control_socket: controller = stem.control.BaseController(control_socket) - response = controller.msg("GETINFO config-file") + response = controller.msg("GETINFO version") - torrc_dst = runner.get_torrc_path() - self.assertEquals("config-file=%s\nOK" % torrc_dst, str(response)) + tor_version = runner.get_tor_version() + self.assertEquals("version=%s\nOK" % tor_version, str(response)) def test_msg_invalid(self): """ diff --git a/test/integ/socket/control_message.py b/test/integ/socket/control_message.py index 72f9174..255e66d 100644 --- a/test/integ/socket/control_message.py +++ b/test/integ/socket/control_message.py @@ -86,6 +86,10 @@ class TestControlMessage(unittest.TestCase): runner = test.runner.get_runner() torrc_dst = runner.get_torrc_path() + chroot_path = runner.get_chroot() + + if chroot_path and torrc_dst.startswith(chroot_path): + torrc_dst = torrc_dst[len(chroot_path):] with runner.get_tor_socket() as control_socket: control_socket.send("GETINFO config-file") diff --git a/test/runner.py b/test/runner.py index 3636b44..6702b8b 100644 --- a/test/runner.py +++ b/test/runner.py @@ -21,6 +21,9 @@ Runner - Runtime context for our integration tests. |- get_test_dir - testing directory path |- get_torrc_path - path to our tor instance's torrc |- get_torrc_contents - contents of our tor instance's torrc + |- get_auth_cookie_path - path for our authentication cookie if we have one + |- get_tor_cwd - current working directory of our tor process + |- get_chroot - provides the path of our emulated chroot if we have one |- get_pid - process id of our tor process |- get_tor_socket - provides a socket to the tor instance |- get_tor_version - provides the version of tor we're running against @@ -122,7 +125,12 @@ def exercise_socket(test_case, control_socket): control_socket (stem.socket.ControlSocket) - socket to be tested """ - torrc_path = get_runner().get_torrc_path() + runner = get_runner() + torrc_path, chroot_path = runner.get_torrc_path(), runner.get_chroot() + + if chroot_path and torrc_path.startswith(chroot_path): + torrc_path = torrc_path[len(chroot_path):] + control_socket.send("GETINFO config-file") config_file_response = control_socket.recv() test_case.assertEquals("config-file=%s\nOK" % torrc_path, str(config_file_response)) @@ -136,7 +144,7 @@ def get_runner(): if not INTEG_RUNNER: INTEG_RUNNER = Runner() return INTEG_RUNNER -class MockChrootFile: +class _MockChrootFile: """ Wrapper around a file object that strips given content from readline() responses. This is used to simulate a chroot setup by removing the restign @@ -161,6 +169,7 @@ class Runner: self._torrc_contents = "" self._custom_opts = None self._tor_process = None + self._chroot_path = None # set if we monkey patch stem.socket.recv_message() @@ -225,9 +234,10 @@ class Runner: # need to set that too self._original_recv_message = stem.socket.recv_message + self._chroot_path = data_dir_path def _chroot_recv_message(control_file): - return self._original_recv_message(MockChrootFile(control_file, data_dir_path)) + return self._original_recv_message(_MockChrootFile(control_file, data_dir_path)) stem.socket.recv_message = _chroot_recv_message @@ -356,9 +366,24 @@ class Runner: test_dir = self._get("_test_dir") return os.path.join(test_dir, "torrc") + def get_torrc_contents(self): + """ + Provides the contents of our torrc. + + Returns: + str with the contents of our torrc, lines are newline separated + + Raises: + RunnerStopped if we aren't running + """ + + return self._get("_torrc_contents") + def get_auth_cookie_path(self): """ Provides the absolute path for our authentication cookie if we have one. + If running with an emulated chroot this is uneffected, still providing the + real path. Returns: str with our auth cookie path @@ -377,18 +402,16 @@ class Runner: return self._get("_tor_cwd") - def get_torrc_contents(self): + def get_chroot(self): """ - Provides the contents of our torrc. + Provides the path we're using to emulate a chroot environment. This is None + if we aren't emulating a chroot setup. Returns: - str with the contents of our torrc, lines are newline separated - - Raises: - RunnerStopped if we aren't running + str with the path of our emulated chroot """ - return self._get("_torrc_contents") + return self._chroot_path def get_pid(self): """ @@ -425,7 +448,7 @@ class Runner: else: raise TorInaccessable("Unable to connect to tor") if authenticate: - stem.connection.authenticate(control_socket, CONTROL_PASSWORD) + stem.connection.authenticate(control_socket, CONTROL_PASSWORD, self.get_chroot()) return control_socket diff --git a/test/unit/connection/authentication.py b/test/unit/connection/authentication.py index 0937439..831ca62 100644 --- a/test/unit/connection/authentication.py +++ b/test/unit/connection/authentication.py @@ -140,9 +140,9 @@ class TestAuthenticate(unittest.TestCase): mocking.mock(auth_function, mocking.raise_exception(raised_exc)) if expect_success: - stem.connection.authenticate(None, "blah", protocolinfo_arg) + stem.connection.authenticate(None, "blah", None, protocolinfo_arg) else: - self.assertRaises(stem.connection.AuthenticationFailure, stem.connection.authenticate, None, "blah", protocolinfo_arg) + self.assertRaises(stem.connection.AuthenticationFailure, stem.connection.authenticate, None, "blah", None, protocolinfo_arg) # revert logging back to normal stem_logger.setLevel(log.logging_level(log.TRACE))

1 0