June 2013 - tor-dev - lists.torproject.org

Haskell packages?
by Nikita Karetnikov 23 Aug '13

23 Aug '13

I'd like to improve my Haskell skills. Are there any opportunities? I've been told there is at least one project that uses Haskell, which is not maintained. (For example, this page [1] mentions TorDNSEL, which was replaced by TorBEL.) [1] https://www.torproject.org/getinvolved/volunteer.html.en

5 21

RFC: obfsproxyssh
by Yawning Angel 30 Jul '13

30 Jul '13

Hello, I have been talking about this in #tor-dev for a while (and pestering people with questions regarding some of the more nuanced aspects of writing a pluggable transport, thanks to nickm, mikeperry and asn for their help), and finally have what I would consider a pre-alpha for the PT implementation. obfsproxyssh is a pluggable transport that uses the ssh wire protocol to hide tor traffic. It uses libssh2 and interacts with a real sshd located on the bridge side. Behaviorally it is identical to a user sshing to a host, authenticating with a RSA public/private key pair and opening a direct-tcp channel to the ORPort of the bridge. It is more aimed at non-technical users (because anyone with an account on a bridge can create a tunnel of their own using existing ssh clients), and thus can be configured entirely through the torrc. It still needs a bit of work before it is ready for deployment but the code is at the point where I can use it for casual web browsing, so if people are interested, I have put a snapshot online at http://www.schwanenlied.me/yawning/obfsproxyssh-20130627.tar.gz Note that it is still at the "I got it working today" state, so some parts may be a bit rough around the edges, and more than likely a few really dumb bugs lurk unseen. Things that still need to be done (in rough order of priority): * It needs to scrub IP addresses in logs. * I need to compare libssh2's KEX phase with popular ssh clients (For the initial "production" release more than likely Putty). It currently also sends a rather distinctive banner since I'm not sure which client(s) it will end up mimicking. * I need to come up with a solution for server side sshd logs. What I will probably end up doing is writing a patch for OpenSSH to disable logging for select users. * In a peculiar oversight, OpenSSH doesn't have a way to disable reverse ssh tunnels (Eg: "PermitOpen 127.0.0.1:6969" allows clients to listen on that port). Not a big deal if Tor starts up before any clients connect, but I'll probably end up writing another patch for this. * Someone needs to test it on Windows. All of my external dependencies are known to work, so the porting effort should be minimal (Famous last words). * The code needs to scrub the private key as soon as a connection succeeds in authenticating instead of holding onto it. Probably not a big deal since anyone that can look at the PT's heap can also look at the bridge line in the torrc. Nice to haves: * Write real Makefiles instead of using CMake (I was lazy). src/CMakeLists.txt currently needs to be edited for anyone compiling it. * It currently uses unencrypted RSA keys. libssh2 supports ssh-agent (on all of the relevant platforms) so key management can be handled that way. I do not think there is currently a mechanism for Tor to query the user for a passphrase and pass it to the PT, but if one gets added, support it would also be easy from my end. * The code does not handle IPv6 since it uses SOCKS 4 instead of 5. When Tor gets a way to pass arguments to PTs that are > 510 bytes, I will change this. * libssh2 needs a few improvements going forward (In particular it does not support ECDSA at all). * Code for the bridge side that makes the tunnel speak the manged PT server transport protocol would be nice (For rate limiting). * libssh2_helpers.c should go away one day. Not sure why the libssh2 maintainers haven't merged the patch that I based the code in there on. Things that need to be done on the Tor side of things: * 0.2.4-14-alpha does not have the PT argument parsing code, so this requires a build out of git to function. * The code currently in Git fails to parse bridge lines with arguments that can't be passed via SOCKS 5 (size restriction). The PT tarball has a crude patch that removes the check, but the config file parser needs to be changed. * The Tor code currently in Git likes to forget PT arguments. asn was kind enough to provide me with a patch that appears to fix this (though the PT has a workaround for when it encounters this situation), but moving forward a formal fix would be ideal. * All the PT related cleverness in the world won't do much vs active probing if there is an ORport exposed on the bridge. Tor should be able to handle "ORPort 127.0.0.1:6969" (It may currently work, I'm not sure. There should be a way to disable the reachability check if only to reduce log spam). Open questions: * Is this useful? * Is it worth biting the bullet and rewriting this to use Twisted Conch instead of being a C app? * Would it be simpler to write a wrapper around existing ssh clients? (Probably not.) * How should people handle distributing bridge information? * How to handle the private key to a given bridge getting compromised? (Correctly configured, all that someone who obtains the key would be able to do is talk to the OR port so it's not a security thing, but it opens the bridge to being blocked). * Does Tor tunneled over SSH look distinctive? No effort is made to change the traffic signature, though this can be added if needed. The tarball contains a more detailed README explaining how to set it up and how it works. obfsproxyssh_client.c has a more in-depth TODO list as a large comment near the top of the file. Comments and feedback will be appreciated. Regards, -- Yawning Angel

5 6

Remote descriptor fetching
by Damian Johnson 23 Jul '13

23 Jul '13

Hi Karsten. I'm starting to look into remote descriptor fetching, a capability of metrics-lib that stem presently lacks [1][2]. The spec says that mirrors provide zlib compressed data [3], and the DirectoryDownloader handles this via a InflaterInputStream [4]. So far, so good. By my read of the man pages this means that gzip or python's zlib module should be able to handle the decompression. However, I must be missing something... % wget http://128.31.0.34:9131/tor/server/all.z % file all.z all.z: data % gzip -d all.z gzip: all.z: not in gzip format % zcat all.z gzip: all.z: not in gzip format % python >>> import zlib >>> with open('all.z') as desc_file: ... print zlib.decompress(desc_file.read()) ... Traceback (most recent call last): File "<stdin>", line 2, in <module> zlib.error: Error -5 while decompressing data: incomplete or truncated stream Maybe a fresh set of eyes will spot something I'm obviously missing. Spotting anything? Speaking of remote descriptor fetching, any thought on the API? I'm thinking of a 'stem/descriptor/remote.py' module with... * get_directory_authorities() List of hardcoded (IP, DirPorts) tuples for tor's authorities. Ideally we'd have an integ test to notify us when our listing falls out of date. However, it looks like the controller interface doesn't surface this. Is there a nice method of determining the present authorities besides polling the authorities array of 'src/or/config.c' [5]? * fetch_directory_mirrors() Polls an authority for the present consensus and filters it down to relays with the V2Dir flag. It then uses this to populate a global directory mirror cached that's used when querying directory data. This can optionally be provided with a Controller instance or cached consensus file to use that instead of polling a authority. * get_directory_cache() Provides a list of our present directory mirrors. This is a list if (IP, DirPort) tuples. If fetch_directory_mirrors() hasn't yet been called this is the directory authorities. * query(descriptor_type, fingerprint = None, retires = 5) Picks a random relay from our directory mirror cache, and attempts to retrieve the given type of descriptor data. Arguments behave as follows... descriptor_type (str): Type of descriptor to be fetched. This is the same as our @type annotations [6]. This raises a ValueError if the descriptor type isn't available from directory mirrors. fingerprint (str, list): Optional argument for the relay or list of relays to fetch the descriptors for. This retrieves all relays if omitted. retries (int): Maximum number of times we'll attempt to retrieve the descriptors. We fail to another randomly selected directory mirror when unsuccessful. Our last attempt is always via a directory authority. If all attempts are unsuccessful we raise an IOError. ======================================== I'd imagine this would make use of the module something like the following... # Simple script to print all of the exits. from stem.descriptor import remote # Populates our directory mirror cache. This does more harm # here than good since we're only making a single request. # However, if this was a longer living script doing this would # relieve load from the authorities. remote.fetch_directory_mirrors() try: for desc in remote.query('server-descriptor 1.0'): if desc.exit_policy.is_exiting_allowed(): print "%s (%s)" % (desc.nickname, desc.fingerprint) except IOError, exc: print "Unable to query the server descriptors: %s" % exc ======================================== Thoughts? Does this cover all of the use cases we'll this module for? Cheers! -Damian [1] https://trac.torproject.org/8257 [2] https://gitweb.torproject.org/metrics-lib.git/blob/HEAD:/src/org/torproject… [3] https://gitweb.torproject.org/torspec.git/blob/HEAD:/dir-spec.txt#l2626 [4] http://docs.oracle.com/javase/6/docs/api/java/util/zip/InflaterInputStream.… [5] https://gitweb.torproject.org/tor.git/blob/HEAD:/src/or/config.c#l780 [6] https://metrics.torproject.org/formats.html#descriptortypes

3 10

atlas.torproject.org question
by me＠rndm.de 08 Jul '13

08 Jul '13

Hi there, I rebuild atlas.torproject.org using Ember.js instead of Backbone. It looks like this right now: search page: http://i.imgur.com/GEM86mk.png details page: http://i.imgur.com/1UwpwlA.png I'd like to share the code and thought about putting it on github but I don't know if it's ok for you with all the Tor assets (tor logo). Would love to hear what you think about it. thank you

7 25

Torsocks development status
by David Goulet 07 Jul '13

07 Jul '13

Hi everyone, For those who don't know, I've been working on a new version of Torsocks in the last three weeks or so. https://lists.torproject.org/pipermail/tor-dev/2013-June/004959.html I just wanted to give a quick status report on the state of the development. The DNS resolution is working for domain name (PTR) and IPv4 address. Currently, Tor does not support IPv6 resolution but the torsocks code support it. Hidden service onion address resolution is also working using a "dead IP range" acting as cookie that is sent back to the user and mapped to the .onion address on the hijacked connect(). I've changed quite a bit the configuration file (torsocks.conf) to fit the style of tor (torrc). At this point, the tor address and port can be configured as well as the "dead IP range" mention above. More is coming but pretty simple for now. Logging is working, connection registry and thread safety as well. There is also a compat layer for mutexes and once I start porting the project to other *nix system (BSD, OS X, ...) probably more subsystem will be added to that compat layer. So, in a nutshell, some libc calls still need to be implemented, *moar* tests and other OS supports. I'm confident to have a beta version to present to the community in a couple of weeks (if nothing goes wrong). Feel free to browse the code, comment on it, contribute!, etc... https://github.com/dgoulet/torsocks/tree/rewrite Cheers! David

2 2

Re: [tor-dev] Migration of Tor Weather to Stem
by Damian Johnson 05 Jul '13

05 Jul '13

> Hi, > > My name is Sreenatha. I've made an attempt to migrate Tor Weather > from TorCtl to Stem as stated in my gsoc proposal(Module 2). > > Can you please take a look at the top 3 commits that I've made at my > github account? > Please suggest any additional changes that you think are necessary. > > Cheers, > Sreenatha > > P.S : I am lucyd@OFTC on IRC. Hi Sreenatha, glad that you've started in on this! We'd much prefer for development discussions to be on tor-dev@ so looping that in. This looks like a great start. I suspect there is a little confusion though around the socket object that the Controller uses. The Controller doesn't take a socket.socket instance, but rather a stem.socket.ControlSocket... https://stem.torproject.org/api/socket.html This comes up a couple places, most notably... https://github.com/lucyd/weather/commit/ff22e74c247bd61163a974329cd2feb2a12… I doubt that function presently works. Rather, it should be... def listen(): controller = Controller.from_port(port = config.control_port) controller.authenticate(config.authenticator) controller.add_event_listener(newconsensus_listener, EventType.NEWCONSENSUS) I'm not sure what 'config.authenticator' is so that part might be wrong. Also note that this creates a control socket then forgets about it (Weather never cleans it up). This isn't a bug you've introduced but rather what Weather already did. Not the end of the world (interpreter shutdown will clean it up, though possibly with a stacktrace about lingering threads). Have you gotten a Weather instance up and running to test your changes? Cheers! -Damian

2 2

IRC meeting to discuss sponsor F progress on Wed July 3, 16:00 to 17:00 UTC in #tor-dev
by Karsten Loesing 04 Jul '13

04 Jul '13

Hi all, I'd like to schedule an IRC meeting to discuss what progress we made on sponsor F deliverables in June. Suggested time and place are: Wed July 3, 16:00 to 17:00 UTC in #tor-dev That time in other timezones is: 9:00 in San Francisco 12:00 in Boston 18:00 in Berlin 19:00 in Athens 21:30 in New Delhi People who should attend are: Roger, Nick, Andrea, George, David, Sathya, Moritz, Linus, Andrew, Tom, and anyone else who wants to attend. If your name is on that list and the suggested date or time doesn't work for you, please let me know, either here or in private mail. An alternative date would be: Tue July 2, 16:00 to 17:00 UTC in #tor-dev I'm going to send out another reminder by the end of next week that will contain the definite date. Thanks, Karsten

3 5

Metrics Plans
by Damian Johnson 01 Jul '13

01 Jul '13

Hi Kostas. Now that we no longer need to worry about accidentally leaking GSoC selection we can talk more openly about your project. Below is an interchange between me and Karsten - thoughts? ---------- Forwarded message ---------- From: Karsten Loesing <karsten(a)torproject.org> Date: Thu, May 23, 2013 at 11:37 AM Subject: Re: Metrics Plans To: Damian Johnson <atagar(a)torproject.org> Cc: Tor Assistants <tor-assistants(a)lists.torproject.org> On 5/23/13 7:22 PM, Damian Johnson wrote: > Hi Karsten. I just finished reading over Kostas' proposal and while it > looks great, I'm not sure if I fully understand the plan. Few > clarifying questions... > > * What descriptor information will his backend contain? Complete > descriptor attributes (ie, all the attributes from the documents), or > only what we need? His proof of concept importer [1] only contains a > subset but that's, of course, not necessarily where we're going. > > If we're aiming for this to be the 'grand unifying backend' for > Onionoo, Exonerator, Relay Search, etc then it seems like we might as > well aim for it to be complete. But that naturally means more work > with schema updates as descriptors change... This GSoc idea started a year back as a searchable descriptor search application, totally unrelated to Onionoo. It was when I read Kostas' proposal that I started thinking about an integration with Onionoo. That's why the plan is still a bit vague. We should work together with Kostas very soon to clarify the plan. > * The present relay search renders raw router status entries. Does it > actually store the text of the router status entries within the > database? With the new relay search I suppose we'll be retrieving the > attributes rather than raw descriptor text, is that right? The present relay search and ExoneraTor store raw text of router status entries in their databases. But that doesn't mean that the new relay search needs to do that, too. > * Kostas' proposal includes both the backend importing/datastore and > also a Flask frontend for rendering the search results. In terms of > the present tools diagram [2] I suppose that would mean replacing > metrics-web-R and having a python counterpart of metrics-db-R (with > the aim of later deprecating the old metrics-db-R). Is that right? Not quite. We cannot replace metrics-db-R yet, because that's the tool that downloads relay descriptors for all other services. It needs to work really stable. Replacing metrics-db-R would be a different project. The good thing though is that metrics-db-R offers its files via rsync, so that's a very clean interface for services using its data. In terms of the tools diagram, Kostas would write a second tool in the "Process" column above Onionoo that would feed two replacement tools for metrics-web-R and metrics-web-E. His processing tool would use data from metrics-db-R and metrics-db-E. If his tool is supposed to replace more parts of Onionoo and not only replace relay search and ExoneraTor, it would use data from metrics-db-B and metrics-db-P, too. > Maybe we should focus on a 'grand unified backend' rather than > splitting Kostas' summer between both a backend and frontend? If he > could replace the backends of the majority of our metrics services > then that would greatly simplify the metrics ecosystem. I'm mostly interested in the back-end, too. But I think it won't be as much fun for Kostas if he can't also work on something that's visible to users. I don't know what he prefers though. In my imagination, here's how the tools diagram looks like by the end of summer: - Kostas has written an Onionoo-like back-end that allows searches for relays or bridges in our archives since 2007 and provides details for any point in the past. Maybe his tool will implement the existing Onionoo interface, so that Atlas and Compass can switch to using it instead of Onionoo. - We'll still keep using Onionoo for aggregating bandwidth and weights statistics per relay or bridge, but Kostas' tool would give out that data. - Thomas has written Visionion and replacements for metrics-web-N and metrics-web-U. You probably saw the long discussion on this list. This is a totally awesome project on its own, but it's sufficiently separate from Kostas' project (Kostas is only interested in single relays/bridges, whereas Thomas is only interested in aggregates). I'm aware that not all of this may happen in one summer. That's why I'm quite flexible about plans. There are quite a lot of missing puzzle pieces in the overall picture, people can start wherever they want and contribute something useful. > I was very, very tempted to start up a thread on tor-dev@ to discuss > this but couldn't figure out a way of doing so without letting Kostas > know that we're taking him on. If you can think of a graceful way of > including him or tor-dev@ then feel free. Let's wait four more days, if that's okay for you. Starting a new discussion there about this together with Kostas sounds like a fine plan. This will be an exciting summer! :) Best, Karsten > [1] https://github.com/wfn/torsearch/blob/master/tsweb/importer.py#L16 > [2] https://metrics.torproject.org/tools.html >

3 9

Run With Limited Capabilities - GSOC
by Cristian-Matei Toader 01 Jul '13

01 Jul '13

Hello, My name is Cristian Toader, and I feel very excited about designing and implementing a capabilities based sandbox for the central Tor project, as part of the GSOC program. ---- About myself: I have been a Linux enthusiast for almost 6 years and have first started using Tor around 3 years ago. I am currently studying in the UK. In approximately one month I will be graduating the Computer Science programme at the University of Surrey, and plan on pursuing a master's degree in Advanced Computer Science at the University of Cambridge for the following academic year. I have completed a placement year at Qualcomm (UK) LTD which involved implementing and testing security solutions for the Linux Android OS. These were based on cryptography and the TrustZone run-mode of the ARM processors. Most of the development during the placement year was performed in C, with some tests written in Java and C++ for the upper layers. ---- About the project: The project I will be working on as part of GSOC is based on the "Run With Limited Capabilities" proposal [1] mentored by Nick Mathewson (nickm) and Andrea Shepard (athena). The project is still in the planning stage. I will start working on an appropriate design as soon as I finish my last exams, which is the 3rd of June. As part of the project I will need to: - investigate research papers regarding capability based sandboxes - get familiar with the Tor code structure - decide on whether there should be different states starting from which the tor program only has a limited set of capabilities, depending on what syscalls it should be able to perform; or maybe have a more complex approach based on a trusted process representing a root of trust (with no network interactions) which controls the capabilities of the processes it launches - integrate an appropriate solution - develop and run tests for the project A constraint agreed with nickm, would be that once the program capabilities are set they should not be modifiable (otherwise a potential attacker could have the option of first enabling capabilities and then execute privileged code). Some additional details can be found in tickets #7005 [2], #5219 [3], and #5220 [4]. I will try to keep everyone updated. I am looking forward to advice and suggestions. If anyone needs to contact me, this is my primary email, my irc.oftc.net username is ctoader, and I am geographically located in GMT+2. Best regards, Cristian Toader. [1] https://www.torproject.org/getinvolved/volunteer.html.en#limitCapabilities [2] https://trac.torproject.org/projects/tor/ticket/7005 [3] https://trac.torproject.org/projects/tor/ticket/5219 [4] https://trac.torproject.org/projects/tor/ticket/5220

5 6

[GSOC] Status report - Tor capabilities
by Cristian-Matei Toader 01 Jul '13

01 Jul '13

Hello tor-dev, As a small reminder the purpose of the project is to create capabilities based sandboxing for Tor, which may only allow the program to execute a number of predefined syscalls. For the past 2 weeks: - I have consulted with Nick Mathewson (nickm) and agreed upon using seccomp2 [1], and more recently a library built on top of that called libseccomp [2]. - I have set up a public remote branch [3]. - We have agreed on a 3 step plan for the project: 1. General sandbox based on a single (permisive) filter which restricts tor to using a number of syscalls. 2. Add configuration option for step 1, if any parts were broken in phase 1 by adding capabilities, they can be re-enabled at the cost of security. 3. Figure out what functionality should be split into separate processes, based on our experience from step 1 and step 2. - So far I have implemented step 1 using both libseccomp and seccomp2 [3]. Step 1 was developed in such a way that nothing from tor should be broken at the moment; What this means is that sandboxing currently exists in the remote branch, but is fairly coarse and will need some fine tuning at a later stage such as only allowing specific files to be open, or allowing the exec syscall to be called with specific parameters. These days I will be adding command line support, which is basically step 2, which will be followed by a code review and merge in the master branch. [1] http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-precise.git;a=blob;f=Documenta… [2] http://sourceforge.net/projects/libseccomp/ [3] https://github.com/cristiantoader/tor-gsoc-capabilities

2 2