May 2013 - tor-dev - lists.torproject.org

Remote descriptor fetching
by Damian Johnson 23 Jul '13

23 Jul '13

Hi Karsten. I'm starting to look into remote descriptor fetching, a capability of metrics-lib that stem presently lacks [1][2]. The spec says that mirrors provide zlib compressed data [3], and the DirectoryDownloader handles this via a InflaterInputStream [4]. So far, so good. By my read of the man pages this means that gzip or python's zlib module should be able to handle the decompression. However, I must be missing something... % wget http://128.31.0.34:9131/tor/server/all.z % file all.z all.z: data % gzip -d all.z gzip: all.z: not in gzip format % zcat all.z gzip: all.z: not in gzip format % python >>> import zlib >>> with open('all.z') as desc_file: ... print zlib.decompress(desc_file.read()) ... Traceback (most recent call last): File "<stdin>", line 2, in <module> zlib.error: Error -5 while decompressing data: incomplete or truncated stream Maybe a fresh set of eyes will spot something I'm obviously missing. Spotting anything? Speaking of remote descriptor fetching, any thought on the API? I'm thinking of a 'stem/descriptor/remote.py' module with... * get_directory_authorities() List of hardcoded (IP, DirPorts) tuples for tor's authorities. Ideally we'd have an integ test to notify us when our listing falls out of date. However, it looks like the controller interface doesn't surface this. Is there a nice method of determining the present authorities besides polling the authorities array of 'src/or/config.c' [5]? * fetch_directory_mirrors() Polls an authority for the present consensus and filters it down to relays with the V2Dir flag. It then uses this to populate a global directory mirror cached that's used when querying directory data. This can optionally be provided with a Controller instance or cached consensus file to use that instead of polling a authority. * get_directory_cache() Provides a list of our present directory mirrors. This is a list if (IP, DirPort) tuples. If fetch_directory_mirrors() hasn't yet been called this is the directory authorities. * query(descriptor_type, fingerprint = None, retires = 5) Picks a random relay from our directory mirror cache, and attempts to retrieve the given type of descriptor data. Arguments behave as follows... descriptor_type (str): Type of descriptor to be fetched. This is the same as our @type annotations [6]. This raises a ValueError if the descriptor type isn't available from directory mirrors. fingerprint (str, list): Optional argument for the relay or list of relays to fetch the descriptors for. This retrieves all relays if omitted. retries (int): Maximum number of times we'll attempt to retrieve the descriptors. We fail to another randomly selected directory mirror when unsuccessful. Our last attempt is always via a directory authority. If all attempts are unsuccessful we raise an IOError. ======================================== I'd imagine this would make use of the module something like the following... # Simple script to print all of the exits. from stem.descriptor import remote # Populates our directory mirror cache. This does more harm # here than good since we're only making a single request. # However, if this was a longer living script doing this would # relieve load from the authorities. remote.fetch_directory_mirrors() try: for desc in remote.query('server-descriptor 1.0'): if desc.exit_policy.is_exiting_allowed(): print "%s (%s)" % (desc.nickname, desc.fingerprint) except IOError, exc: print "Unable to query the server descriptors: %s" % exc ======================================== Thoughts? Does this cover all of the use cases we'll this module for? Cheers! -Damian [1] https://trac.torproject.org/8257 [2] https://gitweb.torproject.org/metrics-lib.git/blob/HEAD:/src/org/torproject… [3] https://gitweb.torproject.org/torspec.git/blob/HEAD:/dir-spec.txt#l2626 [4] http://docs.oracle.com/javase/6/docs/api/java/util/zip/InflaterInputStream.… [5] https://gitweb.torproject.org/tor.git/blob/HEAD:/src/or/config.c#l780 [6] https://metrics.torproject.org/formats.html#descriptortypes

3 10

Metrics Plans
by Damian Johnson 01 Jul '13

01 Jul '13

Hi Kostas. Now that we no longer need to worry about accidentally leaking GSoC selection we can talk more openly about your project. Below is an interchange between me and Karsten - thoughts? ---------- Forwarded message ---------- From: Karsten Loesing <karsten(a)torproject.org> Date: Thu, May 23, 2013 at 11:37 AM Subject: Re: Metrics Plans To: Damian Johnson <atagar(a)torproject.org> Cc: Tor Assistants <tor-assistants(a)lists.torproject.org> On 5/23/13 7:22 PM, Damian Johnson wrote: > Hi Karsten. I just finished reading over Kostas' proposal and while it > looks great, I'm not sure if I fully understand the plan. Few > clarifying questions... > > * What descriptor information will his backend contain? Complete > descriptor attributes (ie, all the attributes from the documents), or > only what we need? His proof of concept importer [1] only contains a > subset but that's, of course, not necessarily where we're going. > > If we're aiming for this to be the 'grand unifying backend' for > Onionoo, Exonerator, Relay Search, etc then it seems like we might as > well aim for it to be complete. But that naturally means more work > with schema updates as descriptors change... This GSoc idea started a year back as a searchable descriptor search application, totally unrelated to Onionoo. It was when I read Kostas' proposal that I started thinking about an integration with Onionoo. That's why the plan is still a bit vague. We should work together with Kostas very soon to clarify the plan. > * The present relay search renders raw router status entries. Does it > actually store the text of the router status entries within the > database? With the new relay search I suppose we'll be retrieving the > attributes rather than raw descriptor text, is that right? The present relay search and ExoneraTor store raw text of router status entries in their databases. But that doesn't mean that the new relay search needs to do that, too. > * Kostas' proposal includes both the backend importing/datastore and > also a Flask frontend for rendering the search results. In terms of > the present tools diagram [2] I suppose that would mean replacing > metrics-web-R and having a python counterpart of metrics-db-R (with > the aim of later deprecating the old metrics-db-R). Is that right? Not quite. We cannot replace metrics-db-R yet, because that's the tool that downloads relay descriptors for all other services. It needs to work really stable. Replacing metrics-db-R would be a different project. The good thing though is that metrics-db-R offers its files via rsync, so that's a very clean interface for services using its data. In terms of the tools diagram, Kostas would write a second tool in the "Process" column above Onionoo that would feed two replacement tools for metrics-web-R and metrics-web-E. His processing tool would use data from metrics-db-R and metrics-db-E. If his tool is supposed to replace more parts of Onionoo and not only replace relay search and ExoneraTor, it would use data from metrics-db-B and metrics-db-P, too. > Maybe we should focus on a 'grand unified backend' rather than > splitting Kostas' summer between both a backend and frontend? If he > could replace the backends of the majority of our metrics services > then that would greatly simplify the metrics ecosystem. I'm mostly interested in the back-end, too. But I think it won't be as much fun for Kostas if he can't also work on something that's visible to users. I don't know what he prefers though. In my imagination, here's how the tools diagram looks like by the end of summer: - Kostas has written an Onionoo-like back-end that allows searches for relays or bridges in our archives since 2007 and provides details for any point in the past. Maybe his tool will implement the existing Onionoo interface, so that Atlas and Compass can switch to using it instead of Onionoo. - We'll still keep using Onionoo for aggregating bandwidth and weights statistics per relay or bridge, but Kostas' tool would give out that data. - Thomas has written Visionion and replacements for metrics-web-N and metrics-web-U. You probably saw the long discussion on this list. This is a totally awesome project on its own, but it's sufficiently separate from Kostas' project (Kostas is only interested in single relays/bridges, whereas Thomas is only interested in aggregates). I'm aware that not all of this may happen in one summer. That's why I'm quite flexible about plans. There are quite a lot of missing puzzle pieces in the overall picture, people can start wherever they want and contribute something useful. > I was very, very tempted to start up a thread on tor-dev@ to discuss > this but couldn't figure out a way of doing so without letting Kostas > know that we're taking him on. If you can think of a graceful way of > including him or tor-dev@ then feel free. Let's wait four more days, if that's okay for you. Starting a new discussion there about this together with Kostas sounds like a fine plan. This will be an exciting summer! :) Best, Karsten > [1] https://github.com/wfn/torsearch/blob/master/tsweb/importer.py#L16 > [2] https://metrics.torproject.org/tools.html >

3 9

Run With Limited Capabilities - GSOC
by Cristian-Matei Toader 01 Jul '13

01 Jul '13

Hello, My name is Cristian Toader, and I feel very excited about designing and implementing a capabilities based sandbox for the central Tor project, as part of the GSOC program. ---- About myself: I have been a Linux enthusiast for almost 6 years and have first started using Tor around 3 years ago. I am currently studying in the UK. In approximately one month I will be graduating the Computer Science programme at the University of Surrey, and plan on pursuing a master's degree in Advanced Computer Science at the University of Cambridge for the following academic year. I have completed a placement year at Qualcomm (UK) LTD which involved implementing and testing security solutions for the Linux Android OS. These were based on cryptography and the TrustZone run-mode of the ARM processors. Most of the development during the placement year was performed in C, with some tests written in Java and C++ for the upper layers. ---- About the project: The project I will be working on as part of GSOC is based on the "Run With Limited Capabilities" proposal [1] mentored by Nick Mathewson (nickm) and Andrea Shepard (athena). The project is still in the planning stage. I will start working on an appropriate design as soon as I finish my last exams, which is the 3rd of June. As part of the project I will need to: - investigate research papers regarding capability based sandboxes - get familiar with the Tor code structure - decide on whether there should be different states starting from which the tor program only has a limited set of capabilities, depending on what syscalls it should be able to perform; or maybe have a more complex approach based on a trusted process representing a root of trust (with no network interactions) which controls the capabilities of the processes it launches - integrate an appropriate solution - develop and run tests for the project A constraint agreed with nickm, would be that once the program capabilities are set they should not be modifiable (otherwise a potential attacker could have the option of first enabling capabilities and then execute privileged code). Some additional details can be found in tickets #7005 [2], #5219 [3], and #5220 [4]. I will try to keep everyone updated. I am looking forward to advice and suggestions. If anyone needs to contact me, this is my primary email, my irc.oftc.net username is ctoader, and I am geographically located in GMT+2. Best regards, Cristian Toader. [1] https://www.torproject.org/getinvolved/volunteer.html.en#limitCapabilities [2] https://trac.torproject.org/projects/tor/ticket/7005 [3] https://trac.torproject.org/projects/tor/ticket/5219 [4] https://trac.torproject.org/projects/tor/ticket/5220

5 6

New flash proxy facilitator domain fp-facilitator.org
by David Fifield 30 Jun '13

30 Jun '13

I moved the flash proxy facilitator to a new domain, fp-facilitator.org. This is to get it away from bamsoftware.com, which also has a lot of unrelated stuff. The old facilitator name tor-facilitator.bamsoftware.com will continue to work (the DNS for both points to the same place). https://trac.torproject.org/projects/tor/ticket/7160 is the ticket. David Fifield

2 3

Vidalia 2.0 - an complete rewrite
by Leo Unglaub 28 Jun '13

28 Jun '13

Hey friends, i want to inform you that i am working on a complete new control tool for Tor. I know some people use Vidalia for it, but i never liked the program and the people i showed vidalia often got confused. So i started to rewrite it from scratch and i am close to the release of the source code. (Currently i need to clean up some classes in there) I had a few things on my feature list for the new control tool and i want to share/discuss it with you. Because maybe some of my ideas are bad or maybe you have some additional ideas for features. A: "native work flow". Thats one of the main goals of my rewrite. The current Vidalia does things a little bit different than people are used to do it. So they get confused, because it does not behave "naturally". So instead of QT i used GTK and followed as always the GNOME HIG. (http://developer.gnome.org/hig-book/3.5/) B: "Multi configuration setup". What do i mean by that? Well, in my case for example i use Tor on my laptop and i am often in different places where i need different settings for tor. For example, if i am at home i can give tor all the bandwith i have, because at home i don't care. In my company on the other hand i cant do that. I am allowed to run Tor in the company, but with a bandwith limit of 2000kbit/s. (Yes, my company allows this ... well, i am the technical directory, so it was my call :) ) But i need a different Tor configuration there. Or if i am giving a talk and only have mobile internet connections i need every bit of bandwith for checking my emails so i want no server, only Tor client support. Currently i have multiple torrc files for that and always move them arround. That works for command line junkies like me, but even i am anoyed to do this multiple times a day. So i added the "multi configuration" support into the tool. This allows you to define multiple configurations and simply change them with a click from the gnome2 notification panel. C: "GUI for config settings". Well, some users are simply not ready to modify config files on there own. So i provide a simple gui (tab based) for all config options i found in the normal torrc file. Always with an explonation (with needs spellchecking, because my spelling sucks in german and is even worse in english :) ) so the user can deside what an option would to. D: "Notifications": My control tool uses the normal desktop notifications to inform the user about changes in the running tor application. For example if there is a problem building the circuit or something you get the notifyd message and can simply react to events without having the arm log open all the time. Thats just a quick overview of what i am duing right now. The source code will be released soon and is of course 100% open source. If you have some addiional ideas, feedback, wishes, ... i would like to hear them. Thanks and greetings Leo PS: I am Germam and very sorry for every spelling error i did above. -- Leo Unglaub Nachreihengasse 39 / Top 7 A-1170 Vienna Austria Phone: 0650 / 6575103 Website: http://www.leo-unglaub.net Twitter: http://twitter.com/LeoUnglaub Send with Mozilla Thunderbird on Linux (Debian)

9 24

[Stegotorus] Fundamental problem with ack/retransmission mechanism
by vmonmoonshine＠gmail.com 24 Jun '13

24 Jun '13

Hello Zack, In testing the retransmission algorithm, I encounter situations that the ack/retransmission algorithm is unable to resolve by design. It happens when the header of a packet gets deliver but the middle part gets dropped. For example in the following scenario Server sends packet 1 len 4k packet 1 gets lost Server sends packet 2 len 64k the last 60kb of packet 2 gets lost client receives packet 2 partially (4k) and waits for the 60k to come. client sends ack to the server saying that packet one is lost server retransmits pack 1 client consider packet 1 as a part of packet 2, and thinks that now it has 8k of packet 2 client again sends ack to the server saying that packet one is lost server retransmits pack 1 client thinks now it has 12K of packet 2 client again sends ack to the server saying that packet one is lost server retransmits pack 1 client thinks now it has 16k of packet 2 etc Soon the transmit queue on the client side will get filled up by the ack packets, cause the client won't realize its mistake before it receives 64k of garbage and be able to compute the mac. Hence the server and the clients will remain in this deadlock for ever. Originally, Stegotorus was dropping any circuit with full transmit queue. But this was causing lots of problem because lots of time the client lags behind the server for a while to process all the packet and sends ack. If ack/retransmit by design wasn't meant to recover from partial packet lost and was only a defense for complete packet lost, then the dropper proxy isn't doing a good job simulating the intended threat model. Otherwise, if the ack/retransmit was meant to withstand partial packet drop, then we need to redesign it so it doesn't get trapped in such a deadlock. Maybe we should have a timer so the circuits with full transmit queue gets a grace period before getting dropped. This is not specific to chop protocol. I'm wondering how TCP is dealing with such a situation. Thanks, Vmon

4 6

Discussion on the crypto migration plan of the identity keys of Hidden Services
by George Kadianakis 19 Jun '13

19 Jun '13

Greetings, I'm supposed to write a Tor proposal for the migration of the long-term identity keys of Hidden Services. When I began writing the proposal, I realized that some of my choices might not be appreciated by Hidden Service operators, and that starting a discussion thread might be a good idea before writing the proposal. The problem with the current long-term HS identity keys is that they are RSA-1024 keys which are considered weak, and they need to be upgraded to a cryptosystem with higher security properties. One of the main issues with this operation, is whether Hidden Services will be accessible using their *old* identity keys even after the migration. That is, when we change the identity keys of a Hidden Service, its onion also changes (since the onion is the truncated hash of its public key). This will be quite problematic for Hidden Services that have a well-established onion address. There are basically two ways to do this: a) After the transition, Hidden Services can be visited _only_ on their _new_ onion addresses. This is quite brutal, but it's the most secure and unambiguous option (might also be easier to implement and deploy). This change can be enforced both on the client-side, by rejecting any old RSA-1024 HS keys, and on the server-side, by only publishing the new keys in HS descriptors. To make the transition easier, we could prepare a tool that generates a new identity keypair before the flag day, so that Hidden Service operators can learn their future onion address beforehand and announce it to their users. b) After the transition, Hidden Services can use both old and new onion addresses. This might result in a more harmonious transition, where Hidden Services advertise their new onion address to users that visit them in their old address. .oO(It would also be interesting to do a redirection on the Tor protocol layer ("I got this descriptor by querying for the old onion address, but it also contains a new onion address. I should probably use the new one."), but I don't think it's possible to redirect the user without knowledge of the application-layer protocol (e.g. 302 for HTTP). Still, a Tor log message might be helpful.) The cons of this approach is that supporting both addresses might make the HS protocol more complicated and painful to implement, and it might also result in some Hidden Services never moving to the new onion addresses since clients can still visit them using the old insecure ones. This approach has a stricter variant, where the old addresses can only be used during a transitionary period (a few months?). After that, clients _have_ to use the new addresses. Of course, this means that we will have to do two flag days, coordinate Tor releases, and other no fun stuff. I'm probably moving towards the latter option because the former will make many people unhappy. Thoughts? (This is not a thread to select the cryptosystem we are going to use. It will derail the discussion, and we might also need to select a specific type of cryptosystem in the end (e.g. a discrete-log-based system) so that schemes like https://trac.torproject.org/projects/tor/ticket/8106 can be possible.).

16 29

Building better pluggable transports - GSoC 2013 project
by Chang Lan 11 Jun '13

11 Jun '13

Hello everyone! I am a Tor GSoC student who will be working on the pluggable transports this summer. My mentor is Steven and my co-mentor is George Kadianakis. It is great to be part of the Tor community! Steven already kicked off the discussion about how to build better transports. The original project proposal[1] discussed the possibility of sending data over UDP with extra efforts to guarantee reliable in-order delivery. However, as George mentioned recently[3], ScrambleSuite[2] may already solve the issue of scanning resistance. Given that ScrambleSuite is being deployed, improving protocol obfuscation will be my main focus. HTTP impersonation is really useful, since there are numerous HTTP proxy outside the censored region, while the number of bridges is quite limited. What I'm gonna be doing during the summer is implementing a good enough HTTP impersonation based on pluggable transports specification. There are still many open questions indeed. Discussions are more than welcome! I'll be keeping my GSoC log here: http://changlan.github.io/obfsproxy I'm also available #tor-dev@oftc with the handle "clan", feel free to ping me there! [1]: http://changlan.info/gsoc.html [2]: http://www.cs.kau.se/philwint/scramblesuit/ [3]: https://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/changla… Cheers, Chang

5 6

Gitian-based Deterministic Build System for TBB (Need MacOS Help!)
by Mike Perry 10 Jun '13

10 Jun '13

Over the past couple weeks I've been redoing the TBB build system to use Gitian to produce alpha TBBs using Tor Launcher instead of Vidalia. I have succeeded in producing deterministic, localized builds of TBB for Linux and Windows. This means that independent people all over the world can now easily produce their own bundles for these platforms fresh from sources, and have their bundles exactly match the bundles the Tor Project releases, down to the SHA256 hash. If we leverage this property wisely, it will allow us to defend against targeted attacks against our bundlers and their build machines, and even ultimately ensure the integrity of our bundles in the event of key compromise of the gpg keys used to sign the bundles. My plan for this is for there to be between 2-3 official signers for each bundle, where each person produces their build independently, and signs the (identical) result files. To further protect against targeted attack, in addition to these 2-3 official signers, we need some people to be "secret verifiers". Ideally these people would not be publicly affiliated with the Tor Project, but would still produce their own bundles anyway. If their SHA256 ever fails to match the signed bundles, that person should anonymously open a trac ticket (using the cypherpunks account) and attach the bundle files that differ for analysis. The differing files can be found easily enough with 'diff -r'. To ensure the existence of these "secret verifiers", I believe that the official signers should occasionally conspire to conduct "Fire Drills", where they all agree to alter the bundle in some innocuous way (such as adding whitespace to a config file or a Firefox JS file), and ensure that a verifiers anonymously report the verification failure. In future versions of Tor, we should probably add a consensus field consisting of a url to a file that lists the current recommended bundle hashes and versions, along with the current SHA256 of that file, to anchor the bundle authentication the Tor's current trust root (the 9 dirauth keys). To try out the new build system, please see the README, and let me know where the system could use clarification or improvement to make it easier to use: https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gi… The build system has some quirks that are worth mentioning: 1. It requires you run it from either an Ubuntu 12.04 or above host with KVM support, *or* you run it from an Ubuntu 12.04 or above chroot/VM. The bundle scripts try to detect your current situation and suggest that you "export USE_LXC=1" from your shell if you need to, to cause the system to use LXC instead of KVM (so that you can build from an Ubuntu VM or on a machine that does not otherwise support KVM). 2. We currently have no MacOS support. To support MacOS, we need to create cross-compilers for it so that we can produce builds from the Gitian VMs (which again are Ubuntu). A few people have done this. I have sent them mail asking for instructions on how to reproduce their compiler packages: http://www.tarnyko.net/en/?q=node/9 https://launchpad.net/~flosoft/+archive/cross-apple/+packages http://wiki.freepascal.org/Cross_compiling_OSX_on_Linux Unfortunately, at least one of those URLs say that to produce a cross-compiler, you need access to an OSX SDK. Since I do not have a Mac that is currently supported by recent OSX SDKs, and since we *really* want to be sure that the cross-compilers we produce use code from a fresh known-good SDK install, I won't be doing this. Please let me know if you'd like to help tackle this problem. In the meantime, I am going to work on the rest of the "Short Term" TODO items, and produce official alpha bundles for Linux and Windows, so we can test Tor Launcher in an official alpha release. Here's the TODO file: https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gi… Happy building! -- Mike Perry

3 6

Censorship Simulator Virtual Machine GSoC
by Arturo Filastò 08 Jun '13

08 Jun '13

Hi Johannes, I wanted to introduce you and your GSoC project to the list. The project is about building a virtual machine based censorship simulator. The goal is to have a way to test if the ooniprobe tests are working properly by simulating the network of a censor. Our goal is to actually run the software that a censor would be running (e.x. squid, bluecoat proxy sg va, etc.) and detect it's presence in an as real as possible environment. Johannes and Klaudiu analyzed a bit of the possible technological solutions to making this here: https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipSimulator. I think that the outcome is that we will be using Vagrant, meaning that this simulator could then be run on OSX, Linux and Windows. The kind of networks we will be creating are those where: * There is a transparent HTTP proxy * There is NAT * Certain sites are blocked thanks to HTTP headers * RST packets are sent when connecting to certain hosts * DNS based censorship is present If you have some ideas of things you would like to see in this, add ideas to this ticket: https://trac.torproject.org/projects/tor/ticket/7233, or create a page in the trac wiki nested under https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipSimulator. ~ Art.

2 1