- tor-dev - lists.torproject.org

[GSoC '13] Status report - Searchable metrics archive
by Kostas Jakeliunas 29 Jul '13

29 Jul '13

Hi everyone, Some clean-ish working code is finally available online [1] (the old PoC code has been moved to [2]); I'll be adding more soon, but this part does what it's supposed to do, i.e.: - archival data import (download, mapping to ORM via Stem, efficiently avoiding re-import of existing data via Stem's persistence path, etc.); what's left for this part is a nice and simple rsync+cron setup to be able to continuously download and import new data (via Metrics archive's 'recent') - data models and Stem <-> ORM <-> database mapping for descriptors, consensuses and network statuses contained in consensuses - models can be easily queried by sqlalchemy's ORM; Karsten suggested that an additional 'query layer' / internal API is not needed until there's actual need for it (i.e., my plan was to provide an additional query API abstracted from ORM (which is itself built on top of database/SQL/python classes), and to build a backend on top of it, as a neat client of that API as it were; I had some simple and ugly PoC's that are now pushed out of priority queue until needed (if ever)) - one example of how this querying (directly atop the ORM) works is provided: a simple (very partial) Onionoo protocol implementation for /summary and /details, including ?search, ?limit and ?offset. Querying takes place over all NetworkStatuses. This is new in the sense that it uses the ORM directly. If there is a need to formulate SQL queries more directly, we'll do that as well. During the Tor developer meetings in Munich, we tried talking over the existing & proposed parts of the system with Karsten. I will be focusing on making sure the Onionoo-like backend (which is being extended) is stable and efficient. I'm still looking into database optimization (with Karsten's advice); an efficient backend for the majority of all archival data available would be a great deliverable in itself, and hopefully we can achieve at least that. I might do well to try and document the database iterations and development, as a lot of thinking now resides in a kind of 'black box' of DB spec, which does not produce code. The large Postgres datasets are residing on a server I'm managing; I'm working on exposing the Onionoo-like API for public queries; doing some simple higher-level benchmarking (simulating multiple clients requesting different data at once, etc.) now. I might need to move the datasets to yet another server (again), but maybe not; it's easy to blame things on limited CPU/memory resources. :) Kostas. [1]: https://github.com/wfn/torsearch [2]: https://github.com/wfn/torsearch-poc

1 0

IRC meeting to discuss sponsor F progress on Wed July 31, 18:00 to 19:00 UTC in #tor-dev
by Karsten Loesing 28 Jul '13

28 Jul '13

Hi all, I'd like to schedule an IRC meeting to discuss what progress we made on sponsor F deliverables in July. Suggested time and place are: Wed July 31, 18:00 to 19:00 UTC in #tor-dev That's in 8 days from today. The time in other timezones is: 11:00 in San Francisco 14:00 in Boston 20:00 in Berlin 21:00 in Athens 23:30 in New Delhi People who should attend are: Roger, Nick, Andrea, George, David, Ximin, Sathya, Nathan, Matt, Colin, Linus, Andrew, Tom, and anyone else who wants to attend. If your name is on that list and the suggested date or time doesn't work for you, please let me know, either here or in private mail. An alternative date would be: Fri August 2, 18:00 to 19:00 UTC in #tor-dev I'm going to send out another reminder in two days from now that will contain the definite date. Thanks, Karsten

2 3

Status report - HTTP pluggable transport
by Chang Lan 28 Jul '13

28 Jul '13

Hi Steven, I deeply apologize for not replying to your emails in the last few weeks. On July 2, I had a bad car accident, and since then I was staying in hospital after a surgery till July 27. I should have let you know earlier to get things less messed up. Sorry again about this. Now I finally get back and will be online for the rest of period. Anyway, I managed to keep up the schedule. Echoing George's request, I wrote a design specification [https://github.com/changlan/http-transport/blob/master/design.md] based on Steven's document. In brief, I apply the BOSH technique to transfer bidirectional-streams over HTTP, which is the skeleton of the design. To disguise Tor's content patterns, we should make the traffic indistinguishable from random bits. Since there is an assumption that user and bridge share a cryptographically secure secret, I simply encrypt the data under AES-128 counter mode. To achieve scanning resistance, the user should send a secure message MAC(IP | Port | Timestamp, K) to the bridge. If the message gets verified on the bridge's side, the user gets accepted. Otherwise, the bridge will refuse to respond the user (or behave like a normal HTTP server). (I am not a crypto guy, so the scheme above may have flaws. Any review?) And finally the traffic should be able to pass through proxies. I guess proxies will not cache POST response; am I correct? That's it. There are some optional components that I am not sure whether to implement or not, such as packet-size obfuscation, Format-Transforming Encryption, etc. I think packet-size obfuscation is necessary, but I would rather put it in a TODO list for now. One thing that is still arguable is that whether the traffic pattern of BOSH is safe enough. In other words, how to evaluate whether the traffic pattern revealed in this HTTP transport is common in other applications (XMPP for example). I have implemented the transport within the pyobfsproxy framework as well as unit tests.[https://github.com/changlan/obfsproxy] Currently scanning resistance and content encryption are not enabled [https://github.com/changlan/obfsproxy/commit/5b5e8b543919c5b3cda756b3a9e7f4…] but the BOSH technique is applied. Tests and code reviews are welcome! Thanks. Best wishes, Chang On Tuesday, July 23, 2013 at 5:27 PM, Steven Murdoch wrote: > Hi Chang, > > On July 29, I will be submitting the Google Summer of Code progress report. Unless I hear from you before this date, I'm afraid I cannot recommend the project continue and Google rules would not allow you to restart at a later stage. I'd rather this not happen, so please do get back in touch. > > Best wishes, > Steven > > >

1 0

Idea regarding active probing and follow-up of SSL connections to TOR bridges
by Lag Inimaineb 27 Jul '13

27 Jul '13

Hi, I've just watched https://www.youtube.com/watch?v=GwMr8Xl7JMQ (old, I know), and found it very interesting. Also, before going any further, I'd like to stress that I'm new to tor-dev and while I've googled my question, I haven't been able to come up with anything, so apologies in advance if this is already solved. Anyway, one of the main topics discussed in that talk was the problem of preventing the blockage of TOR bridges by oppressors. While many "fixes" were mentioned, none of them actually solve the problem of the bridge being probed, by following-up on previously captured SSL sessions (as China does). I was thinking - perhaps instead of making the discovery of the actual bridge addresses "hard", why not setup bridges in places where other *legitimate* SSL connections are also made? What I mean is, maybe we should try and get big sites that cannot be legitimately (in the eyes of the oppressor, of course) blocked (social media, comic sites, whatever), to run a TOR bridge *on the same port* as their regular HTTPS traffic (443), in a way that someone recording the traffic cannot distinguish (in advance, that is) whether a certain SSL connection to that site is a legitimate web browsing session, or a TOR session. That way, even if an address on the internet "speaks" the TOR protocol, it cannot be automatically blocked. Even if this address is known to host a TOR bridge, this might help plausible deniability for people unwilling to disclose that they've been using TOR. I understand this is probably very difficult, for several reasons, but I will try to address some of these. First off - there's the technical issue of binding to port 443 locally on the web servers without disrupting the currently running local client. I see several possible ways around this - a simple one could be local proxying of the SSL connection from the TOR bridge software to the locally running web server, in a way that when the bridge gets an SSL connection that isn't "speaking" the TOR protocol, it will be handed over to the web server. I admit, it's kinda messy, but TBH there are probably more efficient and "cleaner" ways to do this than what I've suggested. Also, long running connections with heavy data usage to a web server address are probably suspicious, but even if these were to be blocked, it would be on a per-client basis, and would also just mean a QoS issue for users but not DoS. Lastly, there's the question of why would a large site agree to this. I still don't have a good answer for this, but maybe you do... (PR, perhaps?) Anyway, I'd like to hear what you think. Maybe this isn't an issue anymore, or maybe you've found a different solution, or mine doesn't work, or you've already implemented it, etc. :) laginimaineb.

3 5

[GSOC 2013] Steganography Browser Extension Status Report
by Hareesan 27 Jul '13

27 Jul '13

Hi, My work plan for last week was to update the locale information and to finish file uploading parts of extension. For now testing purpose, when a file is being selected the byte format will be popped. With this, the locale information has being updated. With this, I have started working on adding public key encryption algorithms to the add on. This task has being planned to be finished before mid term evaluation and for next two weeks, my plan is to start working on image steganography libraries. -- Hareesan

1 0

[GSOC] Status report - Tor capabilities
by Cristian-Matei Toader 26 Jul '13

26 Jul '13

Hello tor-dev, For the last week I was on holiday for the graduation ceremony. I did however manage to get some work done, so for the past 2 weeks I have: - worked on adding parameter filters for the syscall filter; this is done using both a static list of parameters, as well as a dynamic list configurable at runtime; - parameter filters currently support numeric and pointer based parameters such as C strings; pointer based parameters are referenced in the original tor code through a getter (this was implemented for the 'open' syscall). - possibly identified a bug in libseccomp which was causing the accept4 syscall to fail to be added to the filter which was temporarily fixed by accepting all socketcall filters; still need to confirm this with nickm, but for those interested i believe -117 on this [1] line should be at least -120; my local fix makes everything work fine without socketcall. As a general conclusions things are going fine, I am currently trying to figure out what should go in the parameter filter. Another quick link to my remote branch for the ease of those interested: [2]. Looking forward to some feedback, if you happen to have any! References: [1] http://sourceforge.net/p/libseccomp/libseccomp/ci/release-2.1/tree/src/arch… [2] https://github.com/cristiantoader/tor-gsoc-capabilities/tree/gsoc-cap-stage2

1 0

[GSOC2013] EvilGenius status report
by Johannes Fürmann 26 Jul '13

26 Jul '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi *, In addition to attending the tor meeting in Garching (which was really awsesome, by the way) I really got some work on EvilGenius in the past two weeks. Since Arturo and I outlined the rough structure of the EvilGenius command line interface, I was gradually filling in the pieces needed for talking to the vagrant. Starting this tuesday, we built a set of unittests to test the functionality that is already in place. Since yesterday evening, after a bit of hacking, I already implemented the first and most difficult part of network plumbing and got the binary to create valid Vagrantfiles that are actually booting up the network they are supposed to. So, as Arturo said, "the Genius is there, all we have to do now is make it more evil.". We also might have to work on the plumbing a bit and fit some loose pieces together, which is what I shall be doing next week. By the way, it was really nice to meet all of you in Garching. Hope to see you all soon! Have a safe trip home and enjoy your weekend, everyone! Johannes -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJR8m5qAAoJEAgie5o3Q/JpBzcQALiUslixPD5kIL0r+eNvXHU1 BDDDz9JUZhNc/4ZLa1AFXtY2GrJmB4wP6IOGjuj5rAPx0YcHUszVzjaTti5hWflr rlsXbbOgj4VWDsTAcO7AUbhrEVFaH/SlKgQesZKQuLTduDEhzXBSKga6VlCV37AF R//l8iXRBapwq5DzHSW4eYvNXorKF/nKN/ax7XYjZ5+X/P9sXZz1yHqk5/t187Lt mGbnrfcvEC5xkFpLgz/ReWZ7wABUMJRMrgTapI14rau94rGnaWSsVVjbNKRyYlr3 lhvgsEc6bmIRMhaBDn+Ujgt5o75Th3vJQ2T81wXIkWgrTEsZRZTAJGtyjO0nohtI G2UQmy+/U2Xam2tDyMh7xaGhNr4UN1NJpQ17TyCNPd4DzyO5XG6QjHhjb2s4U9ID YhflAo7yEj4P6NsRR+nIh5djveAgUF55sWNpp+fq1SK8VQ2UXMw6Pm76W7uwGIsw uf7icgTbSC7msSBXJf5HgyP7h3ZvlTrblP90/ciVuLS3YPUmwKYqfCJdnrP4PRYI 8p3bV9ZPU59a0Eqc8fhcQazXvaecUjzGfTpW2owkhQYtk2m7jDJ/Rzz6HbYpiToy 7P6N8T4dq9pp4Hb2ANARUPKYmcVYovYJtJZ9GD0zo5hWbi/i7sqPkJCO07RCyqg/ 5ePzmPqQorZP+nawhVbG =THS9 -----END PGP SIGNATURE-----

1 0

Comparing handshake protocols for Tor: Ace vs ntor
by Esfandiar Mohammadi 24 Jul '13

24 Jul '13

Hi, in WPES 2012 [2], Aniket Kate and me introduced Ace, an alternative for Tor's current handshake protocol ntor (which was also briefly discussed on this list [4]). Back then, no implementation for the double scalar multiplication operation (a * b + c * d) on Curve25519 was available. But recently, Robert Ransom implemented in his celator library [1] a highly efficient double scalar multiplication which is suitable for our handshake protocol Ace. In an internship with us, Shivanker Goel implemented and run comparative benchmarks of the Ace protocol, using Robert's celator library [1], and the current ntor implementation. Here are the benchmarks: 64-bit architecture: Ace: 401,129 cycles, ntor: 472,772 cycles, 32-bit architecture: Ace: 2,797,303 cycles, ntor: 5,179,069 cycles, About the benchmarks: * For cleaning the cache, the benchmarks runs the handshake 1000 times prior to starting the benchmarks. Then, we used a tsc_read() function to count the cycles, repeated the handshake 5000 times and computed the average of the cycles. * In order not to be forced to import too many functions from the Tor library, we do not use the function dimap_search, which looks up the secret key for a given public key, in the benchmarks. * We assume that ephemeral keys are precomputed, e.g., in idle cycles. In the benchmarks these ephemeral keys are received as an input. * For the benchmarks hyper-threading and overclocking were disabled and gcc-4.8 with "-O2" optimizations was used. The code is available online [2]. We also tried to formulate a tor-style proposal, which is also available online [3]. Any feedback, comments, or suggestions are welcome. - Esfandiar [1] http://www.infsec.cs.uni-saarland.de/~mohammadi/ace/celator.tar.gz [2] http://www.infsec.cs.uni-saarland.de/~mohammadi/ace/ace-benchmarks.tar.gz [3] http://www.infsec.cs.uni-saarland.de/~mohammadi/ace/ace-handshake.txt [4] It was discussed in the thread "Another key exchange algorithm for extending circuits: alternative to ntor?" from August 2012.

1 0

Remote descriptor fetching
by Damian Johnson 23 Jul '13

23 Jul '13

Hi Karsten. I'm starting to look into remote descriptor fetching, a capability of metrics-lib that stem presently lacks [1][2]. The spec says that mirrors provide zlib compressed data [3], and the DirectoryDownloader handles this via a InflaterInputStream [4]. So far, so good. By my read of the man pages this means that gzip or python's zlib module should be able to handle the decompression. However, I must be missing something... % wget http://128.31.0.34:9131/tor/server/all.z % file all.z all.z: data % gzip -d all.z gzip: all.z: not in gzip format % zcat all.z gzip: all.z: not in gzip format % python >>> import zlib >>> with open('all.z') as desc_file: ... print zlib.decompress(desc_file.read()) ... Traceback (most recent call last): File "<stdin>", line 2, in <module> zlib.error: Error -5 while decompressing data: incomplete or truncated stream Maybe a fresh set of eyes will spot something I'm obviously missing. Spotting anything? Speaking of remote descriptor fetching, any thought on the API? I'm thinking of a 'stem/descriptor/remote.py' module with... * get_directory_authorities() List of hardcoded (IP, DirPorts) tuples for tor's authorities. Ideally we'd have an integ test to notify us when our listing falls out of date. However, it looks like the controller interface doesn't surface this. Is there a nice method of determining the present authorities besides polling the authorities array of 'src/or/config.c' [5]? * fetch_directory_mirrors() Polls an authority for the present consensus and filters it down to relays with the V2Dir flag. It then uses this to populate a global directory mirror cached that's used when querying directory data. This can optionally be provided with a Controller instance or cached consensus file to use that instead of polling a authority. * get_directory_cache() Provides a list of our present directory mirrors. This is a list if (IP, DirPort) tuples. If fetch_directory_mirrors() hasn't yet been called this is the directory authorities. * query(descriptor_type, fingerprint = None, retires = 5) Picks a random relay from our directory mirror cache, and attempts to retrieve the given type of descriptor data. Arguments behave as follows... descriptor_type (str): Type of descriptor to be fetched. This is the same as our @type annotations [6]. This raises a ValueError if the descriptor type isn't available from directory mirrors. fingerprint (str, list): Optional argument for the relay or list of relays to fetch the descriptors for. This retrieves all relays if omitted. retries (int): Maximum number of times we'll attempt to retrieve the descriptors. We fail to another randomly selected directory mirror when unsuccessful. Our last attempt is always via a directory authority. If all attempts are unsuccessful we raise an IOError. ======================================== I'd imagine this would make use of the module something like the following... # Simple script to print all of the exits. from stem.descriptor import remote # Populates our directory mirror cache. This does more harm # here than good since we're only making a single request. # However, if this was a longer living script doing this would # relieve load from the authorities. remote.fetch_directory_mirrors() try: for desc in remote.query('server-descriptor 1.0'): if desc.exit_policy.is_exiting_allowed(): print "%s (%s)" % (desc.nickname, desc.fingerprint) except IOError, exc: print "Unable to query the server descriptors: %s" % exc ======================================== Thoughts? Does this cover all of the use cases we'll this module for? Cheers! -Damian [1] https://trac.torproject.org/8257 [2] https://gitweb.torproject.org/metrics-lib.git/blob/HEAD:/src/org/torproject… [3] https://gitweb.torproject.org/torspec.git/blob/HEAD:/dir-spec.txt#l2626 [4] http://docs.oracle.com/javase/6/docs/api/java/util/zip/InflaterInputStream.… [5] https://gitweb.torproject.org/tor.git/blob/HEAD:/src/or/config.c#l780 [6] https://metrics.torproject.org/formats.html#descriptortypes

3 10

[GSoC '13] Tor status report - Searchable metrics archive
by Kostas Jakeliunas 22 Jul '13

22 Jul '13

Hey all, I apologize for this unusual timing for a status report, but I ended up delaying it beyond measure, so better now than later I guess. I can reiterate it + any updates soon, it's just that I figure I'm long overdue on informing tor-dev on what's going on. I've started my project [1] later than is usual, and more or less immediately ran into what I deemed to be a database / ORM scaling issue (the thing I'd been actually trying to avoid since writing the proposal), or at least a behaviour of the ORM which was suboptimal to what we have in mind: delivering (first and foremost) a searchable metrics archive backend/database which incorporates, as of current plan, server descriptors (relays and bridges, turns out a server descriptor model can happily service both) and server/router statuses across a few year timespan (currently using v3 consensus documents only), and provides querying functionality which can extract relations between the two. The 'querying with relations between the two' part, when tested on a broader span of data, seemed to be causing trouble to me. I ended up allocating probably inefficiently large amounts of time to this problem, rewriting the backend part, and trying to optimize the queries which underlied the ORM (turns out I didn't need to strip off the ORM abstraction - learned a few things about SQLAlchemy that way - I will follow-up with an email pointing to current code (sorry)). * The current iteration of the ORM model / backend (which actually is very simple) solves this problem. * Stem descriptor and network status mapping to ORM works, and is nicely (enough) integrated with the data import (from downloaded metrics archive) tools, as well as an API to make queries on the ORM. * Implemented a partial Onionoo-protocol-adhering (without compression and without some fields) backend for ?summary and ?details Onionoo queries. * Still tidying everything up. And *finally* writing a design document outlining what we actually ended up with, and what is required till full Onionoo integration. Code review will happen pretty soon, and hopefully we'll have some discussion upon where to go from here. Karsten mentioned that it might be possible to use the existing Onionoo incarnation to continue providing bandwidth weight etc. data (basically stuff from extra-info), and it might be possible to join the two systems into an Onionoo-supporting backend which will cover all / majority of archives available. Another (or) further avenue would be to continue with the initial proposed plan to extend the query format; and to build a frontend which would make use of the extended query format. Expect another email with links to (decent) code. [1]: http://kostas.mkj.lt/gsoc2013/gsoc2013.html

1 0