May 2014 - tor-dev - lists.torproject.org

Question to Java developers: what are good frameworks for Tor's server-side Java applications
by Karsten Loesing 25 May '14

25 May '14

Hello devs, I'm seeking advice from people with experience in writing server-side Java applications. Let me give you some background about this request: for the past five years, I have been developing server-side Java applications which all process large amounts of Tor directory data and provide their output via a web interface. Examples: - The metrics data processor (metrics-db) fetches Tor descriptors from the Tor directory authorities, the bridge authority, etc., performs some sanity-checks, and provides descriptors by type as tarballs. We're talking about roughly 7 GiB new bzip2-compressed data per month. - The metrics website (metrics-web) uses the output from the metrics data processor, stuffs everything into a database, computes aggregates, and presents results in graphs and .csv files. - The Onionoo service processes the same data from the metrics data processor, but provides statistics per Tor relay, not for the Tor network as a whole. The processing is done every two hours and may take 30 minutes to 1.5 hours, depending on how overloaded the server is. - The ExoneraTor service, again, uses the same data and puts it in a database to answer whether a certain IP address has been a Tor relay at some point in the past. That's what is done. And here's how it's done under the surface: - There's one or more cronjobs, each of which starts an ant task to process data. Some of these tasks import data into the database, others store results in the file system. - Each application uses a web application deployed in Tomcat to provide results to web users. Most things are written in servlets, some use JSPs. My problem is that this approach is rather fragile and difficult to setup for new volunteers. I'm aware of that, and I'd like to improve it. My question is: what Java frameworks should I be looking at for the applications described above? Bonus points if something is in Debian stable. Note that "switch to $some_other_programming_language" is not a very useful answer to me, at least not for the larger applications. There's just too much existing code and not enough developer time to port it. Thanks in advance! All the best, Karsten

3 3

RFC: obfs4 (Name not final)
by Yawning Angel 24 May '14

24 May '14

Hello, The people that have been following Pluggable Transport development may know that I have been working on something tentatively called "obfs4" recently. It's rapidly approaching the point where I would like to open it up for review and feedback, hence the e-mail. A quick and dirty description would be: obfs4 is ScrambleSuit with djb crypto. Instead of obfs3 style UniformDH and CTR-AES256/HMAC-SHA256, obfs4 uses a combination of Curve25519, Elligator2, HMAC-SHA256, XSalsa20/Poly1305 and SipHash-2-4. The feature set offered by obfs4 is comparable to ScrambleSuit with the following differences (implementation specific changes are noted as such): * The key exchange is based on the ntor handshake, and thus authenticates the server to the client. Both obfs3 and ScrambleSuit depend on the encapsulated protocol to handle this on it's own. * obfs4 always does a full handshake. ScrambleSuit style session ticket handshakes are not supported. Even with Elligator2 mapping taken into account, the obfs4 handshake is significantly faster, so there is less of a need for this. * (Impl) obfs4proxy currently does not offer Inter-Arrival Time obfuscation, but this could easily be added if it is something that a lot of people want. It is worth noting that while obfsproxy and obfsclient both support IAT obfuscation, support for it is currently disabled by default as a performance tradeoff. * (Impl) obfs4proxy is currently written in golang. Neither a clear plus or minus, it does allow people to run bridges on reserved ports significantly easier, is probably faster (native code, can use multiple cores for most things), but is more annoying to build and results in rather large binaries (the go runtime is statically linked). * (Impl) obfs4 supports a lot of the protocol improvements made to ScrambleSuit that are pending merge (See #11271, #11203). This difference is temporary. The code and a draft spec is at: https://github.com/Yawning/obfs4 Open questions: * Is the base design and my implementation sane? * Should obfs4proxy also have (disabled) IAT obfuscation? Adding it later will not require wire protocol changes. * The handshake length mimics ScrambleSuit in terms of maximum padding (< 1500 bytes). Should this be increased to be similar to obfs3 (~8k of maximum padding)? The server side cost for this shouldn't be that high. * Is this different enough from ScrambleSuit to be worth deploying? I would be ok with this ending up as "just a research project" and shelving it if the consensus is otherwise. * Should I have named it ScrambleSuit 2? It's a direct descendant of ScrambleSuit and is an obfs derivative only in name at this point. Future plans: * Implement suggestions/improve the code/fix bugs/write more unit tests. * Improve goptlib. * If this is deployed in meaningful amounts, support it in obfsclient. For the extremely brave, I am running a test bridge with the most recent snapshot of the code: UseBridges 1 ClientTransportPlugin obfs4 exec /path/to/obfs4proxy # Bridge line of doom, all one line. Bridge obfs4 178.209.52.110:52810 67E72FF33D7D41BF11C569646A0A7B4B188340DF node-id=Z+cv8z19Qb8RxWlkagp7SxiDQN8= public-key=vm+w9k57aMIX/o+A9ef5C7o9xKEkhr7kRBj3N4etDn8= As with any PT that requires per-bridge arguments, this also requires tor-0.2.5.x. The node-id in this case happens to be my bridge's fingerprint, the public key is the long term key used to validate my bridge's identity and generate M_[C,S]/MAC in the handshake (The bridge line is a bit of a UI/UX disaster unless people are copy/pasting it). Getting either of them wrong will result in handshake failures and tor not bootstrapping. A few warnings: * The spec assumes familiarity with ntor, ScrambleSuit and NaCl's crypto_secretbox. * The wire protocol is not final and I *will* push builds to the bridge as I break backward compatibility without notifying people (It's been 0 days since a wire protocol change.). The test bridge will randomly be broken/rebooted/etc as I also use it for other things. * Development was done with go1.2.x, older versions of the runtime are not supported. * It would be a terrible idea to use obfs4proxy as anything other than a client at this point. Questions, comments, feedback all appreciated. -- Yawning Angel PS: I also wrote https://github.com/yawning/or-applet recently. It's even less supported/tested than obfs4 is (it is written only for my laptop), but some people may find it cute.

4 5

Pluggable transports meeting today (Friday 23rd of May 2014)
by George Kadianakis 23 May '14

23 May '14

Hello friends, just wanted to remind you that the regular biweekly pluggable transports meeting is going to occur today at 16:00 UTC. Place is the #tor-dev IRC channel in the OFTC network. Please note that we recently *changed* the time from 17:00 to 16:00 UTC. Thanks!

1 0

GSoC: Revamp GetTor - new design
by Israel Leiva 22 May '14

22 May '14

Hi all. One of the decision I've had to take during the community bonding was if rewrite GetTor or fix the current code. IMO the best approach is to rewrite it, and I've been working on a proposal design [1]. I'm currently discussing this new design with my mentors, and I'd really appreciate any other feedback from the dev community. Obviously, several parts of the current code will be reused; what will change is how is organized. Also, there are some things that are not yet covered and need further discussion, such as packages checksums and syncronization. [1] https://github.com/ileiva/gettor/tree/master/spec Regards. -- israel

1 0

A Child's Garden of Pluggable Transports
by David Fifield 21 May '14

21 May '14

Have you ever wondered what makes the Tor protocol fingerprintable, and makes pluggable transports necessary? Have you wondered how obfs3 obscures byte patterns in Tor? What a flash proxy WebSocket connection actually looks like, and why it defeats IP blocking but not DPI? Then I have the wiki page for you: https://trac.torproject.org/projects/tor/wiki/doc/AChildsGardenOfPluggableT… It's a visualization of different pluggable transports, meant to be an aid to understanding. At the top is an ordinary Tor handshake, with some fingerprintable data fields highlighted. The following sections, one for each transport, show how those fields are hidden--or not. I tried to demonstrate aspects of different transports that I think are hard to intuit, such as what flash proxy rendezvous looks like, and how transports look under the encrypted layer that is visible to a censor. There are sections for obfs3, ScrambleSuit, FTE, flash proxy, meek, and Bananaphone. The page is missing a few more from https://www.torproject.org/docs/pluggable-transports. If you know how to run any of those transports, and you know an effective way to visualize it, please add it to the page. David Fifield

2 1

obfsproxy getting scramblesuit password from file in unmanaged mode
by irregulator＠riseup.net 21 May '14

21 May '14

Hey all, when running obfsproxy with scramblesuit in unmanaged mode (e.g. to obfuscate non-Tor traffic) the UniformDH password is passed in command line like this: obfsproxy scramblesuit --password=W3ECD5GOYU5AAW4G35GSH5QXIHSRBU2X The problem with this is that the password is visible in the system's process list. Do you think it would make sense to add an argument like "--password-file", so as scramblesuit can fetch the password from a file? Any caveats? Although this is not related to the Tor ecosystem, i think it would be useful. Cheers, Alex

2 1

Reminder: tor development meetings every Wednesday, 1900UTC
by Nick Mathewson 21 May '14

21 May '14

Hi, all! Here's your regular reminder for the weekly IRC meeting for people working on the program "tor". (This won't cover all the other programs developed under the Tor umbrella.) The next meeting time will be: Wednesday May 21, 19:00 UTC. (That's 3pm EDT and 12:00 noon PDT.) As usual, we'll do it on the #tor-dev IRC channel, unless the meeting turns out to be too disruptive for the rest of the channel or vice versa. I've seen a drop-off in the number of people coming since I stopped sending these out weekly, so I think now's a fine time to start doing them weekly again. cheers, -- Nick

1 0

Software Testing
by Lachlan Schahinger 20 May '14

20 May '14

Hello, For a University Project we have been asked to test software, I personally would like to test parts of the TOR project via methods of unit testing and/or fuzz testing. I was wondering if there are any specific areas of the software that need testing that may present results? Thanks,Lachy

1 0

Code review request for bug #9701
by michael＠schloh.com 18 May '14

18 May '14

Hello list, If anybody's feeling particularly friendly, then it would be nice of you to do the favour of reviewing the proposed solution to bug #9701: https://trac.torproject.org/projects/tor/ticket/9701 It relates to clipboard turds of the Tor Browser in violation of the premise of disk avoidance by design. The code in question is attached to the bug report. Georg (GK) has signalled interest, in case you want to coordinate. Thanks in advance! Cheers, Michael

3 5

Metrics tarballs now available as .tar.xz rather than .tar.bz2
by Karsten Loesing 18 May '14

18 May '14

Hello all, we switched the compression format for metrics tarballs. They are now available as .tar.xz (compressed with `xz -9`), with the previous format being .tar.bz2 (compressed with `bzip2 -1`). The effect is that the total size of metrics tarballs shrinks from 212G to 33G. The savings in disk space and bandwidth were too tempting, even though compression and decompression are going to cost some more CPU and memory. The data page now contains links to .tar.xz tarballs: https://metrics.torproject.org/data.html However, the .tar.bz2 tarballs will still be provided for another two months until July 17, 2014. This includes already existing and newly created tarballs. All the best, Karsten

2 1