March 2014 - tor-dev - lists.torproject.org

Damian's Status Report - March 2014
by Damian Johnson 31 Mar '14

31 Mar '14

Hi all. GSoC coordination and a couple particularly tenacious illnesses pretty well knocked me out of commission this March. The neatest changes I managed to slip in concerned Stem's site... * FAQ entry for how to interact with Tor's controller interface directly. This comes up reasonably frequently on Stack Overflow. In the answer we discuss each connection and authentication scenario with examples... https://stem.torproject.org/faq.html#can-i-interact-with-tors-controller-in… * A few weeks ago Coderman made the great suggestion for a Stem Cookbook. I've put together a few scripts that demonstrate neat things. If you have any suggestions for other things to exemplify then please let me know! https://stem.torproject.org/tutorials/double_double_toil_and_trouble.html#s… * Code reviewed Martin's munin-tor and Lunar's exit-funding scripts. Both are now listed on our examples page, and I also made Exit Map more prominent (thanks Philipp for mentioning Stem in Spoiled Onions!). https://lists.torproject.org/pipermail/tor-relays/2014-March/004173.html https://lists.torproject.org/pipermail/tor-dev/2014-March/006598.html * Handful of other minor improvements such as making FAQ entries more concise, syntax highlighting torrc snippets, and better module docs. Stem now supports HS_DESC events, and fixed an issue with ExitPolicy pickle-ability thanks to Nick Hopper. Final note for March is that as anybody who knows me can attest I make a lot of typos. I spend enough time on irc that I finally invested some effort into getting an Irssi spellchecker working... https://www.atagar.com/transfer/tmp/irssi_spellchecker.png I love Irssi to death but user friendly it is not. If you would like a similar setup then here's my notes... http://paste.debian.net/plain/90855 Cheers! -Damian

1 0

Patch: Hidden service: use inbound bind-address based on circuit ID (effectively giving each circuit a temporarily-unique IP address)
by Harry SeventyOne 31 Mar '14

31 Mar '14

Hi, I've written this (ugly, unconfigurable) patch for Tor which is designed to allow hidden services more information about their users, by giving each inbound circuit its own temporary "IP address" in the 127.x range. This technique works on Linux (I've not tried it on anything else) and allows the application server to do some useful things which were previously difficult: * Identify TCP connections coming from the same client, in a short space of time, for example, for diagnostic log analysis, identifying traffic trends * Rate-limit operations coming from the same client, to defend against some types of DoS attacks * Temporarily block abusive clients (at least, until they make a new Tor circuit) More importantly, it can do this with an unmodified application-server (e.g. web servers typically have these features built-in) because it effectively "spoofs" the client ID as an ip-address, in the 127.x range. The patch is currently not configurable (the feature can't be turned off). It only works with hidden services which are routed to "localhost". Request for comments. What do you think? -- Harry SeventyOne

2 2

Announcing OnionPy, a Python wrapper for OnionOO
by Lukas Erlacher 31 Mar '14

31 Mar '14

Hello, this will most likely only be interesting to the people working on the tor-weather rewrite. I have started work on a Python wrapper for OnionOO with support for transparently caching OnionOO replies in memcached. It is hosted on github for now: https://github.com/duk3luk3/onion-py Please let me know if you have any thoughts on it! Best regards, Luke

3 4

Mr. Hyde, or The guy who runs a Jekyll blog over a Tor Hidden Service
by Karsten Loesing 30 Mar '14

30 Mar '14

Hello everyone, I wrote a first draft of a manual that explains how to run a Jekyll blog over a hidden service. *Disclaimer: The following manual is a draft and may contain severe problems harming your privacy! Do not rely on anything in the manual yet!* http://csxeeumg5ynu2rk7.onion/ https://github.com/kloesing/hyde/blob/master/publisher-manual/index.md The intended audience are bloggers who can handle a terminal window but who don't know the typical pitfalls of securely setting up a web server over a hidden service. The approach taken in the manual is to describe how to assemble existing software for a simple hidden blog in as few steps as possible. Are there any obvious security problems with the described setup? Is it possible to simplify parts of the manual even more? Does anyone want to work on a Jekyll layout to make the manual prettier? Feedback much appreciated! Thanks! All the best, Karsten

1 0

Requesting trac components for some PT projects
by George Kadianakis 30 Mar '14

30 Mar '14

Hello people of the Trac, could you make us some Trac components for some PT projects? Specifically, it would be great if you could make us the following components: a) "meek" b) "FTE" c) "obfsclient" Thank you!

2 1

Exit Funding Script
by Damian Johnson 29 Mar '14

29 Mar '14

Hi Moritz, sorry about the delay. I just took a quick peek at your exit-funding script (https://github.com/torservers/exit-funding) Looks good! Class and function documentation would've been nice, but the scripts reasonably straight forward. As I'm sure you know answering the question 'how fast is relay X' or 'how much is relay X used' is surprisingly thorny. Tor descriptors have two values... * Measurement provided by relays themselves of how much they're used. These are completely gameable since they're self-published. * Heuristic provided by the bandwidth authorities. This is what Tor actually uses for relay selection. However, while this is tricker to game, it's also pretty useless for answering either question since it's purely a unit-less heuristic. Roger discussed this space a bit on... https://blog.torproject.org/blog/lifecycle-of-a-new-relay Iirc the value you're using (the router status entry's 'bandwidth' value) is the later. I suspect both Stem and Tor's control-spec docs about this being kb/s is wrong. I left a question on irc asking for clarification... 15:53 < atagar> No armadev *or* nickm? I think that might be a first... 15:57 < atagar> karsten: Actually, you'd know this offhand... 15:57 < atagar> The dir-spec's description of the 'Bandwidth=' value of w lines in router status entries is confusing me a bit. It's described as... 15:57 < atagar> "An estimate of the bandwidth of this relay, in an arbitrary unit (currently kilobytes per second). Used to weight router selection." 15:57 < atagar> This is the heuristic generated by the bandwidth authorities, right? If it is then it shouldn't have units at all (that is purely a heuristic for relay selection - it has no bearing on a relay's actual capacity or usage). 15:57 < atagar> ... or is this the old heuristic that's based on relay's self-published usage? 15:57 < atagar> I'm guessing the former, and that we should replace "in an arbitrary unit (currently kilobytes per second)" with "as an unit-less heuristic purely used for relay selection. This has no direct relation to a relay's capacity or usage." 15:57 < atagar> (this is a common point of confusion since 'how do I determine the fastest relays?' is a common question) Added your script to Stem's examples page. I don't have much time on my hands so I only took a close peek at the Stem related bits... > class VerboseDescriptorReader(object): You are parsing metrics.torproject.org archives, right? I recall some discussion about Stem performance with archives (... though I don't recall if the trouble was compression or python's tar module). If you decompress them then I suspect that it provides one descriptor per file. If that's the case then you can do the same by registering a read listener instead of wrapping the class... entries_seen = 0 def read_file(path): entries_seen += 1 if entries_seen % 25 == 0: print >> sys.stderr, "%i documents parsed...\n" % entries_seen reader = DescriptorReader([self.descriptors_path]) reader.register_read_listener(read_file) with reader: for relay_desc in reader: ... etc... If you decide to keep the VerboseDescriptorReader then you can drop self._targets (it's unused). > self.controller = Controller.from_port(port=9151) This isn't the usual control port, so you should probably mention it in your readme. > return self.controller.get_info('ip-to-country/%s' % address) This can potentially raise exceptions. You might want to provide a default value... return self.controller.get_info('ip-to-country/%s' % address, 'unknown') In general I suspect we might be able to simplify your script by abstracting away the descriptor handling from the rest. If I'm understanding this correctly the question you're trying to answer is... "I have a set of relays I'm interested in, defined by their contact information. What is the sum of their router status entry 'bandwidth' measurements?" Is that right? Cheers! -Damian

1 0

Linux kernel transproxy packet leak (w/ repro case + workaround)
by Mike Perry 28 Mar '14

28 Mar '14

Hello all, I've discovered that the Linux kernel appears to have a leak in how it applies transproxy rules to the TCP CLOSE_WAIT shutdown condition under certain circumstances. This applies to both the kernels in use by common Android devices (Cyanogenmod 10.x and 11-M4), as well as the Linux kernel in Ubuntu 13.04 (3.8.0-35-generic). The bug can be triggered either by a remote server closing a connection, or by restarting the local tor client. Basically, the bug happens when a transproxy connection shuts down completely before a client application properly closes the socket. This seems to cause the kernel to lose track of the fact that the client application connection was being transproxied, and when the client application finally does close its socket (or exits), the Linux kernel generates a FIN ACK that completely bypasses any transproxy rules you have installed. It sends this packet first as the UID of the app in question, and if that fails, it resends it as a blank UID (the kernel itself). Here's how to reproduce it and see for yourself: First run the attached iptables script, and launch a tor daemon with the attached torrc (edit the iptables script's TOR_UID=`id -u debian-tor` and NETWORK_USER_ID=1000 vars if your setup is different). Then, fire up tcpdump, like so: sudo tcpdump -n -i wlan0 host 74.125.28.104 and tcp port 80 Replace '-i wlan0' with your network interface. If you use '-i any', you will also see transproxied packets (which are not normally leaked). Then, as your transproxied user, paste this python snippet into a python interpreter: import socket s = socket.create_connection(("74.125.28.104", 80)) (That IP handles www.google.com) After the connection is made, you should see something like the following in 'netstat -natp': tcp 0 0 127.0.0.1:9040 192.168.1.23:42235 ESTABLISHED 1121/tor tcp 0 0 192.168.1.23:42235 74.125.28.104:80 ESTABLISHED 977/python At this point, either wait a couple minutes for Google to close that connection on you, or shut down your Tor daemon. In either case, you should see the first connection transition to TIME_WAIT, or FIN_WAIT2, or similar, lose track of its PID+UID, and then finally disappear entirely, but the python program will remain in CLOSE_WAIT indefinitely. Once the first connection is fully gone (this takes 60s from TIME_WAIT state with default TCP settings), issue this in your python shell: s.close() At this point, you will see a FIN ACK or RST ACK packet appear in your tcpdump window. That packet has leaked past the iptables firewall rules, and past the transproxy rules. It went straight to Google. I have noticed several Android apps (including Firefox, F-Droid, and many Google apps and Android services) that allow their sockets to sit in CLOSE_WAIT upon remote close while transproxied, and they all leak packets in this case, which happens frequently in normal usage. I am not sure if this is just a common programming error, an issue with how the Android networking APIs are designed, something specifically exacerbated by the transproxy, or some combination of these. For a workaround, I was able to prevent this issue with the addition of the following rules: iptables -I OUTPUT ! -o lo ! -d 127.0.0.1 ! -s 127.0.0.1 -p tcp -m tcp --tcp-flags ACK,FIN ACK,FIN -j DROP iptables -I OUTPUT ! -o lo ! -d 127.0.0.1 ! -s 127.0.0.1 -p tcp -m tcp --tcp-flags ACK,RST ACK,RST -j DROP None of the transproxy documentation I could find mentions this issue, nor suggests any additional safety rules. This means every transproxied Tor user is unwittingly leaking packets, at least some of the time. Sorry to be the bearer of bad news. Please send workaround discussion to tor-talk, and kernel/TCP state machine discussion to tor-dev. I Cc'd both like a jerk, because I figure each group might have different sets of commentary, and both groups should be aware of this issue. Don't be a jerk like me, please. Use your best judgment to Cc one list or the other. -- Mike Perry

1 0

Call for testing/review: obfsclient-0.0.2
by Yawning Angel 28 Mar '14

28 Mar '14

Hello all, I just tagged obfsclient v0.0.2 (Release) Download at: https://github.com/Yawning/obfsclient/archive/v0.0.2.tar.gz This is mostly a bug fix release that addresses issues found in testing/actual use. The code is quite stable now, and I have been using it for all of my tor for approximately 2 weeks. Changes since 0.0.1: * Change the command line arguments to match the obfsproxy counterparts. This fixes issue #31. * Apply pushback between each side of the proxied connection to limit the amount of data that can be buffered. This fixes issue #3. * Use bufferevent_write_buffer to save allocating/copying in obfs2/obfs3. This fixes issue #24. * Fix assertions primarily seen on abrupt connection close by disabling all reads on transition into the various SOCKS5 FLUSHING states. * Cleaned up the logging, and log SOCKS5 assertions to the log file. * Send a TTL EXPIRED SOCKS5 error on connection timeouts. * Fixed an assertion that occurred if the outgoing connection immediately failed (Eg: Network interface being down). * Darwin build fixes, pointed out by Jeroen Massar. * FreeBSD 8.x/9.0 build fix, pointed out by Fabian Keil. NB: Darwin builds require using liballium from git and not 0.0.1, due to a minor code change to a embedded dependency. It is also entirely possible that I broke this again via changes made since then, but I don't use Darwin at all. Assuming there are no major bugs, I am hoping that 0.0.3 will be a feature release so the pace should be less hectic. Questions, comments, feedback appreciated as always, -- Yawning Angel

2 1

Two torspec proposals
by Virgil Griffith 28 Mar '14

28 Mar '14

I have two proposals to add to the torspec.git. They are: (1) http://dl.dropbox.com/u/3308162/230-quicken-tor2web-mode.txt (2) http://dl.dropbox.com/u/3308162/231-remittance-addresses.txt If someone with commit rights will add them that'd be lovely and we can ignore the rest of this message. If not, I request permission to commit to torspec.git. My PGP is here and we can discuss setting up a password, etc: http://virgil.gr/1+Virgil_Griffith_pub.asc -V

3 2

Starting work on hidden services
by Helder Ribeiro 28 Mar '14

28 Mar '14

Hi everyone, I'm an undergraduate student at University of Campinas, in Brazil, starting now I'll be working all year, >=20h/week on research about and improvements to tor hidden services under the supervision of Prof. Diego F. Aranha. I'm generally very inexperienced both with Tor (I've been reading papers and specs, but don't know my way around the source yet) and with programming C in the real world (contrived class projects are basically all I've done), so there will be a lot to ramp up to. Given the above time and experience constraints, I want to 1. do something that is not urgent/blocking; 2. stay as far away as possible from security-sensitive code, and 3. bite off something that is doable, including ramp-up, within a semester. Going through the "Hidden Services need some love" blog post, I found the item "Analyze Hidden Service Circuit Establishment Timing With Torperf" fitting. From reading the tickets, it seems like I would need to add some instrumentation (#3459) and then do the measurements. Some questions: - Is there anything else that would fit the above constraints that you need done more? I'm starting to write the grant proposal (submission is in 2 weeks) and I can still change it to anything else. (I'll dedicate the same amount of time to the project this year regardless of having the grant approved.) - Who should I report to? - Assuming I stick to the timing analysis above, are there any specific questions that you would like the analysis to answer? Thanks a lot! Cheers, Helder

3 6