- tor-dev - lists.torproject.org

Damian's Status Report - July 2013
by Damian Johnson 03 Aug '13

03 Aug '13

Hi all. This month was mostly spent on non-tor work including a server migration, bad service outage at work, and a full week of cleaning my apartment. Still, plenty of spiffy news in stem land... ====================================================================== Remote Descriptor Fetching ====================================================================== Major feature for this month was the addition of a module to remotely fetch tor descriptors... https://stem.torproject.org/api/descriptor/remote.html https://lists.torproject.org/pipermail/tor-dev/2013-July/005156.html This works much like tor itself does, downloading descriptor content from directory authorities and mirrors. With it we can now easily script against the present state of the tor network without piggybacking on a live tor instance. Curious what you can use it for? See our present monitors for some ideas... https://lists.torproject.org/pipermail/tor-dev/2013-July/005209.html https://gitweb.torproject.org/atagar/tor-utils.git/tree This also included a little work with Nick on the spec and tor side... * Dropped the unimplemented microdescriptor query from the spec. (#9271) * Noting the max queryable fingerprints/hashes in spec. (#9282) * We're getting a high failure rate on the downloads we make. A little more investigation is needed on my part to help narrow this down. (#9379) ====================================================================== Other news includes... * Revised the appearance of stem's frontpage. The blue buttons were pretty jarring, so switched to something that matches the rest of the page... http://www.atagar.com/transfer/stem_frontpage/before.png http://www.atagar.com/transfer/stem_frontpage/after.png * Added Slackware to our download page. Many thanks to Markus for adding us to SlackBuilds! http://slackbuilds.org/repository/14.0/python/stem/ * Worked with Sreenatha to port Tor Weather to stem. Unfortunately Weather does not presently have an active maintainer so I'm not sure how we will proceed on this front. (#8264) https://lists.torproject.org/pipermail/tor-dev/2013-July/005111.html * The Munich dev meeting has attracted quite a few potential volunteers. After discussing prospective projects with seros I tidied up our volunteer page. Changes included... * Added Christian's Globe project https://lists.torproject.org/pipermail/tor-commits/2013-August/060124.html * Dropped Thandy https://lists.torproject.org/pipermail/tor-commits/2013-July/059830.html * Merged TorStatus into the entry for Atlas https://lists.torproject.org/pipermail/tor-commits/2013-July/060003.html * General page corrections https://lists.torproject.org/pipermail/tor-commits/2013-August/060125.html * Our automated Jenkins test runs ran into another regression in tor that caused it to segfault. (#9298) * STREAM events mishandled IPv6 addresses. (caught and patched by soult, #9181) * Thread with Pierre about TorPylle which turned into a discussion with Nick regarding future language direction for the tor codebase. I'm looking forward to seeing where it goes! https://lists.torproject.org/pipermail/tor-dev/2013-July/005161.html * We just finished with midterms for Google Summer of Code. Chang unfortunately did not pass, but the other projects are going well. * Sorted out travel arrangements for the GSoC mentor summit. Nick and I will be going, and Moritz is presently on the waitlist. * Code reviewed ra's rttprober. He provided some great feedback for which I still owe him a reply. https://lists.torproject.org/pipermail/tor-dev/2013-July/005183.html Cheers! -Damian

1 0

Tor-stem in python script - HTTP requests issue
by Eduard Natale 03 Aug '13

03 Aug '13

Hello guys, I had a problem and currently I'm not able to solve it. So, here I am ;) I have a python script that uses python-stem to create and handle a tor instance (on a defined port). What it does is retrieving (using a HTTP GET) a web page and submitting information (using HTTP POST messages). Basically i use tor because I need to test this server from different IP addresses with more requests in parallel. What I also do is keeping trace of Cookies. Here's a sample of the code I use, based on the example on stem website https://stem.torproject.org/tutorials/to_russia_with_love.html (to have more parallel requests, i launch the script many times with different socks_port value): ---------------------------- import socket, socks, stem.process import mechanize, cookielib SOCKS_PORT = 9000 DATA_DIRECTORY = "TOR_%s" % SOCKS_PORT socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', SOCKS_PORT) socket.socket = socks.socksocket tor_process = stem.process.launch_tor_with_config( config = { 'SocksPort': str(SOCKS_PORT), 'ControlPort': str(SOCKS_PORT+1), 'DataDirectory': DATA_DIRECTORY, 'ExitNodes': '{it}', }, ) # initialize python mechanize, with cookies (it works exactly like urllib2, urllib3, etc. already tried...) br = mechanize.Browser() cj = cookielib.LWPCookieJar() br.set_cookiejar(cj) ... for number in num_list: req = br.open_novisit("http://example.com") #_1_ res = req.read() print res req.close() req2 = br.open("http://example.com/post_to_me", data_to_post) #_2_ res2 = req2.read() req2.close() -------------------------------- And that's it. The problem occurs on the lines i marked as _1_ and _2_: basically when it reaches around 200 requests, it seems to block undefinitely, waiting for a response that never comes. Of course, wiresharking doesn't work because it's encrypted. The same stuff, without TOR, works perfectly. So, why does it stuck at about 200 requests!? I tried to: 1. Telnet on control port, forcing to renew circuits with SIGNAL NEWNYM 2. instantiating mechanize (urllib2, 3, whatever) in the loop 3. ...i don't remember what else I thought it could be a local socket connection limit: actually without TOR, i see in wireshark the source port changes every time a request is performed. But actually i don't know if the problem is in using the same source port every time (but i don't think so) and if so, should I close the current socket and open a new one? Should I kill the tor process? I can't exaplain myself why... What I only know is: *when the script stucks, if i kill the python process (ctrl+c) and then re-launch, it starts working again.*. I've seen that it's possible to set the value of TrackHostExitsExpire, is it useful in my case? Thanks in advance to whoever can help me!! Ed

2 2

Re: [tor-dev] Format-Transforming Encryption Pluggable Transport
by George Kadianakis 03 Aug '13

03 Aug '13

Hi Kevin, I tried the bundles in https://kpdyer.com/fte/ . For some reason, when I fire up 'start-tor-browser' I don't get 'fte_relay' listener to bind on '127.0.0.1:8079' (like the torrc expects it to). Hence Tor fails to bootstrap and simply says: "The connection to the SOCKS5 proxy server at 127.0.0.1:8079 just failed. Make sure that the proxy server is up and running." When I run 'start-tor-browser' I'm getting the following message in stdout: """ ./bin/fte_relay: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/_MEI8IBYIj/libz.so.1) """ could this be why fte_relay never sets up a listener? Also, if I try to manually invoke fte_relay by doing: ./bin/fte_relay --mode client --server_ip 128.105.214.241 --server_port 8080 I get the same error. Furthermore, I can't get ./bin/fte_relay to give me some kind of usage information. Any ideas? Thanks!

2 3

Onionoo protocol/implementation nuances / Onionoo-connected metrics project stuff
by Kostas Jakeliunas 31 Jul '13

31 Jul '13

Hi Karsten, (not sure whom to CC and whom not to, I have a couple of fairly specific technical questions / comments (which (again) I should have delved into earlier), but then again, maybe the scope of the tor-dev mailing list includes such cases..) @tor-dev: This is in regard to the Searchable metrics archive project, intersected with Onionoo stuff. I originally wanted to ask two questions, but by the time I reached the middle of the email, I wasn't anymore sure if they were questions, so I think these are simply my comments / observations about a couple of specific points, and I just want to make sure I'm making general sense. :) ..So it turns out that it is very much possible to avoid Postgres sequential scans - to construct such queries/cases which are best executed using indexes (the query planner thinks so, and it makes sense as far as I can tell) and whatever efficient (hm, < O(n)?) search algorithms are deemed best - for the metrics search backend/database; the two things potentially problematic to me seem to be: - when doing ORDER BY, making sure that the resulting SELECT covers / would potentially return a relatively small amount of rows (from what I've been reading / trying out, whenever a SELECT happens which may cover > ~10-15% of all the table's rows, sequential scan is preferred, as using indexes would result in even more disk i/o, whereas the seq.scan can read in massive(r) chunks of data from disk because it's, well, sequential). This means that doing "ORDER BY nickname" (or fingerprint, or any other non-unique but covering-a-relatively-small-part-of-all-the-table column in the network status table) is much faster than doing e.g. "ORDER BY validafter" (where validafter refers to the consensus document's "valid after" field) - which makes sense, of course, when you think about it (ordering a massive amount of rows, even with proper LIMIT etc. is insane.) We construct separate indexes (e.g. even if 'fingerprint' is part of a composite (validafter + fingerprint) primary key), and all seems to be well. I've been looking into the Onionoo implementation, particularly into ResourceServlet.java [1], to see how ordering etc. is done there. I'm new to that code, but am I right in saying that as of now, the results (at least for /summary and /details) are generally fairly unsorted (except for the possibility of "order by consensus_weight"), with "valid_after" fields appearing in an unordered manner? This of course makes sense in this case, as Onionoo is able to return *all* the results for given search criteria (or, if none given, all available results) at once. The obvious problem with the archival metrics search project (argh, we are in need of a decent name I daresay :) maybe I'll think of something.. no matter) is then, of course, the fact that we can't return all results at once. I've been so far assuming that it would therefore make sense to return them, whenever possible (and if not requested otherwise, if "order" parameter is later implemented), in "valid_after" descending order. I suppose this makes sense? This would be ideal, methinks. So far, it seems that we can do that, as long as we have a WHERE clause that is restricting-enough. select * from (select distinct on (fingerprint) fingerprint, validafter, > nickname from statusentry where nickname like 'moria1' order by > fingerprint, validafter desc) as subq order by validafter desc limit 100; , for example, works out nicely, in terms of efficiency / query plan, postgres tells me. (The double ORDER BY is needed, as postgres' DISTINCT needs an ORDER BY, and that ORDER BY's leftmost criterion has to match DISTINCT ON (x)) Now, you mentioned / we talked about what is the (large, archival metrics) backend to do about limiting/cutoff? Especially if there are no search criteria specified, for example. Ideally, it may return a top-most list of statuses (which in /details include info from server descriptors), sorted by "last seen" / "valid after"? Thing is, querying a database for 100 (or 1000) of items with no ORDER BY Is really cheap; introducing ORDER BYs which would still produce tons of results is considerably less so. I'm now looking into this (and you did tell me this, i.e. that, I now think, a large part of DB/backend robustness gets tested at these particular points; this should have been more obvious to me). But in any case, I have the question: what *should* the backend return (when no search parameters/filters specified, or very loose ones (nickname LIKE "a") are? What would be cheap / doable: ORDER BY fingerprint (or digest (== hashed descriptor), etc.) It *might* (well, should) work to order by fingerprint, limit results, and *then* reorder by validafter - with no guarantee that the topmost results would be with highest absolute validafters. I mean, Onionoo is doing this kind of reordering / limiting itself, but it makes sense as it can handle all the data at once (or I'm missing something, i.e. the file I linked to already interacts with a subset of data; granted, I haven't thoroughly read through.) In our/this case, it makes sense to try and outsource all relational logic to ORM (but of course if it turns out we can cheaply get 100 arbitrary results and more easily juggle with them ourselves / on the (direct) python side, then sure.) But would such arbitrary returned results make sense? It would look just like Onionoo results, but - a (small) subset of them. Ramble #2: Onionoo is doing more or less direct (well, compiled, so efficient) regexps on fingerprint, nickname, etc. By default, "LIKE %given_nickname_part%" is again (sequentially) expensive; Postgres does offer full text search extensions (would need to build additional tables, etc.), and I think this makes sense to be done; it would cover all our "can supply a subset of :param" bases. I'll see into this. (It's also possible to construct functional(istic) indexes, e.g. with regexp, but need to know the template - I wonder if using Onionoo's regexpressions would work / make sense - will see.) For now, LIKE/= exact_string is very nice, but of course the whole idea is that it'd be possible to supply substrings. Of note is the fact that in this case, LIKE %substring% is O(n) in the sense that query time correlates with row count, afaict. As of now, full text search extensions would solve the problem I think, even if they may look a bit like an overkill at first. End of an email without a proper direction/point. :) [1]: https://gitweb.torproject.org/onionoo.git/blob/HEAD:/src/org/torproject/oni…

3 6

Help with rebasing arma's n23-5 tor branch to current master
by Karsten Loesing 30 Jul '13

30 Jul '13

Dear tor-devs, is anyone here up for a coding task that could help us research performance improvements of the N23 design more? The situation is that we already have a branch (n23-5 in arma's public repository), but it's based on 0.2.4.3-alpha-dev and needs to be rebased to current master. In theory, it's as simple as the following steps: $ git clone https://git.torproject.org/tor.git $ cd tor/ $ git remote add arma https://git.torproject.org/arma/tor.git $ git fetch arma $ git checkout -b n23-5 arma/n23-5 $ git fetch origin $ git rebase origin/master (clean up the mess) $ git add $ git commit $ git rebase --continue (back to clean-up-the-mess step until git is happy) $ git push public n23-5 Bonus points if you make sure the branch compiles with gcc warnings enabled, appeases make check-spaces, and runs peacefully in a private Chutney network. Unfortunately, the n23-5 branch touches a few places in the tor code that have been refactored in current master, including Andrea's connection/channel rewrite. It might be necessary to dive into the channel thing in order to get this rebase right. Once we have a refactored n23-5 branch, I'll try to simulate it in Shadow. For a tiny bit of context, this is for our sponsor F item 13: https://trac.torproject.org/projects/tor/wiki/org/sponsors/SponsorF/Year3 I'm asking here, because the usual suspects are already overloaded with other stuff. As usual, I guess. Thanks, Karsten

2 5

TorPylle: a Python/Scapy TOR protocol implementation
by pierre.lalet＠cea.fr 30 Jul '13

30 Jul '13

Hi all, To test the Tor program, I though an independent implementation might help. I started writing TorPylle with that in mind. The purpose is NOT to implement a secure or robust implementation that could be an alternative to Tor. It relies on Scapy (http://www.secdev.org/projects/scapy/) and is supposed to be used more or less the same way. The code is here : https://github.com/cea-sec/TorPylle and includes an example file. This is an early development stage. Comments, fixes and questions welcome ! Pierre

4 7

Descriptor Monitors
by Damian Johnson 30 Jul '13

30 Jul '13

Hi all. Over this last weekend I started taking advantage of stem's shiny new remote descriptor fetching module [1] to implement some simple monitors... * Descriptor Checker Hourly task that downloads the server descriptors, extrainfo descriptors, and consensus to check for malformed content. In the case of the consensus this downloads from each of the authorities to also ensure that they're all reachable... https://gitweb.torproject.org/atagar/tor-utils.git/blob/HEAD:/descriptor_ch… * Sybil Checker Replacement for the consensusTracker.py I've been running for the last few years [2]. This checks for sudden influxes of new relays, such as the trotsky relays from 2010 [3]... https://gitweb.torproject.org/atagar/tor-utils.git/blob/HEAD:/sybil_checker… Cheers! -Damian [1] https://stem.torproject.org/api/descriptor/remote.html [2] https://gitweb.torproject.org/atagar/tor-utils.git/blob/e537044:/consensusT… [3] https://trac.torproject.org/projects/tor/wiki/doc/badRelays#trotsky

1 0

RFC: obfsproxyssh
by Yawning Angel 29 Jul '13

29 Jul '13

Hello, I have been talking about this in #tor-dev for a while (and pestering people with questions regarding some of the more nuanced aspects of writing a pluggable transport, thanks to nickm, mikeperry and asn for their help), and finally have what I would consider a pre-alpha for the PT implementation. obfsproxyssh is a pluggable transport that uses the ssh wire protocol to hide tor traffic. It uses libssh2 and interacts with a real sshd located on the bridge side. Behaviorally it is identical to a user sshing to a host, authenticating with a RSA public/private key pair and opening a direct-tcp channel to the ORPort of the bridge. It is more aimed at non-technical users (because anyone with an account on a bridge can create a tunnel of their own using existing ssh clients), and thus can be configured entirely through the torrc. It still needs a bit of work before it is ready for deployment but the code is at the point where I can use it for casual web browsing, so if people are interested, I have put a snapshot online at http://www.schwanenlied.me/yawning/obfsproxyssh-20130627.tar.gz Note that it is still at the "I got it working today" state, so some parts may be a bit rough around the edges, and more than likely a few really dumb bugs lurk unseen. Things that still need to be done (in rough order of priority): * It needs to scrub IP addresses in logs. * I need to compare libssh2's KEX phase with popular ssh clients (For the initial "production" release more than likely Putty). It currently also sends a rather distinctive banner since I'm not sure which client(s) it will end up mimicking. * I need to come up with a solution for server side sshd logs. What I will probably end up doing is writing a patch for OpenSSH to disable logging for select users. * In a peculiar oversight, OpenSSH doesn't have a way to disable reverse ssh tunnels (Eg: "PermitOpen 127.0.0.1:6969" allows clients to listen on that port). Not a big deal if Tor starts up before any clients connect, but I'll probably end up writing another patch for this. * Someone needs to test it on Windows. All of my external dependencies are known to work, so the porting effort should be minimal (Famous last words). * The code needs to scrub the private key as soon as a connection succeeds in authenticating instead of holding onto it. Probably not a big deal since anyone that can look at the PT's heap can also look at the bridge line in the torrc. Nice to haves: * Write real Makefiles instead of using CMake (I was lazy). src/CMakeLists.txt currently needs to be edited for anyone compiling it. * It currently uses unencrypted RSA keys. libssh2 supports ssh-agent (on all of the relevant platforms) so key management can be handled that way. I do not think there is currently a mechanism for Tor to query the user for a passphrase and pass it to the PT, but if one gets added, support it would also be easy from my end. * The code does not handle IPv6 since it uses SOCKS 4 instead of 5. When Tor gets a way to pass arguments to PTs that are > 510 bytes, I will change this. * libssh2 needs a few improvements going forward (In particular it does not support ECDSA at all). * Code for the bridge side that makes the tunnel speak the manged PT server transport protocol would be nice (For rate limiting). * libssh2_helpers.c should go away one day. Not sure why the libssh2 maintainers haven't merged the patch that I based the code in there on. Things that need to be done on the Tor side of things: * 0.2.4-14-alpha does not have the PT argument parsing code, so this requires a build out of git to function. * The code currently in Git fails to parse bridge lines with arguments that can't be passed via SOCKS 5 (size restriction). The PT tarball has a crude patch that removes the check, but the config file parser needs to be changed. * The Tor code currently in Git likes to forget PT arguments. asn was kind enough to provide me with a patch that appears to fix this (though the PT has a workaround for when it encounters this situation), but moving forward a formal fix would be ideal. * All the PT related cleverness in the world won't do much vs active probing if there is an ORport exposed on the bridge. Tor should be able to handle "ORPort 127.0.0.1:6969" (It may currently work, I'm not sure. There should be a way to disable the reachability check if only to reduce log spam). Open questions: * Is this useful? * Is it worth biting the bullet and rewriting this to use Twisted Conch instead of being a C app? * Would it be simpler to write a wrapper around existing ssh clients? (Probably not.) * How should people handle distributing bridge information? * How to handle the private key to a given bridge getting compromised? (Correctly configured, all that someone who obtains the key would be able to do is talk to the OR port so it's not a security thing, but it opens the bridge to being blocked). * Does Tor tunneled over SSH look distinctive? No effort is made to change the traffic signature, though this can be added if needed. The tarball contains a more detailed README explaining how to set it up and how it works. obfsproxyssh_client.c has a more in-depth TODO list as a large comment near the top of the file. Comments and feedback will be appreciated. Regards, -- Yawning Angel

5 6

how much havoc can a compromised baseband do to a Guardian ROM device?
by Eugen Leitl 29 Jul '13

29 Jul '13

Anyone knows whether a Nexus 4 baseband processor has r/w access to system memory? The firmware doesn't seem to be loaded at boot, so I presume it's entirely out of reach/ reversing?

3 4

[GSoC '13] Status report - Searchable metrics archive
by Kostas Jakeliunas 29 Jul '13

29 Jul '13

Hi everyone, Some clean-ish working code is finally available online [1] (the old PoC code has been moved to [2]); I'll be adding more soon, but this part does what it's supposed to do, i.e.: - archival data import (download, mapping to ORM via Stem, efficiently avoiding re-import of existing data via Stem's persistence path, etc.); what's left for this part is a nice and simple rsync+cron setup to be able to continuously download and import new data (via Metrics archive's 'recent') - data models and Stem <-> ORM <-> database mapping for descriptors, consensuses and network statuses contained in consensuses - models can be easily queried by sqlalchemy's ORM; Karsten suggested that an additional 'query layer' / internal API is not needed until there's actual need for it (i.e., my plan was to provide an additional query API abstracted from ORM (which is itself built on top of database/SQL/python classes), and to build a backend on top of it, as a neat client of that API as it were; I had some simple and ugly PoC's that are now pushed out of priority queue until needed (if ever)) - one example of how this querying (directly atop the ORM) works is provided: a simple (very partial) Onionoo protocol implementation for /summary and /details, including ?search, ?limit and ?offset. Querying takes place over all NetworkStatuses. This is new in the sense that it uses the ORM directly. If there is a need to formulate SQL queries more directly, we'll do that as well. During the Tor developer meetings in Munich, we tried talking over the existing & proposed parts of the system with Karsten. I will be focusing on making sure the Onionoo-like backend (which is being extended) is stable and efficient. I'm still looking into database optimization (with Karsten's advice); an efficient backend for the majority of all archival data available would be a great deliverable in itself, and hopefully we can achieve at least that. I might do well to try and document the database iterations and development, as a lot of thinking now resides in a kind of 'black box' of DB spec, which does not produce code. The large Postgres datasets are residing on a server I'm managing; I'm working on exposing the Onionoo-like API for public queries; doing some simple higher-level benchmarking (simulating multiple clients requesting different data at once, etc.) now. I might need to move the datasets to yet another server (again), but maybe not; it's easy to blame things on limited CPU/memory resources. :) Kostas. [1]: https://github.com/wfn/torsearch [2]: https://github.com/wfn/torsearch-poc

1 0

IRC meeting to discuss sponsor F progress on Wed July 31, 18:00 to 19:00 UTC in #tor-dev
by Karsten Loesing 28 Jul '13

28 Jul '13

Hi all, I'd like to schedule an IRC meeting to discuss what progress we made on sponsor F deliverables in July. Suggested time and place are: Wed July 31, 18:00 to 19:00 UTC in #tor-dev That's in 8 days from today. The time in other timezones is: 11:00 in San Francisco 14:00 in Boston 20:00 in Berlin 21:00 in Athens 23:30 in New Delhi People who should attend are: Roger, Nick, Andrea, George, David, Ximin, Sathya, Nathan, Matt, Colin, Linus, Andrew, Tom, and anyone else who wants to attend. If your name is on that list and the suggested date or time doesn't work for you, please let me know, either here or in private mail. An alternative date would be: Fri August 2, 18:00 to 19:00 UTC in #tor-dev I'm going to send out another reminder in two days from now that will contain the definite date. Thanks, Karsten

2 3

Status report - HTTP pluggable transport
by Chang Lan 27 Jul '13

27 Jul '13

Hi Steven, I deeply apologize for not replying to your emails in the last few weeks. On July 2, I had a bad car accident, and since then I was staying in hospital after a surgery till July 27. I should have let you know earlier to get things less messed up. Sorry again about this. Now I finally get back and will be online for the rest of period. Anyway, I managed to keep up the schedule. Echoing George's request, I wrote a design specification [https://github.com/changlan/http-transport/blob/master/design.md] based on Steven's document. In brief, I apply the BOSH technique to transfer bidirectional-streams over HTTP, which is the skeleton of the design. To disguise Tor's content patterns, we should make the traffic indistinguishable from random bits. Since there is an assumption that user and bridge share a cryptographically secure secret, I simply encrypt the data under AES-128 counter mode. To achieve scanning resistance, the user should send a secure message MAC(IP | Port | Timestamp, K) to the bridge. If the message gets verified on the bridge's side, the user gets accepted. Otherwise, the bridge will refuse to respond the user (or behave like a normal HTTP server). (I am not a crypto guy, so the scheme above may have flaws. Any review?) And finally the traffic should be able to pass through proxies. I guess proxies will not cache POST response; am I correct? That's it. There are some optional components that I am not sure whether to implement or not, such as packet-size obfuscation, Format-Transforming Encryption, etc. I think packet-size obfuscation is necessary, but I would rather put it in a TODO list for now. One thing that is still arguable is that whether the traffic pattern of BOSH is safe enough. In other words, how to evaluate whether the traffic pattern revealed in this HTTP transport is common in other applications (XMPP for example). I have implemented the transport within the pyobfsproxy framework as well as unit tests.[https://github.com/changlan/obfsproxy] Currently scanning resistance and content encryption are not enabled [https://github.com/changlan/obfsproxy/commit/5b5e8b543919c5b3cda756b3a9e7f4…] but the BOSH technique is applied. Tests and code reviews are welcome! Thanks. Best wishes, Chang On Tuesday, July 23, 2013 at 5:27 PM, Steven Murdoch wrote: > Hi Chang, > > On July 29, I will be submitting the Google Summer of Code progress report. Unless I hear from you before this date, I'm afraid I cannot recommend the project continue and Google rules would not allow you to restart at a later stage. I'd rather this not happen, so please do get back in touch. > > Best wishes, > Steven > > >

1 0

Idea regarding active probing and follow-up of SSL connections to TOR bridges
by Lag Inimaineb 27 Jul '13

27 Jul '13

Hi, I've just watched https://www.youtube.com/watch?v=GwMr8Xl7JMQ (old, I know), and found it very interesting. Also, before going any further, I'd like to stress that I'm new to tor-dev and while I've googled my question, I haven't been able to come up with anything, so apologies in advance if this is already solved. Anyway, one of the main topics discussed in that talk was the problem of preventing the blockage of TOR bridges by oppressors. While many "fixes" were mentioned, none of them actually solve the problem of the bridge being probed, by following-up on previously captured SSL sessions (as China does). I was thinking - perhaps instead of making the discovery of the actual bridge addresses "hard", why not setup bridges in places where other *legitimate* SSL connections are also made? What I mean is, maybe we should try and get big sites that cannot be legitimately (in the eyes of the oppressor, of course) blocked (social media, comic sites, whatever), to run a TOR bridge *on the same port* as their regular HTTPS traffic (443), in a way that someone recording the traffic cannot distinguish (in advance, that is) whether a certain SSL connection to that site is a legitimate web browsing session, or a TOR session. That way, even if an address on the internet "speaks" the TOR protocol, it cannot be automatically blocked. Even if this address is known to host a TOR bridge, this might help plausible deniability for people unwilling to disclose that they've been using TOR. I understand this is probably very difficult, for several reasons, but I will try to address some of these. First off - there's the technical issue of binding to port 443 locally on the web servers without disrupting the currently running local client. I see several possible ways around this - a simple one could be local proxying of the SSL connection from the TOR bridge software to the locally running web server, in a way that when the bridge gets an SSL connection that isn't "speaking" the TOR protocol, it will be handed over to the web server. I admit, it's kinda messy, but TBH there are probably more efficient and "cleaner" ways to do this than what I've suggested. Also, long running connections with heavy data usage to a web server address are probably suspicious, but even if these were to be blocked, it would be on a per-client basis, and would also just mean a QoS issue for users but not DoS. Lastly, there's the question of why would a large site agree to this. I still don't have a good answer for this, but maybe you do... (PR, perhaps?) Anyway, I'd like to hear what you think. Maybe this isn't an issue anymore, or maybe you've found a different solution, or mine doesn't work, or you've already implemented it, etc. :) laginimaineb.

3 5

[GSOC 2013] Steganography Browser Extension Status Report
by Hareesan 27 Jul '13

27 Jul '13

Hi, My work plan for last week was to update the locale information and to finish file uploading parts of extension. For now testing purpose, when a file is being selected the byte format will be popped. With this, the locale information has being updated. With this, I have started working on adding public key encryption algorithms to the add on. This task has being planned to be finished before mid term evaluation and for next two weeks, my plan is to start working on image steganography libraries. -- Hareesan

1 0

[GSOC] Status report - Tor capabilities
by Cristian-Matei Toader 26 Jul '13

26 Jul '13

Hello tor-dev, For the last week I was on holiday for the graduation ceremony. I did however manage to get some work done, so for the past 2 weeks I have: - worked on adding parameter filters for the syscall filter; this is done using both a static list of parameters, as well as a dynamic list configurable at runtime; - parameter filters currently support numeric and pointer based parameters such as C strings; pointer based parameters are referenced in the original tor code through a getter (this was implemented for the 'open' syscall). - possibly identified a bug in libseccomp which was causing the accept4 syscall to fail to be added to the filter which was temporarily fixed by accepting all socketcall filters; still need to confirm this with nickm, but for those interested i believe -117 on this [1] line should be at least -120; my local fix makes everything work fine without socketcall. As a general conclusions things are going fine, I am currently trying to figure out what should go in the parameter filter. Another quick link to my remote branch for the ease of those interested: [2]. Looking forward to some feedback, if you happen to have any! References: [1] http://sourceforge.net/p/libseccomp/libseccomp/ci/release-2.1/tree/src/arch… [2] https://github.com/cristiantoader/tor-gsoc-capabilities/tree/gsoc-cap-stage2

1 0

[GSOC2013] EvilGenius status report
by Johannes Fürmann 26 Jul '13

26 Jul '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi *, In addition to attending the tor meeting in Garching (which was really awsesome, by the way) I really got some work on EvilGenius in the past two weeks. Since Arturo and I outlined the rough structure of the EvilGenius command line interface, I was gradually filling in the pieces needed for talking to the vagrant. Starting this tuesday, we built a set of unittests to test the functionality that is already in place. Since yesterday evening, after a bit of hacking, I already implemented the first and most difficult part of network plumbing and got the binary to create valid Vagrantfiles that are actually booting up the network they are supposed to. So, as Arturo said, "the Genius is there, all we have to do now is make it more evil.". We also might have to work on the plumbing a bit and fit some loose pieces together, which is what I shall be doing next week. By the way, it was really nice to meet all of you in Garching. Hope to see you all soon! Have a safe trip home and enjoy your weekend, everyone! Johannes -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJR8m5qAAoJEAgie5o3Q/JpBzcQALiUslixPD5kIL0r+eNvXHU1 BDDDz9JUZhNc/4ZLa1AFXtY2GrJmB4wP6IOGjuj5rAPx0YcHUszVzjaTti5hWflr rlsXbbOgj4VWDsTAcO7AUbhrEVFaH/SlKgQesZKQuLTduDEhzXBSKga6VlCV37AF R//l8iXRBapwq5DzHSW4eYvNXorKF/nKN/ax7XYjZ5+X/P9sXZz1yHqk5/t187Lt mGbnrfcvEC5xkFpLgz/ReWZ7wABUMJRMrgTapI14rau94rGnaWSsVVjbNKRyYlr3 lhvgsEc6bmIRMhaBDn+Ujgt5o75Th3vJQ2T81wXIkWgrTEsZRZTAJGtyjO0nohtI G2UQmy+/U2Xam2tDyMh7xaGhNr4UN1NJpQ17TyCNPd4DzyO5XG6QjHhjb2s4U9ID YhflAo7yEj4P6NsRR+nIh5djveAgUF55sWNpp+fq1SK8VQ2UXMw6Pm76W7uwGIsw uf7icgTbSC7msSBXJf5HgyP7h3ZvlTrblP90/ciVuLS3YPUmwKYqfCJdnrP4PRYI 8p3bV9ZPU59a0Eqc8fhcQazXvaecUjzGfTpW2owkhQYtk2m7jDJ/Rzz6HbYpiToy 7P6N8T4dq9pp4Hb2ANARUPKYmcVYovYJtJZ9GD0zo5hWbi/i7sqPkJCO07RCyqg/ 5ePzmPqQorZP+nawhVbG =THS9 -----END PGP SIGNATURE-----

1 0

Comparing handshake protocols for Tor: Ace vs ntor
by Esfandiar Mohammadi 23 Jul '13

23 Jul '13

Hi, in WPES 2012 [2], Aniket Kate and me introduced Ace, an alternative for Tor's current handshake protocol ntor (which was also briefly discussed on this list [4]). Back then, no implementation for the double scalar multiplication operation (a * b + c * d) on Curve25519 was available. But recently, Robert Ransom implemented in his celator library [1] a highly efficient double scalar multiplication which is suitable for our handshake protocol Ace. In an internship with us, Shivanker Goel implemented and run comparative benchmarks of the Ace protocol, using Robert's celator library [1], and the current ntor implementation. Here are the benchmarks: 64-bit architecture: Ace: 401,129 cycles, ntor: 472,772 cycles, 32-bit architecture: Ace: 2,797,303 cycles, ntor: 5,179,069 cycles, About the benchmarks: * For cleaning the cache, the benchmarks runs the handshake 1000 times prior to starting the benchmarks. Then, we used a tsc_read() function to count the cycles, repeated the handshake 5000 times and computed the average of the cycles. * In order not to be forced to import too many functions from the Tor library, we do not use the function dimap_search, which looks up the secret key for a given public key, in the benchmarks. * We assume that ephemeral keys are precomputed, e.g., in idle cycles. In the benchmarks these ephemeral keys are received as an input. * For the benchmarks hyper-threading and overclocking were disabled and gcc-4.8 with "-O2" optimizations was used. The code is available online [2]. We also tried to formulate a tor-style proposal, which is also available online [3]. Any feedback, comments, or suggestions are welcome. - Esfandiar [1] http://www.infsec.cs.uni-saarland.de/~mohammadi/ace/celator.tar.gz [2] http://www.infsec.cs.uni-saarland.de/~mohammadi/ace/ace-benchmarks.tar.gz [3] http://www.infsec.cs.uni-saarland.de/~mohammadi/ace/ace-handshake.txt [4] It was discussed in the thread "Another key exchange algorithm for extending circuits: alternative to ntor?" from August 2012.

1 0

Remote descriptor fetching
by Damian Johnson 23 Jul '13

23 Jul '13

Hi Karsten. I'm starting to look into remote descriptor fetching, a capability of metrics-lib that stem presently lacks [1][2]. The spec says that mirrors provide zlib compressed data [3], and the DirectoryDownloader handles this via a InflaterInputStream [4]. So far, so good. By my read of the man pages this means that gzip or python's zlib module should be able to handle the decompression. However, I must be missing something... % wget http://128.31.0.34:9131/tor/server/all.z % file all.z all.z: data % gzip -d all.z gzip: all.z: not in gzip format % zcat all.z gzip: all.z: not in gzip format % python >>> import zlib >>> with open('all.z') as desc_file: ... print zlib.decompress(desc_file.read()) ... Traceback (most recent call last): File "<stdin>", line 2, in <module> zlib.error: Error -5 while decompressing data: incomplete or truncated stream Maybe a fresh set of eyes will spot something I'm obviously missing. Spotting anything? Speaking of remote descriptor fetching, any thought on the API? I'm thinking of a 'stem/descriptor/remote.py' module with... * get_directory_authorities() List of hardcoded (IP, DirPorts) tuples for tor's authorities. Ideally we'd have an integ test to notify us when our listing falls out of date. However, it looks like the controller interface doesn't surface this. Is there a nice method of determining the present authorities besides polling the authorities array of 'src/or/config.c' [5]? * fetch_directory_mirrors() Polls an authority for the present consensus and filters it down to relays with the V2Dir flag. It then uses this to populate a global directory mirror cached that's used when querying directory data. This can optionally be provided with a Controller instance or cached consensus file to use that instead of polling a authority. * get_directory_cache() Provides a list of our present directory mirrors. This is a list if (IP, DirPort) tuples. If fetch_directory_mirrors() hasn't yet been called this is the directory authorities. * query(descriptor_type, fingerprint = None, retires = 5) Picks a random relay from our directory mirror cache, and attempts to retrieve the given type of descriptor data. Arguments behave as follows... descriptor_type (str): Type of descriptor to be fetched. This is the same as our @type annotations [6]. This raises a ValueError if the descriptor type isn't available from directory mirrors. fingerprint (str, list): Optional argument for the relay or list of relays to fetch the descriptors for. This retrieves all relays if omitted. retries (int): Maximum number of times we'll attempt to retrieve the descriptors. We fail to another randomly selected directory mirror when unsuccessful. Our last attempt is always via a directory authority. If all attempts are unsuccessful we raise an IOError. ======================================== I'd imagine this would make use of the module something like the following... # Simple script to print all of the exits. from stem.descriptor import remote # Populates our directory mirror cache. This does more harm # here than good since we're only making a single request. # However, if this was a longer living script doing this would # relieve load from the authorities. remote.fetch_directory_mirrors() try: for desc in remote.query('server-descriptor 1.0'): if desc.exit_policy.is_exiting_allowed(): print "%s (%s)" % (desc.nickname, desc.fingerprint) except IOError, exc: print "Unable to query the server descriptors: %s" % exc ======================================== Thoughts? Does this cover all of the use cases we'll this module for? Cheers! -Damian [1] https://trac.torproject.org/8257 [2] https://gitweb.torproject.org/metrics-lib.git/blob/HEAD:/src/org/torproject… [3] https://gitweb.torproject.org/torspec.git/blob/HEAD:/dir-spec.txt#l2626 [4] http://docs.oracle.com/javase/6/docs/api/java/util/zip/InflaterInputStream.… [5] https://gitweb.torproject.org/tor.git/blob/HEAD:/src/or/config.c#l780 [6] https://metrics.torproject.org/formats.html#descriptortypes

3 10

[GSoC '13] Tor status report - Searchable metrics archive
by Kostas Jakeliunas 22 Jul '13

22 Jul '13

Hey all, I apologize for this unusual timing for a status report, but I ended up delaying it beyond measure, so better now than later I guess. I can reiterate it + any updates soon, it's just that I figure I'm long overdue on informing tor-dev on what's going on. I've started my project [1] later than is usual, and more or less immediately ran into what I deemed to be a database / ORM scaling issue (the thing I'd been actually trying to avoid since writing the proposal), or at least a behaviour of the ORM which was suboptimal to what we have in mind: delivering (first and foremost) a searchable metrics archive backend/database which incorporates, as of current plan, server descriptors (relays and bridges, turns out a server descriptor model can happily service both) and server/router statuses across a few year timespan (currently using v3 consensus documents only), and provides querying functionality which can extract relations between the two. The 'querying with relations between the two' part, when tested on a broader span of data, seemed to be causing trouble to me. I ended up allocating probably inefficiently large amounts of time to this problem, rewriting the backend part, and trying to optimize the queries which underlied the ORM (turns out I didn't need to strip off the ORM abstraction - learned a few things about SQLAlchemy that way - I will follow-up with an email pointing to current code (sorry)). * The current iteration of the ORM model / backend (which actually is very simple) solves this problem. * Stem descriptor and network status mapping to ORM works, and is nicely (enough) integrated with the data import (from downloaded metrics archive) tools, as well as an API to make queries on the ORM. * Implemented a partial Onionoo-protocol-adhering (without compression and without some fields) backend for ?summary and ?details Onionoo queries. * Still tidying everything up. And *finally* writing a design document outlining what we actually ended up with, and what is required till full Onionoo integration. Code review will happen pretty soon, and hopefully we'll have some discussion upon where to go from here. Karsten mentioned that it might be possible to use the existing Onionoo incarnation to continue providing bandwidth weight etc. data (basically stuff from extra-info), and it might be possible to join the two systems into an Onionoo-supporting backend which will cover all / majority of archives available. Another (or) further avenue would be to continue with the initial proposed plan to extend the query format; and to build a frontend which would make use of the extended query format. Expect another email with links to (decent) code. [1]: http://kostas.mkj.lt/gsoc2013/gsoc2013.html

1 0

Problems. Flash plugin works by default! I can see youtube video with Tor Browser Bundle
by User of Tor 22 Jul '13

22 Jul '13

This was a HTML5 video. Thank you All for answers.

1 0

Problems. Flash plugin works by default! I can see youtube video with Tor Browser Bundle
by User of Tor 18 Jul '13

18 Jul '13

Hi, I've installed Tor Browser Bundle for Windows (tor-browser-2.3.25-2_ru) I didn't change any settings. When I visit any html-page on which there is a flash video i can see it, flash works! Why? It must not work! In settings(plugins) "Flash plugin" is disabled and it is inactive. In Tor Button settings Plugins are turned off. What happens? Is it a BUG? Regards.

5 4

TBB without Vidalia
by Zack Weinberg 15 Jul '13

15 Jul '13

How technically difficult would it be to remove Vidalia from an already-packaged TBB? (Ideally something that could be put into a shell script.) Basically what I want is a more-current-than-ESR10 browser that uses a system-installed & -configured Tor instance. I wouldn't miss the Torbutton UI, either (it should just use Tor all the time).

3 2

Status Report - Steganography browser addon a GSOC 2013 project
by Hareesan 14 Jul '13

14 Jul '13

Hi all, In last two weeks, I worked on detecting video and sound contents from web pages and get them as byte format to be used with algorithms later. To test it, use the test button in context menu over video element and it will alert the byte code of the content selected. ( Might take some time due to the video element file size). My next plan for coming weeks is to finish content uploading to extension from local memory. And more than context menus, all the content handling parts like toolbar button, menu buttons, upload options will be done by the end of coming two weeks. -- Hareesan

1 0

[GSOC] Status report - Tor capabilities
by Cristian-Matei Toader 13 Jul '13

13 Jul '13

Hello tor-dev, Here goes the status report for the past 2 weeks - I have been preparing the code for the first step of the project to be merged in the Tor master branch, which represented a number of changes required by nickm; the full process can be seen here [1], the branch was squashed and should be merged one of these days. - Step 2: - current progress - has changed a bit, and now requires to create a sandbox as 'tight' as possible around the Tor syscalls; - this is achieved my filtering not only syscalls but also syscall parameters, current efforts on my part can be found here [2] in branch "gsoc-cap-stage2"; - the current problem with this stage is filtering string parameters for syscalls which can be done using immutable strings using protected memory regions; - unfortunately some syscalls are out of reach being part of libraries and therefore their parameters cannot be controlled (the pointer used in the seccomp filter cannot be reference in the syscall) which will need to be solved (any suggestions are appreciated). - I have implemented ioerror's suggestion about policy files but is not pushed to git mainly because after talking to nickm we concluded that the implementation effort is not justified especially now when filter configurations are constantly changing which would require also changing how policy files are parsed, having mostly the same end result; furthermore some of the string filter parameters may be generated on runtime, which may be inconsistent with the solution, but I will keep this in mind towards the finishing stage of the development process, especially since it offers an elegant way of configuring and testing different filters. As a small side-note I will mostly be traveling the majority of next week, so I may not be as reachable as before. Looking forward to some feedback, if you happen to have any! References: [1] https://trac.torproject.org/projects/tor/ticket/9168 [2] https://github.com/cristiantoader/tor-gsoc-capabilities

2 1

Status report - Stream-RTT
by ra 12 Jul '13

12 Jul '13

Hi all! I started a little late on this years GSoC project[0] which is about reducing the RTT of preemptively built circuits. Since begin of July I have been working on scripts for Tor path generation and Stream-RTT probing. For the path generation script "path_finder.py" I decided not to reimplement Tor's path generation logic but to ask a Tor client to choose a path without actually building it. I have written two small patches for Tor which implement the following new control commands: -) "DUMPGUARDS" removes all entry guards. (This might also be handy to work around the path selection strangeness when guards are disabled[1]). -) "FINDPATH" tells the Tor client to generate a path but do not build a circuit from it. To verify the correctness of the approach I am currently generating a few million paths and will look into the distribution of nodes. The Stream-RTT-probing script "rttprober.py" is also in a working state. Though it needs some more testing, it is already able to spawn multiple threads where each builds a circuit and runs a number of probes on that circuit, measuring the RTTs of the stream and the circuit build time. I will hopefully be able to get useful Stream-RTT-data soon to tackle the first interesting questions. All scripts and patches are available online[2]. Best, Robert [0] https://www.google-melange.com/gsoc/project/google/gsoc2013/ra_/19001 [1] https://trac.torproject.org/projects/tor/ticket/9188 [2] https://bitbucket.org/ra_/tor-rtt

1 0