- tor-dev - lists.torproject.org

Re: [tor-dev] visionion: defaults for import data
by Karsten Loesing 08 Aug '13

08 Aug '13

(Moving this discussion to tor-dev@ with Thomas' permission.) On 8/7/13 6:04 PM, thomas lörtsch wrote: > hi karsten, > > to reduce complexity in the mapReduce script it would be helpful if every field of import data always had at least a default value. > > for importRelays that would be: > addd "" > node "" > span 0 > date "" > nick "" > role [] > flag [] > bwa 0 > bwc 0 > tsv "" > osv "" > pbr 0 > pbg 0 > pbm 0 > pbe 0 > pex [] > as 0 > cc "" > > > for importBridges: > addd "" > node "" > span 0 > date "" > nick "" > bwa 0 > bwc 0 > tsv "" > osv "" > brp "" > bre 0 > brt [] > > > and for importClients: > addd "" > span 0 > date "" > cb 0 > cbcc [] > cr 0 > crcc [] > cpt { obfs2: 0, obfs3: 0, OR: 0, Unknown: 0 } > cip { v4: 0, v6: 0 } > > > I hope I got that right. if you've got better ideas for default values of arrays and objects please tell me. Done, except that I put in empty objects {} for cbcc, crcc, cpt, and cip as default values, as discussed on IRC. Please find the updated .json files here: https://people.torproject.org/~karsten/volatile/bridges-2013-08-08.json.gz https://people.torproject.org/~karsten/volatile/clients-2013-08-08.json.gz https://people.torproject.org/~karsten/volatile/relays-2013-08-08.json.gz Best, Karsten

1 0

How the pluggable transports factory works (or: How to deploy your very own pluggable transport)
by George Kadianakis 08 Aug '13

08 Aug '13

--How the pluggable transports factory works-- Pluggable transports make Tor harder to detect. They are "pluggable" and Tor should be able to support a variety of transports [0]. For this reason, it should be easy to write new pluggable transports. Writing a pluggable transport involves understanding how Tor bridges work and how the Tor pluggable transport interface works [1] . Software like obfsproxy and pyptlib (both in Python) might help to abstract away the details of the pluggable transport interface [2]. If you plan to write a pluggable transport, here are a few things that you might want to think about: - Before you start coding. If you have a good pluggable transport idea, it would be wise to contact us before spending weeks hacking on it. We might know someone who is already working on a similar concept, or we might be able to help you with the design/implementation of your transport. Send an email to tor-dev(a)lists.torproject.org (note that tor-dev is a public list. Use tor-assistants(a)torproject.org if it's your sekrit research project). Even before coding, you should have a clear threat model in mind for your transport. You should also compose a clean and readable protocol specification document. See [3] for some specs and threat models of already existing pluggable transports. It's up to you to decide on a programming language for your pluggable transport. We have currently deployed transports written in Python, C and Go. We don't really care which language you end up using, as long as it's multiplatform (works on Windows) and you can write secure and comprehensive code with it. A pluggable transport should not make users less safe. Note that this means that if you write your transport in C or C++, we will spend much more time nitpicking on your code so that it feels secure. Worst-case-scenario we might ask you to rewrite your transport in a more confidence-inspiring language. Finally, check out how already existing transports have been deployed. See how their torrc looks like, how Tor communicates with them, etc. Tor expects a certain interface from pluggable transports, and your transport should be able to follow it. If your idea can't work as a Tor pluggable transport, then contact us and we can think of ways to bend it into shape. - While you are coding While you are writing your pluggable transport, try to be conservative on the libraries you use. Are you coding in Python and you want to import PySketchCrypto? Make sure that the library is widely used and its code is clean and secure. Feel responsible for the libraries you choose. Try to think of the end goal while you are writing your pluggable transports. Make sure that the networking architecture you are using is compatible with the one Tor expects from pluggable transports. Don't forget to actually test your pluggable transport with Tor (in the same way that it would get deployed) before you conclude that "it works" (see the ClientTransportPlugin/Bridge torrc options). - After you finish coding When you have finished writing your pluggable transport, put it in a git repository. Don't forget to include the threat model and the protocol specification of your transport in your repository. Send an email to the tor-dev mailing list and let us know of your creation. - After you go public The final step is to get your transport included in the Pluggable Transport Browser Bundle. Before this happens we will nitpick on your idea, your threat model and your specification, review your code, request fixes, etc. This might be a frustrating procedure, but hopefully it will result in better and more maintainable software (most likely we will be the ones fixing any bugs in your code after it gets deployed). After your transort has reached a certain level of legitimacy it will be time to include it in the Pluggable Transport Browser Bundle. This procedure usually involves taking the latest Pluggable Transport Tor Browser Bundle, compiling your transport and your libs, adding them in the bundle, and finally signing it and releasing it to the world. Refs: [0]: https://www.torproject.org/docs/pluggable-transports.html.en [1]: https://gitweb.torproject.org/torspec.git/blob/HEAD:/pt-spec.txt https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/196-transpor… [2]: https://gitweb.torproject.org/pluggable-transports/obfsproxy.git https://gitweb.torproject.org/pluggable-transports/pyptlib.git http://pyptlib.readthedocs.org/en/latest/ [3]: https://gitweb.torproject.org/pluggable-transports/obfsproxy.git/blob/HEAD:… https://gitweb.torproject.org/pluggable-transports/obfsproxy.git/blob/HEAD:… https://gitweb.torproject.org/pluggable-transports/obfsproxy.git/blob/HEAD:… https://gitweb.torproject.org/pluggable-transports/obfsproxy.git/blob/HEAD:… https://gitweb.torproject.org/flashproxy.git/blob/HEAD:/doc/design.txt

1 0

Why not use The Update Framework? (TUF)
by adrelanos 08 Aug '13

08 Aug '13

> For TBB 3.0, we should use the Firefox updater. We should audit the Firefox updater for issues, and triage which of Thandy's features we should merge to it. (For example, we might want to sign the metadata file if it isn't signed; timestamp it if it isn't timestamped, add multiple-signature support, and so on.) [1] That sounds like reinventing the wheel. > Thandy was a good research platform, not a long-term piece of software we want to support. [1] Why not use its predecessor, TUF? [2] [3] TUF is written in python, and after all those years, TUF developers are still maintaining it and actively developing it. I think in future TUF will become a mature and widespread solution. Also work is being done to let pip (the python library installer) internally use TUF. So it can't be so bad after all? If you have discussed this and reasons for rejecting, fine. Just wanted to throw it in, because I think basing this feature on another active project (TUF) works better than reinventing the wheel. [1] Commenting on: https://trac.torproject.org/projects/tor/wiki/org/meetings/2013SummerDevMee… [2] https://www.updateframework.com/ [3] https://github.com/theupdateframework/tuf

2 2

augmenting RSA identities/signatures with ECC and beyond
by Jacob Appelbaum 06 Aug '13

06 Aug '13

Hi, Linus and I had an interesting discussion at IETF 87 this past week in Berlin. We're both concerned about long term Directory Authority identity keys as well consensus signing with RSA keys. We've agreed that we're interested in writing a proposal whereby we add additional identity keys for authorities. Thus, we'll have whatever security may be provided by RSA and the security that should be provided by ECC signatures. The work on ntor should directly assist us in having almost all the required crypto we'll need for such augmentation. I tend to think that every directory authority should generate an additional and new long term ECC identity key. This will require that tor-gencert is extended to understand both ECC and RSA. We'll want to add these fingerprints to src/or/config.c for each respective DA. We'll want each directory authority to sign with both RSA and ECC. We'll also want to extend the consensus format to handle publication of such signatures. Older clients should be able to parse the consensus without worry and they will check RSA signatures as always. Newer clients should check both and report a mismatch into the logs at a high level. When combined with ntor, I believe that we will have significantly improved the cryptography in Tor. It would be nice to be able to add other signature schemes - specifically for pq crypto related undertakings. In an ideal world, I'd like to be able to sign the consensus from my directory authority with RSA, ECC and some kind of djb approved, tanja tested post-quantum computer signature construct. What do you think we should consider as we draft this proposal? All the best, Jacob

2 3

Damian's Status Report - July 2013
by Damian Johnson 03 Aug '13

03 Aug '13

Hi all. This month was mostly spent on non-tor work including a server migration, bad service outage at work, and a full week of cleaning my apartment. Still, plenty of spiffy news in stem land... ====================================================================== Remote Descriptor Fetching ====================================================================== Major feature for this month was the addition of a module to remotely fetch tor descriptors... https://stem.torproject.org/api/descriptor/remote.html https://lists.torproject.org/pipermail/tor-dev/2013-July/005156.html This works much like tor itself does, downloading descriptor content from directory authorities and mirrors. With it we can now easily script against the present state of the tor network without piggybacking on a live tor instance. Curious what you can use it for? See our present monitors for some ideas... https://lists.torproject.org/pipermail/tor-dev/2013-July/005209.html https://gitweb.torproject.org/atagar/tor-utils.git/tree This also included a little work with Nick on the spec and tor side... * Dropped the unimplemented microdescriptor query from the spec. (#9271) * Noting the max queryable fingerprints/hashes in spec. (#9282) * We're getting a high failure rate on the downloads we make. A little more investigation is needed on my part to help narrow this down. (#9379) ====================================================================== Other news includes... * Revised the appearance of stem's frontpage. The blue buttons were pretty jarring, so switched to something that matches the rest of the page... http://www.atagar.com/transfer/stem_frontpage/before.png http://www.atagar.com/transfer/stem_frontpage/after.png * Added Slackware to our download page. Many thanks to Markus for adding us to SlackBuilds! http://slackbuilds.org/repository/14.0/python/stem/ * Worked with Sreenatha to port Tor Weather to stem. Unfortunately Weather does not presently have an active maintainer so I'm not sure how we will proceed on this front. (#8264) https://lists.torproject.org/pipermail/tor-dev/2013-July/005111.html * The Munich dev meeting has attracted quite a few potential volunteers. After discussing prospective projects with seros I tidied up our volunteer page. Changes included... * Added Christian's Globe project https://lists.torproject.org/pipermail/tor-commits/2013-August/060124.html * Dropped Thandy https://lists.torproject.org/pipermail/tor-commits/2013-July/059830.html * Merged TorStatus into the entry for Atlas https://lists.torproject.org/pipermail/tor-commits/2013-July/060003.html * General page corrections https://lists.torproject.org/pipermail/tor-commits/2013-August/060125.html * Our automated Jenkins test runs ran into another regression in tor that caused it to segfault. (#9298) * STREAM events mishandled IPv6 addresses. (caught and patched by soult, #9181) * Thread with Pierre about TorPylle which turned into a discussion with Nick regarding future language direction for the tor codebase. I'm looking forward to seeing where it goes! https://lists.torproject.org/pipermail/tor-dev/2013-July/005161.html * We just finished with midterms for Google Summer of Code. Chang unfortunately did not pass, but the other projects are going well. * Sorted out travel arrangements for the GSoC mentor summit. Nick and I will be going, and Moritz is presently on the waitlist. * Code reviewed ra's rttprober. He provided some great feedback for which I still owe him a reply. https://lists.torproject.org/pipermail/tor-dev/2013-July/005183.html Cheers! -Damian

1 0

Tor-stem in python script - HTTP requests issue
by Eduard Natale 03 Aug '13

03 Aug '13

Hello guys, I had a problem and currently I'm not able to solve it. So, here I am ;) I have a python script that uses python-stem to create and handle a tor instance (on a defined port). What it does is retrieving (using a HTTP GET) a web page and submitting information (using HTTP POST messages). Basically i use tor because I need to test this server from different IP addresses with more requests in parallel. What I also do is keeping trace of Cookies. Here's a sample of the code I use, based on the example on stem website https://stem.torproject.org/tutorials/to_russia_with_love.html (to have more parallel requests, i launch the script many times with different socks_port value): ---------------------------- import socket, socks, stem.process import mechanize, cookielib SOCKS_PORT = 9000 DATA_DIRECTORY = "TOR_%s" % SOCKS_PORT socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', SOCKS_PORT) socket.socket = socks.socksocket tor_process = stem.process.launch_tor_with_config( config = { 'SocksPort': str(SOCKS_PORT), 'ControlPort': str(SOCKS_PORT+1), 'DataDirectory': DATA_DIRECTORY, 'ExitNodes': '{it}', }, ) # initialize python mechanize, with cookies (it works exactly like urllib2, urllib3, etc. already tried...) br = mechanize.Browser() cj = cookielib.LWPCookieJar() br.set_cookiejar(cj) ... for number in num_list: req = br.open_novisit("http://example.com") #_1_ res = req.read() print res req.close() req2 = br.open("http://example.com/post_to_me", data_to_post) #_2_ res2 = req2.read() req2.close() -------------------------------- And that's it. The problem occurs on the lines i marked as _1_ and _2_: basically when it reaches around 200 requests, it seems to block undefinitely, waiting for a response that never comes. Of course, wiresharking doesn't work because it's encrypted. The same stuff, without TOR, works perfectly. So, why does it stuck at about 200 requests!? I tried to: 1. Telnet on control port, forcing to renew circuits with SIGNAL NEWNYM 2. instantiating mechanize (urllib2, 3, whatever) in the loop 3. ...i don't remember what else I thought it could be a local socket connection limit: actually without TOR, i see in wireshark the source port changes every time a request is performed. But actually i don't know if the problem is in using the same source port every time (but i don't think so) and if so, should I close the current socket and open a new one? Should I kill the tor process? I can't exaplain myself why... What I only know is: *when the script stucks, if i kill the python process (ctrl+c) and then re-launch, it starts working again.*. I've seen that it's possible to set the value of TrackHostExitsExpire, is it useful in my case? Thanks in advance to whoever can help me!! Ed

2 2

Re: [tor-dev] Format-Transforming Encryption Pluggable Transport
by George Kadianakis 03 Aug '13

03 Aug '13

Hi Kevin, I tried the bundles in https://kpdyer.com/fte/ . For some reason, when I fire up 'start-tor-browser' I don't get 'fte_relay' listener to bind on '127.0.0.1:8079' (like the torrc expects it to). Hence Tor fails to bootstrap and simply says: "The connection to the SOCKS5 proxy server at 127.0.0.1:8079 just failed. Make sure that the proxy server is up and running." When I run 'start-tor-browser' I'm getting the following message in stdout: """ ./bin/fte_relay: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/_MEI8IBYIj/libz.so.1) """ could this be why fte_relay never sets up a listener? Also, if I try to manually invoke fte_relay by doing: ./bin/fte_relay --mode client --server_ip 128.105.214.241 --server_port 8080 I get the same error. Furthermore, I can't get ./bin/fte_relay to give me some kind of usage information. Any ideas? Thanks!

2 3

Onionoo protocol/implementation nuances / Onionoo-connected metrics project stuff
by Kostas Jakeliunas 31 Jul '13

31 Jul '13

Hi Karsten, (not sure whom to CC and whom not to, I have a couple of fairly specific technical questions / comments (which (again) I should have delved into earlier), but then again, maybe the scope of the tor-dev mailing list includes such cases..) @tor-dev: This is in regard to the Searchable metrics archive project, intersected with Onionoo stuff. I originally wanted to ask two questions, but by the time I reached the middle of the email, I wasn't anymore sure if they were questions, so I think these are simply my comments / observations about a couple of specific points, and I just want to make sure I'm making general sense. :) ..So it turns out that it is very much possible to avoid Postgres sequential scans - to construct such queries/cases which are best executed using indexes (the query planner thinks so, and it makes sense as far as I can tell) and whatever efficient (hm, < O(n)?) search algorithms are deemed best - for the metrics search backend/database; the two things potentially problematic to me seem to be: - when doing ORDER BY, making sure that the resulting SELECT covers / would potentially return a relatively small amount of rows (from what I've been reading / trying out, whenever a SELECT happens which may cover > ~10-15% of all the table's rows, sequential scan is preferred, as using indexes would result in even more disk i/o, whereas the seq.scan can read in massive(r) chunks of data from disk because it's, well, sequential). This means that doing "ORDER BY nickname" (or fingerprint, or any other non-unique but covering-a-relatively-small-part-of-all-the-table column in the network status table) is much faster than doing e.g. "ORDER BY validafter" (where validafter refers to the consensus document's "valid after" field) - which makes sense, of course, when you think about it (ordering a massive amount of rows, even with proper LIMIT etc. is insane.) We construct separate indexes (e.g. even if 'fingerprint' is part of a composite (validafter + fingerprint) primary key), and all seems to be well. I've been looking into the Onionoo implementation, particularly into ResourceServlet.java [1], to see how ordering etc. is done there. I'm new to that code, but am I right in saying that as of now, the results (at least for /summary and /details) are generally fairly unsorted (except for the possibility of "order by consensus_weight"), with "valid_after" fields appearing in an unordered manner? This of course makes sense in this case, as Onionoo is able to return *all* the results for given search criteria (or, if none given, all available results) at once. The obvious problem with the archival metrics search project (argh, we are in need of a decent name I daresay :) maybe I'll think of something.. no matter) is then, of course, the fact that we can't return all results at once. I've been so far assuming that it would therefore make sense to return them, whenever possible (and if not requested otherwise, if "order" parameter is later implemented), in "valid_after" descending order. I suppose this makes sense? This would be ideal, methinks. So far, it seems that we can do that, as long as we have a WHERE clause that is restricting-enough. select * from (select distinct on (fingerprint) fingerprint, validafter, > nickname from statusentry where nickname like 'moria1' order by > fingerprint, validafter desc) as subq order by validafter desc limit 100; , for example, works out nicely, in terms of efficiency / query plan, postgres tells me. (The double ORDER BY is needed, as postgres' DISTINCT needs an ORDER BY, and that ORDER BY's leftmost criterion has to match DISTINCT ON (x)) Now, you mentioned / we talked about what is the (large, archival metrics) backend to do about limiting/cutoff? Especially if there are no search criteria specified, for example. Ideally, it may return a top-most list of statuses (which in /details include info from server descriptors), sorted by "last seen" / "valid after"? Thing is, querying a database for 100 (or 1000) of items with no ORDER BY Is really cheap; introducing ORDER BYs which would still produce tons of results is considerably less so. I'm now looking into this (and you did tell me this, i.e. that, I now think, a large part of DB/backend robustness gets tested at these particular points; this should have been more obvious to me). But in any case, I have the question: what *should* the backend return (when no search parameters/filters specified, or very loose ones (nickname LIKE "a") are? What would be cheap / doable: ORDER BY fingerprint (or digest (== hashed descriptor), etc.) It *might* (well, should) work to order by fingerprint, limit results, and *then* reorder by validafter - with no guarantee that the topmost results would be with highest absolute validafters. I mean, Onionoo is doing this kind of reordering / limiting itself, but it makes sense as it can handle all the data at once (or I'm missing something, i.e. the file I linked to already interacts with a subset of data; granted, I haven't thoroughly read through.) In our/this case, it makes sense to try and outsource all relational logic to ORM (but of course if it turns out we can cheaply get 100 arbitrary results and more easily juggle with them ourselves / on the (direct) python side, then sure.) But would such arbitrary returned results make sense? It would look just like Onionoo results, but - a (small) subset of them. Ramble #2: Onionoo is doing more or less direct (well, compiled, so efficient) regexps on fingerprint, nickname, etc. By default, "LIKE %given_nickname_part%" is again (sequentially) expensive; Postgres does offer full text search extensions (would need to build additional tables, etc.), and I think this makes sense to be done; it would cover all our "can supply a subset of :param" bases. I'll see into this. (It's also possible to construct functional(istic) indexes, e.g. with regexp, but need to know the template - I wonder if using Onionoo's regexpressions would work / make sense - will see.) For now, LIKE/= exact_string is very nice, but of course the whole idea is that it'd be possible to supply substrings. Of note is the fact that in this case, LIKE %substring% is O(n) in the sense that query time correlates with row count, afaict. As of now, full text search extensions would solve the problem I think, even if they may look a bit like an overkill at first. End of an email without a proper direction/point. :) [1]: https://gitweb.torproject.org/onionoo.git/blob/HEAD:/src/org/torproject/oni…

3 6

Help with rebasing arma's n23-5 tor branch to current master
by Karsten Loesing 30 Jul '13

30 Jul '13

Dear tor-devs, is anyone here up for a coding task that could help us research performance improvements of the N23 design more? The situation is that we already have a branch (n23-5 in arma's public repository), but it's based on 0.2.4.3-alpha-dev and needs to be rebased to current master. In theory, it's as simple as the following steps: $ git clone https://git.torproject.org/tor.git $ cd tor/ $ git remote add arma https://git.torproject.org/arma/tor.git $ git fetch arma $ git checkout -b n23-5 arma/n23-5 $ git fetch origin $ git rebase origin/master (clean up the mess) $ git add $ git commit $ git rebase --continue (back to clean-up-the-mess step until git is happy) $ git push public n23-5 Bonus points if you make sure the branch compiles with gcc warnings enabled, appeases make check-spaces, and runs peacefully in a private Chutney network. Unfortunately, the n23-5 branch touches a few places in the tor code that have been refactored in current master, including Andrea's connection/channel rewrite. It might be necessary to dive into the channel thing in order to get this rebase right. Once we have a refactored n23-5 branch, I'll try to simulate it in Shadow. For a tiny bit of context, this is for our sponsor F item 13: https://trac.torproject.org/projects/tor/wiki/org/sponsors/SponsorF/Year3 I'm asking here, because the usual suspects are already overloaded with other stuff. As usual, I guess. Thanks, Karsten

2 5

TorPylle: a Python/Scapy TOR protocol implementation
by pierre.lalet＠cea.fr 30 Jul '13

30 Jul '13

Hi all, To test the Tor program, I though an independent implementation might help. I started writing TorPylle with that in mind. The purpose is NOT to implement a secure or robust implementation that could be an alternative to Tor. It relies on Scapy (http://www.secdev.org/projects/scapy/) and is supposed to be used more or less the same way. The code is here : https://github.com/cea-sec/TorPylle and includes an example file. This is an early development stage. Comments, fixes and questions welcome ! Pierre

4 7