- tor-dev - lists.torproject.org

GSoc update #2 - Stegotorus security enhancement
by Noah Rahman 24 Jun '14

24 Jun '14

Hello tor-dev. over the past two weeks I have - met with the team at SRI who have been developing further enhancements to stegotorus separately from the codebase available on github. We developed a plan to begin integrating the two codebases and to prepare it for integration with tbb and gitian builds as well (isis will hopefully advise us on the latter especially). To this end, I have established an Asana workspace specifically to cover the tasks needed for smooth integration (as opposed to the somewhat out of date individual tracs both sides were keeping) - implemented the Elligator handshake on a circuit level. I need to look into implementing it on a connection level to add greater security, as well as adding OCB encryption (does anyone know for sure whether this is in OpenSSL now? I can't seem to find out) - merged the config file code from SRI into our codebase. Multiple passphrase support should be added. - resolved nagging problems with our SWF Steganographic module architecture; the lessons learned should be helpful when integrating SRI's new JSON modulen. I intend to continue further work on this in person over the next month as well as at HOPE @ NYC in July, and a focused week in person with my mentor there should greatly speed progress. Best Noah R.

2 1

Journey to the core of Tor: Why does Roger has so many guards?
by George Kadianakis 23 Jun '14

23 Jun '14

During our meeting in Iceland, we talked a lot about guard nodes. Some of that discussion eventually turned into proposal 236 [0]. During our discussions, we looked into the state file of Roger, and we noticed that there are 50 or so guard nodes in there. And that made us wonder: "Why does Roger have so many guards?". Roger is not the problem in this case; my state file also has many guards. Most people who don't use bridges or hardcoded EntryNodes have shitloads of guards. This post tries to explain why. So, Tor, in its memory, has an ordered list of entry guards (the global `entry_guards` smartlist in `src/or/entrynodes.c`). This list can be lengthy: it usually contains more than $NumEntryGuards entry guards. You can see this beautiful list just on your right below that beautiful stalagmite: https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… This happens because in its first startup, Tor adds $NumEntryGuards nodes to that list. However if one of them is not Stable and Tor needs to build a Stable circuit, Tor will need to append a Stable guard to the list. Similarly, if one of the guards is down, Tor will need to compensate for that and append [1] one more guard to the list. The same happens if Tor needs to fetch directory documents, but its guards are not directory mirrors. So, if Tor walks to the end of the guard node list and it still hasn't found enough guard nodes with the needed property to make a pick, it picks a random entry guard from the consensus and adds it to the list. It's amazing and yet real, look straight ahead (and don't look directly into the light): https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… But this still does not explain why Roger has so many guards. Usually a list of 5 or 6 nice guards is sufficient to satisfy the needs of any circuit (alive, stable, fast, directory mirror). The reason for Roger's surplus of guards, is the following very interesting functionality of Tor: Consider the following scenario, you start Tor while your network is down, Tor starts picking nodes from your list and attempts to connect to them. All connections fail, since your network is down. So now, Tor needs to add a new guard node to the list. There are two cases now: If Tor fails to connect to this new guard node (your network is still down), Tor removes the new guard node from the entry guard list (that's good; otherwise the list would be full of nodes added while the network is down). Look on your left, you can see this beautiful phenomenon happening here: https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… However, let's say that your network is back up, and Tor manages to connect to this new guard node! That's great! But should Tor keep the connection to this guard? The answer is probably that it shouldn't: Tor should recognize this problem and attempt to reconnect to the primary guards on the top of the list. And that's exactly what Tor does. Nature is truly amazing! Just relax and witness this behavior happening right in front of your eyes: https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… So, when Tor manages to connect to this newly added entry guard, it assumes that the network is back, and walks through the list of entry guards and marks them all as "needs to be retried". It also marks the connection to the new entry guard as rotten and kills it. This to me is very interesting, because it ensures that the primary guards (the ones at the top of the list) are going to be tried again after the network is back up; otherwise we would leak connections to new guards all the time! And all that fluff is related to this post, because this new guard (that made us realise that the network is back up) actually stays in our guard list. So, basically every time the network goes down and Tor does this little dance, a new entry guard is appended to our list and our statefile. And that's why Roger has so many guards! Or at least, that's why *I* have so many guards [2]. Apart from this being wonderful on its own, there are two interesting points here: a) There is always a bug: As this thing happens more times, our guard list gets bigger and the time to walk it increases. Dig this race condition: Tor starts up with the network being down, so the connections to our primary guards fail, but the network comes back while we are walking our entry guard list and trying to connect to the rest of our guards. If we manage to connect to one of the guards in our list (the lucky guard), the code at https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… doesn't get triggered because `first_contact` is not true (that node was already in the guard node list). So, we stick with that lucky guard even though it's not our primary guard, and since the network is back up, a connection to our primary guards would work too. What stinks here is that all the guards above that lucky guard are marked as unreachable, so next time Tor starts up, it will ignore them and jump directly to the lucky guard. This probably needs to be fixed somehow. I opened trac ticket #12450 for this issue [3]. b) While writing proposal 236 we were thinking about how new guard nodes should be picked. Should we pick new guard nodes at the point they are needed? Or should we pick a surplus of guard nodes in the beginning, and then when the primary ones expire, we use the extra ones? You can read more about this behavior here: https://gitweb.torproject.org/torspec.git/blob/2ecd06fcfd883e8c760f0694f359… The insight here is that apparently we are already doing the latter approach, because all these guard nodes that get added when our network goes back up will remain in our guard list. And when our primary guards expire, the ones on the bottom will rise on the top (till they expire themselves). So if you are wondering "when does Tor add new entry guards?", the answer is "when you move your laptop to a new location; just before you connect to the wifi" ;) Greetings from the core, have a good day! [0]: https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/236-single-g… [1]: Note that the word "append" is vital here. The extra guards are appended to the end of the list, and when Tor wants to pick a guard node it walks the list from the top. So, these newly added guards have lower priority so to say (most of them will not even be considered if the ones above are sufficient for building a circuit). [2]: Here is a grep of my logs. Look at how the guard counter increments by one everytime we hit https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… $ zgrep "Marking earlier" /var/log/tor/notices.log.3.gz [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 0/2 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 0/3 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 3/4 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 4/5 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 5/6 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 8/9 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 6/8 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 7/9 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 8/10 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 9/11 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 10/12 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 11/13 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 12/14 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 13/15 entry guards usable/new. [3]: https://trac.torproject.org/projects/tor/ticket/12450#ticket

1 0

GSoC: BridgeDB Twitter Distributor report
by Kostas Jakeliunas 21 Jun '14

21 Jun '14

Hi all, in the past couple of weeks I've been doing more of the same - namely, fleshing out churn control in the bot; finishing a generic challenge-response system (I'm also now considering making it into a Zope Interface); subclassed text-based challenge-response; incorporating isis' IRequestBridges and BridgeRequest's into the bot's bridge request processing part; and fake bridge-line-from-descriptor generator within the bot (didn't really do much re: the latter.) Unfortunately, all those parts are not yet ready for redeployment of the bot, and are either buggy or not finished (inclusion/use of BridgeRequests.) This is partly due to me having a bit less time in the last two weeks (a fault of my own; on the plus side, I've learned to use the soldering iron properly!) My plan is to finish the things that are near completion, and do another midterm-worthy status update "very soon." I'll also be present during the developer meeting hackdays (2th-4th), and hope to use them to flesh out ideas, etc. with isis/sysrqb. -- Kostas. 0x0e5dce45 @ pgp.mit.edu

1 0

PT Combiner GSoC Second Report
by quinn jarrell 21 Jun '14

21 Jun '14

Hi, I'm the GSoC student working on creating a pluggable transport combiner. Progress: 1. I worked on modifying the SOCKS protocol to save the requested address and redirect the requester to a different address. 2. The pt combiner client side now starts a new chain of proxies in between the pts per connection instead of having to create a new set of pts per connection. 3. The pt proxies now shut down their ports after a single connection is established, lowering the number of ports needed. 4. Tested using authenticated pts and added a better logging mechanism. Quinn

1 0

[GSoC 2014] Revamp GetTor status report #2
by Israel Leiva 21 Jun '14

21 Jun '14

Hi all! This is my second status report for the Revamp GetTor GSoC project. I'll summarize my progress since the first report. For more details feel free to ask, or if you have any comments, please tell me. Achievments: * Implemented a script that interacts with Dropbox. * Integrated the dropbox script with GetTor's core module. Now it's possible to automatically upload bundles from a local directory to dropbox and write its download urls to a specific file, which is used by the Core module to obtain links for a certain operating system and locale (e.g. linux-en). The current dropbox.links was generated by dropbox.py, and if you run core_demo.py (with 'linux' and 'es') it will get you those links. * Added logging within the Core module, with one general file (all logs) and separated log files for each level. Others: * Started a discussion [2] about sending links without SSL. I didn't get many answers. My mentors and I agreed to stay with HTTPS links only. Goal(s) until next report: * Do some cleaning, better error detection and better documentation for the current code. * Finish some minor stuff pending on the current code. * Hava a basic SMTP or XMPP module working with the Core module. You can check my progress on [0] and [1]. That's all for now. Have a nice weekend! [0] https://github.com/ileiva/gettor/tree/master/spe <https://github.com/ileiva/gettor/tree/master/spec>c [1] https://github.com/ileiva/gettor/tree/master/sr <https://github.com/ileiva/gettor/tree/master/src>c [2] https://lists.torproject.org/pipermail/tor-dev/2014-June/006977.html -- israel

1 0

GSoC Report - Zack Mullaly - HTTPS Everywhere secure ruleset update mechanism
by Red 20 Jun '14

20 Jun '14

Hello again everyone, I am here once again to report my progress in building a secure ruleset updating mechanism for the HTTPS Everywhere browser extension. This past week's work has involved a lot of updates to the specification for the mechanism[1], and changing a lot of what I had done before to keep up with all of the changes. With regards to the documentation, a great deal of the changes being discussed about the order that tests of authenticity of the update information as well as its relevant have been finalized. There is now a "Pseudocode of the update procedure" section that outlines this order that tests are performed and also gives an idea of what data the mechanism requires, where it's obtained from, and how it changes that data. I have also added extensively to the "update.json and update.json.sig" section to outline everything anyone should need to know from generating update.json to getting a properly-encoded signature of the digest of update.json that can be verified by the extension. The document is now also included as part of the repository so that pull requests and better commenting can be made. I have also been making a lot of improvements to the utility script used for building the content of update.json[2]. The script has been completely revamped to work in accordance with the modified spec. I have also included a few sanity checks to verify that the content provided for update.json make sense, and made it very easy to write more such checks. I spent most of today getting the actual ruleset updater module[3][4] code up to date. The code now reflects the changes to the new standard and has been simplified dramatically. As usual, I have my meeting notes available as a Gist[5]. The notes from the meeting I had today with Yan are already present. My plans for the coming week are to set up a testing environment and begin making sure everything works as expected. I will also begin the discussion about how the extension should handle error reporting- what to do when authenticity tests fail, or the extension can't fetch the update information. Please feel free to take advantage of the update spec being part of the repository now to discuss the details and/or propose changes in the form of a pull request. Either this or the https-everywhere mailing list are also great places to get in touch with myself and everyone who has been involved in my work. Cheers, Zack [1] : https://github.com/redwire/https-everywhere/blob/makeJSONManifest/doc/updat… [2] : https://github.com/redwire/https-everywhere/blob/makeJSONManifest/utils/rul… [3] : https://github.com/redwire/https-everywhere/blob/rulesetUpdating/src/chrome… [4] : https://github.com/redwire/https-everywhere/commit/1a0d5edb8572180854ad620f… [5] : https://gist.github.com/redwire/b62f03905a826e79947a#week-6

1 0

GSoC : Rewrite Tor Weather (Status Update 2)
by Sreenatha Bhatlapenumarthi 20 Jun '14

20 Jun '14

Hi all, As a part of the weather-rewrite project, I've been working on the onionoo-wrapper module[0] since my last update. I managed to write a couple of utility methods to the wrapper like *is_stable*, *is_exit_relay*. Removed the caching feature(using pymemcache) temporarily to ease the testing process on meejah's advice. This will be picked up later from Lukas Erlacher's Onion-Py. Wrote some preliminary unit tests for the wrapper using python's mock module. Also initiated the main backend work with a new implementation of the welcome script using the Onionoo wrapper[1]. It currently fetches details document from Onionoo, checks for new relays, ensures their operators haven't been sent an email yet and then updates the database accordingly before sending them a welcome email. This is my first script regarding the rewrite that interacts with the django components of the previous weather. I'll be finishing up the remaining scripts as well soon enough before the next update and then move on to testing and documentation. As always, if you can take a look at the work and/or provide some feedback that'd be great. [0] - https://trac.torproject.org/projects/tor/ticket/12085 [1] - https://trac.torproject.org/projects/tor/ticket/11081 [2] - https://trac.torproject.org/projects/tor/ticket/11084 Cheers, Sreenatha

1 0

Orbot & Orfox - GSoC bi-weekly report 2
by Amogh Pradeep 20 Jun '14

20 Jun '14

Hey everyone, This is my second bi-weekly report and I am working on Orbot & Orfox. Status Report 2 June 20th, 2014. What I have done: 1) Expanded notification work: Added my finishing touches to the new expanded notification that has been planned to be added to Orbot, got exit IP address working (please do test this if possible, https://github.com/amoghbl1/orbot-1 branch- new_notification_expanded). Found a new layout for the expanded notification to make it look like the original notification that the system shows. 2) Build Fennec: My project timeline now moves on to working on Orfox, as we have decided to make Orfox tor's blend of the firefox for android browser, I am currently working on building firefox so that we may edit their code and make our browser. Hopefully, I can start hacking on the code as of next week, once I get it built. On another note, I will be attending the Summer dev meet in Paris and I'm excited to meet the community :) Best, Amogh Pradeep

1 0

lcov test coverage reports
by David Murray 20 Jun '14

20 Jun '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, Would there be interest in using lcov [0] for test coverage analysis? Obviously, as lcov is an external pacakge, the existing scripts that only use built in gcov commands must remain. However, if I were to create a script that generates a pretty, HTML coverage report using lcov & genhtml, would this be something others would use? And how would you like to see this implemented? As a custom make target (e.g., make htmlcoverage + autotools black magic to silently disable/hide the target if we can't find lcov), or as a standalone script (i.e., ./scripts/test/run_lcov.sh). For what it's worth, I've manually run lcov on tor after enabling coverage through ./configure, and the output seems sane, so I think from a technical standpoint this would work. [0]: http://ltp.sourceforge.net/coverage/lcov.php Cheers, daube -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJTnYryAAoJEBfCjrGktO5q+FgP/26ZgVZOAKSOBvWr6z2HYEku 1Yo+xFqTXk/Fw6yilv6Z4Jwc0Ar2B/VjHfLpOL1Vm+hFZ+VYsObSt9JGG2baItbG IrcedfjCuyjfaP1PloZoDM24FEbXqjseQO2EZ8bXbtErsW52Z4uAR3nYYcyXCCYp 1gi6lEwN/Yn0xm+fGgoz6/Z5FXvETdzbWLfhZoT4jFsnQsp/fByld55JLyft/8gA OrH3QQYGgXZkwIsnTkbLO3xWpXbqC5kU/ngCOL9WXXU99cgnW+bec7DA+cHhTqh/ 7oZD54noQ9XYtOA7AhAEOiTWnJUca2vV1KFPVBKSlqfkOhF3BRANHo2cIpsvgeBK 9+tcTyuwcFGAJVmW9l/AN3Y89CepCxnk6B1SSLFxuUVupAz2ilBM7KVtgj2ilJnC 2GjfuQB0RG/FW0CSEaa+DDtHfzCMfZNwGb6Fs1L41P8NnlGVFOz11NLlMG3t5ZpT 0OrncYaeVz74H+63Ig1ZwOjlfgUr9BHMLeYdwn9FmqIggyFRt1Z5PnNCaMuExs/v WQbLMP0oZ9XK1f51vfFVFYk/n1nh1c4J40/3I5B525GANAZ3fxTDLp0oSp/gNr5X lXPaF32KexhOiup5n1d6x8oJPzeNmex5CfcT29bZMPavF8p8v0OdZrUtU40bG7nO PUlA3EFfva1YW0aTRKY5 =+v+h -----END PGP SIGNATURE-----

2 3

GSoc Status Report Muticore Daemon
by Белоус Михаил 20 Jun '14

20 Jun '14

Hello everyone, This is second Google summer of Code status report, I implemented threadpool for tasks and create simple benchmark for threadpool. It pass test for workqueue, but I will write new test for new threadpool architecture. After my return I will learn more about work of daemon and rewrite bottle necks using my thread and write new benchmarks for this places. Sincerely your, Mikhail

1 0

Pluggable transports meeting today (16:00UTC Friday 20th of June 2014)
by George Kadianakis 20 Jun '14

20 Jun '14

Hello friends, just wanted to remind you that the regular biweekly pluggable transports meeting is going to occur today at 16:00 UTC. Place is the #tor-dev IRC channel in the OFTC network. Thanks!

1 0

getting clock skew on demand rather than by event?
by Patrick Schleizer 19 Jun '14

19 Jun '14

Hi, whonixcheck is a diagnostic utility that can be run on demand. Not a Tor controller, that is supposed to be run continuously. It checks things such as if Tor's pid is running, bootstrap status, DisableNetwork 1, tor --verify-config and more. There is one thing in the logs, that would be worth notify a user about it: [warn] Our clock is 4 hours, 47 minutes behind the time published in the consensus network status document [or clock too fast] Now, parsing the log sounds like an unclean solution. Tor control protocol can tell about CLOCK_SKEW, but it's "only" an event. And "event" sounds to me, only a controller that is continuously running would be capable to learn about it. Is it possible to use stem for something like:, "Tor, tell me right now about your latest info on current clock skew"? Similar to tor_bootstrap_check.py? Cheers, Patrick [1] https://github.com/Whonix/anon-shared-helper-scripts/blob/master/usr/lib/an…

2 1

[GSoC] Consensus diffs - Second report
by Daniel Martí 19 Jun '14

19 Jun '14

Hello everyone, This is the second status report of my Google Summer of Code project, which is to implement consensus diffs for Tor. We hold weekly meetings every Wednesday at 14h UTC with the project mentors Nick and Sebastian. When I sent my first report I was a bit behind schedule as per my own timeline, mainly due to finals. These past two weeks I've been catching up on that timeline. I am now finishing the first part of it, which is to implement the diff generation code. If you read the last report, you might remember that the biggest issue at the time was performance. The code I had worked, but it was very pricy due to the quadratic time it took to compute a consensus diff, translating in running times from 10 to 20 seconds. In that e-mail I explained an idea I had in mind to overcome that. It got implemented and tested, and it works as expected. Now a diff generation between two consensus files takes roughly 0.04 seconds, which is almost the same amount of time that 'diff -e' takes. No precision seems to have been lost, but we'll have to do some edge case tests to verify that. The code was also missing documentation and some tor_asserts to make it more robust and understandable. That is what I'm finishing up at the moment, and then the tests will be the only task left. On to the second timeline item, which is the diff application code - The code sample I wrote for my application was a very simple version of it, supporting only 'delete' ed commands and just writing to stdout line-by-line. Taking that code as a starting point, plus being able to re-use some chunks from the diff generation code, I was able to have a working version in a few hours. So, like the diff generation bit, now that it seems to work properly it's time to document it properly and do some tests. I have until July 5th to finish these two tasks as per my own timeline, so I should be well able to have them done by then. The upcoming timeline item will be the joining of these chunks of code with Tor, at which point I expect some issues to arise. So I would like to get started with that bit the sooner the better, to have more time to get everything working and tested before we run out of time. As usual, any comments or ideas are very welcome. -- Daniel Martí - mvdan(a)mvdan.cc - http://mvdan.cc/ PGP: A9DA 13CD F7A1 4ACD D3DE E530 F4CA FFDB 4348 041C

1 0

Tor Geolocating exit nodes.
by JP Wulf 19 Jun '14

19 Jun '14

So Griffin Boyce is canvasing for some input to improve Tor, specifially for Journalists. https://twitter.com/abditum/status/479052228138119168 1. It is known that various actors are trying to compromise Tor comms by establishing own exit nodes. With enough nodes, they can break Tor (see slides). 2. Idea: Is it possible to allow the end user to determine the geo-location (with various degrees of fine tuning from hemisphere, through continental, to top country domain to regional? (I have NFI about the inner workings of TOR protocol and new work on it) For example. Say a journalist in Russia is using Tor, s/he declares in their tor client, that they only want to use exit nodes in South America and Australia. Thus minimising the chance the nodes are owned. This geolocation could perhaps be used to validate the integrity of the nodes (how I dont know, maybe by establishing TOR honeypots that can only be compromised through traffic through a compromised (owned) exit node). Risk: This is a rats nest, because if implemented incorrectly it may allow hostile actors to direct exit nodes to those that are owned. Thanks for reading my fiction. Maybe its useful in the light of what Griffin is asking about. -- JP Wulf Problem Solution Engineering http://nomeonastiq.com/

4 3

Composing multiple pluggable transports
by Ximin Luo 18 Jun '14

18 Jun '14

Hi Steven, Nikita, I was told that you two are interested in the idea of composing multiple PTs together. Here are our ideas on it. We have a GSoC student, Quinn also at Illinois, working on turning this into reality. ## Concepts On the most abstract level, pt-spec.txt defines an "input interface" to some generic component. It consists of the following: - dest addr, of the Bridge - headers/metadata, such as fingerprint[1] or other PT-level settings - data, the actual application-layer stream, such as OR protocol The concrete form of this is the SOCKS protocol, which allows tor to make a request with the above interface. (Actually, SOCKS does not fully support metadata, which means we've had to extend it ourselves. HTTP might have worked better.) pt-spec.txt does not specify the "output interface". This means it's impossible to chain general PTs, because there's nothing defined to chain. To work around this, we observe that in practise, many PTs follow the below "output interface"; we'll call these "direct PTs": - data, sent directly to the (TCP) address given at the input Direct PTs include obfsproxy, scramblesuit, fteproxy. Indirect PTs are all other PTs, that do something different other than a straight TCP connection *to the endpoint Bridge address*. These include flashproxy and meek. ## Design Our combiner will chain up a sequence of direct PTs, then the last PT can be any PT (either direct or indirect). So for example it could potentially support obfs3|fte|fte|fte|flashproxy and obfs3|fte but not flashproxy|obfs3|obfs3|meek. Not every chain makes sense from a security viewpoint, of course. Because the output interface (TCP) does not exactly match the input interface (SOCKS), we have a component called a "shim", which has an input interface of TCP and an output interface of SOCKS. This is placed between each pair of PTs in the chain. In simple terms, it works like this: pt0 (out) -TCP-> (in) shim1 (out) -SOCKS-to-next-shim-> pt1 -> to shim2 The extra info present in SOCKS (dest addr and metadata) absent from TCP, will be supplied by the combiner, as described in the next section. Also, in practise, these shims are within the same process as the combiner, there is no need to start a new process for these. ## Algorithm Let the ith PT be listening on port pPT[i], done at program start. Later, when tor wants to connect to a PT-chain, we intercept this connection, extracting the following information (the SOCKS in-interface): - dest addr, of the Bridge - headers. generic headers, plus PT-specific headers for each PT in the chain[2] - data, OR protocol Then, the combiner starts a new shim, one for each component in the chain, each listening on pS[i]. Each shim[i] is set-up so that when it receives a connection on pS[i], it tells PT[i] (i.e. the SOCKS client listening on pPT[i]) to connect to pS[i+1], with the metadata set to the generic headers plus specific headers for PT[i], and the data set to whatever it receives from the connection. (In practise the shims are set-up in reverse order, because shim[i] needs to know what pS[i+1] is, and we want to take advantage of OS's feature to "listen on any port".) Special cases: - The last (i.e. (n-1)th) shim tells its SOCKS client to connect to the original dest addr, of the Bridge - there is no pS[last+1]. - The first shim does not need to exist, since the combiner is just sending data to itself so can do this in-process After all the shims are set up, the combiner starts forwarding data from tor over onto PT[0]. Then the magic is complete. ASCII diagram, minus annotations about metadata: +----------+ socks | PT | [ tor ] to >-------> | combiner | bridge +-< | | | +----------+ in-process | v +-------+ socks | direct| tcp to >-> | PT[0] | to >-+ pS[1] | | pS[1] | +-------+ | +-------------------<-------------------+ | +-------+ | socks | direct| tcp +->[ shim[1] ] to >-> | PT[1] | to >-+ pS[2] | | pS[2] | +-------+ | +-------------------<-------------------+ | | [etc] | | +-------------------<-------------------+ | +-------+ | socks | direct| tcp +->[ shim[y] ] to >-> | PT[y] | to >-+ pS[z] | | pS[z] | +-------+ | +-------------------<-------------------+ | +-------+ | socks | any | +->[ shim[z] ] to >-> | PT[z] | whatever it wants --> bridge | | +-------+ Note that, if for whatever reason your chain has the same PT in multiple positions, the chain will re-use the same PT process. Everything should still work, because we have separate *shims* for each *position* that the PT appears within the chain. X [1] fingerprint not actually currently given to PTs by tor; but it should be for reasons argued elsewhere [2] we haven't exactly defined a format for this, but perhaps something like k=v for generic headers to all children (as currently), and x-chain-0-k=v for PT-specific headers, and maybe even something like x-chain-0onwards-k=v. -- GPG: 4096R/1318EFAC5FBBDBCE git://github.com/infinity0/pubkeys.git

2 1

Reminder: tor dev meeting this (and every) Wednesday, 1900 UTC
by Nick Mathewson 17 Jun '14

17 Jun '14

Hi, all! Here's your regular reminder for the weekly IRC meeting for people working on the program "tor". (This won't cover all the other programs developed under the Tor umbrella.) The next meeting time will be: Wednesday June 18, 19:00 UTC. (That's 3pm EDT and 12:00 noon PDT.) As usual, we'll do it on the #tor-dev IRC channel. I've seen a drop-off in the number of people coming since I stopped sending these out weekly, so I'm sending them again for a while. Among other stuff, I expect we'll be looking at tickets for 0.2.6.x. cheers, -- Nick Mathewson

1 0

ANN: OnionPy 0.1.5
by Lukas Erlacher 17 Jun '14

17 Jun '14

Hello all, my OnionOO python wrapper has now matured significantly. While I was gone for around a month due to being otherwise occupied, I have now given OnionPy a lot of polish. The new features: - Python2 compatibility (All it took for that was to change the readme) - Encapsulated caching layer that includes a memcached interface as well as a simple python dict caching mechanism, or you can add glue code for your favourite caching backend yourself - It's on Pypi! [1] - Extended test / demonstration script Many thanks to meejah for his help. If you are planning to make something in python that uses the tor network status, accessing OnionOO [2] using OnionPy might be exactly what you need. If you are interested, check out the repo and readme! [3] Of course, criticism, suggestions, and pull requests are always welcome. Best regards, Luke [1] https://pypi.python.org/pypi/OnionPy [2] https://onionoo.torproject.org/ [3] https://github.com/duk3luk3/onion-py

2 3

Running Tor with static path
by mahdi 16 Jun '14

16 Jun '14

Hi all, I want to execute an experiment on Tor in which i need to fix the ip address of entry,relay and exit onion router. For that i need to determine the IPs and keys of ORs in OP permanently. Is there any idea of what function of Tor's code should be replace?! Thanks a lot

4 3

Re: [tor-dev] [HTTPS-Everywhere] GSoC report - Zack Mullaly - HTTPS Everywhere secure ruleset update mechanism
by Yan Zhu 15 Jun '14

15 Jun '14

(Trying again with my tp.o email address) On 06/15/2014 02:05 PM, Yan Zhu wrote: > It's unclear whether this message went through to tor-dev (can't find it > in the archives), but I've added this update to > https://trac.torproject.org/projects/tor/wiki/doc/gsoc. > > On 06/13/2014 05:06 PM, Red wrote: >> Hello, everyone! >> I apologize for the fact that this is coming in late, but here is a >> summary of my progress and plans thus far in developing a secure ruleset >> update mechanism for the HTTPS Everywhere browser extension. >> >> The specification document detailing how the ruleset updater will >> function has been perhaps the greatest focus for me until now. The >> document is currently hosted on Github as a gist[1], and currently >> details the format for the JSON document the extension will fetch to >> determine whether the update information it receives is authentic and >> relevant. >> >> A second task I have been working on is the creation of a utility[2] >> used to automate much of the process of building the update.json file >> contents outlined by [1]. A lot of the work done here so far has been >> experimental, but it is already providing some utility for composing >> data that can be used for testing purposes. >> >> The third thing I have been working on is the actual implementation of >> the ruleset updater[3]. There are to be some changes to the spec that >> will be reflected in this code in the coming week, but the >> implementation so far is very close to being ready to test. >> >> In the last week, a lot of discussion has occurred centered around >> improving the specification for the ruleset update mechanism and how the >> update.json file and signing thereof should function and be written. I >> have posted my weekly meeting notes to another gist[4] which I will from >> today onwards be keeping up to date with my weekly notes so that they >> will be publicly available and well-formatted. In summary, my upcoming >> work will involve updating the update.json spec to reflect the >> discussion being had on the https-everywhere mailing list and between >> myself and my mentor, Yan. I will then focus on updating the extension >> code as well as the utility I have been working on to reflect the >> changes to the spec. I will then move on to testing the signature >> verification method locally by creating example documents and a Python >> script to verify the signature. I will also be setting up a testing >> environment to properly test my work on the ruleset update mechanism. >> >> My work can be more closely followed on Github- specifically, my fork of >> the official HTTPS-Everywhere repository[5]. The code I have been >> working on resides in my "makeJSONManifest" and "rulesetUpdating" >> branches. You can also follow the discussion on the https-everywhere >> mailing list, and are welcome to join in mine and Yan's weekly meetings >> in #https-everywhere on irc.oftc.net at 11:00AM Pacific Time on >> Fridays. We're happy to have people chime in with ideas, and commentary >> in IRC, the mailing list, and on Github is welcome! >> >> [1]: https://gist.github.com/redwire/2e1d8377ea58e43edb40 >> [2]: >> https://github.com/redwire/https-everywhere/blob/makeJSONManifest/utils/rul… >> [3]: >> https://github.com/redwire/https-everywhere/blob/rulesetUpdating/src/chrome… >> [4]: https://gist.github.com/redwire/b62f03905a826e79947a >> [5]: https://github.com/redwire/https-everywhere >> >> >> >> _______________________________________________ >> HTTPS-Everywhere mailing list >> HTTPS-Everywhere(a)lists.eff.org >> https://lists.eff.org/mailman/listinfo/https-everywhere >> > > -- Yan Zhu <yan(a)eff.org>, <yan(a)torproject.org> Staff Technologist Electronic Frontier Foundation https://www.eff.org 815 Eddy Street, San Francisco, CA 94109 +1 415 436 9333 x134

1 0

Shutting down relay-search service shrinks metrics database by almost 97%
by Karsten Loesing 15 Jun '14

15 Jun '14

Hi devs, I have been planning to shut down the relay-search service, that was part of the metrics website, for about half a year, and now it's gone: https://lists.torproject.org/pipermail/tor-dev/2013-December/005948.html https://metrics.torproject.org/relay-search.html It was a good service. The main reason for shutting it down was that it required us to keep a full database of relays. And the Tor network has grown quite a bit over time. Now that the service is gone I was finally able to clean up the metrics database. Here are some numbers from before and after the cleanup with all sizes being rounded GiB: table | before | after --------------+---------------- bwhist | 13 | 0 descriptor | 6 | 0 dirreq_stats | 10 | 0 statusentry | 67 | 2 ... | | ------------------------------- total | 95 | 3 So, 92 of 95 GiB are gone, simply because we don't need that data anymore. That's much more than I had expected. Almost 97%! Just in case, and I'm going to watch over this, too: if the metrics website shows you funny numbers in the next couple of days, please let me know. All the best, Karsten

1 0

ANN: txtorcon 0.10.0
by meejah 15 Jun '14

15 Jun '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm excited to announce txtorcon 0.10.0 which adds support for Twisted's endpoint strings. This means that ANY Twisted program that uses endpoints can accept "onion:" strings to bring up a hidden services easily (by launching a new Tor instance). Typically, no code changes to the application should be needed (just "pip install txtorcon"). "twistd" supports endpoints, so for example to serve some Web content from ~/public_html as a hidden-serivce, we can do this (with txtorcon installed): twistd web --port onion:80 --path ~/public_html Some examples of other valid "onion:" endpoint strings: onion:80:hiddenServiceDir=/dev/shm/hidsrv onion:80:controlPort=9050:hiddenServiceDir=/srv/hidden The first allows specifying existing hidden service keys and the second says to connect to an already-running Tor instance. Thanks to David Stainton (dawuud) for the initial pull-request (and continued collaboration) that made this happen. There is a complete demonstration of the power of this Fully Operational endpoint-station here: https://txtorcon.readthedocs.org/en/latest/howtos.html#endpoints-enable-tor… You can download the release from PyPI or GitHub (or of course use "pip install txtorcon"): https://pypi.python.org/pypi/txtorcon/0.10.0 https://github.com/meejah/txtorcon/releases/tag/v0.10.0 sha256sum reports: 910ff3216035de0a779cfc167c0545266ff1f26687b163fc4655f298aca52d74 txtorcon-0.10.0-py2-none-any.whl c93f3d0f21d53c6b4c1521fc8d9dc2c9aff4a9f60497becea207d1738fa78279 txtorcon-0.10.0.tar.gz thanks, meejah -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJTnTM5AAoJEMJgKAMSgGmnB5EIAIJACHhd7PEWYfCZmB8lL1pr J4rrhUuXD1iI7zJJL/rmS/SZLL+34JoKke6iQsBRGYzU0GRK2JQkFAeMq++AMdMu QvoTg745OhnMSbZyCyS6buN/NhAVcmD3GYI9h8TK60jfUPEFn7+sDstMG4OePdgq X0QQHBuJq9XtDwEmRWFHq2Aht//7J2DUliPPjtwT3C4FhPT5pXxasvfQA4jOYxsO jTja6UE4/4TnonV5gB/chijp60cKvVxFQi86mKuNnVaKTZu+QCZFJtXkr/DbRSxv CJ+z7jJ9s8xhfQ5Nkv9pkVAreH6w9bemc/iijKlZmQGYvLBTsvISPATyWGtn1f4= =wIVZ -----END PGP SIGNATURE-----

1 0

Re: [tor-dev] Tor Cloud maintenance - Question
by Sina Rabbani 13 Jun '14

13 Jun '14

Dear Team, I started building new images for Amazon EC2, as of last night. I noticed that obf2 and obf3 ports are hard-coded and the same for all Tor instances on Amazon. Is there a specific reason for this? If not, can I randomize the ports? All the best, SiNA On 2014-06-11 23:50, Karsten Loesing wrote: > Hi Sina, > > please find the following conversation below from #tor-project. Any > thoughts? > > Thanks, > Karsten > > > 06:43:15 < Lunar^> We really need a maintainer for Tor Cloud. Thre's > some of the support tickets we are getting I have > no idea how to answer… > 06:45:20 < karsten> > > https://trac.torproject.org/projects/tor/wiki/org/operations/Infrastructure > says inf0 and Runa are the maintainers. > 06:45:55 < karsten> is that not the case? should we mark it as > unmaintained and threaten to turn it off three > months later? > 06:47:01 < arma1> sina said he would pick it up > 06:47:04 < arma1> but i don't know that he ever did > 06:47:13 < karsten> sina = inf0? > 06:47:18 < arma1> perhaps karsten could mail him and ask if he is > maintaining it or if we should turn it off? > 06:47:19 < arma1> yes > 06:47:28 < karsten> sure, I can do that. > 06:47:57 < arma1> thanks! -- "be the change we wish to see in the world." Gandhi

2 3

Re: [tor-dev] Proposal for improving social incentives for relay operators
by Virgil Griffith 11 Jun '14

11 Jun '14

> Also, theconcept of naming authorities is about to be phased out [1], so > better not build new services that rely on nicknames. Karsten I love you. Not only do you have fine ideas, you are the greatest feedback provider in the world. Agreed 100%. Replace key-by-nickname with key-by-fingerprint. Not crazy about having a a separate Torati name space, but meh, you do what you have to. Maybe make the nicknames a version 2.0 feature. (Could always make a namespace integration with google+! Har.) > Okay. See Andrew's concerns about avoiding words having "Tor" in them. I will find a name that doesn't explicitly invoke Tor. -V

1 0

Typo in torspec.git/control-spec.txt
by Frithjof 11 Jun '14

11 Jun '14

Hi, the Tor control protocol spec says in §4.1 about asyncrhronous replies: > These replies can be sent after a corresponding SETEVENTS command has been > received. They will not be interleaved with other Reply elements, but they > can not appear between a command and its corresponding reply. > > C: SETEVENTS CIRC > S: 250 OK > C: GETCONF SOCKSPORT ORPORT > S: 650 CIRC 1000 EXTENDED moria1,moria2 > S: 250-SOCKSPORT=9050 > S: 250 ORPORT=0 If I am not reading the example wrongly, the negation in the last sentence is wrong, e.g. diff --git a/control-spec.txt b/control-spec.txt index 2dde043..70ddf3b 100644 --- a/control-spec.txt +++ b/control-spec.txt @@ -1253,7 +1253,7 @@ These replies can be sent after a corresponding SETEVENTS command has been received. They will not be interleaved with other Reply elements, but they - can not appear between a command and its corresponding reply. For example, + can appear between a command and its corresponding reply. For example, this sequence is possible: Cheers, Frithjof

2 1

Reminder: tor development meetings this (and every) Wednesday, 19:00 UTC
by Nick Mathewson 11 Jun '14

11 Jun '14

Here's your regular reminder for the weekly IRC meeting for people working on the program "tor". (This won't cover all the other programs developed under the Tor umbrella.) The next meeting time will be: Wednesday June 11, 19:00 UTC. (That's 3pm EDT and 12:00 noon PDT.) As usual, we'll do it on the #tor-dev IRC channel. I've seen a drop-off in the number of people coming since I stopped sending these out weekly, so I'm sending them again for a while. Among other stuff, I expect we'll be wrapping up (or planning to wrap up) 0.2.5.5-alpha and maybe doing some initial strategizing on 0.2.6. cheers, -- Nick

1 0