- tor-dev - lists.torproject.org

Python ExoneraTor
by Damian Johnson 25 Jun '14

25 Jun '14

Hi Karsten. This is diving into enough detail that we might as well move this over to tor-dev@. For the list's benefit, Karsten and I are discussing a Python rewrite of ExoneraTor... https://exonerator.torproject.org/ https://gitweb.torproject.org/exonerator.git First I think I need to take a step back to figure out exactly what we're after. From a quick peek at ExoneraTor it looks like it behaves as follows... a. User enters an address (IPv4 or IPv6) and a date (either for a day or an hour). b. ExoneraTor lists router status entries for all relays that match the criteria. These entries link to the consensus they came from and server descriptors they reference. c. The user can then enter a destination address and port to search exit policies in TorDNSEL entres. Step 'a' and 'b' make sense to me. Step 'c' however I'm having a little difficulty groking. Ignoring TorDNSEL entries for a moment, we already have all the ingredients to provide the user with three fields to start with... * Source Address (required) * Timestamp (required) * Destination Address and/or Port (optional) The source address and timestamp come from the consensus, and an optional 'can it exit to destination X' consults the server descriptor's exit policy. So what is TorDNSEL providing us and why is it a separate search on the page? As I understand it the value of TorDNSEL is that we can't trust the address in the router status entries. If that's the case then our present search fields don't make sense to me... * Our initial search consults consensus information for the address and timestamp but not the exit policy. This is weird both because the address this has is faulty, and we have the exit policy so we could trivially include that in our search criteria. * Our second search gives the impression that we're using the earlier consensus results to query exit criteria from TorDNSEL. As I understand it though that's not what it's doing. TorDNSEL is completely independent from the consensus information. I could understand a search that just consults consensus information (ignoring address accuracy, it has everything we need). I could also understand a search that just consults TorDNSEL information (ignoring its inconsistent poll rate, it has everything we need). However, this hybrid approach and how it's presented really confuses me. Unless I'm mistaken with something above what I'd expect from ExoneraTor is... * The three search fields mentioned above. * It shows results based on the consensus information like we presently do. * If we have TorDNSEL entries that either indicate that a relay we're presenting had a different external address or another relay had the address we're searching for then note that. That is to say, the base search is based on consensus information (using server descriptor exit policies if we want to filter by that), and the TorDNSEL results are just appended notes since we can't rely on its poll rate. Thoughts? Cheers! -Damian PS. Congratulations on getting me invested. I just spent the last three hours in front of a whiteboard trying to puzzle out why ExoneraTor works the way it presently does. ;) PPS. Stem's ExitPolicy class has a can_eixt_to() method that would be really handy for this... https://stem.torproject.org/api/exit_policy.html#stem.exit_policy.ExitPolic… PPPS. I'm still hesitant about actually tackling this project. Arm is midway through being rewritten, and considering its sudden uptick in usage probably the most important project on my plate right now. That said, I'm happy to discuss this. Even if we don't implement it right now this thread will be useful so we know where we're going with ticket #8260. Concerning the earlier discussion of 'work with Karsten on a python project' I have a personal bias toward collaborating when the project has few unknowns for me, but working alone when *I'm* learning something. That is to say, I'd love to work with you on a straightforward Stem project and I'd also like to discuss ExoneraTor's design. But when it comes to coding, this has enough unknowns that if I take it on I'd prefer to experiment alone for a while - at least until I know enough about the APIs involved that I can avoid embarrassing myself. :) On Sun, Jun 8, 2014 at 2:56 AM, Karsten Loesing <karsten(a)torproject.org> wrote: > On 08/06/14 06:27, Damian Johnson wrote: >>>> Here's a quick overview of the codebase to facilitate reading through it: >>> >>> Ahhh, very useful - thanks. >> >> Hmmm. Just took a quick peek at the ExoneraTor codebase and, unless >> I'm mistaken, it doesn't actually use metrics-lib, does it? > > You're right, looks like it doesn't. > >> Honestly >> looking over the code is making me a little hesitant to take this on >> after all. I was anticipating a small, quick project of DocTor's scope >> but I've never touched SQLAlchemy or Posgress before. > > I don't think we'll even have to touch the Postgres for moving from Java > to Python. The Python code would simply do SQL calls via its SQL > library just like Java does. > > I just copied all SQL statements that the Python part would have to > prepare and execute: > > CALL insert_descriptor(?, ?); > CALL insert_statusentry(?, ?, ?, ?, ?, ?, ?); > CALL insert_consensus(?, ?); > CALL insert_exitlistentry(?, ?, ?, ?, ?); > SELECT MIN(validafter) AS first, MAX(validafter) AS last FROM consensus; > SELECT validafter FROM consensus WHERE validafter >= ? AND validafter <= ?; > CALL search_statusentries_by_address_date(?, ?); > CALL search_addresses_in_same_24 (?, ?); > CALL search_addresses_in_same_48 (?, ?); > SELECT rawdescriptor FROM descriptor WHERE descriptor = ?; > SELECT descriptor, rawdescriptor FROM descriptor WHERE descriptor LIKE ?; > SELECT rawconsensus FROM consensus WHERE validafter = ?; > > That's it. No further knowledge about Postgres required. > >> Once I wrote this I realized I'm being a damn hypocrite. Here I was >> saying "Karsten, learn Python so we can leverage each other's >> codebases!" but then I hightail it once the project delves into areas >> new to me. New arm users are showing up almost daily on irc and I'm >> anxious to give them a new release... but then this is exactly the >> issue, isn't it? Deliverables you'd like to focus on crowding out time >> to learn new things. >> >> So TL;DR I'm gonna eat my own words and suggest we focus on our >> separate domains for now. I really would like to work on some small >> metrics projects with you. Each month I eyeball your status reports >> asking myself "Is there anything here I can work with Karsten on to >> draw our spaces closer together?" so please let me know if you run >> across anything in Metrics we can collaborate on. > > (Replying below, first replying to the DynamoDB part.) > >> Your hypocritical friend, ~Damian >> >> PS. When we next meet I'd like to discuss ExoneraTor's design a bit. >> First thought I had when looking at the code was 'huh... I wonder if >> this would be a good use case for DynamoDB'. > > I'm wary about moving to another database, especially NoSQL ones and/or > cloud-based ones. They don't magically make things faster, and Postgres > is something I understand quite well by now. And again, I think that we > keep the Postgres part entirely unchanged when moving to Python. Not > saying that DymanoDB can't be the better choice, but switching the > database is not a priority for me. > > > So, regarding the rewrite: rather than canceling the project before it > starts, how about we find a role for you that you're more comfortable with? > > For example, I'd want to try rewriting it step by step based on your > suggestion of frameworks/libraries and with some code review of yours. > > If you're interested, which framework would I use for the new Python > ExoneraTor? It's supposed to do the following tasks: > > - Provide a simple web site with a web form, backed by the PostgreSQL > database. > - Maybe offer a simple RESTful API for lookups that the web form could > use to compose responses, but that could also be used by other > applications directly. > - Return documents from the database by identifier, so without > providing a search functionality. > - Run a scheduled task once per hour that fetches data from CollecTor > and puts it in a database. > > Bonus points if the result is as easy to deploy on Debian Wheezy as > possible. Like, install these few Debian packages, run the setup > script, done. > > Of course, if you'd prefer to focus on other things and not discuss > ExoneraTor stuff, that's perfectly fine, too. :) > > All the best, > Karsten >

4 13

Reminder: tor dev IRC meeting each Wednesday (but not next week)
by Nick Mathewson 25 Jun '14

25 Jun '14

Hi, all! The next tor dev meeting is at the usual time: 19:00 UTC this Wednesday (June 25). This is the weekly IRC meeting for people working on the program "tor". (It won't cover all the other programs developed under the Tor umbrella.) (That's 3pm EDT and 12:00 noon PDT.) As usual, we'll do it on the #tor-dev IRC channel. (Also: we're not going to have one on Wednesday July 3: too many tor developers will be in France.) cheers, -- Nick

1 0

GSoc update #2 - Stegotorus security enhancement
by Noah Rahman 24 Jun '14

24 Jun '14

Hello tor-dev. over the past two weeks I have - met with the team at SRI who have been developing further enhancements to stegotorus separately from the codebase available on github. We developed a plan to begin integrating the two codebases and to prepare it for integration with tbb and gitian builds as well (isis will hopefully advise us on the latter especially). To this end, I have established an Asana workspace specifically to cover the tasks needed for smooth integration (as opposed to the somewhat out of date individual tracs both sides were keeping) - implemented the Elligator handshake on a circuit level. I need to look into implementing it on a connection level to add greater security, as well as adding OCB encryption (does anyone know for sure whether this is in OpenSSL now? I can't seem to find out) - merged the config file code from SRI into our codebase. Multiple passphrase support should be added. - resolved nagging problems with our SWF Steganographic module architecture; the lessons learned should be helpful when integrating SRI's new JSON modulen. I intend to continue further work on this in person over the next month as well as at HOPE @ NYC in July, and a focused week in person with my mentor there should greatly speed progress. Best Noah R.

2 1

Journey to the core of Tor: Why does Roger has so many guards?
by George Kadianakis 24 Jun '14

24 Jun '14

During our meeting in Iceland, we talked a lot about guard nodes. Some of that discussion eventually turned into proposal 236 [0]. During our discussions, we looked into the state file of Roger, and we noticed that there are 50 or so guard nodes in there. And that made us wonder: "Why does Roger have so many guards?". Roger is not the problem in this case; my state file also has many guards. Most people who don't use bridges or hardcoded EntryNodes have shitloads of guards. This post tries to explain why. So, Tor, in its memory, has an ordered list of entry guards (the global `entry_guards` smartlist in `src/or/entrynodes.c`). This list can be lengthy: it usually contains more than $NumEntryGuards entry guards. You can see this beautiful list just on your right below that beautiful stalagmite: https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… This happens because in its first startup, Tor adds $NumEntryGuards nodes to that list. However if one of them is not Stable and Tor needs to build a Stable circuit, Tor will need to append a Stable guard to the list. Similarly, if one of the guards is down, Tor will need to compensate for that and append [1] one more guard to the list. The same happens if Tor needs to fetch directory documents, but its guards are not directory mirrors. So, if Tor walks to the end of the guard node list and it still hasn't found enough guard nodes with the needed property to make a pick, it picks a random entry guard from the consensus and adds it to the list. It's amazing and yet real, look straight ahead (and don't look directly into the light): https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… But this still does not explain why Roger has so many guards. Usually a list of 5 or 6 nice guards is sufficient to satisfy the needs of any circuit (alive, stable, fast, directory mirror). The reason for Roger's surplus of guards, is the following very interesting functionality of Tor: Consider the following scenario, you start Tor while your network is down, Tor starts picking nodes from your list and attempts to connect to them. All connections fail, since your network is down. So now, Tor needs to add a new guard node to the list. There are two cases now: If Tor fails to connect to this new guard node (your network is still down), Tor removes the new guard node from the entry guard list (that's good; otherwise the list would be full of nodes added while the network is down). Look on your left, you can see this beautiful phenomenon happening here: https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… However, let's say that your network is back up, and Tor manages to connect to this new guard node! That's great! But should Tor keep the connection to this guard? The answer is probably that it shouldn't: Tor should recognize this problem and attempt to reconnect to the primary guards on the top of the list. And that's exactly what Tor does. Nature is truly amazing! Just relax and witness this behavior happening right in front of your eyes: https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… So, when Tor manages to connect to this newly added entry guard, it assumes that the network is back, and walks through the list of entry guards and marks them all as "needs to be retried". It also marks the connection to the new entry guard as rotten and kills it. This to me is very interesting, because it ensures that the primary guards (the ones at the top of the list) are going to be tried again after the network is back up; otherwise we would leak connections to new guards all the time! And all that fluff is related to this post, because this new guard (that made us realise that the network is back up) actually stays in our guard list. So, basically every time the network goes down and Tor does this little dance, a new entry guard is appended to our list and our statefile. And that's why Roger has so many guards! Or at least, that's why *I* have so many guards [2]. Apart from this being wonderful on its own, there are two interesting points here: a) There is always a bug: As this thing happens more times, our guard list gets bigger and the time to walk it increases. Dig this race condition: Tor starts up with the network being down, so the connections to our primary guards fail, but the network comes back while we are walking our entry guard list and trying to connect to the rest of our guards. If we manage to connect to one of the guards in our list (the lucky guard), the code at https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… doesn't get triggered because `first_contact` is not true (that node was already in the guard node list). So, we stick with that lucky guard even though it's not our primary guard, and since the network is back up, a connection to our primary guards would work too. What stinks here is that all the guards above that lucky guard are marked as unreachable, so next time Tor starts up, it will ignore them and jump directly to the lucky guard. This probably needs to be fixed somehow. I opened trac ticket #12450 for this issue [3]. b) While writing proposal 236 we were thinking about how new guard nodes should be picked. Should we pick new guard nodes at the point they are needed? Or should we pick a surplus of guard nodes in the beginning, and then when the primary ones expire, we use the extra ones? You can read more about this behavior here: https://gitweb.torproject.org/torspec.git/blob/2ecd06fcfd883e8c760f0694f359… The insight here is that apparently we are already doing the latter approach, because all these guard nodes that get added when our network goes back up will remain in our guard list. And when our primary guards expire, the ones on the bottom will rise on the top (till they expire themselves). So if you are wondering "when does Tor add new entry guards?", the answer is "when you move your laptop to a new location; just before you connect to the wifi" ;) Greetings from the core, have a good day! [0]: https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/236-single-g… [1]: Note that the word "append" is vital here. The extra guards are appended to the end of the list, and when Tor wants to pick a guard node it walks the list from the top. So, these newly added guards have lower priority so to say (most of them will not even be considered if the ones above are sufficient for building a circuit). [2]: Here is a grep of my logs. Look at how the guard counter increments by one everytime we hit https://gitweb.torproject.org/tor.git/blob/d064773595f1d0bf1b76dd6f7439bff6… $ zgrep "Marking earlier" /var/log/tor/notices.log.3.gz [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 0/2 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 0/3 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 3/4 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 4/5 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 5/6 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 8/9 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 6/8 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 7/9 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 8/10 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 9/11 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 10/12 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 11/13 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 12/14 entry guards usable/new. [warn] Connected to new entry guard XXX. Marking earlier entry guards up. 13/15 entry guards usable/new. [3]: https://trac.torproject.org/projects/tor/ticket/12450#ticket

1 0

GSoC: BridgeDB Twitter Distributor report
by Kostas Jakeliunas 21 Jun '14

21 Jun '14

Hi all, in the past couple of weeks I've been doing more of the same - namely, fleshing out churn control in the bot; finishing a generic challenge-response system (I'm also now considering making it into a Zope Interface); subclassed text-based challenge-response; incorporating isis' IRequestBridges and BridgeRequest's into the bot's bridge request processing part; and fake bridge-line-from-descriptor generator within the bot (didn't really do much re: the latter.) Unfortunately, all those parts are not yet ready for redeployment of the bot, and are either buggy or not finished (inclusion/use of BridgeRequests.) This is partly due to me having a bit less time in the last two weeks (a fault of my own; on the plus side, I've learned to use the soldering iron properly!) My plan is to finish the things that are near completion, and do another midterm-worthy status update "very soon." I'll also be present during the developer meeting hackdays (2th-4th), and hope to use them to flesh out ideas, etc. with isis/sysrqb. -- Kostas. 0x0e5dce45 @ pgp.mit.edu

1 0

PT Combiner GSoC Second Report
by quinn jarrell 21 Jun '14

21 Jun '14

Hi, I'm the GSoC student working on creating a pluggable transport combiner. Progress: 1. I worked on modifying the SOCKS protocol to save the requested address and redirect the requester to a different address. 2. The pt combiner client side now starts a new chain of proxies in between the pts per connection instead of having to create a new set of pts per connection. 3. The pt proxies now shut down their ports after a single connection is established, lowering the number of ports needed. 4. Tested using authenticated pts and added a better logging mechanism. Quinn

1 0

[GSoC 2014] Revamp GetTor status report #2
by Israel Leiva 21 Jun '14

21 Jun '14

Hi all! This is my second status report for the Revamp GetTor GSoC project. I'll summarize my progress since the first report. For more details feel free to ask, or if you have any comments, please tell me. Achievments: * Implemented a script that interacts with Dropbox. * Integrated the dropbox script with GetTor's core module. Now it's possible to automatically upload bundles from a local directory to dropbox and write its download urls to a specific file, which is used by the Core module to obtain links for a certain operating system and locale (e.g. linux-en). The current dropbox.links was generated by dropbox.py, and if you run core_demo.py (with 'linux' and 'es') it will get you those links. * Added logging within the Core module, with one general file (all logs) and separated log files for each level. Others: * Started a discussion [2] about sending links without SSL. I didn't get many answers. My mentors and I agreed to stay with HTTPS links only. Goal(s) until next report: * Do some cleaning, better error detection and better documentation for the current code. * Finish some minor stuff pending on the current code. * Hava a basic SMTP or XMPP module working with the Core module. You can check my progress on [0] and [1]. That's all for now. Have a nice weekend! [0] https://github.com/ileiva/gettor/tree/master/spe <https://github.com/ileiva/gettor/tree/master/spec>c [1] https://github.com/ileiva/gettor/tree/master/sr <https://github.com/ileiva/gettor/tree/master/src>c [2] https://lists.torproject.org/pipermail/tor-dev/2014-June/006977.html -- israel

1 0

GSoC Report - Zack Mullaly - HTTPS Everywhere secure ruleset update mechanism
by Red 21 Jun '14

21 Jun '14

Hello again everyone, I am here once again to report my progress in building a secure ruleset updating mechanism for the HTTPS Everywhere browser extension. This past week's work has involved a lot of updates to the specification for the mechanism[1], and changing a lot of what I had done before to keep up with all of the changes. With regards to the documentation, a great deal of the changes being discussed about the order that tests of authenticity of the update information as well as its relevant have been finalized. There is now a "Pseudocode of the update procedure" section that outlines this order that tests are performed and also gives an idea of what data the mechanism requires, where it's obtained from, and how it changes that data. I have also added extensively to the "update.json and update.json.sig" section to outline everything anyone should need to know from generating update.json to getting a properly-encoded signature of the digest of update.json that can be verified by the extension. The document is now also included as part of the repository so that pull requests and better commenting can be made. I have also been making a lot of improvements to the utility script used for building the content of update.json[2]. The script has been completely revamped to work in accordance with the modified spec. I have also included a few sanity checks to verify that the content provided for update.json make sense, and made it very easy to write more such checks. I spent most of today getting the actual ruleset updater module[3][4] code up to date. The code now reflects the changes to the new standard and has been simplified dramatically. As usual, I have my meeting notes available as a Gist[5]. The notes from the meeting I had today with Yan are already present. My plans for the coming week are to set up a testing environment and begin making sure everything works as expected. I will also begin the discussion about how the extension should handle error reporting- what to do when authenticity tests fail, or the extension can't fetch the update information. Please feel free to take advantage of the update spec being part of the repository now to discuss the details and/or propose changes in the form of a pull request. Either this or the https-everywhere mailing list are also great places to get in touch with myself and everyone who has been involved in my work. Cheers, Zack [1] : https://github.com/redwire/https-everywhere/blob/makeJSONManifest/doc/updat… [2] : https://github.com/redwire/https-everywhere/blob/makeJSONManifest/utils/rul… [3] : https://github.com/redwire/https-everywhere/blob/rulesetUpdating/src/chrome… [4] : https://github.com/redwire/https-everywhere/commit/1a0d5edb8572180854ad620f… [5] : https://gist.github.com/redwire/b62f03905a826e79947a#week-6

1 0

GSoC : Rewrite Tor Weather (Status Update 2)
by Sreenatha Bhatlapenumarthi 20 Jun '14

20 Jun '14

Hi all, As a part of the weather-rewrite project, I've been working on the onionoo-wrapper module[0] since my last update. I managed to write a couple of utility methods to the wrapper like *is_stable*, *is_exit_relay*. Removed the caching feature(using pymemcache) temporarily to ease the testing process on meejah's advice. This will be picked up later from Lukas Erlacher's Onion-Py. Wrote some preliminary unit tests for the wrapper using python's mock module. Also initiated the main backend work with a new implementation of the welcome script using the Onionoo wrapper[1]. It currently fetches details document from Onionoo, checks for new relays, ensures their operators haven't been sent an email yet and then updates the database accordingly before sending them a welcome email. This is my first script regarding the rewrite that interacts with the django components of the previous weather. I'll be finishing up the remaining scripts as well soon enough before the next update and then move on to testing and documentation. As always, if you can take a look at the work and/or provide some feedback that'd be great. [0] - https://trac.torproject.org/projects/tor/ticket/12085 [1] - https://trac.torproject.org/projects/tor/ticket/11081 [2] - https://trac.torproject.org/projects/tor/ticket/11084 Cheers, Sreenatha

1 0

Orbot & Orfox - GSoC bi-weekly report 2
by Amogh Pradeep 20 Jun '14

20 Jun '14

Hey everyone, This is my second bi-weekly report and I am working on Orbot & Orfox. Status Report 2 June 20th, 2014. What I have done: 1) Expanded notification work: Added my finishing touches to the new expanded notification that has been planned to be added to Orbot, got exit IP address working (please do test this if possible, https://github.com/amoghbl1/orbot-1 branch- new_notification_expanded). Found a new layout for the expanded notification to make it look like the original notification that the system shows. 2) Build Fennec: My project timeline now moves on to working on Orfox, as we have decided to make Orfox tor's blend of the firefox for android browser, I am currently working on building firefox so that we may edit their code and make our browser. Hopefully, I can start hacking on the code as of next week, once I get it built. On another note, I will be attending the Summer dev meet in Paris and I'm excited to meet the community :) Best, Amogh Pradeep

1 0