April 2013 - tor-dev - lists.torproject.org

GSoC 2013 / Tor project ideas - Searchable Tor descriptor archive - (pre-)proposal
by Kostas Jakeliunas 30 Apr '13

30 Apr '13

Hello Karsten and everyone else :) (TL;DR: would like to work on the searchable Tor descriptor archive project idea; considering drafting up a GSoC application) I'm a student & backend+frontend programmer from Lithuania who'd be very much interested in contributing to the Tor project via Google Summer of Code (well, ideally at least; the plan would be to volunteer some time to Tor in any case, but it's yet to happen, and GSoC is simply too awesome an opportunity not to try) - The 'searchable Tor descriptor/metrics archive' project idea [1] would, I think, best fit in with my previous experience and general interests in terms of contributing to the Tor project. The searchable archive project idea in itself has a rather clear list of goals / generic constraints, and since I haven't contributed any code to the Tor project before, working with an existing general project idea (building a more concrete design proposal on top of it) probably makes most sense. This particular project, I think, would match my previous Python backend programming experience: building backends to work with large datasets / databases -- crafting efficient ORMs and responsive APIs to interact with them. [2] Applying the knowledge/skills learned to something which is ideologically close at heart and the purpose of which is very obvious to me sounds thrilling! (This year, as far as Python frameworks are concerned, I've been mostly exposed and have been working with Flask - have some (limited) experience with Django before that. As far as a proof-of-concept for the searchable archive is concerned, I'm considering trying some things out with Flask, since it allows me to do some quick prototyping.) I'd like to try and work out an implementation/design draft for what I could / would like to do (this is a preliminary email - I know I'm a bit late!) Ideally it (and a simple proof of concept search form -> browseable/clickable results / relay descriptor navigation page) would serve as the base for my GSoC application, but I have to be realistic about me being rather late to apply and not having participated in neither Tor nor GSoC before. I'd like to work out an application draft if possible, though. (Were I to get accepted, I would be able not to do any part-time work this summer, or would only need to take passive care of a couple of already running backends.) I've read into the Tor Metrics portal pages (esp. Data Formats), and am trying to get acquainted with the existing archiving solution (reading into the 'metrics-web' java source (under metrics-web/src/org/torproject/ernie/web) to see how the descriptor etc. archives are currently parsed / imported into Postgres and so on), to first and foremost be able to evaluate the scope of what I'd like to write. I will presently work on a more specific list of constraints for the searchable archive project idea. I can then try producing a GSoC application draft. Just to get an idea of what kind of system I'd be building / working on - at the very least, we'd be looking into: - (re)building the archival / metrics data update system - the proposed method in [1] was a simple rsync over ssh / etc. to keep the data in sync with the descriptor data collection point. If possible, it would help if the rsync could work with uncompressed archives - rsync is intelligent enough not to need to send *that* much excess data - and diffing is more efficient with uncompressed data. A simple rsync script (can be run as a cron job) would work here. - a python script (probably to be run through cron) to import the archives into DB. Can stat files to only need to import new/modified ones, e.g. The good thing about such an approach is that the script could work as a semi-standalone (would still need the DB / ORM design), therefore could be used in conjunction with other, different tools - and it would be built as an atomic target during the implementation process - I heard you guys like modular project design proposals ;) who doesn't like them! We already have metrics-utils/exonerator/exonerator.py (which works as a semantically-aware descriptor archive grep tool) - some archive parsing logic can be reused maybe - the more pertinent thing here would be to - build the ORM for storing all the archival data in DB. Postgres is preferred and could work, especially since probably the a large part of the current ORM logic could be used here (I've taken a glance at the current architecture, it makes good sense to me, but I haven't looked further, neither have I done any benchmarking with the existing ORM (except for some web-based relay search test queries which don't really count.)) - it is very important to build an ORM which would scale well data-wise, and would suit our queries well. - query logic and types - the idea would be to allow to do incremental query-building - on the SQL level, WHERE clauses can be incrementally attached to the higher-level ORM query object. For example, part of fingerprint / relay name -> add a date interval -> additionally, add a day-specific "on date" clause -> etc. ORM efficiency, benchmarking and restrictions on query complexity - this may turn out to very much be a nontrivial matter. - on the user interface level, user query input parsing - the idea would be to allow for flexible user input; date interval / info not mandatory; possibly, the user may provide additional filtering by specifying more descriptor metadata. Would need to figure out the most accessible type of input - a simple input field with info about the flexible syntax with examples and a flexible parser, and/or optional additional fields - a responsive UI (can expand the input form into additional fields/options and try to avoid contradictory / non-sensical input) might prove effective here. - results page - would need to work out what kinds of relay metadata to show, can click on parts of the data on the results page to further narrow results - unless click on relay name / fingerprint / etc., in which case would be taken to that specific relay's page (on a more general sense, this would also be a kind of 'result narrowing.') - => generally create a browsable relay archive, formatting / aggregating specific descriptor fields to provide more info about each relay. <-- may make sense to start working on the particular constraint list starting from this point. Hopefully Karsten can help me with the application, assuming my idea for the project is to make sense. :) I will follow up with more details. Besides my email address < kostas(a)jakeliunas.com>, I can be reached on #tor-dev as 'wfn', or XMPP/Jabber via <phistopheles(a)jabber.org>. Cheers to you all Kostas. [1] https://www.torproject.org/getinvolved/volunteer.html.en#metricsSearch [2] My largest Python backend related project was this winter, building a redis (product likes/dislikes) + mysql (everything else) product recommendation solution to work with a large dataset of (product) metainfo (such as user votes on products), and creating APIs (on top of Flask) (API for the frontend CMS and for a mobile app) which include custom-per-user product recommendation feeds, etc. Large data (nothing close to the Tor descriptor/metrics archives, though!) + Interactive application logic architectures are of interest to me.

2 1

TorBEL migration
by Sreenatha Bhatlapenumarthi 30 Apr '13

30 Apr '13

Hi, My name is Sreenatha Bhatlapenumarthi. I'd like to migrate TorBEL from TorCtl to Stem as a part of GSoC'13 project this summer. Damian pointed out that there is still some improvement to be made to TorBEL to replace DNSEL. I've downloaded the source code from [0] and started looking around for missing functionality and components that use TorCtl(and hence need migration) and here is what I've observed: - TorBEL components that use TorCtl - Controller - control.py - Logger - logger.py - Router - router.py - Tests - tests.py - The main interface - torbel - Exit policies of narrow_exit routers can be checked in Router.will_exit_to[1] - RouterRecord.update_to()[2] can use "Router Status Entry" descriptors[3] to update the router's attributes - I think the following line[4] in network.py has a bug - socket.inet_aton works only for IPv4 addresses[5]. - (ip,) = struct.unpack(">I", socket.inet_aton(peer.host)) Please let me know if I've missed something or got something wrong. I donot have any background on Haskell as such but if you recommend that I learn it inorder to understand TorDNSEL, I am prepared to spend the required amount of time learning it during May before the coding period starts. Also, can you give me a rough estimate of how long you think the migration will take so that I can plan the project timeline accordingly? Thanks for taking your time out to read this. [0] - https://git.torproject.org/torbel.git [1] - https://github.com/lucyd/torbel/blob/master/query.py#L85 [2] - https://github.com/lucyd/torbel/blob/master/router.py#L125 [3] - https://stem.torproject.org/api/descriptor/router_status_entry.html#stem.de… [4] - https://github.com/lucyd/torbel/blob/master/network.py#L75 [5] - http://docs.python.org/2/library/socket.html#socket.inet_aton Cheers, Sreenatha

2 1

Stem Integ tests for Tor
by Ashish Patil 29 Apr '13

29 Apr '13

Hello everyone! I would like to improve on Tor's codebase by writing Integ tests for it in Stem. I would like to know what kind of tests should be included in this. Also, atagar was talking about smoke tests for Tor as a whole. I would love to know everyone's ideas on the same. Also, I would like to start a new project which can automatically download & compile latest Tor. This would greatly help with the Integ tests for Stem & Tor. -Asis

3 2

Fwd: HTTPS Server Impersonation project
by Akshay Hebbar Y.S. 29 Apr '13

29 Apr '13

Hi, I wish to do the project 'HTTPS Server Impersonation' as a GSoC project this year. For that, I propose an approach for implementation. I would like to use Nginx as reverse proxy. Because nginx is very widely used and it is known for its high performance, stability and low resource consumption. Once this is used, this will become the interface for Bridges. All connections to the bridge will be listened on port 443. Two back-end servers for this reverse proxy : 1. Tor itself 2. Web server A special module has to be written for handling the requests. Once the request is authenticated, it will be sent to tor process. Otherwise, it will be sent to web server and suitable response is sent back. Once the new tor client finishes its handshake, its IP address will be mapped to Tor in nginx 'Load Balancer'. Further requests from that IP address will be sent to the Tor directly. How does a bridge distinguish between Tor client and attacker? For this, I need suggestions to select one among the following. 1. Use AUTHORIZE and AUTHORIZED cells as discussed in proposals #187 and #189 2. Provide a public key while giving the bridge address. I.e. When someone requests a bridge address, provide [IP ADDRESS] + [PORT] + [PUBLIC KEY] [ here port = 443 ] Once the client gets the public key of bridge, it can encrypt its data send to bridge. Bridge uses its private key to decrypt the message, and hence only bridge can decrypt the message. If the message received has proper headers it will be sent to tor, otherwise it will go to the default web server. What will the web server contain? Web Server will have some pages which makes the website look like 'online storage site' or 'Online gaming site' or 'social networking site' or something like that. Because a passive attacker can wonder what the client is sending/receiving from a website which looks like 'blog'! Hence from the above discussion, we can achieve the goals of project: a. Impossible for a passive attacker who examines only a few packets at a time to distinguish Tor->Bridge traffic from an HTTPS client talking to an HTTPS server Tor client -> bridge connection will be TLS. So he can't distinguish b. Impossible for an active attacker talking to the server to tell a Tor bridge server from a regular HTTPS server Every ill-formed request will be handled by Web server. c. Against MITM attack If AUTHORIZE and AUTHORIZED cells are used, I need some help in making it preventing MITM attack. If method 2 is used for authentication, MITM attack can be easily prevented. Because even though the attacker receives the packet, he can't decrypt the message since he doesn't have key for decrypting it. Tor client sends this message along with its public key. Hence the bridge can encrypt and send back its response. Once the TLS connection is established then attacker will behave like hop in communication d. Code changes to tor It will be minimum and we just need to implement authorization code. e. Efficiency Once the new client completes its handshake, its IP address will be added to whitelist. Any packets from a client which is in whitelist will not be authenticated by the handlers in nginx. It will be directly sent to Tor process. Please correct me wherever I'm wrong and please give me suggestions to improve it. Thank you -- Akshay Hebbar Y S 6th sem, PESIT-CSE

2 1

About GSOC 2013 Steganography Browser Addon project idea
by Hareesan 29 Apr '13

29 Apr '13

Hey there, I'm Hareesan, undergraduate from University of Moratuwa,Sri Lanka. Interested in working on the project Steganography Browser Addon. I wish to confirm is that a project for GSOC? and Is it a good time to start working on my proposal? I have already gone through few chapters of the paper (Message In A Bottle: Sailing Past Censorship<http://www.cs.kau.se/philwint/censorbib/#Invernizzi2012>) a bit. -- *Hareesan R Undergraduate Department of Computer Science And Engineering University Of Morat**uwa Sri Lanka**** *

3 4

Re: [tor-dev] Stem Integ tests for Tor
by Ashish Patil 29 Apr '13

29 Apr '13

Hi! Thanks atagar, your reply cleared up a lot of things. I am going through the present integ tests in Stem & would contact you if I have any doubts. As for chutney, I suppose nickm's copy of chutney would be the one with most updates (if different from the default copy). I will go through that as well & let you know how much I grasped from it & what all needs to be done further as you mentioned in the reply. Thanks! -Asis

1 0

Re: [tor-dev] Stem GSoC 2013
by Tomasz Kunikowski 29 Apr '13

29 Apr '13

Hey, I favor GSoC projects that are collections of small items rather than a single big one (that way the incremental pieces are helpful, even if the project isn't quite fit into the summer). > We're on the same page here, I think more tasks of a smaller scope is the way to go. The next area I was planning to primary focus was client migrations, > which would involve work with TorBEL and Tor Weather... > I've looked through both TorBEL and Weather and while undoubtedly they could use some love, I think I'll follow my heart and stick with Stem :) Please don't take my ambiguity in regards to any concrete ideas as lack of interest on my side - it's that the more I learn about Stem (and I try to dedicate it most of my free time) the better I can assess its current needs and areas of lacking and come up with something substantial. As far as tutorials go, I was thinking about dealing with manual stream attachment (as a follow up to #8728), exploring more operations on circuits i.e. tearing down, forcing hops through specified countries or address ranges (maybe a more robust example as an excuse to come up with good use cases for REDIRECTSTREAM and TRUNCATE). As for the Stem itself, I'd love to work on adding any missing functionality (against the Tor protocol or expanding the lib itself), but I'd like to hear your suggestions where the real priorities and missing spots are first. Thanks, Tom

3 7

GSoC 2013 - "Compass"
by Emanuel Pascoal 29 Apr '13

29 Apr '13

Goof Night, My name is Emanuel Pascoal, i have 20 years-old and i'm from Portugal. I'm developing studies in the area of computer engineering in the University of Évora. I have knowledge in Python and a little of Java. I would like to make my candidacy in the project named"Compass", that are tagged on your ideas lists for Google Summer Of Code 2013. I would like to discuss the project in order to try to understand the best way to submit the application. I want to know more details about the ideas, so that you can outline what the best ways to make the project as well as the resources that I will need, and be able to think about the time we'll need for the project's development. I would have the most pleasure in work with your community, researching and developing the previous ideas, in any way that i can be helpful. Thanks for the time spent, Emanuel Pascoal

2 1

Source Code Static Analisys
by Ulises Cuñé 28 Apr '13

28 Apr '13

I want colaborate with Tor project. I send a document of analys source code about the lasted version Summary Issues by Fortify Priority Order High 1620 Low 178 Critical 38

3 3

"identity digest" and "hash of a router's identity key"
by Frank Young 28 Apr '13

28 Apr '13

In a consensus, the item after nickname is base64 encoded "hash of a router's identity key" If this item is base64 decoded, is it the same as the "identity digest" of the router?

3 3