- tor-dev - lists.torproject.org

Damian's Status Report - May 2012
by Damian Johnson 01 Jun '12

01 Jun '12

Hi all. As you may have noticed from the elevated traffic on this list a few of us are trying to do more development discussions in public. To that end I'm bucking tradition and sending my monthly status report here too. They've always been public... http://www.atagar.com/arm/log.php ... but since their initial emails have been on a closed list I doubt that most people knew that they existed. On that note, if anyone wants to hack on fun tor related python stuff then let me know! Stem is always looking for new people that want to get involved. ======================================== Hi all. Spring is in the air, and with it many lovely things including the UW Street Fair, Folklife, and an iSec Open Forum. It also included a week of oncall but that doesn't count in the 'lovely things' column. On the upside though, this time it was pleasantly light (first time I've gone a whole week without being woken up by a pager at 3am!). As a GSoC admin I don't have much to report besides a blog posting at the start of the month [1]. However, as a mentor some neat things are in the works... * Ravi is adding SafeCookie support in stem [2]. I've code reviewed the first couple iterations and it's looking great. Given some tests and revising to fit with recent refactoring it should soon be ready to merge. After that he'll start working on the general controller. This last weekend I refactored how response classes are organized and implemented GETINFO as an example, so this will hopefully be an easy project for him to start hacking on. * Beck added descriptor validation to check that the signing key's hash matches its fingerprint [3]. This is the first part of descriptor integrity validation that Karsten suggested, however this project has gotten stuck so he's moving on to other stem tasks. * Investigated a couple issues brought up by Sathyanarayanan [4][5]. * Two Wesleyan students will start working on stem soon! [6] Needless to say helping these projects has occupies much of my time, and will take even more once the Wesleyan students get started. However, after years of trying in vain to attract developers to my projects I wouldn't have it any other way. :) Other stem tasks I finished this month includes... * ExtraInfo descriptor parsing. This took me a couple weeks since it contains so many attributes, but I'm glad it's finished. Now all that remains before we can port Onionoo are the consensus' network status entries. That's now at the top of my todo list for descriptor work, but I'm setting that list aside for now in favor of Sphinx and helping Ravi and Beck with controller work. * Improved the launch_tor() function, making the test instance easily configurable via a temporary torrc (similar to what Vidalia does) and adding integ tests. I also added a workaround so it'll work on Windows [7][8]. * Updated the stem development wiki [9] so it'll be easier for new volunteers to find tasks that interest them. * Discussed descriptor type annotations with Karsten and implemented stem's side of it [10]. We also discussed some changes to bridge descriptor sanatization which led to some minor stem changes too. Cheers! -Damian [1] https://blog.torproject.org/blog/gsoc-2012-projects [2] https://trac.torproject.org/5262 [3] https://trac.torproject.org/5810 [4] https://trac.torproject.org/5917 [5] https://trac.torproject.org/5918 [6] https://lists.torproject.org/pipermail/tor-dev/2012-May/003547.html [7] https://trac.torproject.org/5783 [8] https://trac.torproject.org/5493 [9] https://trac.torproject.org/projects/tor/wiki/doc/stem [10] https://trac.torproject.org/5651

1 0

Fwd: Re: Can we stop sanitizing nicknames in bridge descriptors?
by Karsten Loesing 29 May '12

29 May '12

Forwarding my original answer to Sebastian here. -------- Original Message -------- Subject: Re: [tor-dev] Can we stop sanitizing nicknames in bridge descriptors? Date: Mon, 21 May 2012 19:56:34 +0200 From: Karsten Loesing <karsten(a)torproject.org> To: Sebastian G. <bastik.tor> <bastik.tor(a)googlemail.com> Hi Sebastian, On 5/21/12 7:08 PM, Sebastian G. <bastik.tor> wrote: (Did you intend to send this mail only to me, not to tor-dev? Feel free to move the discussion back to tor-dev if you want.) > Karsten Loesing, 21.05.2012 11:05: >>> >>> Here we go with the similarities of bridge and relay nicknames. >> >> Thanks for spending this much time on the analysis! > > I could have done far worse, but also a lot better in terms of time > spend on extracting the data that I wanted or at least considered that > they'd might be useful. > > Sometimes I'm just slow at things, e.g. writing this reply. > >> Here's what I did with your findings.txt: >> >> - extract unique fingerprint pairs of relays and bridges that you found >> as having similar nicknames, >> >> - look through descriptor archives to see if relay and bridge were >> running in the same /24 at any time in May 2008, and >> >> - determine the absolute and relative number of bridges in a given >> network status that could have been located via nickname similarity. >> >> Results are that 24 of your 81 guesses (30%) were correct in the sense >> that a bridge was at least once running in the same /24 as the relay >> with similar nickname. At any time in May 2008, you'd have located >> between 1 and 6 bridges (2.5% to 18%) with 3 bridges (10%) in the mean >> via nickname similarity. > > Not too bad. I agree. :) >> I think it's acceptable to publish more recent bridge descriptors with >> nicknames in a week from now. Results may look quite different with >> 1000 bridges instead of 30. > > May 2008 was the first month with bridges. I expected lot's of relay > operators that tested a bridge with the same name. Things may have > changed over time. I assume that further comparisons won't have such a > "high" hit ratio. That would be my guess, too. In May 2008, only a few early adopters were running bridges, and most of those probably ran relays at the same time, too. Plus, they were enthusiastic and put some energy in finding cool nicknames. It might be that this has changed since then. To be honest, I didn't look at 2012 tarballs yet. >> Again, thanks for running this analysis! Maybe you're interested in >> automating your comparison and re-running it for a 2012 tarball? > > My claim was you got the data, so you can check. (Not with May 2008) > > To be honest, my first impression was that I wouldn't do anything useful > and did not intend to do that. I guessed it wouldn't turn out that it > doesn't hurt since at least 2011, so I wouldn't find anything good. > > Then you asked and I agreed, but already thought "I couldn't keep my > mouth shut!". I mean I replied to this topic. I surely could have said > no there. I didn't. > > After and while I was doing what I did. I would have said no to the > question if I'm going to do this again. That's valid for up to Sunday > night. Today I'm agreeing again. > > That's a pretty long way to say: Yes! Hah, great! :) I'm going to make the 2012 tarballs available next Wednesday (May 30), assuming that my poor Linux box doesn't run out of $resource. I'll let you know. > Thank you,it's an 2012 tarball. The number of bridges is scary. > > I'm going to upload some files somewhere and explain what I did. Step by > step (somewhat around that). So anyone can check and reproduce what I > did. It would be nice to hear feedback and ways to improve the way I did > what I did. > > Maybe you can tell me if the findings.txt was alright. Yes, the file format was fine. > Unless one objects or you disagree I'm going to upload the files I > created and explain how and maybe I can say even why. No objections at all. Open discussion is good. > I created a Blog, just because I wanted it some when in the past, but > found it silly. That's the channel I planed to use. Maybe it's OK to put > it on a Tor-List as well, but maybe it's considered as noise. I wonder if the Tor wiki would be a better place to collect ideas for reversing the bridge descriptor sanitizing process. Feel free to grab a new page in doc/ and start describing what you did. Best, Karsten

2 4

[GSoC] Update - Safe cookie support in Stem
by Ravi Chandra Padmala 28 May '12

28 May '12

Hello I'm reporting my progress over the last week. I have been working on implementing support for the safe cookie authentication method for Stem[1]. It is in a usable state[2] right now, but, it isn't fully tested. In the process, Stem also helped uncover a bug in the AUTHCHALLENGE error responses[3]. Now, I'll be writing the tests for the safe cookie implementation and getting it merged. Next, I'll begin work on the general controller class. Damien has already implemented an the GETINFO response parser. Beck also wants to help implement the controller class. I/We will be working on the deliverables scheduled for week two in my proposal, i.e. Implementing the wrapper functions and response parsing for the following control commands, GETCONF, SETCONF, LOADCONF, RESETCONF, SAVECONF, SIGNAL, TAKEOWNERSHIP, USEFEATURE and QUIT. 1. https://trac.torproject.org/projects/tor/ticket/5262 2. http://repo.or.cz/w/stem/neena.git/shortlog/refs/heads/safe-cookie-alt 3. https://trac.torproject.org/projects/tor/ticket/5760 -- Regards, neena

1 0

Sanitized bridge descriptor format 1.0
by Karsten Loesing 24 May '12

24 May '12

Hi Damian, I plan to make a few changes to the bridge descriptor sanitizer to implement changes discussed on this list and in various Trac tickets. The result will be format version 1.0. Here's what will change compared to the current (unversioned) format. Can you take a look whether stem would be happy with these descriptors and if there are other tweaks I should do to make it even happier? - Bridge network statuses contain a "published" line containing the publication timestamp, so that parsers don't have to learn that timestamp from the file name anymore. - Bridge network status entries are ordered by hex-encoded fingerprint, not by base64-encoded fingerprint, which is mostly a cosmetic change. - Server descriptors and extra-info descriptors are stored under the SHA1 hashes of the descriptor identifiers of their non-scrubbed forms. Previously, descriptors were (supposed to be; see #5607) stored under the digests of their scrubbed forms. The reason for hashing digests is to prevent looking up an existing descriptor from the bridge authority by its non-scrubbed descriptor digest. With this change, we don't have to repair references between statuses, server descriptors, and extra-info descriptors anymore which turned out to be error-prone (#5608). Server descriptors and extra-info descriptors contain a new "router-digest" line with the hex-formatted descriptor identifier. These lines are necessary, because we cannot calculate the identifier anymore and because we don't want to rely on the file name. - Bridge nicknames (#5684) in all descriptor types and dirreq-* statistics lines (#5807) in extra-info descriptors are not sanitized anymore. - All sanitized bridge descriptors contain @type annotations (#5651). Please let me know what you think about these changes. I plan to start sanitizing descriptors with the described changes tomorrow or the day after and make them available on May 30 (or later if the sanitizing/compressing/uploading takes much longer than expected). Thanks, Karsten

2 5

GSoC Introduction - Pluggable Transports in Python
by Brandon Wiley 22 May '12

22 May '12

Hello fellow Tor developers! Some information about me: *I worked for EFF/Tor Project last year for **GSoC 2011, my project **was a blocking-resistant transport evaluation framework: https://gitweb.torproject.org/user/blanu/blocking-test.git* I am also the author of a pluggable transport written in python: https://github.com/blanu/Dust/tree/master/py/dust/services/socks2 I've been working on censorship resistance technology since 2001. Here are some of my projects: http://blanu.net/Dust-FOCI.pdf http://blanu.net/BayesianClassification.pdf http://blanu.net/Arcadia.pdf http://blanu.net/Freenet2001.pdf Some information about the project: The overall goal of the project is to make it easy for pluggable transports to be written in python. There has been a lot of interest in doing pluggable transports in python, but currently they are all written from scratch. For C transports, obfsproxy can be used to do a lot of the heavy lifting, making it relatively easy to write a new C-based transport. I've heard there is also a port of obfsproxy to C++. A the author of a python transport, I am of course an advocate of writing transports in python. Fortunately, so are some other Tor folks, so soon it will be easy to write python transports! The deliverables for this project are as follows: *A library for parsing pluggable transport configuration options* This will be a python library that authors of SOCKS proxies can use to integrate their proxies with Tor. *A framework (both server and client-side) for writing pluggable transports in python* The framework will provide a SOCKS proxy server already integrated with the pluggable transport library. All the protocol author will need to do is provide the obfuscation and de-obfuscation functions and a main function to do command line parsing and call the framework. *A python implementation of the obfsproxy command line tool* This will be a command line program using the framework that will accept the same command line options as the existing obfsproxy tool. It will support the selection of an obfuscation function, although not all of the protocols currently supported by obfsproxy will initially be available in python. *A python implementation of the obfs2 protocol implemented as an obfsproxy module* The obfs2 protocol will be implemented as a plugin for the framework and made available to the command line tool. *Conversion of Dust to an obfsproxy module* The Dust protocol will be implemented as a plugin for the framework and made available to the command line tool. *py2exe packaging for obfsproxy* The command line tool will be packaged into a standalone executable for Windows. Optional deliverables if there is sufficient time: obfsproxy modules for other protocols, experiment with other packaging systems Current status: I'm working on a spec of the API for the option parsing library. It should be available soon.

2 1

[GSoC] [APAF] Report of 17-21 May hackathon; current status of APAF.
by Michele Orrù 21 May '12

21 May '12

I'm writing this mail as report for the last four days during which, thanks to the supervision of both naif and hellais, I managed to develop the very first basic structure of the APAF project. One nice thing that came out during this weekend was the collaboration and improvement of other python projects, txtorcon[0] and py2app[1]. Especially, I hope to continue improving txtorcon, which will be the most strict dependency for the apaf project. At the end of this hackaton, I reached a state in which: - the basic structure of the apaf has not been only designed, but also developed; - the most important classes have unittests and partial documentation sphynx-compilant; - the file setup.py bulds succesfully an installer for windows, osx, and linux.[2] The very basic structure has been widely described in the documentation, and I will update http://mmaker.github.com/apaf/ according to last changes in the source. The most difficult task during the whole hackathon has been packaging for both windows and osx; pyInstaller didn't manage it, but py2exe + py2app seems to work pretty well, even though with some hacks[3]. Todos. --------- Apart from the design of the panel itself, there are some cool standalone tasks which you may consider as example for other projects. I learned from hellais that the torproject is going to build a sort of package managed in order to safely handle updates of the shipped executables; the APAF, too, is going to download[4] executables from the web as dependencies (gnupg and tor for example). The APAF needs also some icons and graphics for the system tray / icon bar / standalone executable (note that darwin and windows have their own image format); I'm going to use Vidalia's ones, but any adivice/aid here would be strongly appreciated. :) [0] https://github.com/meejah/txtorcon/pull/4 and https://github.com/meejah/txtorcon/issues/6 [1] https://bitbucket.org/ronaldoussoren/py2app/issue/45/psutil-error-importerr… (note: building on osx 10.7 does not work.) [2] at least, it did so yeasterday. :P http://cl.ly/04232z2G1x2K2g1t3s2E [3] https://github.com/mmaker/APAF/blob/master/apaf/blobber.py [4] https://github.com/mmaker/APAF/blob/master/apaf/build.py -- ù

1 0

unscribe
by torbridges.security 21 May '12

21 May '12

unscribe

1 0

Re: [tor-dev] [tor-assistants] Python metrics-lib
by Damian Johnson 17 May '12

17 May '12

Hi Beck, hi Karsten. First I'd like to make sure that I'm clear on what we're trying to do. The javadocs for VerifyDescriptors [1] says that it... > Verify server descriptors using the contained signing key. Verify that > 1) a contained fingerprint is actually a hash of the signing key and > 2) a router signature was created using the signing key. > > Verify consensuses using the separate certs. Verify that > 1) the fingerprint in a cert is actually a hash of the identity key, > 2) a cert was signed using the identity key, > 3) a consensus was signed using the signing key from the cert. Honestly I'm not yet sure what most of this means. The first #2 is simply checking that the descriptor content can be verified using the router-signature and signing-key, right? If so then this sounds like a good place to start since it's entirely self-contained within the descriptor and just involves implementation and testing of... https://gitweb.torproject.org/stem.git/blob/HEAD:/stem/descriptor/server_de… > However, I need some suggestions for the choice of Python cryptography API, since I haven't used any before. Nor have I. At present stem does not have any dependencies outside of python's builtin functions. If we need PyCrypto and it's the best choice then so be it, but be sure to wrap the imports in a try/catch so we only raise an ImportError when executing the function that requires the PyCrypto library. Cheers! -Damian [1] https://gitweb.torproject.org/metrics-tasks.git/blob/HEAD:/task-2768/Verify…

7 20

directory requests
by SiNA Rabbani 16 May '12

16 May '12

I see the same IP sending the same request over and over. Is this normal behaviour? All the best, SiNA XXX.XXX.XXX.XXX - - [16/May/2012:15:15:00 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:01 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:02 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:02 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:02 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:02 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:02 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:02 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:04 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:06 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:08 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:09 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:12 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:13 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:13 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:13 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:16 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:16 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:16 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" XXX.XXX.XXX.XXX - - [16/May/2012:15:15:19 -0400] "GET /tor/status/all.z HTTP/1.0" 200 540655 "-" "-" "-" -- First they ignore you, then they laugh at you, then they fight you, then you win ~ Mahatma Gandhi

1 0

Re: [tor-dev] HFOSS as Wesleyan this summer
by Damian Johnson 16 May '12

16 May '12

Hi Megan, hi Erik, glad to have you working with us this summer! First a bit of a project introduction is probably in order. Stem is a python controller library for tor. Like its predecessor, TorCtl [1], it uses tor's control protocol [2] to help developers program against the tor process, enabling them to build things similar to Vidalia [3] and arm [4]. Most of stem's core bits (process communication with tor, asynchronous message handling, etc) are done, and development is now focused on the wrapper around that which makes for a developer-friendly API. There's still a ton of work to do, and currently three people hacking on the project... * Damian (project author) * Ravi (GSoC student) - Working on a grab bag of stem related tasks including the general controller, some descriptor parsing, and migrating the first client (arm) over to stem. [5] * Beck (volunteer) - Doing a deep dive into the crypto behind descriptor validation. [6][7] Karsten, Ravi, and I just updated stem's development wiki yesterday so it should be reasonably up to date... https://trac.torproject.org/projects/tor/wiki/doc/stem Codebase: https://gitweb.torproject.org/stem.git So my first question is, what are your interests? It seems that most people either lean toward research or application, and we have a lot to do that fit both. To start with please take a look at the wiki's bugs and improvements sections. Those are bite sized tasks that should be reasonably small and start immersing you in the codebase. After that, both the metrics-lib tasks (more research focused) or porting the tor interpretor (more application) would be great projects. Here's a brief overview of the communication channels we use: - #tor and #tor-dev on OFTC: we're most active on irc, so I definitely suggest being in these channels [8] - tor-talk [9] is our user mailing list which is somewhat high volume. - tor-dev [10] is a lower volume development list and a good place for technical discussions. Unless something is private please include this in your email cc. Hope this helps, and again - welcome! -Damian PS. I highly suggest introducing yourself to this list. The more you're involved with the tor community the better, and we'd be thrilled if you stuck around after your project to become core tor developers! [1] https://www.torproject.org/getinvolved/volunteer.html.en#project-torctl [2] https://gitweb.torproject.org/torspec.git/blob/HEAD:/control-spec.txt [3] https://www.torproject.org/getinvolved/volunteer.html.en#project-vidalia [4] http://www.atagar.com/arm/ [5] https://www.torproject.org/about/gsocProposal/gsoc12-proposal-stemImproveme… [6] https://trac.torproject.org/5810 [7] https://lists.torproject.org/pipermail/tor-dev/2012-May/003510.html [8] https://www.torproject.org/about/contact.html.en#irc [9] https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk/ [10] https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev/

1 0