- tor-dev - lists.torproject.org

Andrea's status report for June 2012
by andrea＠persephoneslair.org 30 Jun '12

30 Jun '12

So, here's what I've been up to my first couple weeks with Tor: * Interviewed and got hired * Getting up to speed with everything, setting up git repository, etc. * Met ioerror for coffee and key-signing * Got my first patch merged! (bug 5235) * Fixed several other bugs: * 5452 * 5049 * 6028 * 2385 * 5557 * 5238 * Did code reviews for 6226, 6031, 5285, 4195 and 6255 * Talked to NickM about planning for 0.2.4.x, and made some time estimates. -- Andrea Shepard

1 0

TLS warning using static OpenSSL 1.0.1c
by Gino Badouri 29 Jun '12

29 Jun '12

Hi there, My goal is to run Tor on small cluster of embedded mips devices. Because the platform runs on an older version of OpenSSL and libevent I have chosen to statically link them with Tor. So I went ahead to compile the components. I'm aiming at the beta version: 0.2.3.17 OpenSSL 1.0.1c has been build with: ./Configure debian-mipsel shared enable-tlsext zlib-dynamic no-ssl2 libevent-2.0.19-stable has been built with: ./configure --enable-openssl --disable-debug-mode --with-pic And of course Tor: ./configure --prefix=/usr --libdir=/usr/lib --localstatedir=/var --sysconfdir=/etc \ --with-zlib-dir=/home/mastag/build-mipsel/tmp/staging/mipsel-linux/lib \ --with-openssl-dir=/home/mastag/build-mipsel/tmp/staging/mipsel-linux/lib \ --with-libevent-dir=/home/mastag/build-mipsel/tmp/staging/mipsel-linux/lib \ --disable-asciidoc --enable-static-openssl --enable-static-libevent \ --host=mipsel-linux --with-libnatpmp-dir=/home/mastag/build-mipsel/tmp/staging/mipsel-linux/lib \ --with-libminiupnpc-dir=/home/mastag/build-mipsel/tmp/staging/mipsel-linux/lib \ --enable-upnp --enable-nat-pmp --enable-bufferevents \ CPPFLAGS="-I/home/mastag/openpli/build-dm800/tmp/staging/mipsel-linux/include" This produces the src/or/Tor binary which runs great on the cluster. Only it's giving me the following messages: Jun 21 23:02:12.000 [notice] Tor 0.2.3.17-beta opening new log file. Jun 21 23:02:12.000 [warn] You are running Tor as root. You don't need to, and you probably shouldn't. Jun 21 23:02:13.000 [notice] No AES engine found; using AES_* functions. Jun 21 23:02:13.000 [notice] This OpenSSL has a good implementation of counter mode; using it. Jun 21 23:02:15.000 [notice] OpenSSL OpenSSL 1.0.1c 10 May 2012 looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation Jun 21 23:02:37.000 [notice] Reloaded microdescriptor cache. Found 4807 descriptors. Jun 21 23:02:42.000 [notice] We now have enough directory information to build circuits. Jun 21 23:02:42.000 [notice] Bootstrapped 80%: Connecting to the Tor network. Jun 21 23:02:43.000 [notice] Heartbeat: Tor's uptime is 0:00 hours, with 1 circuits open. I've sent 0 kB and received 0 kB. Jun 21 23:02:43.000 [notice] Bootstrapped 85%: Finishing handshake with first hop. Jun 21 23:02:43.000 [warn] TLS error while creating tor_tls_t object: could not load the shared library (in DSO support routines:DLFCN_LOAD:---) Jun 21 23:02:43.000 [warn] TLS error while creating tor_tls_t object: could not load the shared library (in DSO support routines:DSO_load:---) Jun 21 23:02:47.000 [notice] Bootstrapped 90%: Establishing a Tor circuit. Jun 21 23:02:53.000 [notice] Tor has successfully opened a circuit. Looks like client functionality is working. Jun 21 23:02:53.000 [notice] Bootstrapped 100%: Done. The warning about the AES engine is probably because OpenSSL has no engine for mips. However the following two lines bother me: [warn] TLS error while creating tor_tls_t object: could not load the shared library (in DSO support routines:DLFCN_LOAD:---) [warn] TLS error while creating tor_tls_t object: could not load the shared library (in DSO support routines:DSO_load:---) Any way to get rid of these? I did compile OpenSSL with "enable-tlsext" so I don't understand why it warns me about it. Thanks in advance!

3 7

Re: [tor-dev] Stem Proc Integration Tests
by Norman Danner 29 Jun '12

29 Jun '12

(Sorry, wasn't subscribed to tor-dev@, so these last couple of messages didn't go there...) On 6/29/12 4:09 PM, Damian Johnson wrote: >> I think the question here is that it seems like one should be able to get >> all the descriptors from the running Tor process via >> >> GETINFO desc/all-recent >> >> But there isn't any obvious stem API for this. One could work around it >> (e.g., do a get_info(), process a little, turn into a StringIO, and feed >> that into the functions that want a file), but the real question is whether >> stem expects clients to do that, or whether you think that should be part of >> stem. This is sort of a "how fat should stem be" kind of question. > > Ahhh, gotcha. The controller is still in its infancy and maturing it > is Ravi's GSoC project. Yes, there *should* be methods for 'give me a > descriptor for fingerprint X' or 'give me a list of all the > descriptors' but they have not been implemented yet (that said, they'd > be very, very easy to write). Should we wait for Ravi to implement this, or can we go ahead and do it ourselves as needed? - Norman -- Norman Danner - ndanner(a)wesleyan.edu - http://ndanner.web.wesleyan.edu Department of Mathematics and Computer Science - Wesleyan University

2 1

Nick's status report for June 2012
by Nick Mathewson 29 Jun '12

29 Jun '12

June was a pretty decent month! With help from Mike Perry, I finished a couple of rounds of job interviews, so we could select a new core developer. Let's welcome Andrea Shepard to the Tor project; she's already off to a great start, even though she's only part-time for her first couple of months. I migrated the Free Haven Anonymity Bibliography (http://freehaven.net) to git, and fixed a bunch of longstanding bugs in it, so that its output finally passes the w3 validator. I wrote up a couple of emails about the statuses of current proposals, and proposals implemented in 0.2.3.x, and send them to tor-dev. I hope to do this every month or two. I finally circulated my proposals for improved cell crypto (202), and for impersonating an HTTPS server (203). Weasel and I had a chat about how to avoid backporting all conceivable patches to 0.2.2.x. From now on, I'm going to distinguish between "new stable" and "extended support stable" -- once a Tor series has been stable for long enough, it should really get fixes for grave bugs only. With help from many others, we tagged Tor 0.2.3.17-beta and 0.2.3.18-rc. These fixed a lot of bugs; I think Tor 0.2.3.x is getting close to being ready for a stable release. Once there are packages available for your platform, please try them out! "I reviewed piles of code, merged a bunch of code, and fixed piles of bugs." This is usually the shortest sentence in my status reports with the highest amount of time actually consumed. For details, see the ChangeLogs for Tor 0.2.3.17-beta and 0.2.3.18-rc inclusive for everything that actually got merged. If you like to follow code getting reviewed and merged, and you have a high tolerance for incoming email volume, I recommend the tor-bugs and tor-commits mailing lists. Some highlights include: * Tor clients now declare a less fingerprintable (and actually accurate, we hope!) set of ciphers in their TLS handshakes. In 0.2.4, this will enable us to use more secure TLS ciphersuites. (See bug 4744 and proposal 198.) * Tor now enables compiler-hardening options by default. * We made a quick workaround for a horrible bug in OpenSSL 1.0.1 that prevents TLS 1.1 and TLS 1.2 from renegotiating successfully. I forked a maint-0.2.3 branch from master. Now changes to Tor 0.2.3 go into maint-0.2.3, which gets merged forward into master; changes made only in master will appear in 0.2.4 only. I started a new repository called "tor-next" (at https://gitweb.torproject.org/tor-next.git) It has two main branches, "tor-next" and "tor-next-023". These branches are regenerated periodically; they contain the patches that I'm currently considering merging to master and to maint-0.2.3 respectively. This way, complex code can get a little testing before I actually merge it. If the alpha code just hasn't been alpha enough for your tastes, and you like building from source, you might want to give tor-next a spin. Tor-next is automatically generated by a script; you can see it in our "githax" repository at https://gitweb.torproject.org/githax.git/blob_plain/HEAD:/scripts/make-tor-… I've started merging pending things into 0.2.4.x, and reviewing patch series which had been tagged for Tor 0.2.4.x. I participated in a fun reddit "ask me anything" session with Runa, Karen, and others. I've started playing around with shadow to get it running on my desktop. It worked out okay, but I haven't yet managed to actually do more with it than say, "Yup, that runs." I've scrambled to try to get ready for the developers' meeting and for PETS in early July. Apparently, I'm out of practice at arranging and preparing for travel. Andrea and I started sketching out plans and possible schedules for Tor 0.2.4. We're doing okay making plans at the proposal level, but trying to select and cost out individual tickets seems to be proving more trouble than it might be worth. Further, we're a bit stymied by the state of deliverables tracking; that's going to take some brain-dumps at the dev meeting. Still, I'm hoping we can have a rough plan for 0.2.4 some time before mid-July: I'd like to be able to declare merge deadlines some time around then. best wishes, -- Nick

1 0

Re: [tor-dev] Stem Proc Integration Tests
by Damian Johnson 29 Jun '12

29 Jun '12

> Erik and I have finished the proc.py integration tests Great! I'm looking forward to checking them out. > Tomorrow, we will be starting the next Stem task regarding the Tor Export project. Do you have any recommendations on where we should start exploring in the Stem documentation? Hmm, this concerns translating Descriptor subclasses [1] into a csv (and I suppose loading them back up). First step would be to google around to see if there's a builtin function what'll do this or help with it. If not then this might simply include dumping out their __dict__... >>> class Foo: ... def __init__(self): ... self.foo = 5 ... self.bar = "hello" ... def set_bar(self, value): ... self.bar = value ... >>> f = Foo() >>> f.__dict__ {'foo': 5, 'bar': 'hello'} >>> ",".join([str(v) for v in f.__dict__.values()]) '5,hello' You probably want to sort the keys and use that order, but besides that this should do the trick. Just add a unit test and you're done. :) If we want to convert csvs *back* into Descriptors then that's a bit more work. > Also, what do you anticipate being necessary when it comes to parsing consensus entries? A NetworkStatusEntry class (and its tests) would be similar to the ServerDescriptor [2] and ExtraInfoDescriptor [3], but a fair bit easier (it only has six entries). To implement this you'd... 1. Read over the ServerDescriptor and ExtraInfoDescriptor to figure out how we're tackling the parsing and verification in python. 2. Read over Karsten's NetworkStatusEntryImpl class [4] to see how he's handling this in metrics-lib. 3. Read the spec for these entries [5]. Do we need both a NetworkStatusEntryV2 and NetworkStatusEntryV3 class? Exactly how should we model this? Karsten: thoughts? 4. Go to 'https://metrics.torproject.org/data.html' to get some test data, so you see what the things that you'll be parsing looks like. 5. Write a NetworkStatusEntry class with the goal of "figure out each and every way that data might be malformed". A large part of this class is to verify that the data we're being given is perfectly valid according to the spec, so read the spec and implement exactly what it says. If there's some ambiguity in the spec or you see data that doesn't conform to the spec then let us know. That's a tor bug. 6. Write unit and integ tests similar to what the other descriptors have... - unit tests are the majority of the work, and exercise all the use cases that you can think of against mock objects - integ tests are pretty short, and just run the parser against some test data from the metrics archive and the cached consensus Let me know if you have any questions! -Damian PS. Karsten: Do we want to call this "NetworkStatusEntry" or something else? It seems like "ConsensusEntry" would be more intuitive, but maybe this would just spawn confusion. [1] https://gitweb.torproject.org/stem.git/blob/HEAD:/stem/descriptor/__init__.… [2] https://gitweb.torproject.org/stem.git/blob/HEAD:/stem/descriptor/server_de… [3] https://gitweb.torproject.org/stem.git/blob/HEAD:/stem/descriptor/extrainfo… [4] https://gitweb.torproject.org/metrics-lib.git/blob/HEAD:/src/org/torproject… [5] https://gitweb.torproject.org/torspec.git/blob/HEAD:/dir-spec.txt#l1334

3 3

Karsten's status report June 1--30
by Karsten Loesing 29 Jun '12

29 Jun '12

Hi all, below's what I did in June. Yes, there's still one more day left in June, but I don't expect to do anything useful tomorrow other than packing for Italy and pretending to have a half weekend. Best, Karsten Fixed a few problems with the metrics website: bridge usage statistics graphs were missing 85% of reported bridge statistics due to a truncated file containing raw observations (#6064); download times in Torperf graphs were consistently 1 second too high because of a trivial math bug (#6070); some direct usage graphs showing possible censorship events displayed upturns or downturns for every single data point (first part of #6170); direct usage graphs of countries with little usage displayed missing values where they should display zero users (second part of #6170). Added a new graph to the metrics website that shows number of cloud bridges over time (#5629) [0]. Runa is still investigating the sudden drops in March and April, but it looks like they're related to a problem with updating cloud images. Visualized Torperf phases from starting a download to 1) receiving the CONNECTED cell, 2) receiving the first byte, and 3) receiving the last byte of the response (#6120). Looked more into improved bridge usage statistics (#5807) with little outcome so far. The directory request statistics reported by bridges still look very promising for estimating daily bridge users. Tried to find reasons for missing extra-info descriptors (#3266). There are some interesting patterns, but it's still unclear what's going on. Finished making the new combined Torperf data format the new default format that we provide on the metrics website (#3036) [1, 2]. Added support for parsing GetTor statistics files to metrics-lib and made the metrics website code use that library instead of implementing its own parser. Discussed sanitizing pluggable transport lines in bridge extra-info descriptors with George and implemented the required code in metrics-db (#4957). Deployed George's changes to the censorship detector script to send out daily mails for censorship events [3]. Apparently, there are still plenty of false positives on that list, but the newly gained awareness might help improving the censorship detector script in the future. Fixed a problem in DocTor, the consensus-health checker, which looked at old consensuses for some checks when it shouldn't. Helped prepare the metrics host for a downtime of a few hours for getting its RAID card replaced. Looks like replacing the broken card has solved previously existing performance problems quite well. Made a few smaller extensions to Onionoo, the data provider behind Atlas: added bridge nicknames; added support for searching by country code (#5960); included exit addresses in summary and search (#5251); added support for relays and bridges with multiple OR addresses (#5245); allowed searches for more than 1 search term (#5248); added reverse DNS lookup results (#5247). Also wrote a design document for Onionoo [4] to facilitate writing a Python version of it. [0] https://metrics.torproject.org/network.html#cloudbridges [1] https://metrics.torproject.org/data.html#performance [2] https://metrics.torproject.org/formats.html#torperf [3] https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-censorship-events [4] https://gitweb.torproject.org/onionoo.git/blob/HEAD:/DESIGN

1 0

Proc Unit Testing & Next Project
by Erik I Islo 28 Jun '12

28 Jun '12

Hi Damian, Megan and I have finished a first run at writing unit tests for the proc utilities in stem. The code may be found on Megan's Github at: https://github.com/meganchang/Stem/blob/proc-tests/test/unit/util/proc.py. Until we hear back on a code review, we'll move on to the integration tests. Just for reference, here is a link to the ticket as well: https://trac.torproject.org/projects/tor/ticket/6082. Looking further into the future, we discussed our next project with Stem with Professor Danner today, and agreed that the "Export Tor Desciptors" project would be ideal. ( https://trac.torproject.org/projects/tor/ticket/6171) Do you have any thoughts on this? Otherwise, we just wanted to let you know our plan as we move forward. -Erik & Megan

2 4

Re: [tor-dev] stem test output
by Damian Johnson 27 Jun '12

27 Jun '12

Hi Karsten, hi Sathyanarayanan. Over the last couple weeks I've been making a variety of fixes to stem and its tests. It now runs with python 2.5 and doesn't require tor to be in your path, among other things. However, the hanging issue remains and I'm pretty stuck there. It isn't strictly stem related (I can reproduce the issue with plan sockets), hopefully Nick will have some ideas... https://trac.torproject.org/6235 Karsten, should the 'dirreq-v3-share' percentage values be able to go above 100%? The extra-info descriptors that I just fetched has one such entry, which makes stem's descriptor parser complain... extra-info siltornado 995D0FE5A89563D79A383CCC2444D0E26C6BE625 published 2012-06-24 19:30:31 ... dirreq-v2-share 159.81% dirreq-v3-share 159.81% Cheers! -Damian PS. Including tor-dev@ in case Beck or others are interested.

2 1

Trying to connect to a OR
by Steven Retz 27 Jun '12

27 Jun '12

Everyone, I have been toying with doing something with tor for a long time. I decided to start doing something with it. I wrote this quick POC in C# to connect to a tor node I got from the cache file. I tried to follow the specs in https://gitweb.torproject.org/torspec.git?a=blob_plain;hb=HEAD;f=tor-spec.tx t section 2 connections, but apparently I'm doing something wrong. I'm hoping someone can help me out. Steven

1 0

[GSoC] Stem improvements
by Ravi Chandra Padmala 25 Jun '12

25 Jun '12

Hello I have been working on Stem's general controller class. I have completed implementing the methods that handle the GETCONF, SETCONF and the RESETCONF (for the most part, I haven't put some of the code up for review yet). I'm currently working on implementing the SAVECONF, LOADCONF, SIGNAL methods. I also met Sathyanarayanan at Bangalore this weekend and we spent some time working on Stem. He worked on implementing the Controller's QUIT functionality (among other things). -- neena

1 0