- tor-dev - lists.torproject.org

Damian's Status Report - October 2012
by Damian Johnson 02 Nov '12

02 Nov '12

Ahhh, the beginning of winter. That very special time of the year when Seattle skies are perpetual overcast and gloomy. When crowds flock to the coffee shops and the best thing to do with your time is hook up to a caffeine IV and code. ... on a side note it has been a really productive month. ======================================== * Stem Website Stem now has a far more developer friendly website! https://stem.readthedocs.org/en/latest/index.html Barring the obvious new sections I revised vast swaths of stem's API documentation. Feedback welcome! The tutorial section is just starting, but I have some ideas for expanding it. ======================================== * Network Status Document Parsing After a couple months of work by Ravi and me the network status document handling is *finally* done. It turned out to be a... very big feature branch. Announcement: https://lists.torproject.org/pipermail/tor-dev/2012-October/004092.html Docs: https://stem.readthedocs.org/en/latest/api/descriptor/networkstatus.html Code: https://gitweb.torproject.org/stem.git/commitdiff/ea356b7?hp=5cd457b With this done we're very nearly at feature parity with metrics-lib (and I suspect a bit past it in terms of testing). In finishing this up I also spotted an undocumented oddity with microdescriptor 'directory-signature' lines. [1] ======================================== * GSoC Mentor Summit Met with the developers of numerous other open source projects. I was really impressed at how many worthwhile conversations were crammed into such a short conference. Highlights include... * Arc from Python had distutils suggestions, thanks to which stem and arm will soon have both Python 2.x and 3.x support. * OSU's Open Source Labs are entertaining the thought of running Tor relays. * Discussed TorBirdy and other Tor projects with Sukhbir. He mentioned an issue with trac permissions that is now fixed. [2] * Talked with Terri from Python about Mailman 3, which would be a great answer for the requests we get to have a Tor forum. [3] * Attended a talk led by Marina from Gnome about their outreach program for women. We'll likely be taking part in it this year. For the full trip report see... https://lists.torproject.org/pipermail/tor-dev/2012-October/004115.html ======================================== Other changes include... * Cleaning up orphaned pyc files as a part of running our tests (feature suggested by Ravi) [4] * Ported arm's str_tools module to stem and added unit tests [5][6] * Code reviewed Ravi's attach_stream addition, waiting on the revisions [7] * Arm troubleshooting on irc with zack and ultramage. Also fixed a broken link on arm's site spotted by wh6iQ. * A new contributor, Eoin, spotted some great issues and contributed patches. Tickets include... * python 2.5 compatibility bugs [8] * count of the number of skipped tests [9] * error with the whitespace checker [10] * spelling corrections [11] * Sathyanarayanan spotted some nice bugs including... * on python 2.5-2.6 a missing microdescriptor consensus wasn't causing the related test to be skipped [12] * integ tests for the process module would fail if tor was already running [13] ======================================== Next up: Ravi and I are working on tor event handling, the last major feature stem is missing as a controller library. After that we plan to port arm over, then tidy up loose ends in preparation for stem's initial release. Cheers! -Damian [1] https://trac.torproject.org/7072 [2] https://trac.torproject.org/7186 [3] https://trac.torproject.org/3592 [4] https://trac.torproject.org/7103 [5] https://trac.torproject.org/7158 [6] https://stem.readthedocs.org/en/latest/api/util/str_tools.html [7] https://trac.torproject.org/7152 [8] https://trac.torproject.org/7244 [9] https://trac.torproject.org/7246 [10] https://trac.torproject.org/7263 [11] https://trac.torproject.org/7245 [12] https://trac.torproject.org/7250 [13] https://trac.torproject.org/7251

1 0

circuitmux_ewma.c
by Gisle Vanem 01 Nov '12

01 Nov '12

These gcc extensions: /*** EWMA circuitmux_policy_t method table ***/ circuitmux_policy_t ewma_policy = { .alloc_cmux_data = ewma_alloc_cmux_data, .free_cmux_data = ewma_free_cmux_data, Doesn't work well with MSVC. Here is a patch: --- ..\..\Git-latest\src\or\circuitmux_ewma.c 2012-11-01 15:53:23.125000000 +0100 +++ circuitmux_ewma.c 2012-11-01 17:52:55.371093500 +0100 @@ -200,15 +200,22 @@ /*** EWMA circuitmux_policy_t method table ***/ -circuitmux_policy_t ewma_policy = { .alloc_cmux_data = ewma_alloc_cmux_data, - .free_cmux_data = ewma_free_cmux_data, - .alloc_circ_data = ewma_alloc_circ_data, - .free_circ_data = ewma_free_circ_data, - .notify_circ_active = ewma_notify_circ_active, - .notify_circ_inactive = ewma_notify_circ_inactive, - .notify_set_n_cells = NULL, /* EWMA doesn't need this */ - .notify_xmit_cells = ewma_notify_xmit_cells, - .pick_active_circuit = ewma_pick_active_circuit +#ifdef __GNUC__ + #define STRUCT_INIT(member,val) member = val +#else + #define STRUCT_INIT(member,val) val +#endif + +circuitmux_policy_t ewma_policy = { + STRUCT_INIT (.alloc_cmux_data, ewma_alloc_cmux_data), + STRUCT_INIT (.free_cmux_data, ewma_free_cmux_data), + STRUCT_INIT (.alloc_circ_data, ewma_alloc_circ_data), + STRUCT_INIT (.free_circ_data, ewma_free_circ_data), + STRUCT_INIT (.notify_circ_active, ewma_notify_circ_active), + STRUCT_INIT (.notify_circ_inactive, ewma_notify_circ_inactive), + STRUCT_INIT (.notify_set_n_cells, NULL), /* EWMA doesn't need this */ + STRUCT_INIT (.notify_xmit_cells, ewma_notify_xmit_cells), + STRUCT_INIT (.pick_active_circuit, ewma_pick_active_circuit) }; /*** EWMA method implementations using the below EWMA helper functions ***/ --gv

2 2

Reminder of November, December deadlines for Tor 0.2.4
by Nick Mathewson 01 Nov '12

01 Nov '12

Hi, all! >From the Tor 0.2.4 schedule at https://trac.torproject.org/projects/tor/wiki/org/roadmaps/Tor/024 : """November 20, 2012: Big feature proposal freeze. Any big feature which requires a design proposal must have its design proposal finished by this date. If it needs changes after this date, then maybe we didn't design it well enough, and we probably shouldn't merge it. December 10, 2012: Big feature merge freeze. No big features will be merged after this date. December 15, 2012: Small feature proposal freeze. Any small feature which requires a design proposal must have its tiny little design proposal finished by this date. January 10, 2013: Feature merge freeze. No features after this date. I mean it. ... The definition of "big patch" is deliberately vague. Note that the cutoffs above are merge freezes, not submission freezes.""" (I've just added 10 days to that first deadline, and 5 days to the third.) Also, I'm currently thinking that, to give these deadlines teeth, I shouldn't self-enforce, since I've veered between "pushover" and "insane zealot" in the past in ways that didn't make a lot of sense. Rather, I'm going to say that in cases of doubt ("Is this a little feature") and cases where I'd like to violate freezes ("surely this can go in on December 12"), I'm not going to merge patches on my sole discretion any more. Rather, I'm going to say that at least two of me, Roger, and Andrea ought to agree to violate a freeze before we do so. (I will try not to forget and mess this up.) With any luck, this will let us get closer to a reasonable 6 month release cycle. yrs, -- Nick

1 0

[Patch] tools\tor-fw-helper\tor-fw-helper-natpmp.c
by Gisle Vanem 01 Nov '12

01 Nov '12

The code in tools\tor-fw-helper\tor-fw-helper-natpmp.c does things wrong on Winsock where e.g. errno isn't set on socket-errors and 'fd' is (always?) >= FD_SETSIZE. Patch attached. --gv

2 1

gsathya's October
by Sathyanarayanan Gunasekaran 01 Nov '12

01 Nov '12

I was mostly busy with giving the required exams for grad school and generally worrying about. I did manage to do a bunch of Tor stuff - - Implemented DisableNetwork in Orbot. Orbot stops making new connections and drops existing ones once you lose network connectivity, saving battery life. Also, one of the features I implemented during GSoC11 is now finally merged into master -- Orbot now shows usage statistics(bandwidth). - Wrote a whole bunch of tests for Pyonionoo -- which in hindsight weren't really good. I will mostly need to refactor them and make them more modular. - Added more functionality to Pyonionoo -- /bandwidths and /details work now - Hacked on #1854[0] - Set up a pyobfsproxy bridge to help test. - Found some bugs in stem - #7133, #7250, #7251 - Generally hung around #tor providing support. Hopefully I can do this a lot more. [0] - https://trac.torproject.org/projects/tor/ticket/1854

1 0

use tor controller to detect circuit/path change
by esolve esolve 31 Oct '12

31 Oct '12

Hi, I'm new to Tor controller, Tor client usually open 3 circuits, one for current use and the other 2 for standby. Tor will change the current circuit/path in use from time to time, depending on the network load dynamics I'm wondering whether it is possible to use Tor controller to detect when the circuit/path in use is changed and obtain the list of IP addresses of those relay nodes. thanks!

4 6

[Stegotorus] Re: Do you have time to poke at ST
by vmonmoonshine＠gmail.com 25 Oct '12

25 Oct '12

Hello Zack, Sorry for being bit out of touch as I'm traveling and my access to Internet is sporadic. I'm still working on Stegonotorus. I developed the webpage_tester (now on f--webpage-tester branch on github), that is curl using Stegotorus to download anything on the web and compare it with the original. It is based on assumption that is my best approximation for a browser. I did that in a hope to catch the bug, but I haven't. Basically I failed to browse anything unreliably. I can download 20 megabytes or so with no error. It takes a long time, but It doesn't fail. My payload server is the apache payload server but that shouldn't make a huge difference. I think my payloads even have smaller capacity and less efficient than your fake ones. Anyway, I'm going to tweak the tester proxy to drops packets at random, to force the ack into play, I'll get back to you with more update. Cheers, vmon

1 0

New tech report: Counting daily bridge users
by Karsten Loesing 24 Oct '12

24 Oct '12

Hi everyone, I just finished a new tech report titled "Counting daily bridge users": https://research.torproject.org/techreports/counting-daily-bridge-users-201… >From the abstract: "As part of the Tor Metrics Project, we want to learn how many people use the Tor network on a daily basis. Counting users in an anonymity network is, obviously, a difficult task for which we cannot collect too sensitive usage data. We came up with a privacy-preserving approach for estimating directly connecting user numbers by counting requests to the directory mirrors and deriving approximate user numbers from there. In this report we describe a modified approach for estimating the number of users connecting via bridges by evaluating directory requests made to bridges. We compare this new approach to our current approach that estimates bridge user numbers from total unique IP addresses seen at bridges. We think that results from the new approach are closer to reality, even though that means there are significantly fewer daily bridge users than originally expected." Feedback is much appreciated and could influence our implementation of user number estimates in the future (#7154). Thanks, Karsten

1 0

Re: [tor-dev] Automating Bridge Reachability Testing (#6414)
by Isis 23 Oct '12

23 Oct '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Karsten! Oh sheesh. I did not see it...I will have to figure out why. That is slightly worrying. So, I am rushing to meet the final deadline, but I still think it is doable. I have mostly finished up my OONI work for the month, and I planned to spend the remainder of this month working on the bridge test. I have finished most of the actual Tor connection code, as well as one of the four basic packet level scans (the icmp8 one). Two of the other packet level scans, the TCP SYN and ACK ones, are pretty much copies of the icmp8 one with a couple lines changed, and they shouldn't be a big deal. There is still the vanilla TLS handshake test/scan/thing, which has not been started yet, and will take a bit more time than the others because Python notoriously has problems with SSL bindings and libraries, so I'll need to do a bit of research on newer ones and updates and see which is the best to use now. I hear that tlslite[1] is the current best choice; if anyone else has any input on this it would be very helpful. :) There were a couple minor hups: 1) When George asked me to test pluggable transports, this required significantly more refactoring than I previously thought was necessary. 2) Arturo redesigned the OONI testing framework API again to use a completely different structure, which was supposed to be backwards compatible and turned out not to be (though I believe that my recent OONI commits fixed that). However, I have been fighting the framework already, because the main scripts in OONI (/ooni/oonicli.py and /ooni/ooniprobe.py) control the reactor, and also expect static iterations through single test and single control functions for each asset (an asset in this case would be one bridge address). The bridge testing is rather dynamic (I would like it to be able to evaluate an approximate danger level to running the next test) and so the framework is kind of troublesome. Also, because the framework handles calling the reactor (in Twisted, the reactor is a sort of event scheduler), and it also expects a rather linear progression of defer.Deferreds (in Twisted, those are standin objects which execute callbacks when they get results from some previous deferred/callback), it would be nicer if I had full control of these myself without needing to hack around the parent scripts. I think it's wise that OONI deals with these things for the testwriter in most cases, because the testwriter shouldn't be expected to be an expert in using Twisted. However, I also think that, in the long term, OONI shouldn't prohibit people who know what they are doing or are doing odd things from being able to do so. As a result, I've decided (for now), to use bits are parts of the OONI code before the recent refactoring, and later (after the deliverable) I will work on adding flags to OONI to give the test script full control of the reactor and deferreds, as well as evaluating whether or not the bridge test is even compatible with the new API. I do not want to get caught up in dealing with this right now, I just want to have it all working and deployable in a way that I know will work. 3) The indirect scans are becoming quite complicated to automate in any sane fashion. I still would like to continue working on this, as I'm quite enjoying the difficulty, but due to their temporary and volatile nature (they will change frequently depending on the blocking methods of a particular country and the currently available in-country bounces/proxies/whatever-thing-the-indirect-scan-uses), as well as the fact that many of these methods are still undiscovered, I think it is safe to add them as specialty cases after the fact without impacting overall general testing. There is one in particular that I would like to finish before the deadline because I am quite proud of it and am having a lot of fun working on it, but I'm first going to concentrate on wrapping up the active scans. There are other things which I've marked as helpful things to do, but which are not necessarily part of this deliverable: 1) Having a parser for bridge descriptors to turn them into test inputs, and vice versa. 2) Having some undiscoverable method for setting up lots of IPv6 bridges on one OR (Tor currently only allows up to eight, I believe) and having these be discoverable by bridgedb and no one else. I was thinking of this while talking with Aaron, because he reminded me that people on IPv6 have tons of IPs available, and I was thinking that if we configured some type of one-way hash function, we could say that a bridge descriptor for 2001:db8::1:1 should actually mean multiple descriptors for 2001:db8::fa98:38d2 2001:db8::e099:2188 2001:db8::88aa:3b7 or something, derived from the output of hashing the original descriptor with the OR's key or something else. This would help distribute bridges in the future quite a bit, though it doesn't do much for the current bridge situation. Anyone wanting to help with the above two things, or with an idea for another indirect scan, or with feedback on anything I'm working on, should feel free to contact me and it will be greatly appreciated. :D .-- .. ... .... - -- . .-.. ..- -.-. -.- <(A)3 isis agora lovecruft .-.-. [1] https://github.com/trevp/tlslite On Fri 12 Oct 2012 at 14:56, thus spake Karsten Loesing: > Hi Isis, > > did you see this email? > > Sorry for being pushy, but we're less than three weeks away from the > deadline, which makes me kinda nervous. > > Thanks, > Karsten > > > On 10/8/12 9:14 AM, Karsten Loesing wrote: > > Hi Isis, > > > > can you give me a brief status update on the sponsor F year 2 > > deliverable "Automating Bridge Reachability Testing", ticket #6414? > > > > In particular, how did the two project-internal milestones on September > > 15 and October 7 go? > > > > Do you mind updating #6414 with the status? > > > > The overall deadline is on November 1, which is in 3.5 weeks. How > > realistic do you think it is that we'll have automated bridge > > reachability tests by then? > > > > Thanks, > > Karsten > > > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCgAGBQJQeJ4pAAoJEKOttnos24s1GlkP/1ODucFe9q9a7KaKisgVqBcx Cg1X3yGI+dsWvKtMwzgNqT/hcS+95GAYQiaa8fDlbiLY3nCWEsGYSOTjsbAmp04O qPCdBiSFE4VqiZy4PYUHUkf8qZTYkXqAiMnyuuLVJINJ+BHLAdwEmEWB77ncQUxV PvG6sxQ3GY64ZqRaSHwudf3H8mEPN/tNWRIXubnvAGM16EV0hWb51uORwo6CG86m lKwpBqbSX5uDgG+B6HCeZps0Zv7cDypI3dtdd8hJHncUpFmfuj2y+Oad1M5DVbMC 4UMvCv/4tMPoPyjTYjWgLQRHEFiMdMwQcdPntnF6XqUBMLvd6v1fk1IxaaqNCDah JWxK8ymUu90FLmmiyaSHQE/tfkFoEDCXz3XzR8xQA1zI2aq0hHdxA+G2JmW8yadQ wt0N4EeCzK154r4+YRC1eJLrsSOhmA7SaAE6A+/iEOgasdMY0YWchTDnlUbUB5iv acbMVITSGqmcRMos5xi450V98Qw1nW3UoBj5wi8uh0BWqnhqfomqL6oloQEdgAze MqtCZ91adKKXntIsEZW0C/klBYg5mZIxdtmAW5eO/j+ogLB/L0/hX4/TZ6OH5i/N 7jJWAF0MbSWI2Q1Gk+oFFulib5m3XWzqg5Yq/+8VfE5n1R5xM1FgBwC932SWLOFZ BV5c9ntZ+iJlbhpdL5yJ =jk13 -----END PGP SIGNATURE-----

4 5

Trip Report: GSoC Mentor Summit
by Damian Johnson 22 Oct '12

22 Oct '12

As per conferencing tradition Friday was spent on travel and meeting the other attendees. Some of the highlights for me were... * David from KDE Besides demoing some KDE eye candy we discussed their project infrastructure. KDE is a federation of smaller projects and had over sixty students this year (ten times the number mentored by Tor). Their project's scale has led to some unusual infrastructure decisions. For instance, they have a partly decentralized git infrastructure where pushes go to a single master host and pulls are from any of several mirrors. The config they use to do this leads to some... odd behavior. For instance a 'git pull' updates your tracking branch but not the origin branch reference. The result is that to do a pull for realz you need to call *both* pull and fetch. No doubt they also get fun behavior from mirroring delays... We also talked a bit about post-review and defaults they could set to better support their setup. KDE has the largest public ReviewBoard instance, but the above git setup makes it a bit confusing to use. * Sukhbir from Debian In 2011 Sukhbir applied to us for GSoC to work on TorBirdy. We loved his proposal, but due to prior commitments he ended up working with Debian instead. Since then he has become a GSoC mentor for Debian and involved with the Tor by implementing his earlier proposal for TorBirdy. Sukhbir's interested in getting even more involved with Tor so we discussed other projects that might interest him, and ways that we could better publicize TorBirdy on our site. * Arc from Python When I found out that Python had a mentor at the summit I made a mental note to hunt him down and ask about packaging best practices. After an unexpected discussion about rugby I found out that it's actually easy to support both python's 2.x and 3.x series by including a 2to3 conversion at build time. This can be done via either distutils or distribute... distutils: http://python3porting.com/2to3.html#running-2to3-on-install distribute: http://packages.python.org/distribute/python3.html I also asked him to look into his crystal ball for when python 3 would take over the world and he said 'Next year. Ubuntu and Fedora are ready and willing to make the switch. The last main holdout is Gnome. They tried to migrate but work there isn't finished yet.' ======================================== Saturday was the first day of the unconference. After an amusingly confused attempt to have each of the couple hundred attendees shake each other's hand there were sessions. Some were a little interesting, but I spent more time on the hallway track since that's the real benefit of the summit. The only useful tidbits I got from the talks were... - Do outreach early. The successful GSoC students who stick around tend to be the ones that get involved before the application phase. We should try harder to recruit college students to hack on tor, with the carrot that this'll give them a leg up when applying for the program. OpenHatch (http://openhatch.org/) might be something to look into for this. This would be a nice task for a community manager if we get one. - Google Code In (https://code.google.com/gci/) is a program somewhat similar to GSoC where highschool students become involved with open source. Last year they had 18 organizations and this year they're narrowing it down to 10. I was already highly tentative about having us apply and now that I've heard more I'm sure we don't have enough bandwidth for the hand-holding this would require. As for the hallway track... * Adriano and Luis from Umit Last year Adriano showed me Open Monitor (http://www.openmonitor.org/) a censorship detector written in python. Sounds familiar? I thought so too, and tried a few times to get them to talk with Ooni Probe and vice versa without success. My impression is that they're UI developers (a skillset we sorely lack in the tor project) with a rather unscalable backend, while Ooni Probe's backend is far more mature but lacks any sort of UI for rendering real time censorship information. I made another stab at getting the two projects to talk, after which the meeting took a weird turn with Adriano arguing that 'some censorship is good'. Evidently they decided that Open Monitor won't look for censorship concerning 'porn or terrorism'. I argued that this was a slippery slope and that censorship monitoring shouldn't try to pass a moral judgment on the content being censored, but after a time it was clear that we were talking past each other. I still think that we should leverage their UI expertise, but that's up to the Ooni Probe devs. * Open Source Lab Met with a couple administrators from the OSU's Open Source Labs (https://osuosl.org/) They provide hosting for several of the largest open source projects including Apache and the Linux Foundation. Mostly we talked about amusing legal threats they get for hosting the phpBB project. Evidently lawyers are quite skilled at clicking the 'this is a phpBB forum' link followed by 'hosted by the OSL' before sending their angry emails. We also talked a bit about setting up non-exit relays. They might be pretty receptive to this if we want to follow up. * Sumana and Rob from Wikimedia Unsurprisingly Wikipedia occasionally has issues with spammers using Tor. We talked about some possible options, such as requiring accounts for Tor users to edit with a sort of proof of work in account creation to make ban evasion more of a pita. * Terri from Python Mailman 3 is coming, and with it an interface that *doesn't* look like it came from the 1980s! Most importantly for us, the new version of Mailman provides a forum interface, letting email and forum users communicate by whichever method they prefer. This would be a good answer to our forums ticket (https://trac.torproject.org/3592) She estimates that it'll be ready in six months or so. ======================================== Flying out on Sunday cut my day short, but there was one session that I thought was interesting. Gnome and Wikimedia are launching a program similar to GSoC to encourage more women to get involved with open source. It runs later this winter. One gotcha is that Google's not involved so mentoring orgs need to cover the $5k stipend. I like the idea. Is this something we want to take part in? If so then I'd be happy to administer the non-financial parts of it. Cheers! -Damian

2 1