[tor-reports] Karsten's May

[Karsten Loesing]

Here are the top 5 things I wanted to do in May:

> - Make sure Tom takes over sponsor F year 3 phase 2.

Tom and I decided that I'll continue coordinating sponsor F year 3 phase
2 and that we'll talk about sponsor F and year 4 at the July dev meeting.

> - Get Shadow running with tor master, rebase and test a few tor
> branches that were written for maintained branches, and ask Nick to
> review and merge them into master.

Made good progress on this one.  Shadow works again with tor master
(thanks to Rob), and I wrote a lot more tor code than usual.  Here are
the more detailed status updates that I also tweeted last month:

May 14: Narrowed down #Tor or #Shadow bug that prevents us from
simulating private 0.2.4.10-alpha networks:
https://github.com/shadow/shadow/issues/117#issuecomment-17880126

May 15: Helped make the #Shadow simulator work with #Tor master again,
and simulated alternative scheduling policy.
https://trac.torproject.org/projects/tor/ticket/8810#comment:3

May 16: Rebased and tested branch that will reduce bootstrapping time of
private #Tor networks:
https://trac.torproject.org/projects/tor/ticket/6752#comment:42

May 16: Rebased and tested another #Tor branch that adds controller
events to better understand connection/circuit usage:
https://trac.torproject.org/projects/tor/ticket/7359#comment:8

May 29: Got #Tor branch merged by @nickm_tor that will reduce
bootstrapping time of private Tor networks:
https://trac.torproject.org/projects/tor/ticket/6752

May 31: Wrote unit tests for #Tor branch that adds controller events to
better understand connection/circuit usage in testing network.

> - Make progress on new user statistics, including obfsproxy and IPv6
> statistics.  Maybe deploy on yatei and get some graphs onto the
metrics website.

Made good progress on this one, too.  The metrics website now has new
user statistics.  Tweets contain the details:

May 2: Finally managed to reproduce a nasty bug in our new #PostgreSQL
application that estimates daily #Tor users from directory traffic. Fixing..

May 2: Diagnosed tricky bug, committed tiny 2-line diff, ran lots and
lots of tests, have confidence that code is more correct now. Productive
day.

May 3: Found a potentially interesting pattern in usage statistics
reported by #Tor bridges:
https://people.torproject.org/~karsten/volatile/bridge-ip-transports-2013-05-03.pdf
Now, what does it mean...?

May 5: Wrote first set of tests for #PostgreSQL application to estimate
daily #Tor users from directory traffic. #pgTAP is fun!

May 6: Analyzed how long we have to wait until we have credible user
number estimates from directory traffic.
https://trac.torproject.org/projects/tor/ticket/8462#comment:7

May 6: New graphs on estimated daily #Tor users by country, pluggable
transport, or IP version available:
https://metrics.torproject.org/users.html#userstats

May 7: Spent a few hours thinking about possible issues with estimating
how many censored users use transports like #ofbsproxy and #flashproxy.

> - Write a database importer for Thomas' Visionion project, so that he
> has some real data to visualize.  (This could be the next generation
> metrics website if all goes well.)

Again, good progress.  See details below:

May 1: Discussed format for new #MongoDB #Tor network visualization
project. Key question was how to aggregate data for 1/6/24/168 hour
intervals.

May 8: Started designing a database importer for #Visionion which is
going to visualize #Tor network data using MongoDB, Angular.js, and D3.js.

May 9: Wrote 1000+ Java lines for #Visionion database importer, half of
them new, the rest reused from earlier projects.

May 10: Produced 1.5G of uncompressed JSON data for #Visionion's
#MongoDB database covering 1.25 months; hope it'll handle 5 years of
data = 72G.

> - Improve my Twisted Torperf implementation so that it reaches feature
> parity with the current Torperf.  Also discuss the JSON data format
> with Mike, so that we can use Torperf to gather more data.

Unfortunately, this one slipped through.  Again.

More things I made progress on:

May 1: Tried to install experimental packages into #wheezy, but got
unresolvable errors. apt is only fun as long as one sticks with a single
dist.

May 1: Tweaked a fine set of #unittests for the #Onionoo #servlet,
didn't find any new bugs. Must write more unit tests to find more bugs.

May 3: Number of #Tor cloud bridges doubles in last two weeks. What
happened there, @runasand?
https://metrics.torproject.org/network.html#cloudbridges

May 3: Restricted valid keyword characters in #Tor metrics data
processor to [A-Za-z0-9-]+ to protect poor tools that can only handle
ASCII chars.

May 4: Reviewed first #GSoC proposal for #Tor.

May 6: Spent the first three hours of the week on answering emails and
reviewing #GSoC applications. Now let's do some software development.

May 7: Made a diagram of the various tools that are part of the #Tor
metrics website: https://metrics.torproject.org/tools.html

May 13: Spent too many hours on semi-automatically removing A1
"Anonymous Proxy" entries from MaxMind's May #GeoIP database to be
shipped with #Tor.

May 14: Set up an EC2 instance to test how #IPv6-only #Tor clients would
connect to the Tor network. No luck so far.
https://trac.torproject.org/projects/tor/ticket/6027#comment:16

May 18: Fixed metrics server after upgrade to #Debian #Wheezy. Had to
recompile a few libraries for #R 2.15. #PostgreSQL upgrade to 9.1 worked
fine.

May 22: Fixed problem with #PostgreSQL 9 handling bytea data type
differently than 8. Upgrading JDBC library did the trick.

May 22: Reviewed a paper about simulating #Tor path selection and the
first three chapters of a book about practical anonymity.

May 27: Tweaked and merged patch that makes Compass command-line tool
show #Tor network diversity by /16 network:
https://gitweb.torproject.org/compass.git

May 28: Discussed possible designs of a Python library to download #Tor
descriptors from directory authorities and mirrors:
https://lists.torproject.org/pipermail/tor-dev/2013-May/004925.html

The plan for June is to make progress on these tasks:

- Pick up sponsor F year 3 phase 2 coordination, talk to developers,
make sure we're on track.

- Get feedback from Nick why IPv6-only tor client doesn't work (#6027),
and get fallback directory list into tor master (#8374).

- Tweak my morestats4 branch until Nick likes it enough to merge it (#8462).

- Deploy new user-stats number-crunching code on yatei, rather than on
an EC2 instance.  Keep a close eye on yatei to make sure it still has
enough I/O left to do its many other tasks.

- Improve my Twisted Torperf implementation so that it reaches feature
parity with the current Torperf.  Also discuss the JSON data format with
Mike, so that we can use Torperf to gather more data.

I'm planning to tweet status updates and use StrangeCharm's/weasel's
TeamStatus to keep people up-to-date on what I'm working on:

https://twitter.com/loesing

https://gs.torproject.org/