September 2015 - ooni-dev - lists.torproject.org

Working with OONI reports; or, how we used OONI to find discrimination against Tor exits
by David Fifield 24 Mar '16

24 Mar '16

I am working on a project with Sheharbano Khattak, Sadia Afroz, Mobin Javed, Srikanth Sundaresan, Vern Paxson, Steven Murdoch, and Damon McCoy to measure how often web sites treat Tor users differently (by serving them a block page or a captcha, for example). We used OONI reports for part of the project. This post is about running our code and some general tips about working with OONI data. I hope it can be of some use to the ADINA15 participants :) The source code I'm talking about is here: git clone https://www.bamsoftware.com/git/ooni-tor-blocks.git One of its outputs is here, a big poster showing the web sites with the highest blocking rates against Tor users: https://people.torproject.org/~dcf/graphs/tor-blocker-poster-20150914.pdf I am attaching the README of our code. One of OONI's tests does URL downloads with Tor and without Tor. The code processes OONI reports, compares the Tor and non-Tor HTTP responses, and notes whenever Tor appears to be blocked while non-Tor appears to be unblocked. Our code is focused on the task of finding Tor blocks, but parts of it are generic and will be useful to others who are working with OONI data. The ooni-report-urls program gives you the URLs of every OONI report published at api.ooni.io. The ooni.py Python module provides an iterator over OONI YAML files that deals with encoding errors and compression. The classify.py Python module is able to identify many common types of block pages (e.g. CloudFlare, Akamai). Now some notes and lessons learned about working with OONI data. The repository of all OONI reports is here: http://api.ooni.io/ The web site doesn't make it obvious, but there is a JSON index of all reports, so you can download many of them in bulk (thanks to Arturo for pointing this out). Our source code contains a program called ooni-report-urls that extracts the URLs from the JSON file so you can pipe them to Wget or whatever. (Check before you start downloading, because there are a lot of files and some of them are big!) wget -O ooni-reports.json http://api.ooni.io/api/reports ./ooni-report-urls ooni-reports.json | sort | uniq > ooni-report-urls.txt The choice of a YAML parser really really matters, like 30× performance difference matters. See here: https://bugs.torproject.org/13720 yaml.safe_load_all(f) function is slow. yaml.load_all(f, Loader=yaml.CSafeLoader) is what you want to use instead. yaml.CSafeLoader differs slightly in its handling of certain invalid Unicode escapes that can appear in OONI's representation of HTTP bodies, for example separately encoded UTF-16 surrogates: "\uD83D\uDD07". ooni.py has a way to skip over records like that (there aren't very many of them). With yaml.CSafeLoader, findblocks takes about 2 hours to process 2.5 years of http_requests reports (about 33 GB compressed). There are some inconsistencies and format differences in some OONI reports, particularly very early ones. For example, the test_name field of reports is not always the same for the same test. We were looking for http_requests tests, and we had to match all of the following test_names: http_requests http_requests_test tor_http_requests_test HTTP Requests Test In addition, the YAML format is occasionally different. In http_requests reports, for example, the way of indicating that Tor is in use for a request can be any of: tor: true tor: {is_tor: true} tor: {exit_ip: 109.163.234.5, exit_name: hessel2, is_tor: true} And even in some requests, the special URL scheme "shttp" indicates a Tor request; e.g. "shttp://example.com/". The ooni.py script fixes up some of these issues, but only for the http_requests test. You'll have to figure it out on your own for other tests. A very early version of this processing code appeared here: https://lists.torproject.org/pipermail/ooni-dev/2015-June/000288.html

1 1

Mini hackathon in Rome (1-2 October) instead of ADINA
by Simone Basso 30 Sep '15

30 Sep '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Dear all, Since our tickets are not refundable, I'll travel to Rome in any case with some colleagues and do some hacking. We will probably do some work on visualizations (perhaps World Censorship Report or World Censorship Map) and on MeasurementKit. If you'd like to join us, drop me a line! Cheers, Simone - -- Simone Basso http://nexa.polito.it/people/sbasso -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJWCkBmAAoJEIC2kSd3M9lbVCAP/jhjWqDgS+5pl+iYeHsd5KUN 9+pb1UKLAIZhYEtkM2vPLKv3zXB08Z3GO+re+zqtJbivBddwD0hMDgQ4r/zwDhVx 64K9fX5GM7Kqf54FPkRQs4sr2t1Fv2Hf1qvpWmVkiwp1b8bkTXGvELEDFti/ts0m UIrfSc+KupAUEwj5gEFQrMaSqyvZNwJxop30uM1s0s2EnlTJYI3iDy00KWWZsD2B GUq0KdrsVwIr/9aWxjDsssr2JV8wUMQ1FaaQxUqJBCE6J0k+ruS1XZtOa1R5Ky4b rVVVxOp4dkpg2/ZG3VvLM12xLHOOhYXEk6HMrh6u3KQXLA4h1j2H4+US5RqlyrYm yS33vq10+ggDG19i4HPybDTgHQf4nYTatqcZ9bLvVSaVzrYGIePg6dzPqCvx+De4 +9SFiwz67h+RNJxfEFwoxM/C0Qb9xdeAOEYjG5OrJ7l3HNWw4SpI91cQgNBUXQGD LlZHhTtziYwpetjY9M+XuUr6cx5ZC+NLRHN7U1Eo+B6QkmGSRXLnntJLYQF6yj/z sLAeKWEad8sefTfZuJfFw9SOApFcq18hcI44OtdQdcRJnTBc3EoURKOEkJeif7jb dGUWsjLc+mM7RDnImDh2kgkXgPftbzSjQoku9FkZVJAUknox/aGZNw2rSry/mzRD qmJIUB5U7evxJrgVmSQE =3DSy -----END PGP SIGNATURE-----

1 1

Reminder for weekly OONI dev meeting Monday 2015-09-28 17:00 UTC (19:00 CET)
by Vasilis 28 Sep '15

28 Sep '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hello, This is a reminder that today there will be the weekly OONI meeting. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (19:00 CET, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Vasilis -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJWCVNdAAoJEF+/cLHRJgFiH64P/1+7K7c5kG5gM35lGkSDlDDi 3MB9bWeoaS9+gsLHjBPsZ+RwdbmeW5Ax0hTIbxKla33B9/xpNzLroR8+O+YEc0t1 2l+WdsWjcWrdtI/X0Yr+6oPzNPj1FSt0/7chtPD//GnBsVjG8/frOSbvQoCK2P8a F94mCU0UULkKLdXj+zvLvTG7GrUkTTpn84hzAwFWuDtml+yFuAonvC6h+6Es870y GWI7yQiOKj3I6Wo3vW5ErIuUPnxUPeuO0gX508t8aexKsZEBlQEUmrdZ07Q1Ey0R jaUcCVs9H8gnHjnGZbGBGnqRzX4luhM4tG/c872bVg+z2ha1w4AxXAVKBAu4v7AQ QLBU2sC551H2f5Jp9pBpuRI7iOrfOnTzDHgbm5anK5HQTP10equCWeloyQkBSts1 /6Y1iEZkbIkbnIHqB31++bXFD5MRD3yFWmytU2L2j/HVPa+WvlE1UzxlliZF3sG8 +UC8K/+TlvFHskPUgEnM9o0/eieNKrP6+ADQqHOLCcp99REqwN/YJBb27pgffTxP 77pniMbbqfxF23Itsqj3QA2uBHVMB+2ohY/4LKHkqe2uqw0XNTB5nWGQYNSMFnwN FUZVBeVWyqZLbOMuBBmAl7+eTbQneWG3uzCV63e9xTLfGywDcjcc1ssVFoeoXbWU 2GToOEOb5KF63kheOYmb =PGeZ -----END PGP SIGNATURE-----

1 0

OONI #ADINA15 Hackathon postponed
by Vasilis 27 Sep '15

27 Sep '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Greetings! We are sorry to mention that but unfortunately the OONI hackathon that was planned in to take place on Oct. 1-2 in Rome has been postponed to an unknown (yet) date! It was not an easy decision but we cannot do anything from our side to change this: We had to postpone the hackathon, in fact, due to health issues that would have forced some core OONI developers to skip the hackathon itself. This would have greatly reduced the overall value and usefulness of the event, hence we decided it was better to reschedule it. An announcement will follow when a new possible date for the event will be available. Thank you, - -- Vasilis and Simone On behalf of the OONI Team -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJWCDHpAAoJEF+/cLHRJgFi6FwQAKuUbpg+mVcm5NkfAmymuJoE eXpWvojIVi8fBXWA6BjHhMaBPMc4Rbv3oOeg+5NtLIudPBWpEAHHZR6Ypsgh6CcI 7ZnfbsEt06r47dhLxZRkJTr12mMWUiF33THxee97bRKi7Z8KoqxS2lIM3jmNF4LE 8R4smy81Z2PQ6OJk0JczgupmIMPe5sT0ZikYaD6KWHcYlZbLlWlmeYnmC1ludzBr cCCZeuTrstDZo8Nxc59NBVIUcRRtzMeHYks1OxqVwAcvFPpES7JKwCInqyyq05oG LJByeJRrcZuwnWy4P196dhIJ+rS90MlyRM6KR42g8droNmqznRiFXF4ntVu0u5FM gMlk54NaP/lPOheu/wdJ21Lpr8rn17WyN/iBStTmRdOXA78PpSmFNPtym+B+53Xh hcAdIyG3YhFeeM+5Wd2MvS7CGefu1Y+ibC/Hb5RG62Xbh9xEWn8mEmdCIfzybEr5 XC6MafEBEG/Aon0mCnP1gnm5GYMLzQiTou7OXzrLgzI7krjsB49dfJavlNZteT3P gqxYhyRLpCbd1sEzjlCicn590kL80TJRg98+l9St/qzkzUG4QvXKnOyHJ+SBKdwY U3eI4GVDisBlAzb79+BA/GJ+oWaG7wXJFSGMbO5V4eDW5KyBy9sac+ew79f7E+S6 3EjLUV86n/biEx/13VZB =6oSg -----END PGP SIGNATURE-----

1 0

Re: [ooni-dev] Resuming OONI bridge_reachability tests?
by Arturo Filastò 21 Sep '15

21 Sep '15

Hi David, Apologies for not replying sooner more extensively. Unfortunately the machines on which we had those probes running were shut down due to some unpaid bills and poor sysadmin management on my side. We talked a bit about this during this Tor Measurement meeting: http://meetbot.debian.net/tor-project/2015/tor-project.2015-07-15-14.00.log… At the moment OONI does not have enough resources to dedicate time to making this a priority at the moment, but anybody that considers this to be a priority should absolutely hack on this! To be clear I am not saying that we don't want to do this, it's just that we want to do so many other things (run a hackathon at the Italian Parliament, implement exploratory tools to ALL the OONI data with the help from Brennan Novak, get more stable general purpose ooni-probe in many more countries with lepidopter, setup our big data cluster in Humboldt university, etc.) and have finite resources. Getting them running is basically a matter of having time. If somebody wants to take this up I can send them the information I sent to Jens about setting them back again (logins, accounts, etc.), but I will need to be pinged otherwise it will slide to the bottom of my todo list. ciao! ~ Arturo > On 21 Sep 2015, at 09:58, David Fifield <david(a)bamsoftware.com> wrote: > > I would love to have some ongoing measurement of the default TB obfs4 > bridges. > > However, it seems that there are no bridge_reachability tests since May > 2015 at http://api.ooni.nu/. > > Are these test results being stored somewhere else, or are they not > running? What would it take to get them running again, even on a small > number of ooniprobe installations? >

2 1

Re: [ooni-dev] Cooperation between OONI & RIPE Atlas
by Virgil Griffith 15 Sep '15

15 Sep '15

> I'll look into this & see if someone can make it. > Can is be over... video-conferencing?? > Some of the people who might contribute have family > obligations that stop them from traveling to much... Totally understand that. I feel the primary benefit is making social connections between the Tor Measurements people and the RIPE measurements people, however, if teleconference is the best we can get we will take it. > I asked my colleagues to look into this, I'll get back > to you with more info later. > Any more info? Which countries are you looking into? How > "scriptable" should the solution be? How often do you > have to make decisions? Who will be using the data - > experts or end-users? etc-lots of questions :) This is a little hard to answer. But in short we want to minimize the amount of useful information than an adversary logging Tor-relay traffic receives. This is often thought to be equivalent to "maximizing topographic diversity", but the two are unlikely 100% the same. A precise definition of what we want to minimize will probably have be provided by Nick or Roger (prob in Berlin). But to give you something to go on now, we have two explicit use cases. They are: (1) Torservers.net has money to place a new relay. It wants a list of ASs sorted by the topological diversity that a relay in that AS would add to the Tor network. (2) We are developing a rating system that quantifiers how much a given operator contributes to the Tor network (see www.tor-roster.org) We want to take every existing relay and give it "ASN bonus points" for being on that particular AS. ASs that are relatively distant from the rest of the network are worth more points. Answering your questions... * Application (1): Doesn't have to be automated. Maybe once a month. Only for experts. https://www.torservers.net/ is the obvious user. * Application (2): Needs to be automated. Every day. Only for experts. http://www.tor-roster.org/ is the obvious user. >> The proposal is encourage OONI agents to place an Atlas >> anchor next (to) every major point. > (how many points?) OONI people: Can one of you comment here? How many points could you commit to place Anchors on? Would Anchor measurements be sufficient > 1) current model of placing RIPE Atlas anchors is for > the operators to pay for the HW themselves -- it's a > Soekris box + SSD, total price around 800 Euros. OONI people: Comments? How much is it worth it to you to have traffic disruptions be automatically seen by the wider network infrastructure community? If 800 is too much, here's a proposal... instead of putting the RIPE anchor *next to* the OONI probe, what about the RIPE anchor *being* the OONI probe? If the RIPE anchor becomes the (obviously unofficial) OONI probe... 800 euro seems reasonable... maybe OONI can get a volume discount? OONI people: Are there features the RIPE anchor would need to fully serve as a OONI probe? > Even if some of them are involved in censorship. And they > are also paying for the project -- so if they think that > the results are not favorable towards them, they might > get pissed off. Totally understand that. I do not speak for OONI, but I get that RIPE must maintain neutrality and cannot be directly involved in anti-censorship causes. To support this, I suggest RIPE frame any cooperation as "enjoying the free labor of OONI's 'network disruption enthusiasts' to achieve RIPE's neutral goal of having an accurate up-to-date picture of the Internet. Whatever goals OONI has is their own business. RIPE is just happy to have an unpaid army placing Anchors everywhere." -V On Tue, Sep 15, 2015 at 5:53 PM Vesna Manojlovic <BECHA(a)ripe.net> wrote: > Hi Virgil, all, > > On 14-sep.-15 20:53, Virgil Griffith wrote: > > > When is the "Berlin meeting"? > > Sept 27 - Oct 3. > > > > > https://trac.torproject.org/projects/tor/wiki/org/meetings/2015SummerDevMee… > > I'll look into this & see if someone can make it. > > > Most of the Tor-dev team will be at the Berlin meeting, and we'd love to > > have a presentation on your work! > > Can is be over... video-conferencing?? > > Some of the people who might contribute have family obligations that > stop them from traveling to much... > > > One open-problem we have is how to > > quantify the "AS topological diversity" of the Tor relay network. > > We have two data sets & "tools" that might help you there: > http://ris.ripe.net > http://stat.ripe.net > > and maybe > https://stat.ripe.net/special/bgplay > > > One recent proposal is to use data from: > > http://labs.apnic.net/vizas > > looks good :) > > > And prioritize placing Tor relays on the well-connected ASs that don't > > already have relays. Very interested in any ideas for using your data > > to quantify Tor's topological diversity. > > Me too! > > I asked my colleagues to look into this, I'll get back to you with more > info later. > > Any more info? Which countries are you looking into? How "scriptable" > should the solution be? How often do you have to make decisions? Who > will be using the data - experts or end-users? etc-lots of questions :) > > > > it is now only possible to do HTTP measurements > > > towards RIPE Atlas anchors. > > > > >> doing this will yoke the interests of generic network > > >> infrastructure people with those of Tor and OONI. > > > > > That's certainly one way of "cooperating". > > > > Forgive me, but I can't tell if you are being sarcastic. > > Sorry, it's my English & the medium of email... > > I was not sarcastic, I was serious, but careful. > > > The proposal is encourage OONI agents to place an Atlas anchor next > > (to) every major point. > > (how many points?) > > > RIPE gets dedicated volunteersplacing quality Atlas anchors > > around the world, and OONI gets the pleasure of knowing any detected > > censorship is additionally detected as a network disruption. > > That would be wonderful! > > However... > > 1) current model of placing RIPE Atlas anchors is for the operators to > pay for the HW themselves -- it's a Soekris box + SSD, total price > around 800 Euros. > > Do you think that those dedicated volunteers you mention want to invest > in more HW? > > Otherwise -- there is a proposal to introduce "VM probes" soon (early > next year), and maybe this would be more suitable? > > Forgive me if I assume that the price might be a problem... > > 2) measuring censorship is NOT main goal of RIPE Atlas currently. As it > is, RIPE Atlas is a project of RIPE NCC, which is a Regional registry & > it has to stay neutral towards all its members -- telcos, ISPs, > governments. Even if some of them are involved in censorship. And they > are also paying for the project -- so if they think that the results are > not favorable towards them, they might get pissed off. > > This did not happen yet, althou we did use RIPE Atlas measurements to > point out towards Turkish government meddling with DNS: > > > https://labs.ripe.net/Members/emileaben/a-ripe-atlas-view-of-internet-meddl… > > So, we need to proceed carefully here. Let's keep talking... > > > > We can also somehow do it via MLabs, alhou our cooperation > > > with them is very slow -- two large bureaucracies add lots > > > of inertia ;-) > > > > I don't know this MLabs. Elaborate? > > Apparently OONI already cooperates with MLabs. It's Google's > measurements platform. We also use some of their data. It was a > far-fetched suggestion, so lets stick to our own ideas for now. > > Thanks, looking forward to hear more from you, > Vesna > >

1 0

Continuing bridge_reachability tests?
by David Fifield 15 Sep '15

15 Sep '15

I checked http://api.ooni.io/api/reports for instances of the bridge_reachability test. There are many between 2014-03-01 and 2015-05-28, but they stop after that. Is there a reason for that, or did the operators just decide to stop running the test? About 80% of bridge_reachability tests come from 5 ASes: AS47172, AS29182, AS21341, AS31575, AS4538. I suppose that those represent only 5 instances of ooniprobe. Could it be possible to make the bridge_reachability test run by default?

1 0

One of these people should be invited to the OONI meeting
by Virgil Griffith 14 Sep '15

14 Sep '15

http://atlas.ripe.net/ The idea of putting their probes (or even anchors) next to Tor relays and/or OONI probes seems an immensely good idea because doing this will yoke the interests of generic network infrastructure people with those of Tor and OONI. I'd be happy to bring them to the Tor meeting in Berlin, but they seem a better fit for the OONI meeting in Italy. So I'm inclined to have you invite them. Their community contacts are: becha(a)ripe.net mcb(a)ripe.net Tell them to bring extra probes/anchors to give out to OONI attendees. If for whatever reason they can't come to the Italy meeting I'll invite them to the Berlin meeting. Not as good a fit, but better than nothing. -V

4 6

OONI team status report for August 2015
by Arturo Filastò 14 Sep '15

14 Sep '15

# What we did in August 2015 * Finalise and release the ADINA 2015 Hackathon website https://ooni.torproject.org/event/adina15 https://github.com/TheTorProject/ooni-adina15 * Work on organising and coordinating for the ADINA 2015 hackathon * Attend CCC Camp and discuss potential collaborations for OONI * Brainstorm about a project about monitoring reachability of certain pages on a big website * Engage two new people to work on some OONI related tasks for a current grant we have * Further the development of the raspberry pi image * Get back to many people that got backlogged due to summer * Present the OONI Paper at FOCI * Work on installing hardware in the OONI cluster ~ Arturo

1 0

Blocked.org.uk & ooniprobe
by Daniel Ramsay 13 Sep '15

13 Sep '15

Hi, As part of our work incorporating ooniprobe into the blocked.org.uk scheduling system, we now have a final piece of code that relays ooniprobe results from the scheduling system back to the main OONI collectors. We've currently got about a million individual test results stored, covering 7 ISPs. I have a few questions though: 1) Is it possible to submit results to the OONI collector over HTTP/HTTPS instead of TOR, and if so, is there a public DNS name used for the collector? For the volume of results we've got, it could be a lot more bandwidth efficient and faster to run. 2) Is there a testing or staging collector that you'd like us to use initially? I've been testing against a collector running inside a vagrant VM so far, but wondered if there was any pre-live testing that could be done. 3) Is there a rate limit that you would want enforced? The system is able to relay probe results in real-time, but there's quite a large backlog. Thanks, Daniel.

2 5