- ooni-dev - lists.torproject.org

ADINA15: A Dive Into Network Anomalies. OONI Hackathon 1-2 October Rome, Italy
by Arturo Filastò 23 Jul '15

23 Jul '15

Greeting Oonitarians, We are pleased to announce to you that the dates for the OONI data visualization hackathon are now set and the announcement has been published! This means that you are all invited to the Chamber of Deputees at the Italian Parliament in Rome, Italy on the 1st and 2nd of October 2015 to put your design and data analysis skills to the test! You can read more about this here: https://ooni.torproject.org/event/adina15/ If you use twitter be sure to also retweet this: https://twitter.com/OpenObservatory/status/624254930321305600 Have fun! ~ Arturo

1 0

OONI team status report for June 2015
by Arturo Filastò 09 Jul '15

09 Jul '15

# What we did in June 2015 * Restore the processing of all the gathered reports with the next generation OONI pipeline: https://github.com/TheTorProject/ooni-pipeline-ng * Implement most basic heuristics for identifying interesting http_requests measurements: https://github.com/TheTorProject/ooni-pipeline-ng/commit/e314a97587eebef1c7… * Implement map of the world with measurements gathered per country: https://github.com/TheTorProject/ooni-api/commit/1d41c45818b632f8964d17eb51… * Implement proxy of OONI reports download from S3: https://github.com/TheTorProject/ooni-api/commit/79d0527c5a33680b847f027d1f… * Publish article about greek gambling website censorship: https://ooni.torproject.org/post/eeep-greek-censorship/ * Update the ubuntu PPA with the latest version of ooniprobe: https://github.com/TheTorProject/ooni-probe/commit/edb5c1e22d9de6c50d678029… * Work on the roadmap for measurement-kit and move the project to a different github organisation: https://github.com/measurement-kit * Define the roadmap and next steps for the citizenlab test-lists project: https://github.com/citizenlab/test-lists/milestones/test-lists%201.0.0%20-%… * Review the a legal memo given to use by the HLS CyberLaw Clinic and define the next steps with them * Submit OTF proposal * Attend the CitizenLab summer institute and connect with many potential probe operators We also did the following weekly dev meetings: http://meetbot.debian.net/ooni/2015/ooni.2015-06-01-17.01.log.html http://meetbot.debian.net/ooni/2015/ooni.2015-06-08-17.02.log.html http://meetbot.debian.net/ooni/2015/ooni.2015-06-15-17.00.log.html http://meetbot.debian.net/ooni/2015/ooni.2015-06-22-17.00.log.html http://meetbot.debian.net/ooni/2015/ooni.2015-06-29-16.59.log.html Overall I would say this was a very fruitful month. Much <3 to everyone who helped make all of this happen! ~ Arturo

1 0

Reminder for weekly OONI dev gathering Monday 2015-07-06 17:00 UTC (19:00 CEST)
by Arturo Filastò 06 Jul '15

06 Jul '15

Hello Oonitarians and prospect Oonitarians! This is a reminder that today there will be the weekly OONI gathering. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (19:00 CEST, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0

Reminder for weekly OONI dev gathering Monday 2015-06-29 17:00 UTC (19:00 CEST)
by Arturo Filastò 29 Jun '15

29 Jun '15

Hello Oonitarians, We are travelling in Canada, so got a bit confused with the time zones. Nonetheless we will be around today for the OONI weekly gathering. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (19:00 CEST, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0

Testing HTTPS URLs and certificate chain
by David Fifield 28 Jun '15

28 Jun '15

It would be good to have ongoing tests for the domains we use as fronts for anticensorship, e.g.: https://www.google.com/ https://a0.awsstatic.com/ https://ajax.aspnetcdn.com/ I would love to have periodic checks that 1) each domain is accessible, and 2) the certificate chain is what we expect, to find MITM attempts. I suppose the existing nettests/blocking/http_requests.py can handle simple HTTPS connectivity. Is it easy to add the URLs above to the standard tests? I'm less sure about how to get the certificate chain. I did some searching and didn't find a way to get the certificate chain from the twisted.web.client.Agent that templates/httpt.py uses (maybe you provide it a twisted.internet.ssl.ContextFactory somehow?). nettests/experimental/tls_handshake.py doesn't seem to be quite what I want. What do you suggest?

4 4

Mining OONI reports to find server-side Tor blocking (e.g. CloudFlare captchas)
by David Fifield 23 Jun '15

23 Jun '15

I want to search OONI reports for cases of Tor exits being blocked by the server (things like the CloudFlare 403 captcha). The http_requests test is great for that because it fetches a bunch of web pages with Tor and without. The attached script is my first-draft attempt at finding block pages. Its output is at the end of this message. You can see it finds a lot of CloudFlare captchas and other blocks. First, I ran ooniprobe to get a report-http_requests.yamloo file. Then I ran the script, which does this: Skip the first YAML document, because it's a header. For all other documents: Skip it if it has non-None control_failure or experiment_failure--there are a few of these. Look for exactly two non-failed requests, one with is_tor:false and one with is_tor:true. Skip it if it lacks these. Classify the blocked status of the is_tor:false and is_tor:true responses. 400-series and 500-series status codes are classified as blocked and all others are unblocked. Print an output line if the blocked status of is_tor:false does not match the blocked status of is_tor:true. I have a few questions. Is this a reasonable way to process reports? Is there a more standard way to do e.g. YAMLOO processing? I know there are many reports at https://ooni.torproject.org/reports/. Is that all of them? I think I heard from Arturo that some reports are not online because of storage issues. What's the best way for me to get the reports for processing? Just download all *http_requests* files from the web server? Here is the output of the script. 403-CLOUDFLARE is the famous "Attention Required!" captcha page. I investigated some of the others manually and they are mostly custom block pages or generic web server 403s. (There are also a couple of CloudFlare pages that have a different form.) Overall, almost 4% of the 1000 URLs scanned by ooniprobe served a block page over Tor. I'm not sure what's up with the non-Tor 503s from Amazon. They just look like localized internal service error pages ("ist ein technischer Fehler aufgetreten", "une erreur de système interne a été décelée"). The one for blog.com is a generic Nginx "Bad Gateway" page. non-Tor Tor domain 302 403-OTHER yandex.ru 302 403-OTHER craigslist.org 301 403-CLOUDFLARE thepiratebay.se 503-OTHER 301 amazon.de 200 403-CLOUDFLARE adf.ly 301 403-OTHER squidoo.com 301 410-OTHER myspace.com 303 503-OTHER yelp.com 302 403-CLOUDFLARE typepad.com 503-OTHER 301 amazon.fr 301 403-CLOUDFLARE digitalpoint.com 301 403-CLOUDFLARE extratorrent.com 200 403-OTHER ezinearticles.com 200 403-OTHER hubpages.com 200 403-OTHER 2ch.net 200 403-OTHER hdfcbank.com 302 403-CLOUDFLARE meetup.com 302 403-CLOUDFLARE 1channel.ch 200 403-CLOUDFLARE multiply.com 301 403-CLOUDFLARE clixsense.com 301 403-OTHER zillow.com 301 403-CLOUDFLARE odesk.com 301 403-CLOUDFLARE elance.com 301 403-CLOUDFLARE youm7.com 200 403-CLOUDFLARE jquery.com 200 403-CLOUDFLARE sergey-mavrodi.com 301 403-CLOUDFLARE templatemonster.com 302 403-CLOUDFLARE 4tube.com 301 403-CLOUDFLARE mp3skull.com 301 403-CLOUDFLARE porntube.com 200 403-OTHER tutsplus.com 200 403-CLOUDFLARE bitshare.com 301 403-OTHER sears.com 200 403-CLOUDFLARE zwaar.net 502-OTHER 200 blog.com 302 403-CLOUDFLARE myegy.com 301 400-OTHER mercadolibre.com.ve 302 403-OTHER jabong.com 301 403-CLOUDFLARE free-tv-video-online.me 302 403-CLOUDFLARE traidnt.net

4 5

Summary of ooni-pipeline-ng AWS costs 2015-06
by Arturo Filastò 22 Jun '15

22 Jun '15

Following the example led by David Fifield in publishing the meek transport cloud service costs here is the summary of costs we incurred for running the ooni-pipeline-ng up until now. Mar Apr May June* Total Monthly Total $0.00 $130.20 $89.58 $266.05 $485.83 ------------------------------------------------------------------ EC2 $105.58 $30.61 $158.57 $294.76 Tax $23.49 $16.15 $47.98 $87.62 RDS $12.80 $34.06 $46.86 S3 $0.99 $28.28 $18.09 $47.37 Elastic MapReduce $0.13 $7.35 $7.48 * Partial monthly expenses We are working on migrating the infrastructure over to dedicated hardware and should no longer face such high fees in the future, but perhaps this is still of interest to some of you. It’s also worth pointing out that in May 2015 the cost would have been much higher, since in testing the deployment of the pipeline we left some machines running for too long by mistake, but luckily Amazon was kind enough to give us $650.00 in credit for unexpected usage. The projection of costs for June is to be at around $330.00. ~ Arturo

1 0

Reminder for weekly OONI dev gathering Monday 2015-06-22 17:00 UTC (19:00 CEST)
by Arturo Filastò 22 Jun '15

22 Jun '15

Hello Oonitarians, This is a reminder that today there will be the weekly OONI gathering. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (19:00 CEST, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0

Reminder for weekly OONI dev gathering Monday 2015-06-15 17:00 UTC (19:00 CEST)
by Arturo Filastò 15 Jun '15

15 Jun '15

Hello Oonitarians, This is a reminder that today there will be the weekly OONI gathering. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (19:00 CEST, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0

OONI team status report for May 2015
by Arturo Filastò 09 Jun '15

09 Jun '15

# What we did in May 2015 * Attended an event at Stony Brook about network measurement and met up with various internet censorship researchers * Worked on GUI for network measurement instruments called NetworkMeter: https://github.com/TheTorProject/network-meter * Produce some statistics on current OONI user base. The jist of it is that we estimate to have at least 221 users (these are the unique ASNs we count in the reports, so it could as well be more). In recent times (the last couple of months) we have reports coming in from at least 10 different users on a weekly basis. Given the fact that since 2 months some of our personal probing machines have gone offline this is even more surprising. * We automatized the building of libight libraries in the Xcode app * The first iteration of the new data pipeline was almost completed and web based API interface started to be developed. Here is a chart of the architecture of it so far: https://raw.githubusercontent.com/TheTorProject/ooni-pipeline-ng/master/doc… code for the pipeline can be found here: https://github.com/TheTorProject/ooni-pipeline-ng code for the API (and Javascript frontend) can be found here: https://github.com/TheTorProject/ooni-api * The ubuntu ooniprobe package has been updated to the latest version (1.3.1). * We applied for a AWS grant to have some free amazon credits to pay for our infrastructure costs (the last 2 bills I got were in the 100s of $ range: 130 USD, 89 USD and 200 USD for April, May and June respectively). * Progress was made on the raspberry-pi image for ooniprobe. * Planning of hadoop cluster deployment. * Installation/configuration of the management server for hadoop (7U kotetsi) * We held 2 weekly meetings (2 were skipped this month due to travel): http://meetbot.debian.net/ooni/2015/ooni.2015-05-11-16.59.html http://meetbot.debian.net/ooni/2015/ooni.2015-05-18-17.00.html ~ Arturo

1 0