July 2015 - ooni-dev - lists.torproject.org

Blocked.org.uk & ooniprobe
by Daniel Ramsay 13 Sep '15

13 Sep '15

Hi, As part of our work incorporating ooniprobe into the blocked.org.uk scheduling system, we now have a final piece of code that relays ooniprobe results from the scheduling system back to the main OONI collectors. We've currently got about a million individual test results stored, covering 7 ISPs. I have a few questions though: 1) Is it possible to submit results to the OONI collector over HTTP/HTTPS instead of TOR, and if so, is there a public DNS name used for the collector? For the volume of results we've got, it could be a lot more bandwidth efficient and faster to run. 2) Is there a testing or staging collector that you'd like us to use initially? I've been testing against a collector running inside a vagrant VM so far, but wondered if there was any pre-live testing that could be done. 3) Is there a rate limit that you would want enforced? The system is able to relay probe results in real-time, but there's quite a large backlog. Thanks, Daniel.

2 5

Fact-check my understanding of ooniprobe URL testing lists
by David Fifield 04 Aug '15

04 Aug '15

I'm working on a project where we're using OONI http_requests reports to find servers that block Tor users. I'd like you to check my understanding of where the URL lists for testing come from. I wrote: A single OONI report typically tests many URLs. For the most part, reports use the Citizen Lab URL testing lists (https://github.com/citizenlab/test-lists) which contain about 1,200 "global" URLs, plus up to about about 900 additional country-specific URLs that are tested depending on the country in which ooniprobe is run. Users may run ooniprobe with their own custom URL lists; the results are uploaded to the global report pool. (There are several one-off reports in the data that test a single URL.) Is this true? Has it always been like this? How do ooniprobe users get updates when the list changes?

3 3

South Korea dedicated server provider UNDER restriction
by Clodo 27 Jul '15

27 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 I got a server from ehostidc.cn . They give servers UNDER the Korean restriction: hundreds of thousands destination (including porn and gambling) redirected here: http://warning.or.kr/ Also if i address an IP directly (it's not dns-based censorship). Now i need to dismiss that server, i cannot install an OONI probe on it. I write here because if anyone want to investigate about South Korea, this info can be an useful resource. Clodo -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJVsNVPAAoJEC/ixHrG0m4LriAH/RSY+1+mh5ncirS6UiF1k7EH LtXkJyF3OMl29NCAFNjbMu13EmgJQ22Fp45raX4Xh5Cjkez8CzR70b/nR/VA6qVt DgF2De26vSQjvvt8y74+ANdZqeWtLtjOJNB4kplbem2ifjlN+Q4GFf3TFyl8tUuq 5ck5jZDhDAq6O6XHsUO7TSUO/VPkW6Y0k/zhT17ShnQb/sLgfgfx8fxYX22z7rs8 0Ue4GGSTuJR6xshWDSa2liTsTQE0xiEcj8/OeKPUKio538uwQXBQ1zt9LvkMAR4I Ri7rC64ctlQ7uMffomdGFUsct8wD4v4xzKgVWDKfb5cPp9UM2krrM4BxhqAlxMc= =Uvn0 -----END PGP SIGNATURE-----

2 1

Detection of DNS used by probe
by Clodo 27 Jul '15

27 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi, i recently do some maintenance on a website called ipleak.net. I added a json/api feature, and i think can be useful in a OONI probe to detect DNS spoofing/injection. For example, fetch this: (change the third-level domain to a random hash ): https://a_long_random_hash_for_every_request.ipleak.net?mode=json The domain are resolved by ISP, they DNS query ask the resolution of the random domain to our authoritative server, our server collect the IP address of the latest ISP DNS that request the domain and report it in the http response. Note that if the ISP have more DNS server (load-balancing), doing multiple requests (every with a new hash) can return many DNS IP. For example, here in Italy doesn't matter if i try to use Google DNS 8.8.8.8, my ISP (Vodafone) always do a 'Transparent DNS', they capture any request over port 53 and redirect to their DNS. If a country do the same thing for censorship reason, you can detect it with this technique. If this feature is interesting for OONI, feel free to use it on ipleak.net throught our API. Otherwise, if you prefer to implement yourself, i'm here for free support. You need a domain with NS record that point to a server you control (dns authoritative), a bind9, a named-pipe between bind9 and a script, and a wildcard SSL certificate if you want all under SSL. Ciao! Clodo -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJVpRXGAAoJEC/ixHrG0m4LVQoH/RAJ4dPCTcCsV0tWpgjOprGk CQJ2BLuqqc+1tr9sUansLYhPjTM3MVmsBwqrE18k4jLx/wb9CxJicSWR3BUFCtUt eZANQPWjo+wr9xrH2G448pHosjsUKUJH+45f05M/RL2ucCwHw39JSh8vrz3SBxiv QOOrbC5SXFY20kOVtX4uDPjsSyf5e1cpwDmawNUE/anaM7TtOWYMtSQADYozl7/1 6cl1sCfeH0uR4mIshgnIevDf4BYOwUrzVtsxuNh3Z4FCwxX0qcVCgVVv6Iur6O3H BuMUPAiy5GruD+T95AlHYu5mkh2/0tkCQHu3+xgq4hp0s9IyzL5Bv5sfy9hRVzU= =20ib -----END PGP SIGNATURE-----

2 1

Reminder for weekly OONI dev gathering Monday 2015-07-27 17:00 UTC (19:00 CEST)
by Arturo Filastò 27 Jul '15

27 Jul '15

Hello Oonitarians, This is a reminder that today there will be the weekly OONI gathering. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (19:00 CEST, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0

Missing http_requests reports
by David Fifield 23 Jul '15

23 Jul '15

I tried to download all the http_requests reports as of today. Out of 2795 URLs, however, 8 failed to download, with status 404: http://api.ooni.io/reportFiles/2015-01-06/20150106T160035Z-AS22773-http_req… http://api.ooni.io/reportFiles/2015-04-13/20150413T140425Z-AS22773-http_req… http://api.ooni.io/reportFiles/2015-04-13/20150413T153956Z-AS3352-http_requ… http://api.ooni.io/reportFiles/2015-04-21/20150421T140026Z-AS22773-http_req… http://api.ooni.io/reportFiles/2015-04-27/20150427T022702Z-AS3352-http_requ… http://api.ooni.io/reportFiles/2015-05-05/20150505T000230Z-AS29141-http_req… http://api.ooni.io/reportFiles/2015-05-08/20150508T002621Z-AS48452-http_req… http://api.ooni.io/reportFiles/2015-05-19/20150519T000031Z-AS29141-http_req… Here is what I did. First, downloaded the JSON index of reports. $ wget -O reports.json http://api.ooni.io/api/reports Filtered out just the http_requests ones (using the attached script). $ ./ooni-report-urls -t http_requests reports.json | sort | uniq > ooni-report-urls.txt Downloaded with Wget. $ wget -c -i ooni-report-urls.txt

1 0

ADINA15: A Dive Into Network Anomalies. OONI Hackathon 1-2 October Rome, Italy
by Arturo Filastò 23 Jul '15

23 Jul '15

Greeting Oonitarians, We are pleased to announce to you that the dates for the OONI data visualization hackathon are now set and the announcement has been published! This means that you are all invited to the Chamber of Deputees at the Italian Parliament in Rome, Italy on the 1st and 2nd of October 2015 to put your design and data analysis skills to the test! You can read more about this here: https://ooni.torproject.org/event/adina15/ If you use twitter be sure to also retweet this: https://twitter.com/OpenObservatory/status/624254930321305600 Have fun! ~ Arturo

1 0

OONI team status report for June 2015
by Arturo Filastò 09 Jul '15

09 Jul '15

# What we did in June 2015 * Restore the processing of all the gathered reports with the next generation OONI pipeline: https://github.com/TheTorProject/ooni-pipeline-ng * Implement most basic heuristics for identifying interesting http_requests measurements: https://github.com/TheTorProject/ooni-pipeline-ng/commit/e314a97587eebef1c7… * Implement map of the world with measurements gathered per country: https://github.com/TheTorProject/ooni-api/commit/1d41c45818b632f8964d17eb51… * Implement proxy of OONI reports download from S3: https://github.com/TheTorProject/ooni-api/commit/79d0527c5a33680b847f027d1f… * Publish article about greek gambling website censorship: https://ooni.torproject.org/post/eeep-greek-censorship/ * Update the ubuntu PPA with the latest version of ooniprobe: https://github.com/TheTorProject/ooni-probe/commit/edb5c1e22d9de6c50d678029… * Work on the roadmap for measurement-kit and move the project to a different github organisation: https://github.com/measurement-kit * Define the roadmap and next steps for the citizenlab test-lists project: https://github.com/citizenlab/test-lists/milestones/test-lists%201.0.0%20-%… * Review the a legal memo given to use by the HLS CyberLaw Clinic and define the next steps with them * Submit OTF proposal * Attend the CitizenLab summer institute and connect with many potential probe operators We also did the following weekly dev meetings: http://meetbot.debian.net/ooni/2015/ooni.2015-06-01-17.01.log.html http://meetbot.debian.net/ooni/2015/ooni.2015-06-08-17.02.log.html http://meetbot.debian.net/ooni/2015/ooni.2015-06-15-17.00.log.html http://meetbot.debian.net/ooni/2015/ooni.2015-06-22-17.00.log.html http://meetbot.debian.net/ooni/2015/ooni.2015-06-29-16.59.log.html Overall I would say this was a very fruitful month. Much <3 to everyone who helped make all of this happen! ~ Arturo

1 0

Reminder for weekly OONI dev gathering Monday 2015-07-06 17:00 UTC (19:00 CEST)
by Arturo Filastò 06 Jul '15

06 Jul '15

Hello Oonitarians and prospect Oonitarians! This is a reminder that today there will be the weekly OONI gathering. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (19:00 CEST, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0