August 2015 - ooni-dev - lists.torproject.org

Blocked.org.uk & ooniprobe
by Daniel Ramsay 13 Sep '15

13 Sep '15

Hi, As part of our work incorporating ooniprobe into the blocked.org.uk scheduling system, we now have a final piece of code that relays ooniprobe results from the scheduling system back to the main OONI collectors. We've currently got about a million individual test results stored, covering 7 ISPs. I have a few questions though: 1) Is it possible to submit results to the OONI collector over HTTP/HTTPS instead of TOR, and if so, is there a public DNS name used for the collector? For the volume of results we've got, it could be a lot more bandwidth efficient and faster to run. 2) Is there a testing or staging collector that you'd like us to use initially? I've been testing against a collector running inside a vagrant VM so far, but wondered if there was any pre-live testing that could be done. 3) Is there a rate limit that you would want enforced? The system is able to relay probe results in real-time, but there's quite a large backlog. Thanks, Daniel.

2 5

Reminder for weekly OONI dev meeting today 2015-08-31 17:00 UTC (19:00 CEST)
by Simone Basso 31 Aug '15

31 Aug '15

Dear all, This is a reminder that today there will be the weekly OONI meeting. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (19:00 CEST, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, -Simone

1 0

Reminder for weekly OONI dev gathering Monday 2015-08-24 17:00 UTC (19:00 CEST)
by Arturo Filastò 24 Aug '15

24 Aug '15

Hello Oonitarians, This is a reminder that today there will be the weekly OONI gathering. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (19:00 CEST, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0

OONI, A Dive Into Network Anomalies Hackathon registrations open! #ADINA15
by Arturo Filastò 20 Aug '15

20 Aug '15

Greetings! We are very pleased to announce that the registrations for the OONI Open Data hackathon “A Dive Into Network Anomalies” are now open! If you would like to come to Rome, Italy October 1st-2nd 2015 and hack on some internet censorship data in the Italian Parliament NOW is the moment to register. By visiting https://ooni.torproject.org/event/adina15/ at the end of the page you will see a list of projects that we have proposed ourselves. We strongly encourage people to submit their own ideas and don’t want to put any limits on what you can do. We are also very happy to announce that AirVPN has offered to sponsor the prizes, so there will be some interesting prizes for the awarded teams. Unfortunately due to particular house rules it is a requirement that people submit their full legal name and we need to submit these names to the Italian Parliament before the event. Moreover the style police will be making sure that people are wearing jackets at all time while inside of the Italian Parliament (again, sorry this is not our rule, but be creative, what is the REAL definition of a jacket? ;)). If you need travel and accommodation support do not hesitate to contact me as we do have some (small) budget for that and would be very happy if it were used by somebody that would not be able to attend otherwise. Nothing is left, but to invite you to register and see you soon in Italy! More information at: https://ooni.torproject.org/event/adina15 #ADINA15 https://twitter.com/OpenObservatory/status/634416448820092929 Have fun! The OONI team & Arturo

1 0

Hello list
by Jun Matsushita 06 Aug '15

06 Aug '15

Hi there, I had an email exchange with Arturo and he recommended I post this on the list. > Arturo, have you looked at https://zeppelin.incubator.apache.org/ ? Also remember I mentioned SlamData ? It claims to do very fast MongoDB analytics (http://slamdata.com/use-cases/#operational-analytics) - and Michele, they are functional heads (using Purescript for the front-end https://github.com/slamdata/slamdata and talking about relational algebra http://slamdata.com/murray/) Also I think you mentioned that you tried to have your tera byte of data in Mongo but that there were size leaks? Can you talk more about that? Also have you looked at http://gopulsar.io/ ? What is the best place to ask questions about 1/ the data processing pipeline systems 2/ the type of analytics that could be done on the data? These are all great suggestions and I think the only one I had briefly glanced at is zeppelin. Being still an incubator project it may not be something good to rely on, but perhaps something interesting to play around with (for doing similar tasks we currently import a partition of the data into elastic search and then use kibana to explore it). Makes sense. Haven’t tried any of them myself, and just stumbled upon the Slamdata one because of my interest in functional front-end approaches and they seem to be doing interesting stuff with Mongo and a kind of Python notebook approach to creating analysis documents which I thought was cool. It would be great if this information were posted to the ooni-dev mailing list (https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-dev) as perhaps some people there with more knowledge than me can comment on them too. Done! Regarding the size leaks of mongoDB I didn’t do much investigation into it, though I still have a copy of the original database if you would like to check it out. Basically the problem was that we had something like ~100GB of raw data (~10GB compressed) that when placed inside of the database would increase to a size of ~500GB. I didn’t dig too much into the issue as that iteration of the pipeline was presenting also other scaling issues (the processing of the batch jobs was not being distributed, but just done on a single machine) so we just ditched the mongodb approach and went for elastic search since a community member had recommended we use that. From what I hear, elastic search is generally doing the job. I’d be interested in understanding what’s the current and planned view on what is done by elastic search (I guess simple full text or maybe also faceted queries?) from what is done by the hadoop cluster from what is done by the storm cluster. One of the main questions I have with regards to the overall analysis capability of the data pipeline is: is there an approach that would allow analysts (rather than developers) to deploy rapidly complex queries into the data pipeline? In other words, can we shorten the feedback loop between thinking “hey that would be a cool analysis to try” and “look here’s the result!”. It seems that this would help iterate and be more creative with analysing the data. Maybe its something like a staging/analysis cluster which is a replica of the production cluster but with additional ‘analysis’ processes running and write access for analysts to deploy their experiments? I guess something where you can deploy “queries” as map reduce jobs, to both a batch processing cluster and a streaming processing cluster? Anyhow. This is surely not a priority but maybe it’ll be interesting to some folks here. Cheers, Jun -- Jun Matsushita Founder, CEO - information innovation lab - iilab.org - @iilab +49 157 530 86174 - jun(a)iilab.org - skype: junjulien

1 0

Fact-check my understanding of ooniprobe URL testing lists
by David Fifield 04 Aug '15

04 Aug '15

I'm working on a project where we're using OONI http_requests reports to find servers that block Tor users. I'd like you to check my understanding of where the URL lists for testing come from. I wrote: A single OONI report typically tests many URLs. For the most part, reports use the Citizen Lab URL testing lists (https://github.com/citizenlab/test-lists) which contain about 1,200 "global" URLs, plus up to about about 900 additional country-specific URLs that are tested depending on the country in which ooniprobe is run. Users may run ooniprobe with their own custom URL lists; the results are uploaded to the global report pool. (There are several one-off reports in the data that test a single URL.) Is this true? Has it always been like this? How do ooniprobe users get updates when the list changes?

3 3

Reminder for weekly OONI dev gathering Monday 2015-08-03 17:00 UTC (19:00 CEST)
by Arturo Filastò 02 Aug '15

02 Aug '15

Hello Oonitarians, This is a reminder that tomorrow there will be the weekly OONI gathering. It will happen as usual on the #ooni channel on irc.oftc.net at 17:00 UTC (19:00 CEST, 13:00 EST, 10:00 PST). Everybody is welcome to join us and bring their questions and feedback. See you later, ~ Arturo

1 0