<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><div id="bloop_customfont" style="margin: 0px;"># OONI Monthly Report: September 2017</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">The OONI team completed all deliverables for its current contract with the OTF.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">## Stable release of Measurement API</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We released the stable version of OONI's new measurement API, which is available here: https://api.ooni.io/.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">The OONI API is based on the stable release of our data processing pipeline, which has been re-engineered over the last year to detect censorship events around the world faster and more accurately.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We added the following to the OONI API:</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* API endpoints for listing and filtering anomalous measurements</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* API endpoints for downloading full reports</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We changed the following:</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* Reverse sorting in `by_date` view and hide measurements from time travellers</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* Better API documentation thanks to redoc based on OpenAPI 2.0</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* Improve request validation thanks to connexion base on OpenAPI</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* Oonify the UI</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* Better testing</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">## Stable release of the OONI data processing pipeline</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We now have a much stronger data processing pipeline (https://github.com/TheTorProject/ooni-pipeline/pull/62).</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">The scheduling of the pipeline is based on Airflow which is more robust than our prior luigi based solution. By doing this we have been able to reduce the number of silent outages going undetected for weeks.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">The new version of the pipeline allows for much quicker feedback and interation, allowing us to run meaningful SQL queries that return result within minutes (not hours). This is also the foundation for building our new OONI measurements API.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We feel confident that the pipeline is going to easily scale to up to 100 GB of network measurement data per day. The current rate is of ~20GB of measurements per day and we are able to process this amount of data within an hour (~10 min for compression, ~10 min for sanitisation, ~30 min for inserting it into the DB), so we have some margin for scaling up to 100GB a day without significant modifcations to our infrastructure.</div><div id="bloop_customfont" style="margin: 0px;">Currently the data is being processed in 24 hour batches, but based on the stats above, we are aiming to reduce that to 12h, 8h and eventually 1h.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">The pipeline has been designed to be modular and allow us to easily ingest network measurement data from other projects too.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">Currently the pipeline is analysing the data looking for blockpages and signs of DNS manipulation. We have plans for adding heuristics for detecting also more advanced insights in future versions.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">For the future of the pipeline see: https://github.com/TheTorProject/ooni-pipeline/milestone/2</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">## Implement visualisations for the measurements interface</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">Elio Qoshi has started working for OONI as a designer. With him we have started to define the overall design and style of all OONI related content.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">Some of this work has been integrated into our living style guide available at the following URL: http://openobservatory.github.io/design.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">Part of the work also includes data visualisations that are used to visualise OONI data (see, for example, the BarChart component).</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">## Data visualisations for reports and blogs</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">Thanks to the work on the design front, we have started creating templates and guidelines for making charts that much such a task easier in the future. See: https://github.com/OpenObservatory/design/issues/2.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We created data visualizations based on our new style guide that are included in the following reports:</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">1. Cuba: https://ooni.torproject.org/post/cuba-internet-censorship-2017/#findings</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">2. Iran: https://ooni.torproject.org/post/iran-internet-censorship/#findings</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">## Published research report on internet censorship in Iran</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">In collaboration with ASL19, ARTICLE 19, and Small Media, we published a research report on internet censorship in Iran.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">The report is available here: https://ooni.torproject.org/post/iran-internet-censorship/</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We also published a summary of the report on the Tor blog: https://blog.torproject.org/internet-censorship-iran-findings-2014-2017</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">## Stable release of Orchestrator</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We released the stable version of Probe Orchestration.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">This features:</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* Support for sending push notifactions to all mobile probes that include a message</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* Support for having them run custom tests via the OONI Run custom URI scheme</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* Better unittesting that also speak to the database to ensure everything is working as expected</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">See: https://github.com/TheTorProject/proteus/pull/28</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">For the time being we have decided to postpone including support for instrumenting probes to run measurements, following possible security concerns, see: https://github.com/TheTorProject/proteus/issues/24. Once we have come up with a more solid specification and a proof of concept implementation of this scheme, we would greatly appreciate feedback and maybe a third party security audit of our proposed approach.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">## OONI Run</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We released OONI Run, a custom URI scheme for OONI Probe mobile apps that enables users to:</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* Engage others with testing the sites and running the OONI Probe tests (included in OONI mobile apps) of their choice</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">* Embed an OONI Probe widget to monitor the blocking of their site around the world</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">OONI Run is available here: https://run.ooni.io/</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We also published a blog post that explains OONI Run: https://ooni.torproject.org/post/ooni-run/</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">## Community</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We hosted our monthly community meeting on https://slack.openobservatory.org on 29th September 2017 at 14:00 UTC.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">We discussed the following:</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">1. OONI Run: Questions and feedback</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">2. Announcement of OONI API release: Questions</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">## Userbase</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">In September 2017 ooniprobe was run 81,258 times from 1,595 different vantage points across 149 countries around the world.</div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">This information can also be found through our stats here: <a href="https://api.ooni.io/stats">https://api.ooni.io/stats</a></div><div id="bloop_customfont" style="margin: 0px;"><br></div><div id="bloop_customfont" style="margin: 0px;">~ The OONI team</div></div><br><div id="bloop_sign_1506717142275939840" class="bloop_sign"></div></body></html>