Hello,
Throughout March 2020, the OONI team worked on the following 2 (two-week) sprints:
* Flounder Sprint (2nd March 2020 - 15th March 2020) * Ariel Sprint (16th March 2020 - 29th March 2020)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Details and highlights are available in this report below.
## OONI Probe mobile app
We released OONI Probe mobile 2.3.0!
* Android: https://github.com/ooni/probe-android/releases/tag/v2.3.0 * iOS: https://github.com/ooni/probe-ios/releases/tag/v2.3.0
Highlights include:
1. You can re-test URLs. This new feature can be useful if you want to collect more evidence on potential blocking, or if a test fails.
2. You can test many more websites in one go! By disabling the “Limit test duration” option (of the website settings), you can test all of the URLs included in the Citizen Lab's global and country-specific test lists.
This release also fixed a bug that led to WhatsApp false positive test results (once WhatsApp made changes to its infrastructure), and includes a series of other improvements documented here: https://github.com/ooni/probe-android/releases/v2.3.0
To address certain infrastructure issues that emerged following the release of OONI Probe mobile 2.3.0, we released OONI Probe mobile 2.3.1 with a fix: https://github.com/ooni/probe-android/releases/tag/v2.3.1
We also released OONI Probe mobile 2.3.2 with further UX improvements and bug fixes: https://github.com/ooni/probe-android/releases/tag/v2.3.2
In March 2020, we also started working on adding support for automated testing: https://github.com/ooni/probe-android/pull/304
Finally, we planned the backlog and roadmap of OONI Probe mobile to better determine the next steps of OONI Probe mobile development: https://github.com/ooni/probe/issues/977
## Research on push notifications
In order to add support to the OONI Probe mobile app to send push notifications, we have been researching the use of Countly as a potential frontend and backend solution for push notifications.
As part of this, we:
* Enabled analytics on countly: https://github.com/ooni/probe/issues/1002 * Tested countly to see whether we can use push notifications for sending OONI Run links: https://github.com/ooni/probe/issues/1055 * Checked how to set-up and use the countly rich notification plugin: https://github.com/ooni/probe/issues/1056 * Explored whether the countly community edition is sufficient to meet our needs: https://github.com/ooni/probe/issues/1057
We also researched how to design the push notification backend, as documented through the following ticket: https://github.com/ooni/backend/issues/339
## OONI Probe desktop app
Throughout March 2020, we continued to make improvements to the OONI Probe desktop app in preparation for the upcoming public launch.
More specifically, we:
* Made a new release candidate for the OONI Probe desktop app (including newly translated strings for the circumvention tool tests): https://github.com/ooni/probe-desktop/releases/tag/v3.0.0-rc.6 * Made UX improvements to the Dashboard and measurement pages: https://github.com/ooni/probe/issues/1047 & https://github.com/ooni/probe/issues/1042 * Integrated the animation for the circumvention tool tests: https://github.com/ooni/probe/issues/1063 * Investigated the failures in the auto-update system and added better exception logging: https://github.com/ooni/probe/issues/1008 * Aligned the middlebox test results in the Test Results page: https://github.com/ooni/probe/issues/1049 * Fixed the markdown in the Psiphon measurement pages: https://github.com/ooni/probe/issues/1051 * Edited the copy in the Websites card: https://github.com/ooni/probe/issues/1048 * Fixed OONI Explorer links: https://github.com/ooni/probe/issues/1041 * Fixed the z-index and draggability of Hero on measurement pages: https://github.com/ooni/probe/issues/1033 * Fixed the scrolling on measurement pages: https://github.com/ooni/probe/issues/959 * Fixed the sorting of columns in the Tor test details page: https://github.com/ooni/probe/issues/1007
## Creating official Linux packages for OONI Probe
To better support the Linux OONI community (which contributes the largest volume of measurements from stable vantage points), we aim to create official Linux packages for OONI Probe.
To this end, we started doing some initial Debian packaging for Linux in March 2020 and we created a prototype (see: https://github.com/ooni/probe-cli/pull/117), which is available here: https://github.com/ooni/probe/issues/1061
## OONI Probe measurement engine
OONI Probe has relied on the MaxMind database (https://www.maxmind.com/) for IP geolocation, but they recently changed their license.
In March 2020, we continued to work on replacing the MaxMind ASN database following the license change: https://github.com/ooni/probe-engine/issues/336
## Making the OONI Probe apps rely entirely on the OONI Probe golang engine
We continued to make progress on making the OONI Probe mobile and desktop apps rely entirely on the golang based probe-engine.
To this end, we completed the work in the following tickets:
https://github.com/ooni/probe-engine/issues/348 https://github.com/ooni/probe-engine/issues/350 https://github.com/ooni/probe-engine/issues/351 https://github.com/ooni/probe-engine/issues/353 https://github.com/ooni/probe-engine/issues/362 https://github.com/ooni/probe-engine/issues/399%C2%A0 https://github.com/ooni/backend/issues/369 https://github.com/ooni/probe/issues/1030
## Improving censorship circumvention tool methodologies
We are improving our censorship measurement methodologies by including metrics that are related to measuring the performance of circumvention tools.
We aim to add support for running the NDT speed and performance test over circumvention tools (Tor, Psiphon, obfs4proxy). This will allow us to determine at what throughput (for example, a 1Mbit download) the tool is able to perform on the network in question, and to better characterize the circumvention tool’s performance.
In March 2020, we started off by replacing NDT with the next generation ndt7 speed test in the OONI Probe Command Line Interface (CLI). This work is available through the following pull request: https://github.com/ooni/probe-cli/pull/119
## Expanding OONI Probe measurement methodologies
As part of our ongoing efforts to expand OONI Probe measurement methodologies, we made improvements to the implementation of our SNI experiment, which involved addressing feedback shared by an engineer at Jigsaw (Vinicius Fortuna): https://github.com/ooni/probe-engine/issues/309
## Prototype of system for probe URL selection
Throughout March 2020 -- during the OONI Team Meeting and during our weekly backend and frontend development calls -- we did a lot of brainstorming in order to determine what an MVP for the URL selection prototype should look like.
Our plan for the MVP is documented in the following github ticket: https://github.com/ooni/backend/issues/361
We also documented several issues related to the creation of the prototype: https://github.com/ooni/backend/issues/320 https://github.com/ooni/backend/issues/361 https://github.com/ooni/backend/issues/321 https://github.com/ooni/ooni.org/issues/431 https://github.com/ooni/ooni.org/issues/430 https://github.com/ooni/backend/issues/322 https://github.com/ooni/backend/issues/221 https://github.com/ooni/backend/issues/207 https://github.com/ooni/probe/issues/930 https://github.com/ooni/explorer/issues/392 https://github.com/ooni/explorer/issues/391
## Data analysis and pipeline work necessary for extracting per site metrics
In March 2020, we started experimenting with ClickHouse to evaluate whether it could be suitable for the more advanced batch analysis required for extracting website metrics. So far, the results are quite promising. The cursory findings of this research are documented here: https://github.com/ooni/backend/issues/380
We also did a lot of database tuning and troubleshooting related to adding counter tables which support aggregating per website metrics: https://github.com/ooni/backend/issues/371
Furthermore, we:
* Wrote an analysis script to close active database connections and fix a connections issue: https://github.com/ooni/pipeline/pull/306 * Made progress on keeping the fast-path rows in the measurement table: https://github.com/ooni/backend/issues/233
## OONI Probe orchestration logic for circumvention tool testing
As part of our work on developing OONI Probe orchestration logic that is specific to circumvention tool testing, we:
* Added backend support for provisioning circumvention tool test configurations to probes: https://github.com/ooni/orchestra/issues/68 * Made it possible to instruct probes in near real-time to test a specific circumvention tool with a particular configuration: https://github.com/ooni/probe/issues/889
## OONI Explorer
Throughout March 2020, we made a few improvements to OONI Explorer.
More specifically, we:
* Fixed the anomaly indicator to ensure that the search results are consistent with the anomaly status in the measurement pages: https://github.com/ooni/explorer/issues/360 * Made measurement pages resilient to the absence of nullable data: https://github.com/ooni/explorer/issues/414 * Handled cases where tcp_connect test keys are missing: https://github.com/ooni/explorer/pull/412 * Wrote copy for the new Psiphon test results: https://github.com/ooni/explorer/issues/419 * Wrote copy for the new Tor test results: https://github.com/ooni/explorer/issues/418 * Opened tickets based on community feedback: https://github.com/ooni/explorer/issues?q=is%3Aopen+is%3Aissue+label%3A%22us...
## Quality assurance and testing of OONI software
As part of our ongoing efforts to increase the quality of OONI software components, we carried out the following activities in March 2020:
* Created an end-to-end testing strategy for OONI Explorer: https://github.com/ooni/explorer/issues/159 * Added end-to-end tests for the OONI Probe desktop app: https://github.com/ooni/probe/issues/1062 * Tested the OONI Probe mobile app 2.3 release: https://github.com/ooni/ooni.org/issues/398
We also made the improvements to the probe-engine, as documented through the following tickets:
https://github.com/ooni/probe-engine/issues/310 https://github.com/ooni/probe-engine/issues/315 https://github.com/ooni/probe-engine/issues/317 https://github.com/ooni/probe-engine/issues/327
## OONI infrastructure
As part of our efforts to improve our server infrastructure and the performance of reliant tools, we:
* Dealt with an incident affecting API slowdown and frequent 500 errors: https://github.com/ooni/backend/issues/371 * Dropped some unused hosts once they stopped receiving traffic: https://github.com/ooni/backend/issues/309 * Investigated slow OONI Explorer / OONI API queries: https://github.com/ooni/backend/issues/327 * Investigated why certain OONI Explorer measurements were unreachable: https://github.com/ooni/backend/issues/329 * Added supported for automatically building docker images as part of the CI process: https://github.com/ooni/backend/issues/346 * Performed VACCUM of the metadb: https://github.com/ooni/backend/issues/354 * Stopped doing full table scans in some places to improve API performance: https://github.com/ooni/backend/issues/356
## OONI website
In preparation for the upcoming public launch of the OONI Probe desktop app, we worked on re-designing the Install page of our website. A community member generously volunteered to write the code for implementing the new Install page based on our design mockups. This work is available here: https://github.com/ooni/ooni.org/pull/446
We also wrote and published the following:
* Tor test description: https://ooni.org/nettest/tor/ * Psiphon test description: https://ooni.org/nettest/psiphon/
Furthermore, we made improvements to how images are handled in the OONI website: https://github.com/ooni/ooni.org/issues/390
## Published report on the temporary blocking of the Farsi edition of Wikipedia in Iran
We published a report on the temporary blocking of the Farsi language edition of Wikipedia in Iran: https://ooni.org/post/2020-iran-blocks-farsi-wikipedia/
OONI measurements showed that the Farsi edition of Wikipedia (fa.wikipedia.org) was blocked on at least 6 networks in Iran between 2nd to 3rd March 2020.
We also found the Farsi edition of Wikinews (fa.wikinews.org) blocked in Iran, though we're being told that it's been blocked for several years now.
Both Farsi Wikipedia and Wikinews were blocked by means of DNS tampering and SNI filtering. In our report, we share relevant data.
## Published blog post encouraging the testing of COVID-19 websites
Ensuring a free and open internet during emergencies is important, but access to COVID-19 information has already been censored in China and Venezuela.
We therefore published a blog post encouraging community participation in a global measurement campaign to increase testing coverage of COVID-19 relevant websites.
This blog post (providing relevant instructions for testing) is available here: https://ooni.org/post/covid-19-ooni-measurement/
We also embedded a “COVID-19” button on the OONI website to enable OONI Probe testing of COVID-19 websites: https://ooni.org/get-involved/run/
## Year in Review: OONI in 2019
We published a blog post sharing highlights from the OONI-verse over the last year, as well as some of the great ways that community members made use of OONI Probe and OONI data as part of their work.
The “Year in Review: OONI in 2019” post is available here: https://ooni.org/post/ooni-in-2019/
OONI highlights from 2019 were also discussed in a separate blog post written and published by VPNCompare: https://www.vpncompare.co.uk/ooni-censorship-2020s/
## Published blog post on OONI highlights between 2017-2019
We wrote a blog post that shares OONI highlights from 2017 to 2019 that were made possible thanks to OTF support.
This blog post was published on the OTF website: https://www.opentech.fund/news/ooni-highlights-2017-2019/
We also cross-posted this blog post on the OONI website: https://ooni.org/post/ooni-highlights-2017-2019/
## OONI Team Meeting
We held a 4-day OONI Team Meeting between 10th to 13th March 2020.
We were originally planning on facilitating this meeting in person in Rome. However, due to the escalating impact of COVID-19 (coronavirus) and the national lockdown of Italy (which started on 10th March 2020), we cancelled travel arrangements (for OONI team members not based in Rome) and facilitated the entire 4-day meeting remotely, online.
During our team meeting, we refined our roadmap for the next 6 months and held various meetings/sessions that enabled us to discuss plans and next steps for the various projects in the OONI-verse.
Below we share the sessions that we held during our 4-day team meeting.
### Day 1
* Discussion of team meeting agenda * Time for reflection: What worked well and what didn’t work well over the last 6 months * Test vectors and technical documentation * Expanding OONI Probe methodologies * Data analysis needs: Building capacity * Data analysis required for extracting website metrics
### Day 2
* What do you want OONI to be? * Adding support for OONI Probe push notifications * Brainstorming on URL prioritization * Server infrastructure: Next steps * Presenting website metrics on OONI Explorer * OONI Explorer: Next steps
### Day 3
* Improving OONI Run * OONI Probe Mobile: Next steps * OONI Probe Desktop: Next steps * Identifiers of measurements (OOID) and in defense of the report ID * OONI data processing pipeline design
### Day 4
* OONI Team Roadmap for 2020 * Fundraising priorities
Most of Day 4 of the team meeting was spent creating a detailed roadmap for all of our projects until December 2020.
We expect to revise our roadmap during our next team meeting which we plan to have in about 6 months from now (September or October 2020).
## Fundraising
In March 2020, we were invited by the Ford Foundation to submit a proposal to seek support for hosting the next OONI Partner Gathering and other community-related work.
We previously submitted a proposal to the OTF Community Lab to seek support for hosting the OONI Partner Gathering, and in March 2020 we addressed their latest feedback.
## Test list updates
Throughout March 2020, we:
* Updated the Venezuelan test list to include reportedly blocked coronavirus-related sites: https://github.com/citizenlab/test-lists/pull/591 * Updated the Global test list to include many coronavirus-related websites (particularly data maps): https://github.com/citizenlab/test-lists/pull/594 * Updated the Global & Saudi Arabian test lists to include LGBTQI sites shared by OutRight International: https://github.com/citizenlab/test-lists/pull/593 * Updated the Myanmar test list to include URLs from a new blocklist (shared by an OTF Information Controls Fellow): https://github.com/citizenlab/test-lists/pull/596 & https://github.com/citizenlab/test-lists/pull/597
## Data analysis
Throughout March 2020, we performed OONI data analysis for the following:
* Azerbaijan Internet Watch: Analysis of measurements collected from Azerbaijan * Small Media: Analysis of OONI measurements collected from Lebanon & Rwanda, in support of relevant stakeholder submissions to the Universal Periodic Review (UPR): https://github.com/ooni/ooni.org/issues/435
Based on these analyses, we wrote reports explaining and interpreting the findings, which we shared with Azerbaijan Internet Watch and Small Media.
## Supported OTF Fellow
Over the last year, we have hosted an OTF Information Controls Fellow who has been researching internet censorship in certain states of India.
In preparation for the publication of their research report, we continued to review it, share suggested copy edits and other feedback. We also made improvements to the data visualizations included in their research report: https://github.com/ooni/ooni.org/issues/441
## Google Summer of Code (GSoC)
OONI is participating in Google Summer of Code (GSoC) under The Tor Project and the DIAL Open Source Center (OSC). In March 2020, prospect GSoC students interested in working on OONI projects over the summer reached out to us.
Below are some of the projects that we suggested to the prospect GSoC students:
* OONI Probe Desktop support for OONI Run and custom URLs, https://community.torproject.org/gsoc/ooni-probe-desktop-custom-urls/ * OONI Probe network experiments, https://community.torproject.org/gsoc/ooni-probe-experiments/ * OONI Explorer: Improvements to testing, https://community.torproject.org/gsoc/ooni-explorer-testing/ * OONI Explorer findings: Improvements related to social media sharing, https://community.torproject.org/gsoc/ooni-explorer-findings/
## IEEE article discussing OONI
OONI’s Arturo was interviewed by IEEE to explain and discuss OONI.
IEEE subsequently published an article that presents OONI and discusses how our project measures internet censorship around the world.
This article is available here: https://spectrum.ieee.org/tech-talk/telecom/internet/how-to-detect-a-governm...
## Community activities
### RIPE hackathon
We contributed project ideas to the online RIPE hackathon: https://labs.ripe.net/Members/becha/hackathons-in-the-time-of-corona
We also participated in the hackathon video-chat (23rd March 2020) and offered the #ooni-dev channel (on Slack and IRC) as a space for online communication and coordination during the hackathon.
### OONI Community Meeting
On 31st March 2020, we facilitated the monthly OONI Community Meeting on our Slack channel (https://slack.ooni.org/), during which we discussed the following topics:
1. Updates on blocked Biafra websites in Nigeria 2. Testing a custom list of URLs with the OONI Probe CLI 3. The future of automatic OONI Probe testing on desktop and Raspberry Pis 4. Storage data format that is easier to consume 5. Faster event analysis
## Userbase
In March 2020, 8,214,072 OONI Probe measurements were collected from 5,719 networks in 212 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): http://explorer.ooni.org/
~ The OONI team.