Hello,
Throughout July 2020, the OONI team worked on the following sprints:
* Sprint 16 - Neon (1st July 2020 - 5th July 2020) * Sprint 17 - Θέτις (6th July 2020 - 19th July 2020) * Sprint 18 - Colomber (20th July 2020 - 31st July 2020)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
## OONI Probe mobile app
In July 2020 we released:
* OONI Probe Mobile 2.5.1 on Android (https://github.com/ooni/probe-android/releases/tag/v2.5.1) and iOS (https://github.com/ooni/probe-ios/releases/tag/v2.5.1)
* OONI Probe Mobile 2.5.2 on Android (https://github.com/ooni/probe-android/releases/tag/v2.5.2) and iOS (https://github.com/ooni/probe-ios/releases/tag/v2.5.2)
The OONI Probe Mobile 2.5.1 release automatically deletes uploaded measurements (creating more local storage space) and enables users to stop a test while it’s running.
The OONI Probe Mobile 2.5.2 release ships the WhatsApp, Facebook Messenger, and middlebox tests rewritten in golang.
## Adding support for configuring push notifications
As part of our ongoing efforts to add support for configuring push notifications, we:
* Implemented countly on iOS: https://github.com/ooni/probe/issues/1209 * Implemented countly on Android: https://github.com/ooni/probe/issues/1208
## OONI Probe desktop app
Throughout July 2020, we continued to make progress on improving the OONI Probe desktop app, as documented through the following pull requests:
https://github.com/ooni/probe-desktop/pull/164 https://github.com/ooni/probe-desktop/pull/163 https://github.com/ooni/probe-desktop/pull/162 https://github.com/ooni/probe-desktop/pull/161 https://github.com/ooni/probe-desktop/pull/159
We also created mockups for new settings to be included in the OONI Probe desktop app. See: https://github.com/ooni/probe/issues/1225
## Making the OONI Probe apps rely entirely on the golang engine
We now have all OONI Probe tests written in golang!
As part of our work on making the OONI Probe apps rely entirely on the golang engine, we:
* Rewrote the OONI WhatsApp experiment in Go: https://github.com/ooni/probe-engine/issues/55 * Rewrote the OONI Facebook Messenger experiment in Go: https://github.com/ooni/probe-engine/issues/736 * Removed the “All endpoints” option in the WhatsApp test: https://github.com/ooni/probe/issues/1226 * Discussed how to improve the web interface of WhatsApp measurements: https://github.com/ooni/probe-engine/issues/740 * Rewrote the OONI HTTP Invalid Request Line experiment in Go: https://github.com/ooni/probe-engine/issues/761 * Rewrote the OONI HTTP Header Field Manipulation experiment in Go: https://github.com/ooni/probe-engine/issues/763%C2%A0 * Rewrote the OONI Web Connectivity experiment in Go: https://github.com/ooni/probe-engine/issues/776 * Rewrote our SNI blocking experiment using urlgetter: https://github.com/ooni/probe-engine/issues/684 * Updated to the latest version of probe-cli/probe-engine: https://github.com/ooni/probe/issues/1178
As mentioned previously, we shipped WhatsApp, Facebook Messenger, and middlebox tests rewritten in golang as part of the OONI Probe Mobile 2.5.2 release.
## Expanding OONI Probe measurement methodologies
As part of our ongoing efforts to improve and expand upon our measurement methodologies, we enabled the measurement of private Tor bridges and wrote an integration test, as documented through the following tickets: https://github.com/ooni/probe-engine/issues/721 and https://github.com/ooni/probe-engine/issues/629.
We also worked on investigating and remediating TLS false positives: https://github.com/ooni/probe-engine/issues/769
## URL prioritization
We wrote an analysis script to add coverage metrics for URL prioritization (https://github.com/ooni/backend/issues/425). This is implemented in the following pull request: https://github.com/ooni/pipeline/pull/319
This script will be used to monitor test list improvements (https://github.com/ooni/backend/issues/418).
## Data analysis and pipeline work necessary for extracting per site metrics
We made progress on implementing API endpoints for returning top-level measurement metadata: https://github.com/ooni/backend/issues/341
## Improving our server infrastructure
Throughout July 2020, we made important progress on improving the monitoring of our infrastructure.
We created an analysis tool (“analysis runbook”) that runs a number of threads to monitor the slow query summary and more. We also built a grafana dashboard to receive alerts based on the monitoring of our infrastructure. See: https://github.com/ooni/backend/wiki/Runbooks:Analysis
We also worked on setting up a new database host in Amsterdam: https://github.com/ooni/backend/issues/400
## Testing and quality assurance
We implemented our own ASN database on the backend. This involved investigations around data quality, automated generation and testing, and deploying the ASN database in the pipeline and using it when processing measurements. This work is tracked here: https://github.com/ooni/backend/issues/406
We also:
* Worked on repairing a broken CI build in Measurement Kit: https://github.com/measurement-kit/measurement-kit/issues/1926 * Improved upon oonimkall: https://github.com/ooni/probe-engine/issues/816 * Made sure we compiled a static miniooni for Linux so that it could be run from any recent Linux distribution without worrying about the libc ABI: https://github.com/ooni/probe-engine/issues/815 * Set up automatic, periodic CI runs: https://github.com/ooni/probe-engine/issues/819 * Improved upon the OONI HTTP Header Field Manipulation test: https://github.com/ooni/probe-engine/issues/821 * Wrote quality assurance tests for the OONI instant messaging and middlebox experiments: https://github.com/ooni/probe-engine/issues/829 * Fixed a bug related to certain tests not running: https://github.com/ooni/probe/issues/1233 * Fixed a bug in the OONI Telegram test: https://github.com/ooni/probe-engine/issues/827 * Backported the Telegram test bugfix to a stable branch: https://github.com/ooni/probe-engine/issues/828
Following a series of improvements, we released Measurement Kit v0.10.12: https://github.com/ooni/probe-engine/issues/809
## Updated OONI Data Policy
On 22nd July 2020, we released the 1.4.0 version of the OONI Data Policy.
The updated OONI Data Policy is available here: https://ooni.org/about/data-policy
The latest version (1.4.0) of the OONI Data Policy includes the following changes:
* We now use our own ASN database (following the changes to MaxMind's license).
* You can no longer opt out of submitting country code information (e.g. "IT" for Italy), as this significantly limits the value of measurements (i.e. we have no idea where a test was run).
* We use a self-hosted analytics platform on our websites and OONI Probe apps to better understand how our apps are used, improve our understanding of user needs, improve OONI tools, and better serve the OONI community. You can opt out of our use of analytics by disabling the collection of app usage statistics in the OONI Probe app settings (and you can opt out of our use of analytics on ooni.org and explorer.ooni.org by unchecking the opt-out box at the end of our Data Policy).
* In our Data Policy, we have added a section on the collection of app usage statistics.
* In our Data Policy, we have added a section where we disclose what information we will collect if you opt in to push notifications.
* OONI measurements are openly published by default, but we will not publish data related to app analytics and push notification support (which is securely stored separately from the public measurement metadb).
## Report on network performance in Northern Italy amid the COVID-19 pandemic
We collaborated with M-Lab on the publication of a research report that shares measurement observations on network performance in Northern Italy during the peak of the COVID-19 pandemic.
This report is available here: https://ooni.org/post/2020-network-performance-covid19-italy/
In summary, we found:
* Significant increase in the number of NDT tests run per day around the beginning of Italy’s nationwide lockdown (which started on 10th March 2020);
* Reduced NDT download and upload speed results during the nationwide lockdown in Italy (between March 2020 to May 2020);
* Improved NDT download and upload speed results from early May 2020 onwards, when the Italian government gradually started to lift the strict lockdown measures.
As the available measurements suggest that there may be a correlation between increased internet use and reduced network performance, they highlight the need to invest in internet infrastructure that is more robust and sustainable in the long-term. They also highlight the need for more longitudinal measurements collected from more stable vantage points to improve our understanding of network performance issues.
OONI’s Simone also collaborated with M-Lab on a blog post that introduces ndt7, which is available here: https://www.measurementlab.net/blog/ndt7-introduction/#ndt7-introduction
## Internet Measurement Village 2020 recap blog post
Throughout June 2020, up until 3rd July 2020, we hosted the first Internet Measurement Village (IMV), an online community event aimed at sharing skills, knowledge, and resources on internet measurement.
As all the IMV sessions were live-streamed and will continue to live on the OONI YouTube channel (https://www.youtube.com/c/OONIorg), we hope that these recordings will serve as a valuable resource on internet measurement for the internet freedom community.
In July 2020, we published a blog post to provide a recap of the Internet Measurement Village 2020 and to share the slides and video recordings of each presentation. This blog post is available here: https://ooni.org/post/2020-imv-slides-recordings/
## Statement in support of the OTF
In July 2020, we published a second statement in support of the OTF amid recent developments (our first statement is available here: https://ooni.org/post/2020-06-19-save-internet-freedom-support-the-open-tech...).
In our statement, titled “Support the OTF: Support a Free and Open Internet”, we share our experience working with the OTF, and discuss how they have played a crucial role in supporting the fight against internet censorship worldwide.
This statement is available here: https://ooni.org/post/2020-support-otf/
## Created a new OONI partnerships page
We created a new page for the OONI website which features all of our partners, shares their work, and the projects and/or research reports that we have collaborated on.
This work is available here: https://github.com/ooni/ooni.org/pull/528
We will publish this page once it has been reviewed and approved by all of our partners.
## Collaboration with Netalitica
As part of our ongoing collaboration with Netalitica, we continued to review their test list updates.
In July 2020, we reviewed the updates to the Ethiopian, South African, and Thai test lists and shared feedback. We also reviewed and shared feedback on the Netalitica updates to the Algerian, Iraqi, Indonesian, and Zimbabwean test lists.
## Data analysis
As part of our partnership with Azerbaijan Internet Watch, we analyzed all OONI measurements collected from Azerbaijan between 1st January 2020 to 31st July 2020, based on which we produced relevant charts and wrote a report.
We also performed data analysis for an OTF Information Controls Fellow (https://github.com/ooni/ooni.org/issues/552) examining internet censorship in Myanmar through the use of OONI Probe. Based on this analysis, we produced a CSV file and several charts for the research fellow.
## Community activities
### NetGain Partnership webinar
On 8th July 2020, OONI’s Maria was invited to participate in the NetGain Partnership online event, specifically in the webinar titled “Surging Demand and the Global Internet Infrastructure”.
Information about the webinar is available here: https://www.netgainpartnership.org/events/covid19/global-infrastructure
As part of her participation in this webinar, Maria presented OONI’s work and discussed how increased internet use during the COVID-19 pandemic was correlated with decreased network performance (based on NDT measurement analysis).
### RightsCon Online 2020
The OONI team attended RightsCon 2020, which took place entirely online between 27th-31st July 2020. As part of our participation, we attended many sessions, as well as the meetup of the KeepItOn campaign.
On 29th July 2020, OONI’s Maria co-facilitated the following session along with Rama and Alberto from CAIDA: “Investigating internet shutdowns with open data”.
As part of this session, Maria introduced participants to OONI for measuring the blocking of websites and apps, while Rama and Alberto introduced participants to IODA for measuring internet outages.
Maria was also on the RightsCon Online Program Committee, which involved reviewing session proposals and providing advice for the “Network Connectivity and Internet Shutdowns” track.
### Kill Switch Podcast
On 30th July 2020, OONI’s Maria was invited to speak on Access Now’s Kill Switch podcast (https://www.volume.africa/kill-switch) that discussed internet shutdowns around the world. As part of her participation in the podcast, Maria discussed OONI and its role in the KeepItOn campaign.
The podcast (Episode 4: “The Fight to Get Back Online”) is available here: https://player.captivate.fm/episode/61ad48c6-c67b-42ec-b8ca-9191915ec746
## Userbase
In July 2020, 7,567,361 OONI Probe measurements were collected from 5,765 networks in 209 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/
~ The OONI team.