Hello,
Throughout November 2020, the OONI team worked on the following sprints:
* Sprint 25 - Näkki (October 26 - November 8, 2020) * Sprint 26 - Neerali (November 9 - 22, 2020) * Sprint 27 - Hexacorallia (Nov 23 - Dec 6, 2020)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
## OONI Probe Mobile
### Released OONI Probe Mobile 2.7.1
We released OONI Probe Mobile 2.7.1 for:
* Android: https://github.com/ooni/probe-android/releases/tag/v2.7.1 * iOS: https://github.com/ooni/probe-ios/releases/tag/v2.7.1
This release includes the following noteworthy changes:
* We have removed the "Include Network Info" setting (which enabled you to opt-out of ASN collection). Now ASN information (https://ooni.org/support/glossary/#asn) is collected by default. We decided to do this because measurements that don't include ASN information are non-actionable.
* We have removed the setting which enabled you to share your IP address (on an opt-in basis). We decided to remove this setting because we think the potential risk associated with sharing IP addresses is greater than the benefit. Furthermore, when users did choose to share their IP addresses over the last years, we did not find them particularly useful.
Following these changes, the "Sharing" setting is no longer available in the OONI Probe mobile app (since the above options have been removed), and the "Publish Results" setting (which is enabled by default) has been moved under the Privacy tab of the app.
Through these changes (in OONI Probe Mobile 2.7.1), we aim to make OONI Probe measurements both more secure and more actionable.
### Progress on adding support for automated regular testing
We continued to make progress towards adding support in the OONI Probe mobile app for automated regular testing, as documented through this ticket: https://github.com/ooni/probe/issues/916
We also worked on activities pertaining to the following tickets: https://github.com/ooni/probe/issues/1269 https://github.com/ooni/probe/issues/1271 https://github.com/ooni/probe/issues/970 https://github.com/ooni/probe/issues/1249
In addition, we released the beta version of OONI Probe Android 2.8.0: https://github.com/ooni/probe-android/releases/tag/v2.8.0-beta.1
## Updated OONI Data Policy
We updated the OONI Data Policy to reflect the changes made in the OONI Probe Mobile 2.7.1 release. The updated version (1.4.1) of the OONI Data Policy is available here: https://ooni.org/about/data-policy
This is a minor update, mainly aiming to reflect that OONI Probe mobile app users can no longer opt-out from ASN collection, nor opt-in to IP address collection. These options are currently available through the OONI Probe desktop app (though future releases will remove these options too).
## OONI Probe Desktop
### Released OONI Probe Desktop 3.1.0
We released OONI Probe Desktop 3.1.0 for macOS and Windows: https://github.com/ooni/probe-desktop/releases/tag/v3.1.0
With the latest version (3.1.0) you can:
* Choose the categories of websites that you want to test (by enabling/disabling website categories in the settings)
* Change the language of your OONI Probe desktop app (thanks to support from the Localization Lab community!)
The latest version also features a new settings screen layout.
In preparation for the OONI Probe Desktop 3.1.0 release, the Localization Lab coordinated the translation of new OONI Probe strings. We also translated the new strings to Greek ourselves.
Towards the end of November 2020, we released OONI Probe Desktop 3.1.1 with some bug fixes: https://github.com/ooni/probe-desktop/releases/tag/v3.1.1
We also worked on the activities documented through the following tickets and pull requests: https://github.com/ooni/probe/issues/1243 https://github.com/ooni/probe-desktop/pull/185 https://github.com/ooni/probe/issues/1274 https://github.com/ooni/probe/issues/1290 https://github.com/ooni/probe/issues/1291
### Updated OONI Probe macOS homebrew package
We worked on updating the OONI Probe macOS homebrew package with the OONI Probe Command Line Interface (CLI).
This work is documented through the following tickets: https://github.com/ooni/probe/issues/1270 https://github.com/ooni/probe/issues/1289
We have opened a pull request for the new OONI Probe homebrew package here: https://github.com/Homebrew/homebrew-core/pull/66474
## OONI Probe Command Line Interface (CLI)
We continued to improve upon the OONI Probe CLI through the following releases:
* 3.0.10: https://github.com/ooni/probe-cli/releases/tag/v3.0.10 * 3.0.11: https://github.com/ooni/probe-cli/releases/tag/v3.0.11
## Building Debian package for OONI Probe
We made progress on building a Debian package for OONI Probe. Our work on this is available here: https://github.com/ooni/probe-cli/pull/162
The Debian package for OONI Probe could optionally be periodically invoked by systemd to perform automatic testing.
## OONI Probe Engine
In November 2020, we made the following improvements to the OONI Probe measurement engine:
* Implemented fallback for lookup IP: https://github.com/ooni/probe-engine/issues/1026 * Routine releases: https://github.com/ooni/probe-engine/issues/1004 * Removed the ASN, country code, and IP settings (in line with the OONI Probe Mobile 2.7.1 release): https://github.com/ooni/probe-engine/issues/974 * Reverted the sanitization of the resolver IP in the Web Connectivity test: https://github.com/ooni/probe-engine/issues/1047 * Bug fix: https://github.com/ooni/probe-engine/issues/994
## Expanding OONI Probe measurement methodologies
As part of our ongoing efforts to expand our measurement capabilities, we:
* Wrote the Web Connectivity test helper in Go: https://github.com/ooni/probe-engine/pull/1078 * Split TLS ClientHello to better characterise SNI blocking (as part of research on how to measure TLS middleboxes): https://github.com/ooni/probe-engine/issues/622 * Added support to the http3 experiment (contributed by community members) for using an alternative resolver: https://github.com/ooni/probe-engine/issues/1024 * Wrote a basic specification for the `urlgetter` experiment: https://github.com/ooni/spec/pull/205 * Made `urlgetter` experiment measurements public through the OONI API: https://github.com/ooni/probe-engine/issues/1056
We also fixed a bug which occurred when the new RiseupVPN experiment (contributed by community members) ran after the OONI Psiphon experiment, resulting in the failure of the RiseupVPN experiment. This bug occurred because the RiseupVPN experiment was mistakenly forced to pass through the tunnel created by Psiphon. This was caused by an implementation detail where the tunnel created by Psiphon was associated with the measurement session and subsequently reused by other experiments (in this case, the RiseupVPN experiment). The fix therefore entailed changing the codebase so that the OONI Psiphon experiment uses its own private tunnel and does not share it with the measurement session. This work is documented through the following ticket: https://github.com/ooni/probe-engine/issues/1050
## OONI Explorer
### Website-centric stats page
We made progress on creating a website-centric stats page for OONI Explorer: https://github.com/ooni/explorer/pull/475. The goal of this page is to present users with OONI measurement stats and visualizations based on a tested website around the world.
We created an initial version of this page, which is available here: https://explorer.ooni.org/experimental/website
We encourage community feedback! We are collecting feedback through the following ticket: https://github.com/ooni/explorer/issues/525
### Other OONI Explorer improvements
We also made progress on the activities documented through the following tickets:
* Adding more end-to-end testing of OONI Explorer: https://github.com/ooni/explorer/issues/490 & https://github.com/ooni/explorer/pull/493 * UI improvements: https://github.com/ooni/explorer/issues/511 * Upgrading to the latest ooni-components: https://github.com/ooni/explorer/issues/509 * Adding nivo-toy to OONI Explorer: https://github.com/ooni/explorer/pull/514 * Fixing issues in WhatsApp measurement pages: https://github.com/ooni/explorer/issues/406 * Linking to measurements on OONI Explorer Country pages: https://github.com/ooni/explorer/issues/338
## OONI backend
Throughout November 2020, we worked on the following backend activities:
* Created documentation, diagrams, dashboards, and runbooks * Monitored 500 status code errors * Looked into missing Telegram measurements * Changed the rate limiting * Wrote tooling to manage CI/CD deployments on test and prod hosts * Implemented a backup/export of the fastpath table into S3 * Wrote a measurement_forwarder * Deployed an event detector on test * Monitored coverage before switching all traffic to test-list/urls from ams-pg * Fixed Nginx logging and ipaddr truncation * Implemented API that allows OONI Probes to check-in before running tests * Implemented and deployed the reactive test-list * Added a dashboard to monitor the reactive test-list * Added 2 new test types to the OONI API * Tested the persistent journald * Implemented a deployer to support OSX
## Internet Shutdown Measurement Training for Advocates
Throughout November 2020, we continued to facilitate (as a lead partner) Internews’ Internet Shutdown Measurement Training for human rights advocates in Sub-Saharan Africa.
During the week of 2nd November 2020, we helped facilitate the synchronous training provided by M-Lab for the module on “Measuring Internet Performance”.
During the week of 9th November 2020, we helped facilitate the synchronous training for the module on “Contextualizing Your Data: Using Local Insight and Qualitative Research”.
During the week of 16th November 2020, we helped facilitate the synchronous training provided by Access Now for the module on “Using Measurements in Advocacy”.
During the final week of November 2020, we helped training participants prepare for their group presentations (at the end of the training program on 1st December 2020).
Throughout the training program, we supported the training participants by sharing relevant resources, addressing questions, reviewing their homework assignments, and helping them prepare their projects.
### Created post-training survey
We created a (second, overall) survey for the participants of Internews’ Internet Shutdown Measurement Training program.
This survey was distributed towards the end of the 6-week training program with the goal of collecting participant feedback on:
* The overall training program * Each of the training modules (speaker and guest speaker sessions) * The quality of the training mentorship * Their intra-group collaboration * The homework assignments * How future training programs can be improved
Through this post-training survey, we also aimed to gauge participant knowledge and skills acquired throughout the training program, in order to evaluate its effectiveness.
## Created videos for micro-course on internet censorship
We created 3 short videos (as well as relevant slides and resources) for an online micro-course on internet censorship.
Each of these videos cover the following topics:
* What is Internet Censorship? * The Problem of Internet Censorship * How to Detect Internet Censorship with OONI Probe
We expect to see these resources published by community members in 2021.
## Worked on research report on LGBTIQ website blocking
In collaboration with OutRight Action International and the Citizen Lab, we have been working on a joint research report which examines the blocking of LGBTIQ websites in 6 countries based on OONI data. Throughout November 2020, we continued to edit the report.
## Collaboration with Netalitica on test lists
Netalitica researchers continued to update the Citizen Lab test lists, and we reviewed (and shared feedback on) the updates to the Zimbabwean and Algerian test lists.
We also made updates to several other test lists based on URLs shared by community members:
https://github.com/citizenlab/test-lists/pull/682 https://github.com/citizenlab/test-lists/pull/694 https://github.com/citizenlab/test-lists/pull/684
## Data analysis for Azerbaijan Internet Watch
In support of our partner, Azerbaijan Internet Watch, we analyzed OONI measurements collected from Azerbaijan between September 2020 to November 2020.
In recent months, OONI measurements show that a number of social media services started presenting signs of blocking in Azerbaijan (in addition to the ongoing blocking of media websites) over the last months as well.
We documented the findings of our analysis in a report (including relevant charts), which we shared with Azerbaijan Internet Watch.
## Preparing new Partnerships page for the OONI website
We have created a new page for the OONI website which features all of our partners, highlights their work, links to their websites, and shares some of the research reports and projects that we have collaborated on. We plan to publish this page in December 2020.
In preparation, we reached out to our partners requesting their feedback on this new page and we edited accordingly.
## Creating a new Donate page for the OONI website
We started working on creating a new Donate page for the OONI website. Our work on this front involved the design and creation of relevant mockups and frontend development.
## Community use of OONI data
### OTF Fellow research report on information controls in Myanmar
Over the last year, we had the opportunity to collaborate with and support Phyu Phyu Kyaw, an OTF Information Controls Fellow, who investigated surveillance and internet censorship (through the use of OONI Probe and OONI data) in Myanmar.
More specifically, we supported Phyu Phyu Kyaw with OONI data analysis (and relevant charts) on cases of internet censorship in Myanmar.
Phyu Phyu Kyaw’s report is published here: https://www.opentech.fund/news/information-controls-unprotected-legal-landsc...
### Internews community needs assessment report
Internews carried out a study that aimed to document community needs with respect to advocating against internet shutdowns. Their report, based on this study, is available here: https://static1.squarespace.com/static/5a9efdd2f2e6b149480187ea/t/5fbd37cda2...
Internews’ research found that the vast majority of advocates who participated in this study reported that (out of several network measurement tools) they are mainly familiar with (and have experience using) OONI Probe. The community feedback collected through this study is valuable as it will help inform the development and improvement of our work.
## Community activities
### Internet Governance Forum (IGF)
We participated in the Internet Governance Forum (IGF) 2020 (which took place entirely online) by hosting an OONI e-booth between 2nd to 13th November 2020. We also presented OONI during the IGF Village virtual tours.
Through the OONI e-booth, we aimed to engage participants with OONI’s censorship measurement tools and datasets, share resources, address questions, and receive community feedback (through live consultations).
Information about the OONI e-booth at the IGF Village is available here: https://www.intgovforum.org/multilingual/content/igf-2020-village-booth-60-o...
### State of the Onion 2020
On 16th November 2020, OONI’s Maria presented OONI during the Tor Project’s annual State of the Onion 2020: https://blog.torproject.org/state-of-the-onion-2020
More specifically, we shared OONI highlights from 2020, as well as upcoming OONI projects for 2021. The State of the Onion 2020 event can be viewed here: https://www.youtube.com/watch?v=IyWyTypRGWQ
### OONI Community Meeting
On 24th November 2020, we hosted the monthly OONI Community Meeting on our Slack channel (https://slack.ooni.org/), during which we discussed the following topics:
1. Brief updates from the OONI-verse
2. Open source and reliable auditing frameworks
3. Integrating the new DNSCheck experiment into the OONI Probe mobile and desktop apps
4. Requirements for running and shipping new experimental tests: Request for community feedback
## Userbase
In November 2020, 3,368,016 OONI Probe measurements were collected from 4,679 networks in 195 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/
We examined this drop in measurement coverage (in comparison to previous months), and we found that it is explained primarily by the following 2 reasons:
* We recently stopped processing and publishing AS0 measurements, particularly because measurements without an ASN have very limited value; * Several legacy probes (which contributed large volumes of stable measurements) were discontinued over the last few months.
That said, we aim to boost global measurement coverage over the next months once regular automatic testing is shipped as part of the OONI Probe mobile app. We also plan to release official Linux and macOS packages for OONI Probe, which will serve as a replacement for legacy probe users.
~ OONI team.