Hello,
Hope you're all doing well.
These days we're primarily supported by the DRL, which is why we do quarterly reporting. That said, we'd like to share monthly updates from the OONI team with the community, hence our April 2021 report shared below. I'll also follow-up to share our May and June 2021 reports.
*# OONI Monthly Report: April 2021*
Throughout April 2021, the OONI team worked on the following sprints:
* Sprint 36 - Rhodophyta (1-11 April, 2021) * Sprint 37 - Vampire squid (12-25 April, 2021) * Sprint 38 - Umbrella octopus (26-30 April, 2021)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
*## Released new OONI Probe test for Signal Private Messenger App*
The OONI Probe apps now feature a brand new test for the Signal Private Messenger app! You can test the Signal app on your network and check if it's blocked.
The specification of the OONI Probe Signal test is available here: https://github.com/ooni/spec/blob/master/nettests/ts-029-signal.md
We also wrote a Signal test description (for a more general audience), which we published on our website: https://ooni.org/nettest/signal
Through the OONI Probe Signal test, users can check if it’s possible to establish a connection to the various backend services used by the Signal app or if Signal is being interfered with on their network.
Following the implementation of our Signal test ( https://github.com/ooni/probe-cli/pull/230), we created a UI for displaying the test results inside of the OONI Probe apps. We then worked on integrating the Signal test inside of the OONI Probe mobile and desktop apps in preparation for the new app releases.
Our new Signal test was shipped as part of OONI Probe Desktop 3.4.0 ( https://github.com/ooni/probe-desktop/pull/202) and OONI Probe Mobile 2.11.0 (https://github.com/ooni/probe-android/pull/422, https://github.com/ooni/probe/issues/1415).
All Signal test results from around the world are openly published in near real-time on OONI Explorer: https://explorer.ooni.org/search?since=2021-03-15&test_name=signal
*## Updated the user guides for OONI Probe Mobile and Desktop Apps*
In April 2021, we published updated versions of our user guides for the OONI Probe apps:
1) User Guide: OONI Probe Mobile: https://ooni.org/support/ooni-probe-mobile
2) User Guide: OONI Probe Desktop: https://ooni.org/support/ooni-probe-desktop
Following relevant OONI Probe mobile and desktop releases, we updated the guides to include new features, settings, step-by-step instructions, and screenshots (https://github.com/ooni/ooni.org/issues/844).
*## Beta for automated testing on OONI Probe Android*
In April 2021, we released a beta version for automated testing on OONI Probe Android! This means that users can have OONI Probe test websites automatically multiple times per day (without having to remember to run OONI Probe or do anything).
We reached out to community members, requesting that they help test the beta and report any issues they encounter. Users could help test the beta version by:
1) Joining the beta program: https://play.google.com/apps/testing/org.openobservatory.ooniprobe
2) Updating (through the Play Store) to OONI Probe 3.0.0-beta.1 (Android)
3) Through the OONI Probe 3.0.0-beta.1 app, tapping Settings -> Automated testing and enabling the "Run tests automatically" setting.
The beta included settings for having automated tests run only when users are connected to WiFi and while their phone is charging (though they can configure this in the "Automated testing" settings).
To avoid cluttering the UI of the OONI Probe Test Results screen, the test results from automated testing will only be made available on OONI Explorer (published automatically in real-time): https://explorer.ooni.org/
*## Improvements to the testing of OONI Probe Android*
We onboarded an external contractor (Bloco) to assist us with the development of integration and unit tests for OONI Probe Android. This has led to improving the overall code quality of the app, as well as ensuring that it’s better tested so that we can have higher confidence in what we ship to end users.
This work is documented through the following pull request: https://github.com/ooni/probe-android/pull/426
*## Research on implementing automated testing on iOS*
We carried out research to explore the feasibility of adding support to the OONI Probe iOS app to perform background tasks (required for automated testing). We also created a prototype for OONI Probe iOS background tasks: https://github.com/ooni/probe/issues/1459
*## OONI Probe backend proxy support*
We worked on adding backend proxy support to the OONI Probe apps in order to make them more resilient to intentional or accidental blocking. More specifically, we added support that will enable OONI Probe users to use Psiphon or a custom proxy to communicate with OONI backend services.
In April 2021, we released an alpha version with Psiphon proxy support: https://github.com/ooni/probe-cli/releases/tag/v3.10.0-alpha. We also created a UI for the new OONI backend proxy settings in the OONI Probe apps.
All of this work is documented through the following ticket: https://github.com/ooni/probe/issues/985
*## Preview release of OONI Probe Desktop with RTL support*
In April 2021, we made a preview release of OONI Probe Desktop with RTL support: https://github.com/ooni/probe-desktop/releases/tag/v3.5.0-rtl.1
This experimental release includes support for languages that use Right-to-Left (RTL) scripts, such as Arabic and Farsi. We shared this preview release with community members to get their feedback on how to improve it before the public release. Based on their feedback, we made a series of improvements.
*## OONI Probe Command Line Interface (CLI)*
We released OONI Probe CLI 3.9.0 (https://github.com/ooni/probe/issues/1369) which includes a series of improvements and end-to-end testing. We continued to work on redesigning the probeservices package ( https://github.com/ooni/probe/issues/1405) to add circumvention support, making OONI Probe more resilient to censorship. We also completed work related to implementing the check-in API in OONI Probe CLI ( https://github.com/ooni/probe/issues/1299).
*## Changes to the OONI Probe Android TLS fingerprint*
We noticed that OONI Probe Android was not working properly on some networks in Iran. We therefore investigated the issue and discovered that the block had to do with the fact that a specific ordering of the TLS cipher suites was leading to fingerprinting and blocking on this network.
As a result of this, we patched the golang TLS code to maintain the same cipher suite ordering as it does on desktop, which resulted in us managing to unblock the app in Iran. More details about this issue are available through the following ticket: https://github.com/ooni/probe/issues/1444 (and our relevant blog post: https://ooni.org/post/making-ooni-probe-android-more-resilient/#changing-our... ).
*## Research on implementing a new Snowflake experiment in OONI Probe*
We carried out research (and testing) necessary for evaluating the feasibility of implementing a new Snowflake experiment in the OONI Probe apps and concluded that it is feasible. Our work on this research prototype is documented through the following ticket: https://github.com/ooni/probe-engine/issues/283
*## Data quality improvements*
We improved upon the quality of OONI Probe RiseupVPN measurements by fixing a bug to reduce the ratio of false positives. This work is documented through the following ticket: https://github.com/ooni/probe/issues/1354
Based on the RiseupVPN test, RiseupVPN was flagged as “likely blocked” even if a single gateway was blocked, without taking into account the built-in fallback mechanisms. Moreover, the gateways were indicated as “likely blocked” if they were malfunctioning for other reasons (such as misconfiguration, DDoS, reboot on security updates etc.). As a result, this produced a lot of false positives. To address this, we made a series of improvements in order to improve the accuracy of the RiseupVPN test and to reduce the ratio of false positives.
*## Improvements to OONI Explorer*
We added support for grouping test names by the relative test group under the “Test Name” filter of the OONI Explorer Search page: https://github.com/ooni/explorer/pull/573
We fixed an issue related to how we were displaying the timestamps in the summary text of the OONI Explorer measurement pages: https://github.com/ooni/explorer/issues/541
Thanks to a community contribution, we now have support in the measurement pages for expanding all the measurement data JSON keys: https://github.com/ooni/explorer/pull/567
*## Creating a real-time incident response dashboard*
We made considerable progress on the Measurement Aggregation Toolkit, which allows users to plot aggregate charts of measurements in response to ongoing censorship events. In particular, we added support for plotting charts on multiple axes, allowing us to display anomaly counts of measurements over time per URL.
This work is documented through the following pull request: https://github.com/ooni/explorer/pull/579
*## Proof of concept for new URL submission web platform*
To enable community members to contribute to the Citizen Lab test lists (which include URLs measured by OONI Probe), we are creating a web platform that will enable users to review and contribute to the test lists without using GitHub (contributions from this new web platform will still end up as pull requests on the Citizen Lab test list repository for review).
In April 2021, we created a proof of concept backend implementing the full URL submission functionality. We Integrated the URL submission/editing backend into the OONI API: https://github.com/ooni/backend/issues/490. This allows users to contribute URLs to the Citizen Lab test lists without needing to create a GitHub account or knowing how to format a pull request. This work is documented through the following ticket: https://github.com/ooni/backend/issues/487
*## OONI backend*
Throughout April 2021, we worked on the following backend activities:
* Added support to the OONI API for the new OONI Probe Signal test * Investigated a check-in test list bug (which wasn’t returning entries), added metrics, fixed the bug, and added fallback options: https://github.com/ooni/probe/issues/1328 * Monitored the growth of incoming OONI measurements from unattended runs (to ensure that our infrastructure can handle the increased load and to check if measurement coverage increases as expected) * We started evaluating DuckDB as an alternative database solution by building a test DB and performing a quick benchmark * Server infrastructure improvements related to VPS that we use for running OONI Probe CLI experiments * Worked on switching ooniprobe and backend Debian packages to S3: https://github.com/ooni/probe/issues/1359 * Made progress on the OONI API registration and login, deployed it on our testing infrastructure and performed testing with probes: https://github.com/ooni/backend/issues/491 * Integrated the URL submission/editing backend into the OONI API: https://github.com/ooni/backend/issues/490 * Discussed backend/db requirements for storing new dimensions around OONI Web Connectivity measurements (server IP address, multiple DNS resolvers, blocking taxonomy): https://github.com/ooni/backend/issues/492
*## Dealt with the discontinuation of Bintray*
We used to rely on Bintray for hosting the Debian package repositories for probe-cli, as well as several internal tooling used in our infrastructure. Bintray has announced that they are going to discontinue the service ( https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcen...) and as a result of this, we have had to migrate our infrastructure away from Bintray and replace it with an s3 based solution. This work is documented through the following ticket: https://github.com/ooni/probe/issues/1437
We also moved the debian package generation over to github actions so that it’s following the same pattern used in the rest of our repositories. Moreover we added support for signing of packages and added support for the detection of duplicate packages.
*## Updated OONI Data Policy*
On 13th April 2021, we published an updated version of the OONI Data Policy: https://ooni.org/about/data-policy
This updated documentation reflects the changes we made in terms of our use of analytics in the OONI Probe apps ( https://github.com/ooni/ooni.org/pull/838).
Following a careful audit of various analytics tools (as reported in March 2021), we removed Firebase analytics and app usage metrics entirely from the OONI Probe mobile app, while ensuring that no analytics are used on OONI Probe F-Droid. Rather, we limited the OONI Probe mobile app to the use of Sentry for crash reports (when users opt-in) and to the use of Countly for sending push notifications on Android. On OONI Probe desktop, we removed Matomo app usage metrics, and limited the app to the use of Sentry for crash reports (when users opt-in). We made these changes as part of our data minimization efforts.
*## Academic paper investigating internet censorship in Myanmar*
We collaborated with CAIDA, Censored Planet, and other researchers on an academic research paper investigating the censorship events in Myanmar following the military coup (on 1st February 2021). This paper builds upon our previous study on censorship events in Myanmar ( https://ooni.org/post/2021-myanmar-internet-blocks-and-outages/), offering additional data analysis and new insights.
In April 2021, we worked on (and completed) OONI data analysis required for this academic paper (https://github.com/ooni/ooni.org/issues/855). Specifically, we analyzed OONI measurements collected from Myanmar with the aim of examining whether and how censorship events changed in the country between January 2021 to April 2021, how censorship varied across AS networks, and we investigated potential cases of collateral damage (as a result of IP blocks).
Based on our analysis, we produced multiple charts (communicating key findings) and started working on writing the paper.
*## Data analysis for IPYS Venezuela*
We provided data analysis support to our Venezuelan partner, IPYS Venezuela (in support of their annual digital rights report). This involved a year-long analysis of OONI measurements collected from Venezuela, primarily aimed at identifying the blocking of websites (and how that varies across AS networks). As a result of this analysis, we produced a relevant chart and CSV file (https://github.com/ooni/ooni.org/issues/823), which we shared with IPYS Venezuela in support of their research.
*## Data analysis for Azerbaijan Internet Watch*
In support of our partner, Azerbaijan Internet Watch, we analyzed OONI measurements collected from Azerbaijan between 1st January 2021 to 30th April 2021. As part of this analysis, we documented the (ongoing) blocking of independent news media and circumvention tool websites (based on OONI data).
Based on our analysis, we produced relevant charts (that summarize key findings) and a report: https://www.az-netwatch.org/wp-content/uploads/2021/04/2021-04-Azerbaijan-In...
Azerbaijan Internet Watch also published a summary of our report findings on their site: https://www.az-netwatch.org/news/azerbaijan-keeps-blocking-access-to-key-ind...
*## Data analysis for the Tor Project*
We collaborate with the Tor Project’s anti-censorship team on monitoring Tor blocking around the world. As part of these efforts, we analyzed OONI measurements collected from Belarus over the last year to examine torproject.org reachability.
Based on this analysis, we produced and shared relevant CSV files and charts with the Tor Project’s anti-censorship team.
*## Test list updates*
In April 2021, we updated the Citizen Lab test list for Togo ( https://github.com/citizenlab/test-lists/pull/766) by adding media websites shared by Reporters Without Borders.
*## Planning the OONI Partner Training 2021*
In April 2021, we started planning the logistics surrounding the organization of an OONI Partner Training. We were initially hoping to host an in-person event for all our partners, but in light of the COVID-19 pandemic (and associated travel restrictions), we decided to facilitate the event entirely online.
Given that our partners are located around the world (from Latin America to Asia), we needed to consider training formats that would accommodate different time zones and different community needs. As such, we started organizational work related to hosting online OONI Partner Trainings in April 2021 (the events were scheduled for late June 2021 and early July 2021), and created a Concept Note.
*## Ford Foundation Financial Innovation & Resilience Training*
As part of our participation in the Ford Foundation’s Financial Innovation and Resilience training program, we attended a relevant workshop on 14th April 2021, and a relevant webinar on 22nd April 2021.
We were offered this great opportunity because we are a grantee of the Ford Foundation, who support OONI’s community-related work. The knowledge and skills gained through this training program have helped us rethink and improve OONI’s fundraising strategies.
*## Community activities### 87th Nexa Lunch Seminar*
On 28th April 2021, OONI’s Simone was a speaker at the Nexa Center for Internet and Society’s (online) 87th Lunch Seminar (titled “The risks of digital sovereignty”). Information about the event is available here: https://nexa.polito.it/lunch-87
*### Research on Twitter throttling in Russia*
Censored Planet produced a great research report examining the throttling of Twitter in Russia: https://censoredplanet.org/throttling. OONI’s Simone collaborated with the Censored Planet team and contributed to this research.
*### OONI Community Meeting*
On 27th April 2021, we hosted the monthly OONI Community Meeting on our Slack channel (https://slack.ooni.org/), during which we discussed the following topics:
1. Updates from the OONI team: (1) New Signal test, (2) Automated testing on OONI Probe Android 3.0.0-beta.1, (3) Switching from legacy ooniprobe to OONI Probe CLI.
2. Updates on internet censorship in Uganda (social media and VPN blocking) and Tanzania (Twitter blocking).
3. Updates on OONI’s new URL submission web platform.
4. The research benefit of potentially including dead/expired URLs in test lists.
*## Userbase*
In April 2021, 8,539,001 OONI Probe measurements were collected from 5,141 AS networks in 200 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/
~ OONI team.
ooni-talk@lists.torproject.org