Hello,
This email shares OONI's monthly report for June 2023.
*# OONI Monthly Report: June 2023*
Throughout June 2023, the OONI team worked on the following sprints:
* Sprint 93 (5th - 18th June 2023) * Sprint 94 (19th June - 2nd July 2023)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
*## Launch of OONI Explorer supporting localization*
We are thrilled to share that OONI Explorer (https://explorer.ooni.org/) now supports localization! OONI Explorer is an open data resource on internet censorship around the world.
As of June 2023, OONI Explorer is available in 8 new languages: * Chinese: https://explorer.ooni.org/zh-cn * Farsi: https://explorer.ooni.org/fa * French: https://explorer.ooni.org/fr * German: https://explorer.ooni.org/de * Russian: https://explorer.ooni.org/ru * Spanish: https://explorer.ooni.org/es * Thai: https://explorer.ooni.org/th * Turkish: https://explorer.ooni.org/tr
We thank the Localization Lab (https://www.localizationlab.org/) community for making OONI Explorer – and real-time open data on internet censorship worldwide – available in so many new languages!
*## OONI Probe Mobile*
We made some improvement to the code quality of OONI Probe Android by making progress on de-duplicating the code used for running background tests (https://github.com/ooni/probe-android/pull/572) and we made improvements to some of the unit tests ( https://github.com/ooni/probe-android/pull/575). We also added the ability to “select all”, “deselect all” category codes in the websites categories to test settings (https://github.com/ooni/probe-android/pull/576).
*## OONI Probe user guides translated to German*
Thanks to a community member (Curtis Baltimore), the OONI Probe user guides are now available in German!
The German user guides are available through the following links: * OONI Probe Mobile User Guide: https://ooni.org/de/support/ooni-probe-mobile * OONI Probe Desktop User Guide: https://ooni.org/de/support/ooni-probe-desktop
*## OONI Run*
We have started adding support to OONI Probe Android for parsing and displaying OONI Run v2 descriptors ( https://github.com/ooni/probe-android/pull/577). We also made progress on the web editor for OONI Run v2 links (https://github.com/ooni/run/pull/131) and on the backend for adding support for storing and serving them, by deploying it to the testing infrastructure ( https://github.com/ooni/backend/pull/678).
*## OONI Probe CLI*
We released OONI Probe CLI v3.17.4 ( https://github.com/ooni/probe-cli/releases/tag/v3.17.4). This release updates to OpenSSL 1.1.1u (https://github.com/ooni/probe-cli/pull/1148), includes a test helper fix to correctly measure Twitter ( https://github.com/ooni/probe/issues/2488), and updates the GeoIP database to 2023-06 to address a Facebook Messenger issue ( https://github.com/ooni/probe/issues/2485). We also released OONI Probe v3.17.5 (https://github.com/ooni/probe-cli/releases/tag/v3.17.5), which updates to go1.19.10.
Moreover, we upgraded the main development branch to go1.20.5 ( https://github.com/ooni/probe/issues/2478) and we released OONI Probe CLI v3.18.0-alpha (https://github.com/ooni/probe-cli/releases/tag/v3.18.0-alpha) and v3.18.0 (https://github.com/ooni/probe-cli/releases/tag/v3.18.0).
In addition to the aforementioned patches that we also included in v3.17.x, the v3.18.x series releases the following major features (please, refer to the v3.18.0-alpha and v3.18.0 release notes for the complete changelog):
* dslx ( https://pkg.go.dev/github.com/ooni/probe-cli/v3@v3.18.0/internal/dslx): an internal DSL for simplifying the process of writing and maintaining network experiments;
* netemx ( https://pkg.go.dev/github.com/ooni/probe-cli/v3@v3.18.0/internal/netemx): support for writing integration testing using Gvisor (a TCP/IP stack in userspace that allows us to simulate censorship and throttling in a predictable environment).
We started working on dslx in December 2022 and we started working on netemx in March 2023. We are excited to see these features finally landing in a stable release!
While working on the v3.18.x release, we also updated the underlying ooni/netem library (https://github.com/ooni/netem) used by the netemx feature to emulate DNS spoofing (https://github.com/ooni/netem/pull/27).
By releasing v3.18.0 (https://github.com/ooni/probe-cli/releases/tag/v3.18.0), which uses go1.20.5, we also addressed a community raised issue ( https://github.com/ooni/probe/issues/2478) where it was impossible to compile OONI Probe using the go1.20.x compiler series.
*## Expanding OONI’s testing model to support richer testing input*
We continued to work on supporting richer testing input ( https://github.com/ooni/ooni.org/issues/1291). We tagged version v0.1.0 ( https://github.com/ooni/2023-05-richer-input/releases/tag/v0.1.0) of our ooni/2023-05-richer-input prototype ( https://github.com/ooni/2023-05-richer-input). This release saves the stage of the repository at the end of May 2023. In this initial prototype, we implemented richer input as a list of flat “mini nettests” that an existing nettest (e.g., Facebook Messenger) should execute to obtain a complete measurement of all the relevant targets.
While this design was adequate for an initial prototype, in June 2023 we spent additional time trying to engineer and simplify this design. In particular, we investigated whether we could reuse the DSL introduced in ooni/probe-cli v3.18.0-alpha ( https://github.com/ooni/probe-cli/releases/tag/v3.18.0-alpha) to express richer inputs in a more flexible and composable way ( https://github.com/ooni/probe/issues/2494). This culminated in releasing v0.2.0 (https://github.com/ooni/2023-05-richer-input/releases/tag/v0.2.0) of the prototype, which included a possible richer input implementation that was using a DSL instead of flat nettest targets.
A DSL-based design is more flexible because we can naturally compose existing building blocks (e.g., DNS lookups, TCP connect, TLS handshake) with other blocks whose task is that of calculating a nettest’s top level keys (e.g., the `blocking` key of Web Connectivity).
We introduced a complete DSL for richer input and adapted Facebook Messenger to use such a DSL ( https://github.com/ooni/2023-05-richer-input/pull/5, https://github.com/ooni/2023-05-richer-input/pull/6).
*## Creating a throttling measurement methodology*
Regarding measuring throttling (https://github.com/ooni/ooni.org/issues/1296), we focused on adding (to the DSL introduced in ooni/probe-cli v3.18.0-alpha (https://github.com/ooni/probe-cli/releases/tag/v3.18.0-alpha)) prototype support for collecting extra statistics during long downloads to Web Connectivity LTE (the current experimental version of Web Connectivity) and to the ooni/2023-05-richer-input ( https://github.com/ooni/2023-05-richer-input). Currently, the underlying framework that we use for these three pieces of code collects up to 64 network events during and after the TLS or the QUIC handshake. While these events naturally include the initial download of the body, we wanted to extend this model to collect download speed snapshots.
To this end, we imported memoryless from m-lab/go ( https://github.com/ooni/probe-cli/pull/1163) to have support for sampling at exponentially distributed (and truncated) intervals, which provides PASTA properties (https://en.wikipedia.org/wiki/Arrival_theorem) when observing network events. We then used this functionality to implement collecting download speed snapshots ( https://github.com/ooni/probe-cli/pull/1166) and we added initial support for measuring the download speed ( https://github.com/ooni/2023-05-richer-input/pull/4).
*## Creating a Social Media Censorship Alert System*
In May 2023, we continued to make progress towards building a Social Media Censorship Alert System. Specifically, we added support for the reprocessing of existing measurements, and we are currently running the event detector on the backend host. As part of improvements to the overall event detector logic, we fixed a bug affecting how columns are handled in Pandas.
This work is documented through the following pull request: https://github.com/ooni/backend/pull/651
*## Creating a Censorship Incident Reporting Platform*
We’re excited to share that we’re building a Censorship Incident Reporting Platform!
New censorship events frequently emerge worldwide on an ongoing basis, but producing an in-depth report and analysis on each event (in a timely manner) is not easily scalable. Moreover, rapid response efforts benefit from censorship findings that are easier and quicker to consume.
We therefore aim to create a Censorship Incident Reporting Platform which will regularly provide short reports with summary information on emergent censorship events based on OONI data and charts. This will enable journalists and human rights defenders to more easily learn about and respond to censorship events. Over time, the platform will become an archive of known censorship incidents from around the world based on OONI network measurement data.
In June 2023, we made significant progress towards creating a Censorship Incident Reporting Platform. Specifically, we developed the backend API for creating, updating and retrieving censorship incidents through OONI Explorer (https://github.com/ooni/backend/pull/678). This will allow logged in OONI Explorer users to create incidents that can then be retrieved by any user on the OONI Explorer platform, allowing us to develop views for them. The backend API feature was deployed on testing and production infrastructure, allowing us to start working on the frontend for displaying and editing the censorship incidents.
On the frontend part, we started working on the editor ( https://github.com/ooni/explorer/pull/862), which supports an enriched markdown format that allows us to embed MAT charts within markdown formatted text. It also allows us to define some metadata which can be used for retrieving incidents matching a specific search criterion (e.g. all censorship incidents pertaining to a particular country).
*## OONI backend*
In June 2023, we implemented the extraction of test helper metadata (URL and type) in the fastpath (https://github.com/ooni/backend/pull/682), which will enable us to track which test helper was used in Web Connectivity measurements and to create dashboards to detect correlation between test helper locations and test failures. We updated the test helper rotation tool to deploy Vector and configure log forwarding on test helpers ( https://github.com/ooni/backend/pull/692), which will allow collecting logs from the test helpers centrally and storing them. We also implemented an initial form of alarming based on the test helper failure rate.
We added support for GeoIP lookup for the probes ( https://github.com/ooni/backend/pull/678), and we implemented an experimental tool and dashboard to monitor the accessibility of our website (ooni.org) through Tor. We created an experimental STUN server to perform initial connectivity tests from countries that block many other protocols.
*## Automating censorship detection and characterization based on OONI measurements*
In June 2023, we added support for storing IP metadata related to control measurements (e.g. AS and country information) to be used as part of the DNS analysis (https://github.com/ooni/data/pull/30). We also added support for making it easier to perform migrations of the database SQL schema, allowing us to more quickly enrich the database table with more features ( https://github.com/ooni/data/pull/30/commits/7c9b6a5c68841c86e553c2d60d5fbf4... ).
We also added support for correctly mapping unknown_failures (caused by software bugs) to known failure strings, which allows us to consider more kinds of measurements as part of the analysis (which would have otherwise been considered as “failed”). This work is documented here: https://github.com/ooni/data/pull/29
Finally, we added support for skipping the use of the dask parallel processing engine, when no parallelism is needed, thereby increasing performance (https://github.com/ooni/data/pull/31).
*## Interviewed Project Manager candidates*
In June 2023, we completed the first round of interviews for the OONI Project Manager position ( https://ooni.org/post/2023-job-opening-ooni-project-manager/). Based on these interviews, we created a shortlist for the second (and final) round of interviews and coordinated with candidates. We also started the second round of interviews with shortlisted candidates.
*## Blocking of social media in Senegal*
Following unrest erupted over the sentencing of opposition leader Ousmane Sonko, OONI data from Senegal shows that ISPs started blocking access to several social media platforms on 1st June 2023.
Specifically, OONI data shows the blocking of: * WhatsApp: https://explorer.ooni.org/chart/mat?probe_cc=SN&since=2023-05-03&unt... * Telegram: https://explorer.ooni.org/chart/mat?probe_cc=SN&since=2023-05-03&unt... * Facebook, Instagram, YouTube, Twitter: https://explorer.ooni.org/chart/mat?probe_cc=SN&since=2023-05-03&unt...
OONI data collected from Senegal shows that access to social media and messaging platforms is blocked by means of TLS interference (i.e. TLS handshakes timing out).
In response to the blocks, we shared relevant OONI data (and charts) on our social media platforms ( https://twitter.com/OpenObservatory/status/1664611174854303745), with the #KeepItOn advocacy community, and with Senegalese advocacy groups.
*## Blocking of F-Droid in China*
As of 22nd June 2023, OONI data collected from China shows the blocking of F-Droid by means of DNS injection, where an unrelated IP address (e.g. DropBox) is returned to the client ( https://explorer.ooni.org/m/20230622211826.539282_CN_webconnectivity_87bc828...). This technique is commonly used in China to restrict access to services, and OONI data shows that F-Droid is blocked on at least 4 networks in China.
OONI data on the blocking of F-Droid in China is available here: https://explorer.ooni.org/chart/mat?probe_cc=CN&since=2023-06-13&unt...
In response to the block, we shared OONI data (and charts) in both English and Chinese on our social media platforms ( https://twitter.com/OpenObservatory/status/1673659531639894019). We also reached out to F-Droid developers to share relevant data that could potentially help with circumvention.
*## Community use of OONI data### Civil society response to the social media blocks in Senegal*
As soon as the social media blocks emerged in Senegal on 1st June 2023, Senegalese civil society groups ran OONI Probe and independently reported on the blocks, sharing OONI data ( https://twitter.com/aliamsi/status/1664374471849902080). Similarly, other internet freedom community members, such as Deutsche Welle’s Oliver Linow, responded to the blocks by sharing OONI data ( https://twitter.com/OliverLinow/status/1664397344111042565).
*### Access Now statement in response to social media blocks in Senegal*
In response to the social media blocks in Senegal, Access Now published a statement (citing OONI data): https://www.accessnow.org/press-release/keep-people-connected-senegal/
*## Community activities### RightsCon 2023*
Between 5th-8th June 20213, OONI’s Elizaveta, Maria, and Arturo traveled to Costa Rica to attend RightsCon 2023 (https://www.rightscon.org/).
As part of their participation, they presented OONI as part of the following 5 sessions:
1) Session: “KeepItOn in 2022: successes, opportunities, struggles, and lessons learned in the fight against election-related shutdowns” (hosted by Access Now and KICTANET), 6th June 2023, OONI speaker: Elizaveta Yachmeneva
2) Session: “Data and Action: Using data analysis for incident response and threat prevention” (hosted by the Center for Digital Resilience), 7th June 2023, OONI speaker: Arturo Filastò
3) Workshop: “Learn to investigate internet shutdowns through an Iran 2022 case study” (hosted by OONI and IODA), 7th June 2023, OONI speaker: Maria Xynou
4) Session: “Internet blocking and the consequences for human rights?” (hosted by Cloudflare), 8th June 2023, OONI speaker: Arturo Filastò
5) Lightning talk: “Investigating Internet shutdowns through open data” (hosted by OONI, IODA, M-Lab), 8th June 2023, OONI speaker: Maria Xynou
We also participated in a private #KeepItOn meeting to discuss how telcos can support the fight against internet shutdowns.
Overall, participating at RightsCon provided a great opportunity to reconnect with many of our partners and community members in person, and to learn from many of the sessions that we attended!
*### OONI workshop for journalists and civil society in Sierra Leone*
On 21st June 2023, OONI’s Elizaveta facilitated an online OONI workshop for journalists and civil society organizations in Sierra Leone as part of the hybrid Stakeholders Dialogue on Internet Shutdowns event organized by Access Now and our Sierra Leone partner, CHRDI ( https://ooni.org/partners/chrdi/), ahead of Sierra Leone’s 2023 elections.
Information about the event is available through the following links: https://twitter.com/chrdiorg/status/1671661565068623877 https://twitter.com/accessnow/status/1670860527541600256
The event (along with the OONI workshop) can be watched here: https://www.youtube.com/watch?v=ho1wTgvIE-M
*### OONI webinar for civil society in Sierra Leone*
On 22nd June 2023, OONI’s Elizaveta presented OONI’s tools and methods to civil society groups in Sierra Leone as part of Paradigm Initiative’s “Defend, Connect, Report: Navigating Internet Censorship and Defending Digital Rights in Sierra Leone’s Election” webinar in preparation for Sierra Leone’s 2023 elections.
Information about the webinar is available here: https://twitter.com/ParadigmHQ/status/1671091608840990722
*### OONI workshops at DSAConnectCon 2023*
Between 28th-29th June 2023, OONI’s Maria traveled to Johannesburg, South Africa, to participate at DSAConnectCon, a 2-day community convening organized by our partner, Digital Society of Africa (DSA). Learn about DSA’s important work here: https://digitalsociety.africa/
As part of her participation, Maria co-facilitated 2 training sessions in collaboration with IODA’s Amanda. These training sessions (“Investigating Internet shutdowns through open data”) introduced participants to both OONI and IODA tools, methods, and datasets, including hands-on exercises based on real-world internet shutdown scenarios.
Maria also assisted participants with localizing OONI Probe into 7 additional African languages (Zulu, Ndebele, Nyanja, Swati, Mozambican Portuguese, Tumbuka, Shona) during the Localization Lab’s ( https://www.localizationlab.org/) hands-on workshop on localizing internet freedom tools.
*### OONI Community Meeting*
On 27th June 2023, we hosted the monthly OONI Community Meeting on our Slack channel (https://slack.ooni.org/), during which we briefly discussed:
1) Social media blocks in Senegal. 2) F-Droid block in China. 3) Monitoring internet censorship in Sierra Leone during the country’s 2023 elections. 4) OONI Explorer being released in 8 new languages (encouraging further localization efforts).
*## Measurement coverage*
In June 2023, 56,948,483 OONI Probe measurements were collected from 2,897 networks in 165 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/
~ OONI team.