Hello,
OONI's August 2022 status report is shared below.
*# OONI Monthly Report: August 2022*
Throughout August 2022, the OONI team worked on the following sprints:
* Sprint 71 (1st-14th August 2022) * Sprint 72 (15th-28th August 2022)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
*## New OONI Team member*
In August 2022, Elizaveta Yachmeneva joined the OONI team as our new Community Coordinator! Elizaveta will support the global OONI community and help expand our community work.
*## OONI Probe Mobile*
In August 2022, we released OONI Probe Android 3.7.1 with bug fixes and improvements (https://github.com/ooni/probe-android/releases/tag/v3.7.1).
We continued to work on a prototype of the OONI Probe app using Flutter, which will allow us to have a shared codebase between the OONI Probe mobile and desktop apps. Specifically, we experimented with connecting Flutter with the OONI Probe engine (https://github.com/ooni/probe/issues/2197). In particular, we experimented with three distinct approaches ( https://github.com/ooni/probe-cli/pull/849, https://github.com/ooni/probe-cli/pull/850, https://github.com/ooni/probe-cli/pull/851) to expose the OONI Probe engine to code written in Dart (which is the language used by Flutter). Once that work was done, we adapted our app prototypes to use the recent OONI Probe engine changes and generate PoC apps for mobile and desktop using Flutter ( https://github.com/ooni/probe-cli/pull/880).
We also looked into Sentry issues (https://github.com/ooni/probe/issues/2187 , https://github.com/ooni/probe/issues/2186), Google Play errors, and failing CI on OONI Probe Android.
*## OONI Run*
Following the implementation of OONI Run v2 for miniooni users (in July 2022), we wrote a design document which describes the miniooni OONI Run v2 functionality (https://github.com/ooni/probe-cli/pull/916).
*## OONI Probe Desktop*
In response to community feedback, we fixed the summary table for Tor measurements on OONI Probe Desktop ( https://github.com/ooni/probe/issues/2209), and we looked into fixing inconsistencies when displaying results ( https://github.com/ooni/probe/issues/2210).
*## Worked on updating the OONI Probe user guides*
Throughout August 2022, we worked on extensive updates to the OONI Probe mobile and desktop user guides to ensure that they include information and screenshots for features based on the latest releases. This work is documented through the following pull request: https://github.com/ooni/ooni.org/pull/1202
*## Test Lists Editor*
Following the launch of OONI’s Test Lists Editor in July 2022 ( https://ooni.org/post/2022-test-lists-editor/), we continued to work on improvements to the platform. Specifically, we improved the UI of the “Add new URL” section (https://github.com/ooni/test-lists-ui/issues/61), we fixed an issue with repeated background requests to account metadata endpoint (https://github.com/ooni/test-lists-ui/issues/62), and we adapted the login request to work with the new API changes.
*## Expanding censorship measurement methodologies*
In previous months, we created Websteps ( https://github.com/bassosimone/websteps-illustrated/) which includes significant improvements to how we measure website blocking. But shipping Websteps would create a wide range of complexities for interdependent OONI software components and data structures. We therefore decided that rather than shipping Websteps, we will instead improve upon our Web Connectivity experiment based on Websteps.
To this end, we drafted and implemented a plan ( https://github.com/ooni/probe/issues/2237) for improving upon our Web Connectivity experiment by incrementally integrating selected features and experiments from our Websteps prototype ( https://github.com/bassosimone/websteps-illustrated/) into Web Connectivity.
During August 2022, we created a new version of Web Connectivity (“Web Connectivity v0.5”) based on the “step-by-step” style ( https://github.com/ooni/probe-cli/pull/814), which aims to incrementally integrate features from Websteps. The initial implementation of Web Connectivity v0.5 was introduced through the following pull request: https://github.com/ooni/probe-cli/pull/882
Subsequent changes focused on collecting late, duplicate DNS replies, which often provide a clear indication of censorship ( https://github.com/ooni/probe-cli/pull/878, https://github.com/ooni/probe-cli/pull/883), as well as on improving the Web Connectivity test helper to support v0.5 measurements ( https://github.com/ooni/probe-cli/compare/c3964e43b33d36b7cba371e9906d5ed0bc...). One of these patches (https://github.com/ooni/probe-cli/pull/878) was contributed by Mehul Gulati, one of our Google Summer of Code (GSoC) students.
We improved the code used by Web Connectivity v0.5 to simplify the task of splitting measurements into observations ( https://github.com/ooni/probe-cli/pull/879, https://github.com/ooni/probe-cli/pull/873), thus enabling efforts aimed at automating the detection and characterization of more forms of website censorship. We also improved DNS measurement code to collect CNAMEs from DNS responses (https://github.com/ooni/probe-cli/pull/875, https://github.com/ooni/probe-cli/pull/876, https://github.com/ooni/probe-cli/pull/877).
To write this new version of Web Connectivity in a more automated way, we specified and drafted a code generation tool called “boilerplate” ( https://github.com/ooni/probe-cli/pull/853). This tool should simplify maintaining code used for performing specific measurements using the “step-by-step” style.
As part of research on censorship events in Indonesia, we added support for running measurements in parallel (https://github.com/ooni/probe-cli/pull/848), which helps with running our new experiment (“DNS Scan”) for scanning open DNS resolvers to detect blocking. ( https://github.com/ooni/probe-cli/pull/847).
Lastly, the work on allowing Dart clients to use the OONI Probe engine (described above) highlighted the need to cleanup OONI Probe CLI build scripts. To this end, we refactored the Makefile to defer most of its work to support scripts (https://github.com/ooni/probe/issues/2217, https://github.com/ooni/probe/issues/2218).
*## OONI Explorer### Domain-centric pages*
Throughout August 2022, we made progress on developing domain-centric pages for OONI Explorer (https://github.com/ooni/explorer/pull/762). We also implemented a new API endpoint for returning domain metadata that we can display in the new domain-centric pages ( https://github.com/ooni/api/pull/300).
*### OONI accounts*
We wrote an internal design document which outlines our plans for implementing authentication to OONI services for OONI accounts. As part of this, we documented the primary use cases (user stories) for these accounts and the implementation details.
Generally, the OONI accounts are being created to support the following use cases: * Presenting website measurements to account users; * Crowdsourcing false positives and fingerprints for blocking detection; * Enabling OONI Probe users to view their own measurements; * Monitoring the success of measurement campaigns; * Creating and monitoring OONI Run links; * Enabling users to request higher testing priority for specific URLs; * URL submission and management.
Based on our design document, we implemented a modal for the login system and worked on relevant code refactoring. We also added relevant backend support for authentication (https://github.com/ooni/api/pull/298).
*### User feedback reporting system*
We created a design document for the (upcoming) OONI Explorer user feedback reporting system, which will enable users to share feedback via OONI Explorer to report false positives, or share other information that is useful for evaluating measurements (such as information about blocked websites).
Based on this, we implemented logic functionality in OONI Explorer as a base for our (upcoming) user feedback reporting system ( https://github.com/ooni/explorer/issues/774). We also implemented an initial prototype in the backend which enables registered users to submit, view, and edit feedback for measurements ( https://github.com/ooni/api/pull/302).
*### Other OONI Explorer improvements*
We continued to work towards adding localization support to OONI Explorer by updating Transifex strings, coordinating with the Localization Lab, and fixing relevant issues (https://github.com/ooni/explorer/pull/778). We also improved upon OONI Explorer by removing links to external pages on measurement pages (https://github.com/ooni/explorer/issues/783), bug fixing (https://github.com/ooni/explorer/pull/777), including failed measurements in the MAT tooltip (https://github.com/ooni/explorer/issues/779), and by upgrading dependencies.
*## OONI backend*
Notably, we switched all our infrastructure over to the new backend host and we completed the migration to the Clickhouse database ( https://github.com/ooni/pipeline/pull/380).
We continued to test and make improvements to the new ASN-based URL prioritization system, we monitored the availability of our test helpers and improved internal dashboards, we fixed measurement bugs, we improved upon the API uploader, and we continued to improve upon JSONL tables ( https://github.com/ooni/api/pull/297/).
We handled an incident that involved an attack on our infrastructure. Specifically, we added IP-address based filtering as a temporary mitigation, then more strict filtering on inputs on API calls ( https://github.com/ooni/api/pull/301), and also extended API caching to more entry points.
We fixed a backend bug (https://github.com/ooni/pipeline/pull/396) that was impacting the correct annotation of measurements ( https://github.com/ooni/explorer/issues/764). We added DNS fingerprints to our database to automatically detect and confirm more cases of website blocking in Iran (https://github.com/ooni/pipeline/pull/390, https://github.com/ooni/backend/issues/543). We also created a diagram which shows the process through which measurements are processed and published (https://github.com/ooni/pipeline/pull/391).
*## Advancing OONI data analysis capabilities*
We have been working on a set of tools (https://github.com/ooni/data) that make it easier for us and third parties to analyze OONI data as part of research and to expand the capabilities of our data processing pipeline.
In particular, we wrote parsers for several common data formats used in OONI measurements and evaluated the performance ( https://github.com/hellais/oonidata-bench) of several approaches to building python data class objects based on these definitions.
One of the benefits of having a well-defined schema for all OONI probe tests is that people interested in analyzing OONI data can benefit from type hints and code completion for accessing all the necessary attributes of a particular test, without needing to refer to OONI specifications. Moreover, it takes care of handling some inconsistencies across various revisions of the data generated by older probes so that the access patterns are as smooth as possible. Lastly, the data is being validated to be conformant with the specification, so it’s easier to identify issues affecting data quality.
Other aspects worked on include adding the ability to break up some common types of tests into distinct observations and writing them to either a Clickhouse database backend or a flat CSV file on disk. We also started outlining a system for taking these observations and generating verdicts related to censorship around the world.
Since we are also using it heavily internally for doing research with OONI data, our goal is to eventually consolidate all the knowledge and patterns we develop into it, so that it can be beneficial to the wider public.
We also created a new GitHub repository for the collection of blocking fingerprints (https://github.com/ooni/blocking-fingerprints), with the goal of enhancing our capabilities to automatically confirm more website blocks.
*## Data requests*
In response to community requests, we shared relevant OONI data (and interpretation) on censorship events. Specifically, we responded to Freedom on the Net’s data requests, providing relevant OONI data and information pertaining to 11 censorship cases. We also coordinated with community members in the Philippines on interpreting relevant OONI data, and we continued to coordinate with members of the #KeepItOn campaign on monitoring blocks.
*## Creating a new OONI Outreach Kit*
To better meet community needs (particularly around OONI community engagement), we started working towards creating new materials and resources that will eventually get published as part of a new OONI Outreach Kit. In August 2022, we worked on several OONI fact sheets and related brochures.
*## OONI blog post on encrypted-DNS censorship published by RIPE Labs*
On 4th August 2022, RIPE Labs published our blog post where we discuss our research on encrypted-DNS censorship. This blog post is available here: https://labs.ripe.net/author/bassosimone/measuring-encrypted-dns-censorship-...
*## Test list updates*
In August 2022, we contributed updates to the following test lists:
* Philippines (https://github.com/citizenlab/test-lists/pull/1055); * Kenya (https://github.com/citizenlab/test-lists/pull/1051); * Russia (https://github.com/citizenlab/test-lists/pull/1032); * Tajikistan (https://github.com/citizenlab/test-lists/pull/1034); * Georgia (https://github.com/citizenlab/test-lists/pull/1038); * Lebanon and Global (https://github.com/citizenlab/test-lists/pull/1059).
*## Started planning the OONI Team Meeting for 2022*
In August 2022, we started logistical preparations for organizing the first (since the start of the pandemic) in-person OONI Team Meeting, which will take place between 2nd-4th November 2022 in Rome, Italy. This meeting will bring all OONI team members and OTF research fellows hosted by OONI together for a 3-day event to plan strategic priorities and roadmaps.
*## Created a new OONI Code of Conduct Incident Response Committee*
In August 2022, we created a new Incident Response Committee with the goal of improving OONI’s Code of Conduct (CoC) and addressing any CoC violations when they occur.
We are excited to share that our new CoC Incident Response Committee comprises of the following members:
1) Elizaveta Yachmeneva (OONI) 2) Amanda Meng (IODA, Georgia Tech) 3) Felicia Anthonio (Access Now) 4) Kelly Koh (Sinar Project) 5) Maria Xynou (OONI) 6) Gurshabad Grover (OTF Information Controls Fellow) 7) Federico Ceratto (OONI)
Throughout August 2022, we organized several meetings to discuss the goals and responsibilities of the committee, how to improve upon OONI’s Code of Conduct (CoC), and how to handle CoC violation reports.
We also worked towards re-writing OONI’s Code of Conduct based on initial community feedback collected during the June 2022 OONI Community Meeting, as well as feedback shared by Code of Conduct experts in our field.
*## Community activities### OONI training session for election observers*
On 4th August 2022, OONI’s Maria facilitated an OONI training session for election observers. This training introduced participants to OONI Probe and OONI data, particularly with the goal of monitoring emerging censorship events during elections around the world.
*### OONI training session for researchers in Egypt*
On 10th August 2022, OONI’s Maria facilitated a 3-hour OONI training session for researchers in Egypt. This workshop provided an introduction to internet censorship, as well as to OONI tools and data for measuring internet censorship.
*## Userbase*
In August 2022, 60,352,386 OONI Probe measurements were collected from 2,798 ASNs in 173 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/
~ OONI team.