Hello,
Throughout September 2021, the OONI team worked on the following sprints:
* Sprint 47 - Caribbean Reef Octopus (1st - 12th September 2021) * Sprint 48 - Amazon river dolphin (13th - 26th September 2021)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
*## New OONI Probe experiment for website testing*
We completed the implementation of our new Websteps experiment for measuring the blocking of websites ( https://github.com/ooni/probe/issues/1733).
This new experiment is the successor to our Web Connectivity network test ( https://github.com/ooni/spec/blob/master/nettests/ts-017-web-connectivity.md). The research question that this experiment tries to address is that of enumerating all the possible ways by which a specific URL can be blocked. This means that this experiment does not stop when it detects the first type of blocking, but rather drills deeper to discover all the ways by which blocking is implemented.
The main differences compared to Web Connectivity that are worth highlighting include:
* We use a new test helper mechanism for enumerating all the endpoints that need to be tested for a target URL. This includes all the IPs that a particular domain resolves to, as well as the full redirect chain from an uncensored vantage point.
* We also use QUIC to measure websites.
* We expand the taxonomy for classifying the various means by which blocking is implemented.
* We add support for measuring all IP:URL pairs to detect IP blocking that only targets certain addresses.
In writing this new test, we also made significant improvements to our measurement engine. In particular, we implemented several new primitives for performing the various stages of a URL request (that can eventually be reused by other tests as well): https://github.com/ooni/probe-cli/pull/528. In doing so, we also documented (in our tutorial on writing OONI Probe tests, see the following section) how third parties can go about using these functions for writing their own tests.
Once the improvements to the measurement engine were made, we wrote the new implementation of Websteps using these new functions: https://github.com/ooni/probe-cli/pull/530
Currently, it’s only possible to run the Websteps experiment through the miniooni researcher tool (https://github.com/ooni/probe-cli#miniooni).
Our next steps for rolling the experiment out in production include:
* Comparing Websteps to Web Connectivity in terms of bandwidth used; * Comparing Websteps to Web Connectivity in terms of overall test runtime; * Comparing Websteps to Web Connectivity in terms of accuracy at detecting blocking events; * Deploying the new Websteps test helper in production; * Integrating the Websites experiment into the OONI Probe experimental card; * Collecting real world data and analyzing the results for accuracy; * Fully replacing the Web Connectivity test with Websteps and implementing all the needed UI changes in the user facing tools.
*## Published new tutorial for writing OONI Probe tests*
We published a new tutorial which explains how to write OONI Probe tests.
The starting point of the tutorial is here: https://github.com/ooni/probe-cli/tree/master/internal/tutorial
This in-depth tutorial is meant for developers who are interested in contributing new experiments to OONI Probe. The tutorial includes multiple chapters that explain how to practically write a network measurement experiment (using the torsf experiment as an example), how to use the measurex package to write network experiments, and how to use the netxlite networking library. The tutorial includes code based on existing network measurement experiments.
OONI Probe has always been free and open source to encourage independent third-party review of our methodologies. We have always hoped that the openness of OONI Probe would also encourage talented developers to contribute new experiments.
Now, with the help of our new tutorial, community members can contribute their own network measurement tests. If integrated into OONI Probe, their test will be run in around 200 countries and territories every month, with test results published as open data (https://ooni.org/data/).
We have already integrated the RiseupVPN test developed by the LEAP collective, and we have worked with M-Lab on integrating the NDT and DASH tests. We look forward to integrating more tests from the community!
*## Published new OONI Probe CLI guide*
We published a new user guide for the OONI Probe Command Line Interface (CLI): https://ooni.org/support/ooni-probe-cli
Our new user guide provides step-by-step instructions on how to use OONI Probe CLI on macOS, Debian/Ubuntu Linux, and on Raspberry Pis.
In particular, the user guide explains how to install and run OONI Probe CLI, enable automated testing, and view OONI Probe CLI test results. We also provide a reference for every command supported by the OONI Probe CLI.
*## Published a research report on the blocking of Gutenberg site in Italy*
In collaboration with Davide Brunello (independent researcher), we published a new report which examines the blocking of the Gutenberg book publishing website across networks in Italy based on OONI data.
Read the report here: https://ooni.org/post/2021-italy-blocks-gutenberg-book-publishing-website/
Since May 2020, access to the Gutenberg book publishing website has been blocked in Italy (in compliance with a decree of the court of Rome) over copyright violation.
OONI measurement analysis shows that access to www.gutenberg.org is blocked on at least 7 AS networks in Italy. Most ISPs primarily block access by means of DNS tampering.
In blocking www.gutenberg.org, some ISPs in Italy (e.g Fastweb & Tiscali) return an NXDOMAIN, others (e.g. TIM, Telecom Italia, Iliad) return the IP address 127.0.0.1, while Vodafone Italia returns the IP address 83.224.65.170.
Regardless of what is returned in the DNS response, what all ISPs in Italy (at least those included in this study) appear to have in common is the lack of transparency of the fact that access to www.gutenberg.org is intentionally blocked. Internet users just see an error message.
*## Job Opening for OONI Mobile Developer*
We are currently looking for a dedicated mobile developer to lead the development of the OONI Probe mobile app.
In September 2021, we published the job opening, providing information about the job description, qualification requirements, and how to apply for the OONI Mobile Developer position.
Our job opening is available here: https://ooni.org/post/2021-job-opening-ooni-mobile-developer/
Following the publication of this job opening, we worked on outreach efforts to help ensure that mobile developers from diverse backgrounds learn about the opportunity and feel encouraged to apply. To this end, we shared the job opening extensively on all OONI social media channels, and with relevant mailing lists and community spaces. We also posted the job opening on a number of job boards, such as FOSS Jobs ( https://www.fossjobs.net/job/10668/mobile-developer-for-ooni-probe-at-open-o...), the Digital Rights Board (https://www.digitalrights.community/job-board), and Ada’s list, among others.
As applications started to trickle in, we worked on organizing applications and following-up with applicants to communicate next steps.
*## OONI Probe Mobile*
We released OONI Probe Mobile 3.3.0 for Android ( https://github.com/ooni/probe-android/releases/tag/v3.3.0) and iOS ( https://github.com/ooni/probe-ios/releases/tag/v3.3.0).
With this release, we improved the layout of the measurement screens with the goal of improving the accessibility of information.
In particular, we removed the menu settings that used to exist in each test result page (to avoid nesting information too much), and we moved those settings (log, data, OONI Explorer link) to the main measurement screen of each test result. We also included a share button on the top right corner of each test result, enabling users to easily share each measurement directly with their contacts. Thanks to support from Zaina Foundation and the Localization Lab community, the latest version of the OONI Probe mobile app now supports 2 new languages: Swahili and Dutch.
We also implemented a modal to ask users to update their OONI Probe mobile app to the latest version, as documented through the following ticket and pull requests: https://github.com/ooni/probe/issues/1237, https://github.com/ooni/probe-android/pull/449 and https://github.com/ooni/probe-ios/pull/447.
Over the last months, we had the opportunity to work with Bloco on improving the code quality and testing of the OONI Probe Android app. Bloco document their collaboration with us through their blog post: https://www.bloco.io/blog/building-test-suite-ooni-probe-android
On OONI Probe Linux, we increased the frequency of automated testing to 12h.
*## OONI Explorer*
We added support to OONI Explorer for displaying measurements (and filtering by anomaly status) from our new Tor Snowflake experiment: https://github.com/ooni/explorer/commit/d75fa69fbcfe40ed95a240a25e6c023b83c3...
We also added support to the OONI Explorer search listing for filtering and displaying the anomaly status of OONI Probe Signal, Psiphon, and RiseupVPN test results: https://github.com/ooni/explorer/commit/35aac4a23b9394e205894060e2aa5b9a7b3d...
*## OONI Measurement Aggregation Toolkit (MAT)*
We made several important improvements to the OONI Measurement Aggregation Toolkit (MAT). Specifically, we added support for ensuring that there are no holes in the charts so that multi-axis charts are properly aligned. Moreover, we added support for a custom tooltip that contains a link to the raw measurements. This work is documented through the following pull request: https://github.com/ooni/explorer/pull/600
*## Building a web platform for test list updates*
We made some progress on the web platform that we’re building to enable community contributions to the Citizen Lab test lists ( https://github.com/citizenlab/test-lists). As part of these UI improvements, we added a descriptive warning when a user adds a website that is already included in test lists ( https://github.com/ooni/test-lists-ui/issues/3), and we ensured that comments are only enabled when users edit or delete entries (but not when they add new entries).
*## OONI backend*
In September 2021, we worked on the following OONI backend activities:
* URL prioritization in the OONI API: Deployed prioritization fixes, monitored URL prioritization and created relevant internal dashboards, sorted the prioritization rules listing; * Looked into missing measurements in the jsonl table; * Added support for returning the current time as a UTC timestamp via the check-in API, which is useful for detecting when the probe system clock is out of sync; * Fixed URL prioritization further and added more end-to-end testing ( https://github.com/ooni/api/pull/264): Re-checked the test list coverage and created new internal dashboards; * URL prioritization: Debugged missing “NEWS” category prioritization rule, configured prioritization rules for Cuba and Afghanistan, added support for refraining to test “risky” category codes (see: https://github.com/ooni/backend/issues/530); * Continued to work on updating our fast-path pipeline; * Continued to monitor OONI measurement coverage from unattended runs on OONI Probe Mobile and Desktop; * Added support to the data processing pipeline for selectively reprocessing data from a specific country or of a specific experiment; * Added scoring for the Signal test to the fastpath pipeline and reprocessed OONI data; * Continued to work on updating the logic used to populate the counter tables which are used for performing aggregation queries (counters_table_updater); * Worked on deploying an upgraded version of our monitoring infrastructure.
We investigated alternative database solutions that we can potentially use to boost the performance of our services and better meet our data needs. As part of this investigation, we looked into the following database solutions:
* BigQuery: We dumped the fastpath in JSONL and ran benchmark queries against the whole dataset; * ScyllaDB (we had a call with them to discuss our use cases and needs); * PostgreSQL 13 (we setup a testing instance of it to test migrating our data over); * Clickhouse (we setup a testing instance of Clickhouse and ran some experiments to estimate the performance); * DuckDB: We ran benchmark queries and looked into packaging the library.
We also created and plotted database-heavy queries that we can run against the database solutions that we are testing, and we looked into alternative hosting platforms.
*## OONI data analysis*
Leading up to Russia’s 2021 legislative election, we analyzed OONI measurements collected from Russia with the goal of understanding which platforms were already blocked in the country.
Based on our analysis, we produced 2 charts on the blocking of news media and circumvention tool websites in Russia, which we shared on Twitter: https://twitter.com/OpenObservatory/status/1438913975752740866
We also shared OONI measurements which show that ISPs in Russia continued to block access to opposition leader Alexei Navalny’s website: https://twitter.com/OpenObservatory/status/1438912653607555077
*## Research reports*
Throughout September 2021, we worked on 2 research reports (in addition to our report on the blocking of the Gutenberg site in Italy, which we published in September: https://ooni.org/post/2021-italy-blocks-gutenberg-book-publishing-website/) that required relevant OONI data analysis and writing. We completed these research reports by the end of September 2021, but we will publish them in October 2021, once relevant external review is complete.
*## Notable community use of OONI Probe and OONI data### iThena integration of OONI Probe*
In September 2021, OONI Probe was integrated into iThena ( https://root.ithena.net/), a volunteer distributed computing initiative running on the Berkeley Open Infrastructure for Network Computing. This integration is documented by iThena here: https://root.ithena.net/usr/forum_thread.php?id=151
Thanks to iThena’s integration of OONI Probe, OONI measurement coverage has increased significantly! We created some charts based on OONI measurement coverage (illustrating the volume of OONI measurement coverage by platform), which clearly show the spike in coverage by iThena’s OONI Probe integration. We shared these charts on Twitter: https://twitter.com/OpenObservatory/status/1443944726235856901
*### NetDAN assistive tool*
On 30th September 2021, Unwanted Witness (a digital rights organization in Uganda) launched the NetDAN assistive tool which notifies people with visual disabilities of network disruptions in Uganda based on OONI data ( https://twitter.com/OpenObservatory/status/1443560903073538053).
The NetDAN assistive tool can be accessed here: https://app.netdan.org/
Alternatively, users can interact with NetDAN via phone by calling a toll-free number (+256 800 113 269).
NetDAN is an assistive tool that detects and audibly informs the users about the connectivity status of the most commonly used websites and applications in Uganda, using OONI data.
*### OONI data cited in Freedom on the Net 2021 reports*
In September 2021, Freedom House published its annual Freedom on the Net country reports ( https://freedomhouse.org/report/freedom-net/2021/global-drive-control-big-te... ).
OONI data (primarily through OONI research reports) was cited in the following reports:
* Freedom on the Net 2021: Uganda, https://freedomhouse.org/country/uganda/freedom-net/2021 * Freedom on the Net 2021: Italy, https://freedomhouse.org/country/italy/freedom-net/2021 * Freedom on the Net 2021: Myanmar, https://freedomhouse.org/country/myanmar/freedom-net/2021 * Freedom on the Net 2021: Azerbaijan, https://freedomhouse.org/country/azerbaijan/freedom-net/2021 * Freedom on the Net 2021: Iran, https://freedomhouse.org/country/iran/freedom-net/2021 * Freedom on the Net 2021: Malaysia, https://freedomhouse.org/country/malaysia/freedom-net/2021 * Freedom on the Net 2021: Saudi Arabia, https://freedomhouse.org/country/saudi-arabia/freedom-net/2021 * Freedom on the Net 2021: Zambia, https://freedomhouse.org/country/zambia/freedom-net/2021 * Freedom on the Net 2021: Brazil, https://freedomhouse.org/country/brazil/freedom-net/2021 * Freedom on the Net 2021: Colombia, https://freedomhouse.org/country/colombia/freedom-net/2021 * Freedom on the Net 2021: Jordan, https://freedomhouse.org/country/jordan/freedom-net/2021 * Freedom on the Net 2021: Ethiopia, https://freedomhouse.org/country/ethiopia/freedom-net/2021
*## Community activities### Participation in Access Now Twitter Space on internet shutdowns*
On 15th September 2021, OONI’s Maria participated as a speaker in Access Now’s Twitter Space discussion titled “Global Rise of Internet Shutdowns #KeepItOn” (https://twitter.com/accessnow/status/1437893160856870915). As part of her participation in this discussion, Maria discussed how OONI tools and data can be used for measuring “partial internet shutdowns” (involving the blocking of websites and apps), as well as OONI’s role in Access Now’s #KeepItOn campaign fighting internet shutdowns worldwide. *### Participation on Kenya Internet Governance Forum (KIGF) 2021 panel*
On 21st September 2021, OONI’s Maria participated as a speaker in a (pre-event) panel session on “Leveraging the Philanthropic Sector” at the Kenya Internet Governance Forum (KIGF) 2021 (organized by KICTANet). General information about KIGF 2021 is available here: https://kigf.or.ke/kenya-igf-2021/
This panel discussion highlighted the digital rights issues in the Eastern Africa region and Africa, exploring ways through which partners can leverage resources (financial, technical, networks, expertise, etc.) to better respond to threats on digital rights in the region.
As part of her participation, Maria discussed OONI’s partnership program and explained how OONI collaborates with digital rights groups in Africa to measure and respond to internet censorship events. Maria also highlighted the types of resources that such groups can make use of to support their digital rights work in the region.
*### OutRight Action International Webinar on launch of LGBTIQ website censorship report*
On 29th September 2021, OONI’s Maria participated in OutRight Action International’s webinar ( https://twitter.com/OutRightIntl/status/1438251801212596227) to discuss the recent launch of our joint research report “No Access: LGBTIQ Website Censorship in Six Countries” ( https://ooni.org/post/2021-no-access-lgbtiq-website-censorship-six-countries... ).
The webinar panelists included prominent LGBTIQ researchers and advocates from Russia and Indonesia, as well as the founder of My.Kali (an online pan-Arab LGBT magazine), who shared important insights on the impact of LGBTIQ website censorship in their regions. OONI’s Maria and the Citizen Lab’s Irene Poetranto attended to help address any questions related to our research report.
The webinar recording is available here: https://www.youtube.com/watch?v=-W_Py9uXpz0
*### OONI Community Meeting*
On 28th September 2021, we hosted the monthly OONI Community Meeting on our Slack channel (https://slack.ooni.org/), during which we discussed the following topics:
1. OONI updates: (1) Job Opening for OONI Probe Mobile Developer, (2) New User Guide for OONI Probe CLI.
2. VE sin Filtro’s public image for running OONI Probe on Raspberry Pis.
3. Measuring server side-blocking and DNS poisoning.
*## Userbase*
In September 2021, 16,887,184 OONI Probe measurements were collected from 4,765 AS networks in 192 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/
~ OONI team.