Hello,
Hope you're all doing well.
These days we're primarily supported by the DRL, which is why we do quarterly reporting. That said, we'd like to share monthly updates from the OONI team with the community, hence our May 2021 report shared below. I'll also follow-up to share our June 2021 report.
*# OONI Monthly Report: May 2021*
Throughout May 2021, the OONI team worked on the following sprints:
* Sprint 38 - Umbrella octopus (1-9 May, 2021) * Sprint 39 - Vaquita (10-23 May, 2021) * Sprint 40 - Manatee (24-31 May, 2021)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
*## Released OONI Probe Android 3.0.0 with automated testing, backend proxy support, and new experimental card*
We released OONI Probe Android 3.0.0 which is a major release because it includes 3 exciting new features:
1) Automated regular testing: By enabling automated testing in the settings, OONI Probe Android will test websites automatically every hour!
2) New Experimental card: Users can now run our new STUN reachability test.
3) Backend proxy support: Users can enable Psiphon or use a custom proxy through the new OONI backend proxy settings.
Further information is provided below.
*### Automated testing*
Following the beta release in April 2021 ( https://github.com/ooni/probe/issues/1451), we made a public release of OONI Probe Android 3.0.0 with support for automated testing: https://github.com/ooni/probe-android/releases/tag/v3.0.0
Instead of having to remember to manually run OONI Probe, users can now have OONI Probe test websites automatically every hour! All they have to do is enable the "Run tests automatically" option in the new "Automated testing" setting of the OONI Probe Android app.
Through these settings, users can choose whether they would prefer to have OONI Probe only run tests while they’re connected to WiFi and their phone is charging (to avoid consuming their data and battery). By enabling "automated testing", the OONI Probe Android app will test websites automatically (in the background) every hour, without requiring users to do anything.
We started off with only automated website testing, but we plan to add support for automatically running other tests as well in future releases (similarly to automated testing on OONI Probe Desktop: https://ooni.org/support/ooni-probe-desktop#automated-ooni-probe-testing).
To avoid over-cluttering the "Test Results" section of the app with too many test results (from automated runs), they are only made available on the OONI API (https://api.ooni.io/) and OONI Explorer ( https://explorer.ooni.org/), where they are openly published in near real-time. We strongly recommend enabling automated testing, as this will help ensure regular testing (which is crucial for detecting censorship events).
*### New Experimental card*
As censorship becomes more sophisticated, we need to ship new censorship detection tests faster. To this end, we added a new Experimental card in the OONI Probe Android app, where we plan to release our latest experiments.
This card is meant to provide us agility in adding and removing new experiments on an ongoing basis (particularly in response to emergent censorship events). With OONI Probe Android 3.0.0, users can run our new STUN reachability test: https://github.com/ooni/spec/blob/master/nettests/ts-025-stun-reachability.m...
Our work on adding support for running experimental tests is documented through the following pull request and tickets: https://github.com/ooni/probe-android/pull/429, https://github.com/ooni/probe/issues/1388, https://github.com/ooni/probe/issues/1185, https://github.com/ooni/probe/issues/1473.
*### Backend proxy support*
In some places, such as Iran, communication to the OONI backend services is being interfered with. We have therefore added support for configuring a circumvention proxy for speaking to OONI backend services.
OONI Probe Android 3.0.0 includes new backend proxy settings where users can:
* Enable Psiphon (https://psiphon.ca/)
* Use a custom proxy
This will help with circumventing any accidental or deliberate OONI Probe blocking.
Our work on this is documented through the following tickets: https://github.com/ooni/probe/issues/1469, https://github.com/ooni/probe/issues/1470, https://github.com/ooni/probe/issues/1477.
*## Published a blog post on making OONI Probe Android more resilient*
We published a blog post where we describe what we did to make OONI Probe Android more resilient to deliberate or accidental blocking: https://ooni.org/post/making-ooni-probe-android-more-resilient/
As part of this blog post, we explain why we decided to add backend proxy support to OONI Probe, how and why we changed our Android TLS fingerprint, and we document future improvements.
*## Improvements to the code quality of OONI Probe Android*
In collaboration with Bloco, we improved the way in which the OONI Probe Android app is being tested by adding UI and domain tests, as documented through the following pull requests: https://github.com/ooni/probe-android/pull/431 and https://github.com/ooni/probe-android/pull/437.
We also refactored the app to make use of the dependency injection pattern: https://github.com/ooni/probe-android/pull/430
*## Added RTL support to OONI Probe Desktop*
We continued to work on adding support for Right-To-Left (RTL) languages (such as Arabic and Farsi) in OONI Probe Desktop. This allows us to integrate translations (provided by the Localization Lab community) in the OONI Probe desktop app. This work is documented through the following ticket: https://github.com/ooni/probe/issues/1087
*## Prototype of browser-based OONI Probe*
Community members have been requesting a browser-based version of OONI Probe that they can run. To this end, we started fleshing out a browser-based OONI Probe which allows for testing the accessibility of websites by loading the favicon of the target websites. This work is documented through the following ticket: https://github.com/ooni/ooni.org/issues/854
*## OONI Probe for Raspberry Pis*
We added support for creating builds for armv7 and started publishing them on our Debian repository. This allows Raspberry Pi users to install OONI Probe CLI on their Raspberry pi devices. This is documented through the following ticket: https://github.com/ooni/probe/issues/807
*## OONI Probe Command Line Interface (CLI)*
We added support to OONI Probe CLI for exposing the is_uploaded status inside of the result listing. This will allow us to expose an icon at the resulting listing level indicating that a measurement has not been uploaded and eventually support re-uploading them. This is documented through the following ticket: https://github.com/ooni/probe/issues/1457
*## Developing a new OONI Probe test for the Snowflake pluggable transport*
We made progress on developing a new OONI Probe test for measuring the Snowflake pluggable transport. Specifically, we researched the feasibility of developing such a test and we started fleshing it out in probe-cli, as documented through the following tickets: https://github.com/ooni/probe-engine/issues/283 and https://github.com/ooni/probe/issues/1565.
*## Building a web platform for test list updates and URL prioritization*
We made progress on building a web platform that will serve two purposes:
* Enabling community contributions to the Citizen Lab test lists (without a GitHub account); * Enabling the OONI team to make adjustments to URL priorities for testing.
To avoid unsolicited traffic/activity, we will be issuing OONI accounts to community members so that they can login to this web platform. Once logged in, community members will be able to review the Citizen Lab test lists, add new URLs, propose the deletion or edit of existing URLs, and propose different category codes. All of this will be done through a web platform, without requiring community members to use GitHub. Once they have submitted their contributions through this web platform, pull requests will automatically be opened on the Citizen Lab test list GitHub repository (to ensure that test list contributions continue to go through the usual community review process on GitHub).
The same web platform will also be used by the OONI team to adjust the testing prioritization of URLs, based on our URL prioritization policy: https://ooni.org/post/ooni-smart-url-list-system/
In May 2021, we made progress on building the URL priority management interface, the login interface, and the URL submission interface. The URL prioritization management interface is an internal tool which allows members of the OONI team to adjust the priority of URLs or of Citizen Lab category codes for testing (https://github.com/ooni/url-prioritization).
Specifically, we added support to the backend and the frontend of the platform that will enable users to login. This login workflow functions by sending a one-time login token to the registered user’s email address. Once the token is presented to the backend, a cookie is registered in the user’s browser, allowing them to login to the URL submission and prioritization backend. Our work on this is documented through the following ticket: https://github.com/ooni/backend/issues/491
*## OONI Measurement Aggregation Toolkit*
We made progress on OONI’s new Measurement Aggregation Toolkit ( https://explorer.ooni.org/experimental/mat) which allows us to display measurements in aggregate over any arbitrary axis (for example, measurements over time). In particular, we started adding support for creating multi-dimensional charts which allows us to display all measurements for all inputs over time.
Since the charts are very large, we needed to measure the performance of various approaches to plotting them and came up with a solution that allows us to achieve acceptable performance.
This performance boost was achieved through the use of a technique called windowing, where only the visible area of the chart is rendered in the browser at a given time. This work is documented through the following pull request: https://github.com/ooni/explorer/pull/579
*## OONI Explorer*
We added support to OONI Explorer for displaying the reason for the anomaly (DNS, TCP/IP, HTTP-failure, HTTP-diff) in the search listing. This allows users to quickly look at the different types of anomalies that occur over time for a given search query (enabling the confirmation of censorship events and ruling out false positives). This work is tracked through the following GitHub issue: https://github.com/ooni/explorer/issues/438.
We also added support for filtering the WhatsApp IM test results by anomaly (https://github.com/ooni/explorer/pull/589), and we added support for running end-to-end tests of legacy YAML measurements inside of the continuous integration of OONI Explorer ( https://github.com/ooni/explorer/pull/594).
*## OONI backend*
We added backend support for the scoring (i.e. marking them as anomalies or OK) of Signal experiment measurements: https://github.com/ooni/backend/issues/500
Due to the discontinuation of the Bintray service, we migrated all our internal Debian repositories away from the service and switched to using AWS S3 instead. This is documented through the following tickets: https://github.com/ooni/probe/issues/1440 and https://github.com/ooni/backend/issues/504.
*## Academic research paper on internet censorship in Myanmar*
In collaboration with CAIDA, Censored Planet, and other researchers, we worked on an academic research paper which investigates internet censorship in Myanmar following the February 2021 military coup.
As part of this paper, we analyzed relevant OONI measurements from Myanmar and documented the blocking of websites across AS networks in the country since the coup. In collaboration with Censored Planet, we discovered several cases of collateral damage (as a result of IP blocks), which we document in the paper. Researchers from CAIDA also document the nightly internet outages in Myanmar following the coup, providing detailed analysis and new findings.
We completed the writing of this paper by mid-May and submitted it to the FOCI academic journal ( http://conferences.sigcomm.org/sigcomm/2021/workshop-foci.html), where it has been approved for publication.
*## Research report on internet censorship in Azerbaijan*
In collaboration with Azerbaijan Internet Watch, we worked on writing a joint research report that examines internet censorship in Azerbaijan over the last year. The first step though was to analyze OONI measurements collected from Azerbaijan, and so we performed relevant year-long OONI data analysis (https://github.com/ooni/ooni.org/issues/851 ). Based on this data analysis, we produced multiple charts (communicating key findings) and we wrote the technical findings.
*## Improving OONI Run based on community feedback*
Last year, we carried out an extensive OONI Run usability study (involving both a survey and interviews with community members) to collect community feedback on how to improve the OONI Run platform (https://run.ooni.io/). Based on this study, we published a detailed report which includes all of the feedback and findings from this study ( https://ooni.org/post/2020-06-09-ooni-run-usability-study-findings/).
In May 2021, we had several dedicated team meetings during which we went through all of the community feedback, opened relevant GitHub tickets, and documented the next steps (and a relevant roadmap) for improving the OONI Run platform based on community feedback. This has been quite a lengthy process because we have had to evaluate several interdependencies (with other platforms we’re currently building), as well as the feasibility of implementing many of the community feature requests. This is an ongoing process.
*## Community outreach related to the discontinuation of legacy-probe*
Throughout May 2021, we continued relevant community outreach efforts in relation to the upcoming discontinuation of the legacy version of ooniprobe, which relies on v2 onion services (which are being discontinued by October 2021). As legacy ooniprobe still accounts for a relatively large percentage of global OONI measurements, we have been trying to reach these users to encourage them to switch over to the OONI Probe Command Line Interface (https://ooni.org/install/cli).
To this end, we have been sharing relevant information (including how to switch to OONI Probe CLI) with multiple community mailing lists and on our social media channels. We have also been reaching out to our partners and other community members, in an effort to reach legacy probe users.
*## Test list updates*
In May 2021, we contributed the following updates to the Citizen Lab test lists:
* Israel: https://github.com/citizenlab/test-lists/pull/770 (we added Palestinian news media and human rights websites to check if they’re blocked in Israel) * Colombia: https://github.com/citizenlab/test-lists/pull/775 (we added a site which OONI data indicated was blocked, but which was not already included in the test list)
*## Collaboration with Netalitica on test list updates*
Netalitica researchers continued to do excellent work in reviewing and updating more Citizen Lab test lists.
In May 2021, we opened a pull request based on Netalitica updates to the Malaysian test list (https://github.com/citizenlab/test-lists/pull/772) and we reviewed their updates to the Indian test list, based on which we shared feedback for further possible improvements.
We also provided relevant data analysis support to Netalitica ( https://github.com/ooni/ooni.org/issues/887), sharing statistical information on how many new censorship cases were discovered following their updates to the Citizen Lab test lists.
*## Created resources for OPTIMA 2.0 training program*
We previously (in late 2020) supported Internews’ OPTIMA Network Measurement Training program for advocates by creating relevant OONI resources and facilitating OONI training sessions. Information about this training program is available here: https://internews.org/resource/optima-network-measurement-training/
In support of Internews’ OPTIMA 2.0 program (the continuation of the training program), we created (in May 2021) a new video recording and a relevant OONI quiz for training participants. We also created a resource on network measurement tools for internet shutdown advocacy.
*## Organizing the 2 online OONI Partner Training events*
In preparation for the 2 online OONI Partner Training events, we continued to do relevant organizational work.
In May 2021, we sent out personal invitations to more than 100 prospect participants and we prepared a draft outline of the training sessions. To ensure that participant needs are met during the training events, we created (and shared) a pre-training survey ( https://forms.gle/YU4TW62LrLKmCC7N9) to collect participant feedback. The feedback shared through this survey helped inform the creation of the final training agenda.
*## Updated the OONI Partners page*
On our website, we feature all of our partners and their important work through our Partners page: https://ooni.org/partners
In May 2021, we updated this page to add 2 more of our partners (with whom we’ve been collaborating closely over the last few years):
* IODA: https://ooni.org/partners/ioda/ * OutRight Action International: https://ooni.org/partners/outright-action-international/
*## OONI Team Meeting Day: Strategic Planning*
On 5th May 2021, we had an (online) OONI Team Meeting Day focused on strategic planning.
As part of this day-long meeting, our team had the following 4 sessions:
* Session 1: Time for Reflection * Session 2: Communications Strategies * Session 3: What do you want OONI to be? * Session 4: Strategic fundraising
As part of these sessions (particularly sessions 2 and 4), we had the opportunity to share knowledge and skills (with the whole OONI team) gained through the Ford Foundation Communications and Financial Resilience training programs we had participated in over the last months.
These sessions offered our team an opportunity to take a step back and brainstorm on big-picture strategic decisions and goals moving forward. Over the years, we generally have such team meetings on an annual or bi-annual basis.
In preparation for this OONI Team Meeting Day, we prepared relevant slides, resources, and hands-on exercises to ensure that the discussions are as interactive and useful as possible. As an outcome, all of our team members gained new skills and knowledge in relation to communications and fundraising strategies, and based on our brainstorming discussions and exercises, we refined our upcoming milestones.
*## Ford Foundation Financial Innovation & Resilience Training*
As part of our participation in the Ford Foundation’s Financial Innovation and Resilience training program, we attended a relevant workshop on 12th May 2021, and a relevant webinar on 27th May 2021.
We were offered this great opportunity because we are a grantee of the Ford Foundation, who support OONI’s community-related work. The knowledge and skills gained through this training program have helped us rethink and improve OONI’s fundraising strategies.
*## Community use of OONI data### 2020 Annual Report on Digital Rights in Venezuela*
On 16th May 2021, our Venezuelan partners, IPYS Venezuela ( https://ooni.org/partners/ipys-venezuela/), published their 2020 Annual Report on Digital Rights in Venezuela. As part of this report, they made use of OONI data ( https://twitter.com/VEinteligente/status/1412532307248877570).
Their report is available here: https://ipysvenezuela.org/2021/05/16/32196/
*### Report on the blocking of websites in Egypt*
On 24th May 2021, our Egyptian partners, Masaar - Technology and Law Community (https://ooni.org/partners/masaar/), published a research report documenting the techniques and laws around the blocking of websites in Egypt. As part of this report, they made use of OONI data.
Their report is available here: https://masaar.net/en/blocking-websites-in-egypt-techniques-and-laws/
*## Community activities### Stockholm Internet Forum 2021*
On 12th May 2021, OONI’s Maria participated as a speaker on a panel discussion (“#KeepItOn: Governments weaponizing Internet shutdowns”) at the Stockholm Internet Forum 2021 (https://stockholminternetforum.se/).
As part of her participation on the panel, Maria presented OONI’s work and discussed how OONI has supported the advocacy efforts of the #KeepItOn campaign (https://www.accessnow.org/keepiton/) in fighting internet shutdowns around the world.
*### OONI workshops at TechUp event*
On 15th May 2021, OONI’s Maria facilitated 2 online OONI workshops for civil society communities in India and in the Philippines during the TechUp event, organized by the IO Foundation ( https://www.theiofoundation.org/techup/).
As part of these 2 workshops, Maria presented OONI’s tools, methodologies, and dataset, and provided live demos explaining how participants can use OONI Explorer (https://explorer.ooni.org/) to track censorship events worldwide in real-time.
As part of outreach efforts, we created an OONI e-booth on TechUp’s Airmeet platform. This provided OONI resources and information for civil society communities in Asia.
*### Localization Lab Summit 2021*
On 21st and 28th May 2021, we attended the Localization Lab Summit 2021 ( https://www.localizationlab.org/localization-summit-2021-sessions), during which we had the opportunity to meet with the Localization Lab community, learn about emerging digital rights threats around the world, and unmet community needs. We also had the opportunity to connect with funders and the developers of other internet freedom tools, and to learn more about localization needs and challenges.
Given that the translation and overall localization of the OONI Probe apps is supported by the Localization Lab community, this was a very important opportunity for us to learn more about community needs and challenges, which helps inform the development and improvement of our tools.
*### OONI training facilitated by Cameroonian partners*
In May 2021, our Cameroonian partner, COMPSUDEV ( https://ooni.org/partners/compsudev/), collaborated with AfroLeadership ( https://afroleadership.org/) on organizing and facilitating a 2-day training event (“Building Evidence for Internet Rights Advocacy”) for civil society groups in Cameroon.
As part of this training event, they facilitated OONI workshops with the goal of introducing participants to OONI’s tools, methodologies, and dataset. More specifically, they walked through the OONI Probe apps (explaining how participants can use them to measure internet censorship), demoed OONI Explorer (explaining how they can find and use OONI data as part of advocacy), and collected participant feedback. We are very grateful for these community-initiated OONI workshops!
*### OONI feedback collected from civil society in Tanzania*
In May 2021, Tanzania’s Zaina Foundation carried out a Feedback Collection project for several open source tools, including OONI. As part of this, they asked community members in Tanzania to share their feedback on OONI Probe and OONI Explorer.
Following this project, Zaina Foundation shared with us the feedback they collected from human rights defenders in Tanzania for the improvement of OONI Probe and OONI Explorer.
*### OONI guide by Fundación Karisma*
Starting on 26th May 2021, OONI measurements collected from Colombia started to show:
* Blocking of ghostbin.co: https://explorer.ooni.org/search?until=2021-06-02&since=2021-05-03&d... * Blocking of archive.org: https://explorer.ooni.org/search?until=2021-06-02&since=2021-05-03&d...
To encourage further testing, our Colombian partner, Fundación Karisma ( https://ooni.org/partners/karisma-foundation/), wrote and published a short OONI guide (in Spanish): https://web.karisma.org.co/micro-guia-para-el-uso-de-ooni/. They also promoted their OONI guide (and OONI Probe testing) on their social media channels: https://twitter.com/Karisma/status/1398372339918721030
*### OONI Community Meeting*
On 25th May 2021, we hosted the monthly OONI Community Meeting on our Slack channel (https://slack.ooni.org/), during which we discussed the following topics:
1. Updates regarding the new web platform for suggesting updates/edits to the Citizen Lab test lists. How shall we deal with dead URLs that still remain blocked in a given country?
2. Pluggable transport implementation in any networking system, and the current state of obfs4 support in OONI Probe.
*## Userbase*
In May 2021, 10,261,241 OONI Probe measurements were collected from 5,132 AS networks in 191 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/
~ OONI team.
tor-project@lists.torproject.org