Hello,
Below is OONI's progress report from February 2020.
## OONI Probe support for running circumvention tool tests
During February 2020, we added support to OONI Probe for testing the availability of the Tor, obfs4proxy, and Psiphon circumvention tools.
These tests have been integrated into the OONI Probe engine (https://github.com/ooni/probe-engine) and are available to run via the OONI Probe desktop app.
Below are the specifications for each test:
* Tor test: https://github.com/ooni/spec/blob/master/nettests/ts-023-tor.md * Psiphon test: https://github.com/ooni/spec/blob/master/nettests/ts-015-psiphon.md
As part of our work on integrating these tests into OONI Probe, we:
* Implemented the Tor test UI in the OONI Probe desktop app: https://github.com/ooni/probe/issues/975 * Improved the UX of the Tor test results in the OONI Probe desktop app: https://github.com/ooni/probe/issues/990 * Created an animation that appears when OONI Probe users run circumvention tool tests: https://github.com/ooni/probe/issues/1029 * Configured the OONI Probe CLI (which the desktop app relies on) to submit measurements, even if an experiment fails: https://github.com/ooni/probe-engine/issues/316
## Analysis of circumvention tool test data and presentation in OONI Explorer
We added support for analyzing the new OONI Probe circumvention tool test results, and we integrated them into OONI Explorer and the OONI API.
In February 2020, we worked on presenting the measurements in OONI Explorer for the following new OONI Probe tests:
* Psiphon: https://github.com/ooni/explorer/issues/332 * Tor: https://github.com/ooni/explorer/issues/333
We also investigated how the batch pipeline deals with the new OONI Probe experiments for Tor and Psiphon: https://github.com/ooni/backend/issues/301
## Making OONI Probe reporting logic more resilient to censorship
To collect OONI measurements from around the world, OONI Probe clients need to upload measurements to OONI servers. As some censors may seek to block connections to OONI servers (in an attempt to prevent the collection of measurements), we have adopted various strategies over the years (such as using onion services), but they have presented various limitations and challenges.
As part of our ongoing efforts to make OONI Probe’s reporting logic more resilient to censorship, we researched whether requesting OONI Probe users in censored environments to submit their measurements via instant messaging app messages would be a feasible and desirable option. This is documented through the following ticket: https://github.com/ooni/probe/issues/976
We also explored the possibility of enabling OONI Probe clients to submit their measurements via email, as documented through the following ticket: https://github.com/ooni/probe/issues/992
## OONI Probe orchestration logic for circumvention tool testing
As part of our work on developing OONI Probe orchestration logic that is specific to circumvention tool testing, we updated the canonical location of Tor test targets: https://github.com/ooni/backend/issues/299
These Tor testing targets are then given out to OONI Probe clients by using the OONI Probe orchestration system.
## OONI Probe desktop app
We released the 4th release candidate of the OONI Probe desktop app 3.0.0: https://github.com/ooni/probe-desktop/releases/tag/v3.0.0-rc.4
Throughout February 2020, we researched how to best run OONI Probe automatically on desktop (https://github.com/ooni/probe/issues/1011), and we made a series of improvements to the OONI Probe desktop app.
https://github.com/ooni/probe-desktop/pull/97 https://github.com/ooni/probe-desktop/pull/98 https://github.com/ooni/probe-desktop/pull/99 https://github.com/ooni/probe-desktop/pull/100 https://github.com/ooni/probe-desktop/pull/101 https://github.com/ooni/probe-desktop/pull/102 https://github.com/ooni/probe-desktop/pull/103 https://github.com/ooni/probe-desktop/pull/104 https://github.com/ooni/probe-desktop/pull/105 https://github.com/ooni/probe-desktop/pull/106 https://github.com/ooni/probe-desktop/pull/107
We also designed mock-ups for the UI elements of the OONI Probe desktop app: https://github.com/ooni/probe/issues/1009#issuecomment-590459771 https://github.com/ooni/probe/issues/966#issuecomment-585128461 https://github.com/ooni/probe/issues/1006#issuecomment-590354947
## OONI Probe Command Line Interface (CLI)
In February 2020, we made 2 OONI Probe CLI releases:
* OONI Probe CLI v3.0.0-rc.9: https://github.com/ooni/probe-cli/releases/tag/v3.0.0-rc.9 * OONI Probe CLI v3.0.0-rc.10: https://github.com/ooni/probe-cli/releases/tag/v3.0.0-rc.10
These releases enable users to stop running experiments, specify test list categories, and they address a critical bug in how measurements are stored on disk.
Throughout the month, we made a series of improvements to the OONI Probe CLI:
https://github.com/ooni/probe-cli/pull/104 https://github.com/ooni/probe-cli/pull/106 https://github.com/ooni/probe-cli/pull/107 https://github.com/ooni/probe-cli/pull/108 https://github.com/ooni/probe-cli/pull/109 https://github.com/ooni/probe-cli/pull/110 https://github.com/ooni/probe-cli/pull/111 https://github.com/ooni/probe-cli/pull/112 https://github.com/ooni/probe-cli/pull/113 https://github.com/ooni/probe-cli/pull/114 https://github.com/ooni/probe-cli/pull/115 https://github.com/ooni/probe-cli/pull/116
## OONI Probe mobile app
Throughout February 2020, we made a series of improvements to the OONI Probe mobile app.
On Android:
https://github.com/ooni/probe-android/pull/289 https://github.com/ooni/probe-android/pull/290 https://github.com/ooni/probe-android/pull/291 https://github.com/ooni/probe-android/pull/292 https://github.com/ooni/probe-android/pull/293 https://github.com/ooni/probe-android/pull/295 https://github.com/ooni/probe-android/pull/296 https://github.com/ooni/probe-android/pull/297
On iOS:
https://github.com/ooni/probe-ios/pull/336 https://github.com/ooni/probe-ios/pull/337 https://github.com/ooni/probe-ios/pull/338 https://github.com/ooni/probe-ios/pull/339 https://github.com/ooni/probe-ios/pull/340 https://github.com/ooni/probe-ios/pull/341
These improvements were in preparation for the release of OONI Probe 2.3.0, including a fix to the OONI Probe WhatsApp test which was presenting false positives following recent changes to WhatsApp’s infrastructure.
## Research on push notifications
We researched how to best acquire push notification permissions from OONI Probe mobile app users (see https://github.com/ooni/probe/issues/813).
As part of this, we wrote a document which outlines the problem that we are trying to solve, the core questions that we need to answer, and it proposes a potential implementation solution. This document is available here: https://docs.google.com/document/d/1x1jMjMq4-e8Xb4USegBSUV6V_cpNUWb2GwfBE5jh...
We also deployed a test backend for countly (https://github.com/ooni/sysadmin/commit/d69f9975a16fbb7c4b7a29ec0a0be28c135e...) and did some testing to check how well it could fit our use case as a push notification backend.
## OONI Probe measurement engine
We implemented padding for DNS over HTTPS (DoH) and DNS over TLS (DoT) in the OONI Probe measurement engine (https://github.com/ooni/probe-engine/issues/285). We also explored test helper options for the OONI Probe SNI blocking experiment (https://github.com/ooni/probe-engine/issues/303).
OONI Probe has relied on the MaxMind database (https://www.maxmind.com/) for IP geolocation, but they recently changed their license. As part of our efforts to manage this change (https://github.com/ooni/probe-engine/issues/269), we explored the possibility of generating our own ASN database that would be compatible with the MaxMindDB (https://github.com/ooni/probe-engine/issues/336).
We made a series of improvements to the OONI Probe engine throughout the month, as documented through the following pull requests.
https://github.com/ooni/probe-engine/pull/313 https://github.com/ooni/probe-engine/pull/318 https://github.com/ooni/probe-engine/pull/319 https://github.com/ooni/probe-engine/pull/322 https://github.com/ooni/probe-engine/pull/323 https://github.com/ooni/probe-engine/pull/324 https://github.com/ooni/probe-engine/pull/329 https://github.com/ooni/probe-engine/pull/333 https://github.com/ooni/probe-engine/pull/342 https://github.com/ooni/probe-engine/pull/343 https://github.com/ooni/probe-engine/pull/344 https://github.com/ooni/probe-engine/pull/345 https://github.com/ooni/probe-engine/pull/346 https://github.com/ooni/probe-engine/pull/347 https://github.com/ooni/probe-engine/pull/354 https://github.com/ooni/probe-engine/pull/360 https://github.com/ooni/probe-engine/pull/361 https://github.com/ooni/probe-engine/pull/364 https://github.com/ooni/probe-engine/pull/368 https://github.com/ooni/probe-engine/pull/369 https://github.com/ooni/probe-engine/pull/370 https://github.com/ooni/probe-engine/pull/373 https://github.com/ooni/probe-engine/pull/374 https://github.com/ooni/probe-engine/pull/375
## Making the OONI Probe apps rely entirely on the OONI Probe golang engine
We made considerable progress on making the OONI Probe mobile and desktop apps rely entirely on the golang based probe-engine.
To this end, we:
* Investigated the use of certifi/gocertifi to support shipping SSL certificates using Go: https://github.com/ooni/probe-engine/issues/296 * Replaced most of the HTTP code in measurement-kit with the golang based ooni/netx: https://github.com/ooni/probe-engine/issues/302 * Implemented a golang-based replacement for the measurement-kit method called MKAsyncTask: https://github.com/ooni/probe-engine/issues/339 * Ensured we never emit empty fields in the golang code: https://github.com/ooni/probe-engine/issues/348
## OONI Explorer
Throughout February 2020, we made a series of improvements to OONI Explorer based on GitHub tickets opened by community members and our team over the last months.
Our work on improving OONI Explorer is available through the following pull requests. https://github.com/ooni/explorer/pull/387 https://github.com/ooni/explorer/pull/393 https://github.com/ooni/explorer/pull/394 https://github.com/ooni/explorer/pull/395 https://github.com/ooni/explorer/pull/397 https://github.com/ooni/explorer/pull/398 https://github.com/ooni/explorer/pull/401 https://github.com/ooni/explorer/pull/404 https://github.com/ooni/explorer/pull/405 https://github.com/ooni/explorer/pull/408 https://github.com/ooni/explorer/pull/409 https://github.com/ooni/explorer/pull/411 https://github.com/ooni/explorer/pull/412 https://github.com/ooni/explorer/pull/415
We also started implementing a strategy for performing end-to-end testing of OONI Explorer to ensure that everything works as expected. See: https://github.com/ooni/explorer/issues/159 & https://github.com/ooni/explorer/pull/417.
This is a big step in the direction of ensuring better quality of the OONI software ecosystem end-to-end.
## OONI API
In order to improve the performance of the OONI API and of the services that rely on it (such as OONI Explorer), we investigated how to better log long-running queries (https://github.com/ooni/backend/issues/325). As part of this, we created an internal dashboard that generates metrics on query runtime and enables us to better track heavy queries (https://github.com/ooni/api/pull/162).
We also implemented limits to the database queries (https://github.com/ooni/api/pull/165), and made a series of other improvements, as documented through the following pull requests.
https://github.com/ooni/api/pull/163 https://github.com/ooni/api/pull/164 https://github.com/ooni/api/pull/166 https://github.com/ooni/api/pull/167 https://github.com/ooni/api/pull/168 https://github.com/ooni/api/pull/169 https://github.com/ooni/api/pull/170 https://github.com/ooni/api/pull/171 https://github.com/ooni/api/pull/172
## OONI data processing pipeline
In order for OONI Explorer to present “confirmed blocked” cases, we need to add the fingerprints of block pages to the OONI database. Thanks to the support of community members -- and, especially, Vasilis Ververis from the Magma project -- many GitHub tickets have been filed providing block page fingerprints.
Throughout February 2020, we completed the task of adding all the remaining fingerprints to the OONI database: https://github.com/ooni/backend/issues/281. This improves the analytics needed to extract per-website metrics.
We implemented a watchdog system that monitors the amount of rows generated in our fast-path pipeline and compares them with the measurement table. This enables us to see if we’re missing any measurements and to otherwise better understand issues related to OONI data processing. We also implemented an internal dashboard that generates charts and alarms based on deltas (https://github.com/ooni/backend/issues/219). All of this work is available through the following pull request: https://github.com/ooni/pipeline/pull/292
We also made several other improvements to the OONI data processing pipeline throughout February 2020, as documented through the following pull requests.
https://github.com/ooni/pipeline/pull/290 https://github.com/ooni/pipeline/pull/291 https://github.com/ooni/pipeline/pull/292 https://github.com/ooni/pipeline/pull/295 https://github.com/ooni/pipeline/pull/296 https://github.com/ooni/pipeline/pull/297 https://github.com/ooni/pipeline/pull/299
## OONI infrastructure
We made several important improvements to the OONI server infrastructure.
Specifically, we:
* Consolidated the pattern used to deploy web-connectivity test helpers: https://github.com/ooni/backend/issues/300 * Upgraded our monitoring software to the latest version, prometheus 2.15.2: https://github.com/ooni/backend/issues/302 * Dealt with several incidents affecting our infrastructure and discussed how to better document them going forward: https://github.com/ooni/backend/issues/343 * Added support for logging long-running queries and plotted them in an internal dashboard: https://github.com/ooni/backend/issues/325
## OONI website
In collaboration with community members, we:
* Improved the SEO of the OONI website: https://github.com/ooni/ooni.org/issues/314 * Replaced the CSS framework of the OONI website: https://github.com/ooni/ooni.org/issues/343 * Fixed the HTML <title> tag for certain pages on our website, improving how our website appears in search engines: https://github.com/ooni/ooni.org/issues/296
## Google Summer of Code
We submitted the following OONI project ideas for the Google Summer of Code (GSoC):
* OONI Probe network experiments: https://community.torproject.org/gsoc/ooni-probe-experiments/ * OONI Explorer Advanced Search: https://community.torproject.org/gsoc/ooni-explorer-advanced-search/ * Privacy aware geo-lookup: https://community.torproject.org/gsoc/privacy-aware-geo-lookup/
## Report on censorship in Togo
We published a research report on the blocking of instant messaging apps in Togo amid the 2020 presidential election.
Our report is available here: https://ooni.org/post/2020-togo-blocks-instant-messaging-apps/
Thanks to measurements contributed by OONI Probe users in Togo, we found that access to the WhatsApp mobile app, Telegram Web, and Facebook Messenger were blocked on 2 networks (Togo Telecom & Atlantique Telecom) in Togo on election day. They were, however, accessible on the Canalbox network, suggesting that internet censorship varies across ISPs in Togo.
VPNCompare published a blog post discussing our report on Togo: https://www.vpncompare.co.uk/internet-censorship-togo/
## Supported OTF Fellow
Over the last year, we have hosted an OTF Information Controls Fellow who has been researching internet censorship in certain states of India.
Throughout February 2020, we worked closely with the fellow to analyze relevant OONI measurements and document the findings.
## Collaboration with Netalitica
Netalitica researchers continue to do an excellent job updating the Citizen Lab test lists!
In February 2020, we reviewed Netalitica updates to the Citizen Lab test lists for Kazakhstan (https://github.com/citizenlab/test-lists/pull/585), Turkey, and Ukraine.
In response to a data analysis request from Netalitica, we extracted and analyzed Web Connectivity measurements from 19 countries and generated charts, all of which are available through the following ticket: https://github.com/ooni/ooni.org/issues/333.
## Other test list updates
We opened pull requests with several other test list updates as well. More specifically, we:
* Updated the global and Chinese test lists to include coronavirus-related websites: https://github.com/citizenlab/test-lists/pull/571 & https://github.com/citizenlab/test-lists/pull/572 * Updated the global test list to add a circumvention tool site: https://github.com/citizenlab/test-lists/pull/569 * Updated the Iranian test list: https://github.com/citizenlab/test-lists/pull/575 & https://github.com/citizenlab/test-lists/pull/574
## Reviewed RightsCon session proposals
This year, OONI’s Maria is a Program Committee Co-Chair for the Network Connectivity and Internet Shutdowns track at RightsCon 2020.
As part of this capacity, Maria reviewed 27 RightsCon session proposals submitted under the Network Connectivity and Internet Shutdowns track (excluding session proposals submitted by OONI).
## Organization of IFF Internet Measurement Village
Throughout February 2020, we continued to coordinate efforts around the organization of the Internet Measurement Village at the Internet Freedom Festival (IFF).
Our coordination efforts involved reaching out to community members to encourage them to submit session proposals, discussing session ideas with community members, collecting session ideas in order to create an agenda, and participating in bi-weekly calls with other village organizers.
Unfortunately though the IFF 2020 has been cancelled as a result of the escalating impact of COVID-19 (coronavirus) in Europe and around the world.
Nonetheless, we hope to co-organize the Internet Measurement Village at the IFF next year. Meanwhile, we would like to explore opportunities to facilitate measurement sessions remotely online (beyond the online monthly OONI community meetings).
## Community use of OONI data
### Censored Planet & Citizen Lab research paper
Censored Planet and the Citizen Lab collaborated on a research paper, titled “Measuring the Deployment of Network Censorship Filters at Global Scale”, which makes use of OONI data.
Their research paper (available here: https://censoredplanet.org/assets/filtermap.pdf) will appear in Network and Distributed System Security Symposium (NDSS), 2020.
Censored Planet also published a report summarizing the findings: https://censoredplanet.org/filtermap
As part of their research, Censored Planet and the Citizen Lab used OONI data (along with Censored Planet data) to gather blockpages from filter deployments around the world.
### Blocking of Tutanota in Russia
In mid-February 2020, OONI measurements collected from Russia revealed the blocking of email provider tutanota.com.
Once Tutanota was notified of the blocking of their site in Russia, they independently coordinated OONI Probe testing on Twitter to gather more measurements: https://twitter.com/TutanotaTeam/status/1228735217768292352
Roskomsvoboda followed up to share the relevant blocklist (https://reestr.rublacklist.net/search/?q=tutanota) and they published a post on the blocking of tutanota.com in Russia: https://roskomsvoboda.org/55478/
A news article, citing OONI Explorer measurements on the blocking of tutanota.com in Russia, was also published: https://www.clubic.com/pro/legislation-loi-internet/donnees-personnelles/act...
## Community activities
### FOSDEM
OONI’s Federico traveled to Brussels to attend FOSDEM (https://fosdem.org/2020/) on 1st & 2nd February 2020.
### Community meeting
We facilitated the monthly OONI Community Meeting on 25th February 2020 on our Slack channel (https://slack.ooni.org/), during which we discussed:
1. Measuring throttling
2. Measuring the blocking of IP addresses
3. Mining OONI data to examine the blocking of the Tor network
## Userbase
In February 2020, 7,669,694 OONI Probe measurements were collected from 5,511 networks in 214 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on monthly coverage worldwide): http://explorer.ooni.org/
~ The OONI team.