[tor-project] OONI Monthly Report: February 2020
maria at openobservatory.org
Mon Mar 9 19:11:03 UTC 2020
Below is OONI's progress report from February 2020.
## OONI Probe support for running circumvention tool tests
During February 2020, we added support to OONI Probe for testing the
availability of the Tor, obfs4proxy, and Psiphon circumvention tools.
These tests have been integrated into the OONI Probe engine
(https://github.com/ooni/probe-engine) and are available to run via the
OONI Probe desktop app.
Below are the specifications for each test:
* Tor test: https://github.com/ooni/spec/blob/master/nettests/ts-023-tor.md
* Psiphon test:
As part of our work on integrating these tests into OONI Probe, we:
* Implemented the Tor test UI in the OONI Probe desktop app:
* Improved the UX of the Tor test results in the OONI Probe desktop app:
* Created an animation that appears when OONI Probe users run
circumvention tool tests: https://github.com/ooni/probe/issues/1029
* Configured the OONI Probe CLI (which the desktop app relies on) to
submit measurements, even if an experiment fails:
## Analysis of circumvention tool test data and presentation in OONI
We added support for analyzing the new OONI Probe circumvention tool
test results, and we integrated them into OONI Explorer and the OONI API.
In February 2020, we worked on presenting the measurements in OONI
Explorer for the following new OONI Probe tests:
* Psiphon: https://github.com/ooni/explorer/issues/332
* Tor: https://github.com/ooni/explorer/issues/333
We also investigated how the batch pipeline deals with the new OONI
Probe experiments for Tor and Psiphon:
## Making OONI Probe reporting logic more resilient to censorship
To collect OONI measurements from around the world, OONI Probe clients
need to upload measurements to OONI servers. As some censors may seek to
block connections to OONI servers (in an attempt to prevent the
collection of measurements), we have adopted various strategies over the
years (such as using onion services), but they have presented various
limitations and challenges.
As part of our ongoing efforts to make OONI Probe’s reporting logic more
resilient to censorship, we researched whether requesting OONI Probe
users in censored environments to submit their measurements via instant
messaging app messages would be a feasible and desirable option. This is
documented through the following ticket:
We also explored the possibility of enabling OONI Probe clients to
submit their measurements via email, as documented through the following
## OONI Probe orchestration logic for circumvention tool testing
As part of our work on developing OONI Probe orchestration logic that is
specific to circumvention tool testing, we updated the canonical
location of Tor test targets:
These Tor testing targets are then given out to OONI Probe clients by
using the OONI Probe orchestration system.
## OONI Probe desktop app
We released the 4th release candidate of the OONI Probe desktop app
Throughout February 2020, we researched how to best run OONI Probe
automatically on desktop (https://github.com/ooni/probe/issues/1011),
and we made a series of improvements to the OONI Probe desktop app.
We also designed mock-ups for the UI elements of the OONI Probe desktop app:
## OONI Probe Command Line Interface (CLI)
In February 2020, we made 2 OONI Probe CLI releases:
* OONI Probe CLI v3.0.0-rc.9:
* OONI Probe CLI v3.0.0-rc.10:
These releases enable users to stop running experiments, specify test
list categories, and they address a critical bug in how measurements are
stored on disk.
Throughout the month, we made a series of improvements to the OONI Probe
## OONI Probe mobile app
Throughout February 2020, we made a series of improvements to the OONI
Probe mobile app.
These improvements were in preparation for the release of OONI Probe
2.3.0, including a fix to the OONI Probe WhatsApp test which was
presenting false positives following recent changes to WhatsApp’s
## Research on push notifications
We researched how to best acquire push notification permissions from
OONI Probe mobile app users (see https://github.com/ooni/probe/issues/813).
As part of this, we wrote a document which outlines the problem that we
are trying to solve, the core questions that we need to answer, and it
proposes a potential implementation solution. This document is available
We also deployed a test backend for countly
and did some testing to check how well it could fit our use case as a
push notification backend.
## OONI Probe measurement engine
We implemented padding for DNS over HTTPS (DoH) and DNS over TLS (DoT)
in the OONI Probe measurement engine
(https://github.com/ooni/probe-engine/issues/285). We also explored test
helper options for the OONI Probe SNI blocking experiment
OONI Probe has relied on the MaxMind database (https://www.maxmind.com/)
for IP geolocation, but they recently changed their license. As part of
our efforts to manage this change
(https://github.com/ooni/probe-engine/issues/269), we explored the
possibility of generating our own ASN database that would be compatible
with the MaxMindDB (https://github.com/ooni/probe-engine/issues/336).
We made a series of improvements to the OONI Probe engine throughout the
month, as documented through the following pull requests.
## Making the OONI Probe apps rely entirely on the OONI Probe golang engine
We made considerable progress on making the OONI Probe mobile and
desktop apps rely entirely on the golang based probe-engine.
To this end, we:
* Investigated the use of certifi/gocertifi to support shipping SSL
certificates using Go: https://github.com/ooni/probe-engine/issues/296
* Replaced most of the HTTP code in measurement-kit with the golang
based ooni/netx: https://github.com/ooni/probe-engine/issues/302
* Implemented a golang-based replacement for the measurement-kit method
called MKAsyncTask: https://github.com/ooni/probe-engine/issues/339
* Ensured we never emit empty fields in the golang code:
## OONI Explorer
Throughout February 2020, we made a series of improvements to OONI
Explorer based on GitHub tickets opened by community members and our
team over the last months.
Our work on improving OONI Explorer is available through the following
We also started implementing a strategy for performing end-to-end
testing of OONI Explorer to ensure that everything works as expected.
See: https://github.com/ooni/explorer/issues/159 &
This is a big step in the direction of ensuring better quality of the
OONI software ecosystem end-to-end.
## OONI API
In order to improve the performance of the OONI API and of the services
that rely on it (such as OONI Explorer), we investigated how to better
log long-running queries (https://github.com/ooni/backend/issues/325).
As part of this, we created an internal dashboard that generates metrics
on query runtime and enables us to better track heavy queries
We also implemented limits to the database queries
(https://github.com/ooni/api/pull/165), and made a series of other
improvements, as documented through the following pull requests.
## OONI data processing pipeline
In order for OONI Explorer to present “confirmed blocked” cases, we need
to add the fingerprints of block pages to the OONI database. Thanks to
the support of community members -- and, especially, Vasilis Ververis
from the Magma project -- many GitHub tickets have been filed providing
block page fingerprints.
Throughout February 2020, we completed the task of adding all the
remaining fingerprints to the OONI database:
https://github.com/ooni/backend/issues/281. This improves the analytics
needed to extract per-website metrics.
We implemented a watchdog system that monitors the amount of rows
generated in our fast-path pipeline and compares them with the
measurement table. This enables us to see if we’re missing any
measurements and to otherwise better understand issues related to OONI
data processing. We also implemented an internal dashboard that
generates charts and alarms based on deltas
(https://github.com/ooni/backend/issues/219). All of this work is
available through the following pull request:
We also made several other improvements to the OONI data processing
pipeline throughout February 2020, as documented through the following
## OONI infrastructure
We made several important improvements to the OONI server infrastructure.
* Consolidated the pattern used to deploy web-connectivity test helpers:
* Upgraded our monitoring software to the latest version, prometheus
* Dealt with several incidents affecting our infrastructure and
discussed how to better document them going forward:
* Added support for logging long-running queries and plotted them in an
internal dashboard: https://github.com/ooni/backend/issues/325
## OONI website
In collaboration with community members, we:
* Improved the SEO of the OONI website:
* Replaced the CSS framework of the OONI website:
* Fixed the HTML <title> tag for certain pages on our website, improving
how our website appears in search engines:
## Google Summer of Code
We submitted the following OONI project ideas for the Google Summer of
* OONI Probe network experiments:
* OONI Explorer Advanced Search:
* Privacy aware geo-lookup:
## Report on censorship in Togo
We published a research report on the blocking of instant messaging apps
in Togo amid the 2020 presidential election.
Our report is available here:
Thanks to measurements contributed by OONI Probe users in Togo, we found
that access to the WhatsApp mobile app, Telegram Web, and Facebook
Messenger were blocked on 2 networks (Togo Telecom & Atlantique Telecom)
in Togo on election day. They were, however, accessible on the Canalbox
network, suggesting that internet censorship varies across ISPs in Togo.
VPNCompare published a blog post discussing our report on Togo:
## Supported OTF Fellow
Over the last year, we have hosted an OTF Information Controls Fellow
who has been researching internet censorship in certain states of India.
Throughout February 2020, we worked closely with the fellow to analyze
relevant OONI measurements and document the findings.
## Collaboration with Netalitica
Netalitica researchers continue to do an excellent job updating the
Citizen Lab test lists!
In February 2020, we reviewed Netalitica updates to the Citizen Lab test
lists for Kazakhstan
(https://github.com/citizenlab/test-lists/pull/585), Turkey, and Ukraine.
In response to a data analysis request from Netalitica, we extracted and
analyzed Web Connectivity measurements from 19 countries and generated
charts, all of which are available through the following ticket:
## Other test list updates
We opened pull requests with several other test list updates as well.
More specifically, we:
* Updated the global and Chinese test lists to include
* Updated the global test list to add a circumvention tool site:
* Updated the Iranian test list:
## Reviewed RightsCon session proposals
This year, OONI’s Maria is a Program Committee Co-Chair for the Network
Connectivity and Internet Shutdowns track at RightsCon 2020.
As part of this capacity, Maria reviewed 27 RightsCon session proposals
submitted under the Network Connectivity and Internet Shutdowns track
(excluding session proposals submitted by OONI).
## Organization of IFF Internet Measurement Village
Throughout February 2020, we continued to coordinate efforts around the
organization of the Internet Measurement Village at the Internet Freedom
Our coordination efforts involved reaching out to community members to
encourage them to submit session proposals, discussing session ideas
with community members, collecting session ideas in order to create an
agenda, and participating in bi-weekly calls with other village organizers.
Unfortunately though the IFF 2020 has been cancelled as a result of the
escalating impact of COVID-19 (coronavirus) in Europe and around the world.
Nonetheless, we hope to co-organize the Internet Measurement Village at
the IFF next year. Meanwhile, we would like to explore opportunities to
facilitate measurement sessions remotely online (beyond the online
monthly OONI community meetings).
## Community use of OONI data
### Censored Planet & Citizen Lab research paper
Censored Planet and the Citizen Lab collaborated on a research paper,
titled “Measuring the Deployment of Network Censorship Filters at Global
Scale”, which makes use of OONI data.
Their research paper (available here:
https://censoredplanet.org/assets/filtermap.pdf) will appear in Network
and Distributed System Security Symposium (NDSS), 2020.
Censored Planet also published a report summarizing the findings:
As part of their research, Censored Planet and the Citizen Lab used OONI
data (along with Censored Planet data) to gather blockpages from filter
deployments around the world.
### Blocking of Tutanota in Russia
In mid-February 2020, OONI measurements collected from Russia revealed
the blocking of email provider tutanota.com.
Once Tutanota was notified of the blocking of their site in Russia, they
independently coordinated OONI Probe testing on Twitter to gather more
Roskomsvoboda followed up to share the relevant blocklist
(https://reestr.rublacklist.net/search/?q=tutanota) and they published a
post on the blocking of tutanota.com in Russia:
A news article, citing OONI Explorer measurements on the blocking of
tutanota.com in Russia, was also published:
## Community activities
OONI’s Federico traveled to Brussels to attend FOSDEM
(https://fosdem.org/2020/) on 1st & 2nd February 2020.
### Community meeting
We facilitated the monthly OONI Community Meeting on 25th February 2020
on our Slack channel (https://slack.ooni.org/), during which we discussed:
1. Measuring throttling
2. Measuring the blocking of IP addresses
3. Mining OONI data to examine the blocking of the Tor network
In February 2020, 7,669,694 OONI Probe measurements were collected from
5,511 networks in 214 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on monthly coverage worldwide):
~ The OONI team.
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 819 bytes
Desc: OpenPGP digital signature
More information about the tor-project