[tor-project] OONI Monthly Report: February 2020
Maria Xynou
maria at openobservatory.org
Mon Mar 9 19:11:03 UTC 2020
Hello,
Below is OONI's progress report from February 2020.
## OONI Probe support for running circumvention tool tests
During February 2020, we added support to OONI Probe for testing the
availability of the Tor, obfs4proxy, and Psiphon circumvention tools.
These tests have been integrated into the OONI Probe engine
(https://github.com/ooni/probe-engine) and are available to run via the
OONI Probe desktop app.
Below are the specifications for each test:
* Tor test: https://github.com/ooni/spec/blob/master/nettests/ts-023-tor.md
* Psiphon test:
https://github.com/ooni/spec/blob/master/nettests/ts-015-psiphon.md
As part of our work on integrating these tests into OONI Probe, we:
* Implemented the Tor test UI in the OONI Probe desktop app:
https://github.com/ooni/probe/issues/975
* Improved the UX of the Tor test results in the OONI Probe desktop app:
https://github.com/ooni/probe/issues/990
* Created an animation that appears when OONI Probe users run
circumvention tool tests: https://github.com/ooni/probe/issues/1029
* Configured the OONI Probe CLI (which the desktop app relies on) to
submit measurements, even if an experiment fails:
https://github.com/ooni/probe-engine/issues/316
## Analysis of circumvention tool test data and presentation in OONI
Explorer
We added support for analyzing the new OONI Probe circumvention tool
test results, and we integrated them into OONI Explorer and the OONI API.
In February 2020, we worked on presenting the measurements in OONI
Explorer for the following new OONI Probe tests:
* Psiphon: https://github.com/ooni/explorer/issues/332
* Tor: https://github.com/ooni/explorer/issues/333
We also investigated how the batch pipeline deals with the new OONI
Probe experiments for Tor and Psiphon:
https://github.com/ooni/backend/issues/301
## Making OONI Probe reporting logic more resilient to censorship
To collect OONI measurements from around the world, OONI Probe clients
need to upload measurements to OONI servers. As some censors may seek to
block connections to OONI servers (in an attempt to prevent the
collection of measurements), we have adopted various strategies over the
years (such as using onion services), but they have presented various
limitations and challenges.
As part of our ongoing efforts to make OONI Probe’s reporting logic more
resilient to censorship, we researched whether requesting OONI Probe
users in censored environments to submit their measurements via instant
messaging app messages would be a feasible and desirable option. This is
documented through the following ticket:
https://github.com/ooni/probe/issues/976
We also explored the possibility of enabling OONI Probe clients to
submit their measurements via email, as documented through the following
ticket:
https://github.com/ooni/probe/issues/992
## OONI Probe orchestration logic for circumvention tool testing
As part of our work on developing OONI Probe orchestration logic that is
specific to circumvention tool testing, we updated the canonical
location of Tor test targets:
https://github.com/ooni/backend/issues/299
These Tor testing targets are then given out to OONI Probe clients by
using the OONI Probe orchestration system.
## OONI Probe desktop app
We released the 4th release candidate of the OONI Probe desktop app
3.0.0: https://github.com/ooni/probe-desktop/releases/tag/v3.0.0-rc.4
Throughout February 2020, we researched how to best run OONI Probe
automatically on desktop (https://github.com/ooni/probe/issues/1011),
and we made a series of improvements to the OONI Probe desktop app.
https://github.com/ooni/probe-desktop/pull/97
https://github.com/ooni/probe-desktop/pull/98
https://github.com/ooni/probe-desktop/pull/99
https://github.com/ooni/probe-desktop/pull/100
https://github.com/ooni/probe-desktop/pull/101
https://github.com/ooni/probe-desktop/pull/102
https://github.com/ooni/probe-desktop/pull/103
https://github.com/ooni/probe-desktop/pull/104
https://github.com/ooni/probe-desktop/pull/105
https://github.com/ooni/probe-desktop/pull/106
https://github.com/ooni/probe-desktop/pull/107
We also designed mock-ups for the UI elements of the OONI Probe desktop app:
https://github.com/ooni/probe/issues/1009#issuecomment-590459771
https://github.com/ooni/probe/issues/966#issuecomment-585128461
https://github.com/ooni/probe/issues/1006#issuecomment-590354947
## OONI Probe Command Line Interface (CLI)
In February 2020, we made 2 OONI Probe CLI releases:
* OONI Probe CLI v3.0.0-rc.9:
https://github.com/ooni/probe-cli/releases/tag/v3.0.0-rc.9
* OONI Probe CLI v3.0.0-rc.10:
https://github.com/ooni/probe-cli/releases/tag/v3.0.0-rc.10
These releases enable users to stop running experiments, specify test
list categories, and they address a critical bug in how measurements are
stored on disk.
Throughout the month, we made a series of improvements to the OONI Probe
CLI:
https://github.com/ooni/probe-cli/pull/104
https://github.com/ooni/probe-cli/pull/106
https://github.com/ooni/probe-cli/pull/107
https://github.com/ooni/probe-cli/pull/108
https://github.com/ooni/probe-cli/pull/109
https://github.com/ooni/probe-cli/pull/110
https://github.com/ooni/probe-cli/pull/111
https://github.com/ooni/probe-cli/pull/112
https://github.com/ooni/probe-cli/pull/113
https://github.com/ooni/probe-cli/pull/114
https://github.com/ooni/probe-cli/pull/115
https://github.com/ooni/probe-cli/pull/116
## OONI Probe mobile app
Throughout February 2020, we made a series of improvements to the OONI
Probe mobile app.
On Android:
https://github.com/ooni/probe-android/pull/289
https://github.com/ooni/probe-android/pull/290
https://github.com/ooni/probe-android/pull/291
https://github.com/ooni/probe-android/pull/292
https://github.com/ooni/probe-android/pull/293
https://github.com/ooni/probe-android/pull/295
https://github.com/ooni/probe-android/pull/296
https://github.com/ooni/probe-android/pull/297
On iOS:
https://github.com/ooni/probe-ios/pull/336
https://github.com/ooni/probe-ios/pull/337
https://github.com/ooni/probe-ios/pull/338
https://github.com/ooni/probe-ios/pull/339
https://github.com/ooni/probe-ios/pull/340
https://github.com/ooni/probe-ios/pull/341
These improvements were in preparation for the release of OONI Probe
2.3.0, including a fix to the OONI Probe WhatsApp test which was
presenting false positives following recent changes to WhatsApp’s
infrastructure.
## Research on push notifications
We researched how to best acquire push notification permissions from
OONI Probe mobile app users (see https://github.com/ooni/probe/issues/813).
As part of this, we wrote a document which outlines the problem that we
are trying to solve, the core questions that we need to answer, and it
proposes a potential implementation solution. This document is available
here:
https://docs.google.com/document/d/1x1jMjMq4-e8Xb4USegBSUV6V_cpNUWb2GwfBE5jhAV4/edit#
We also deployed a test backend for countly
(https://github.com/ooni/sysadmin/commit/d69f9975a16fbb7c4b7a29ec0a0be28c135e25d2)
and did some testing to check how well it could fit our use case as a
push notification backend.
## OONI Probe measurement engine
We implemented padding for DNS over HTTPS (DoH) and DNS over TLS (DoT)
in the OONI Probe measurement engine
(https://github.com/ooni/probe-engine/issues/285). We also explored test
helper options for the OONI Probe SNI blocking experiment
(https://github.com/ooni/probe-engine/issues/303).
OONI Probe has relied on the MaxMind database (https://www.maxmind.com/)
for IP geolocation, but they recently changed their license. As part of
our efforts to manage this change
(https://github.com/ooni/probe-engine/issues/269), we explored the
possibility of generating our own ASN database that would be compatible
with the MaxMindDB (https://github.com/ooni/probe-engine/issues/336).
We made a series of improvements to the OONI Probe engine throughout the
month, as documented through the following pull requests.
https://github.com/ooni/probe-engine/pull/313
https://github.com/ooni/probe-engine/pull/318
https://github.com/ooni/probe-engine/pull/319
https://github.com/ooni/probe-engine/pull/322
https://github.com/ooni/probe-engine/pull/323
https://github.com/ooni/probe-engine/pull/324
https://github.com/ooni/probe-engine/pull/329
https://github.com/ooni/probe-engine/pull/333
https://github.com/ooni/probe-engine/pull/342
https://github.com/ooni/probe-engine/pull/343
https://github.com/ooni/probe-engine/pull/344
https://github.com/ooni/probe-engine/pull/345
https://github.com/ooni/probe-engine/pull/346
https://github.com/ooni/probe-engine/pull/347
https://github.com/ooni/probe-engine/pull/354
https://github.com/ooni/probe-engine/pull/360
https://github.com/ooni/probe-engine/pull/361
https://github.com/ooni/probe-engine/pull/364
https://github.com/ooni/probe-engine/pull/368
https://github.com/ooni/probe-engine/pull/369
https://github.com/ooni/probe-engine/pull/370
https://github.com/ooni/probe-engine/pull/373
https://github.com/ooni/probe-engine/pull/374
https://github.com/ooni/probe-engine/pull/375
## Making the OONI Probe apps rely entirely on the OONI Probe golang engine
We made considerable progress on making the OONI Probe mobile and
desktop apps rely entirely on the golang based probe-engine.
To this end, we:
* Investigated the use of certifi/gocertifi to support shipping SSL
certificates using Go: https://github.com/ooni/probe-engine/issues/296
* Replaced most of the HTTP code in measurement-kit with the golang
based ooni/netx: https://github.com/ooni/probe-engine/issues/302
* Implemented a golang-based replacement for the measurement-kit method
called MKAsyncTask: https://github.com/ooni/probe-engine/issues/339
* Ensured we never emit empty fields in the golang code:
https://github.com/ooni/probe-engine/issues/348
## OONI Explorer
Throughout February 2020, we made a series of improvements to OONI
Explorer based on GitHub tickets opened by community members and our
team over the last months.
Our work on improving OONI Explorer is available through the following
pull requests.
https://github.com/ooni/explorer/pull/387
https://github.com/ooni/explorer/pull/393
https://github.com/ooni/explorer/pull/394
https://github.com/ooni/explorer/pull/395
https://github.com/ooni/explorer/pull/397
https://github.com/ooni/explorer/pull/398
https://github.com/ooni/explorer/pull/401
https://github.com/ooni/explorer/pull/404
https://github.com/ooni/explorer/pull/405
https://github.com/ooni/explorer/pull/408
https://github.com/ooni/explorer/pull/409
https://github.com/ooni/explorer/pull/411
https://github.com/ooni/explorer/pull/412
https://github.com/ooni/explorer/pull/415
We also started implementing a strategy for performing end-to-end
testing of OONI Explorer to ensure that everything works as expected.
See: https://github.com/ooni/explorer/issues/159 &
https://github.com/ooni/explorer/pull/417.
This is a big step in the direction of ensuring better quality of the
OONI software ecosystem end-to-end.
## OONI API
In order to improve the performance of the OONI API and of the services
that rely on it (such as OONI Explorer), we investigated how to better
log long-running queries (https://github.com/ooni/backend/issues/325).
As part of this, we created an internal dashboard that generates metrics
on query runtime and enables us to better track heavy queries
(https://github.com/ooni/api/pull/162).
We also implemented limits to the database queries
(https://github.com/ooni/api/pull/165), and made a series of other
improvements, as documented through the following pull requests.
https://github.com/ooni/api/pull/163
https://github.com/ooni/api/pull/164
https://github.com/ooni/api/pull/166
https://github.com/ooni/api/pull/167
https://github.com/ooni/api/pull/168
https://github.com/ooni/api/pull/169
https://github.com/ooni/api/pull/170
https://github.com/ooni/api/pull/171
https://github.com/ooni/api/pull/172
## OONI data processing pipeline
In order for OONI Explorer to present “confirmed blocked” cases, we need
to add the fingerprints of block pages to the OONI database. Thanks to
the support of community members -- and, especially, Vasilis Ververis
from the Magma project -- many GitHub tickets have been filed providing
block page fingerprints.
Throughout February 2020, we completed the task of adding all the
remaining fingerprints to the OONI database:
https://github.com/ooni/backend/issues/281. This improves the analytics
needed to extract per-website metrics.
We implemented a watchdog system that monitors the amount of rows
generated in our fast-path pipeline and compares them with the
measurement table. This enables us to see if we’re missing any
measurements and to otherwise better understand issues related to OONI
data processing. We also implemented an internal dashboard that
generates charts and alarms based on deltas
(https://github.com/ooni/backend/issues/219). All of this work is
available through the following pull request:
https://github.com/ooni/pipeline/pull/292
We also made several other improvements to the OONI data processing
pipeline throughout February 2020, as documented through the following
pull requests.
https://github.com/ooni/pipeline/pull/290
https://github.com/ooni/pipeline/pull/291
https://github.com/ooni/pipeline/pull/292
https://github.com/ooni/pipeline/pull/295
https://github.com/ooni/pipeline/pull/296
https://github.com/ooni/pipeline/pull/297
https://github.com/ooni/pipeline/pull/299
## OONI infrastructure
We made several important improvements to the OONI server infrastructure.
Specifically, we:
* Consolidated the pattern used to deploy web-connectivity test helpers:
https://github.com/ooni/backend/issues/300
* Upgraded our monitoring software to the latest version, prometheus
2.15.2: https://github.com/ooni/backend/issues/302
* Dealt with several incidents affecting our infrastructure and
discussed how to better document them going forward:
https://github.com/ooni/backend/issues/343
* Added support for logging long-running queries and plotted them in an
internal dashboard: https://github.com/ooni/backend/issues/325
## OONI website
In collaboration with community members, we:
* Improved the SEO of the OONI website:
https://github.com/ooni/ooni.org/issues/314
* Replaced the CSS framework of the OONI website:
https://github.com/ooni/ooni.org/issues/343
* Fixed the HTML <title> tag for certain pages on our website, improving
how our website appears in search engines:
https://github.com/ooni/ooni.org/issues/296
## Google Summer of Code
We submitted the following OONI project ideas for the Google Summer of
Code (GSoC):
* OONI Probe network experiments:
https://community.torproject.org/gsoc/ooni-probe-experiments/
* OONI Explorer Advanced Search:
https://community.torproject.org/gsoc/ooni-explorer-advanced-search/
* Privacy aware geo-lookup:
https://community.torproject.org/gsoc/privacy-aware-geo-lookup/
## Report on censorship in Togo
We published a research report on the blocking of instant messaging apps
in Togo amid the 2020 presidential election.
Our report is available here:
https://ooni.org/post/2020-togo-blocks-instant-messaging-apps/
Thanks to measurements contributed by OONI Probe users in Togo, we found
that access to the WhatsApp mobile app, Telegram Web, and Facebook
Messenger were blocked on 2 networks (Togo Telecom & Atlantique Telecom)
in Togo on election day. They were, however, accessible on the Canalbox
network, suggesting that internet censorship varies across ISPs in Togo.
VPNCompare published a blog post discussing our report on Togo:
https://www.vpncompare.co.uk/internet-censorship-togo/
## Supported OTF Fellow
Over the last year, we have hosted an OTF Information Controls Fellow
who has been researching internet censorship in certain states of India.
Throughout February 2020, we worked closely with the fellow to analyze
relevant OONI measurements and document the findings.
## Collaboration with Netalitica
Netalitica researchers continue to do an excellent job updating the
Citizen Lab test lists!
In February 2020, we reviewed Netalitica updates to the Citizen Lab test
lists for Kazakhstan
(https://github.com/citizenlab/test-lists/pull/585), Turkey, and Ukraine.
In response to a data analysis request from Netalitica, we extracted and
analyzed Web Connectivity measurements from 19 countries and generated
charts, all of which are available through the following ticket:
https://github.com/ooni/ooni.org/issues/333.
## Other test list updates
We opened pull requests with several other test list updates as well.
More specifically, we:
* Updated the global and Chinese test lists to include
coronavirus-related websites:
https://github.com/citizenlab/test-lists/pull/571 &
https://github.com/citizenlab/test-lists/pull/572
* Updated the global test list to add a circumvention tool site:
https://github.com/citizenlab/test-lists/pull/569
* Updated the Iranian test list:
https://github.com/citizenlab/test-lists/pull/575 &
https://github.com/citizenlab/test-lists/pull/574
## Reviewed RightsCon session proposals
This year, OONI’s Maria is a Program Committee Co-Chair for the Network
Connectivity and Internet Shutdowns track at RightsCon 2020.
As part of this capacity, Maria reviewed 27 RightsCon session proposals
submitted under the Network Connectivity and Internet Shutdowns track
(excluding session proposals submitted by OONI).
## Organization of IFF Internet Measurement Village
Throughout February 2020, we continued to coordinate efforts around the
organization of the Internet Measurement Village at the Internet Freedom
Festival (IFF).
Our coordination efforts involved reaching out to community members to
encourage them to submit session proposals, discussing session ideas
with community members, collecting session ideas in order to create an
agenda, and participating in bi-weekly calls with other village organizers.
Unfortunately though the IFF 2020 has been cancelled as a result of the
escalating impact of COVID-19 (coronavirus) in Europe and around the world.
Nonetheless, we hope to co-organize the Internet Measurement Village at
the IFF next year. Meanwhile, we would like to explore opportunities to
facilitate measurement sessions remotely online (beyond the online
monthly OONI community meetings).
## Community use of OONI data
### Censored Planet & Citizen Lab research paper
Censored Planet and the Citizen Lab collaborated on a research paper,
titled “Measuring the Deployment of Network Censorship Filters at Global
Scale”, which makes use of OONI data.
Their research paper (available here:
https://censoredplanet.org/assets/filtermap.pdf) will appear in Network
and Distributed System Security Symposium (NDSS), 2020.
Censored Planet also published a report summarizing the findings:
https://censoredplanet.org/filtermap
As part of their research, Censored Planet and the Citizen Lab used OONI
data (along with Censored Planet data) to gather blockpages from filter
deployments around the world.
### Blocking of Tutanota in Russia
In mid-February 2020, OONI measurements collected from Russia revealed
the blocking of email provider tutanota.com.
Once Tutanota was notified of the blocking of their site in Russia, they
independently coordinated OONI Probe testing on Twitter to gather more
measurements: https://twitter.com/TutanotaTeam/status/1228735217768292352
Roskomsvoboda followed up to share the relevant blocklist
(https://reestr.rublacklist.net/search/?q=tutanota) and they published a
post on the blocking of tutanota.com in Russia:
https://roskomsvoboda.org/55478/
A news article, citing OONI Explorer measurements on the blocking of
tutanota.com in Russia, was also published:
https://www.clubic.com/pro/legislation-loi-internet/donnees-personnelles/actualite-886124-protonmail-russie-interdit-usage-emails-chiffres-tutanota.html
## Community activities
### FOSDEM
OONI’s Federico traveled to Brussels to attend FOSDEM
(https://fosdem.org/2020/) on 1st & 2nd February 2020.
### Community meeting
We facilitated the monthly OONI Community Meeting on 25th February 2020
on our Slack channel (https://slack.ooni.org/), during which we discussed:
1. Measuring throttling
2. Measuring the blocking of IP addresses
3. Mining OONI data to examine the blocking of the Tor network
## Userbase
In February 2020, 7,669,694 OONI Probe measurements were collected from
5,511 networks in 214 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on monthly coverage worldwide):
http://explorer.ooni.org/
~ The OONI team.
--
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20200309/b3ea7bdd/attachment.sig>
More information about the tor-project
mailing list