[tor-project] OONI Monthly Report: September 2020

[Maria Xynou]

Hello,

Below we share OONI's September 2020 status report. Sincere apologies
for the huge delay (our team has been quite swamped these past few
months)! Our October 2020 report will follow soon too.

# OONI Monthly Report: September 2020

Throughout September 2020, the OONI team worked on the following sprints:

* Sprint 21 - Steve Z. (31st August 2020 - 13th September 2020)
* Sprint 22 - Ποσειδάων (14th September 2020 - 27th September 2020)

Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni

Highlights are shared in this report below.

## Migrating OONI infrastructure to Amsterdam

One of our top priorities in September 2020 was the migration of
critical OONI infrastructure (hosting the OONI data processing pipeline)
to servers in Amsterdam, and updating the OONI PostgreSQL metaDB to
adapt it to the requirements of our new data processing pipeline.

We worked towards replacing the batch OONI data processing pipeline with
our new fastpath pipeline, which analyzes and publishes OONI
measurements from around the world in real-time (the measurements are
published as soon as the report is closed).

To this end, we built a new PostgreSQL metaDB, which is powered by the
latest version of PostgreSQL (11) available in debian stable, and which
uses a different set of tables (based on our improved, fastpath pipeline).

More specifically, our work related to migrating OONI infrastructure
involved:

* Implementing a fast-path based API to support OONI Explorer:
https://github.com/ooni/backend/issues/442
* Refactoring the API codebase: https://github.com/ooni/api/pull/192
* Implementing some form of canning to use the current collector and/or
the new API to receive measurements without relying on the batch
pipeline: https://github.com/ooni/backend/issues/284
* Setting up a new database host in Amsterdam:
https://github.com/ooni/backend/issues/400
* Implementing measurement_meta lookup: autoclaved, minican, fastpath host
* Implementing measurement_uid
* Implementing jsonl in the uploader
* Updating the fastpath pipeline to support jsonl cans
* Updating the fastpath pipeline to support API POST submissions
* Implementing measurement upload into the API with new mini-canning
* End-to-end testing of all probes
* Investigating, implementing, testing Tor measurement submission
* Adding software_name and software_version to the fastpath pipeline and
rerunning it
* Fixing the S3 bucket public permissions

All of the above work is tracked through the following master ticket:
https://github.com/ooni/backend/issues/437

## Improving the OONI Probe engine

### Improved our golang probe-engine to support calling geolocate and
measurement resubmission APIs

To improve the interaction between our probe-engine and the OONI Probe
mobile app, we designed new Go based APIs. The core design choice was to
expose Go objects directly to Java/ObjC code, rather than continuing to
implement a set of JSON based wrappers.

This work is tracked here: https://github.com/ooni/probe-engine/issues/893

### Improving the data quality of OONI measurements

As part of our ongoing efforts to improve the data quality of OONI
measurements, we worked on:

* Understanding DNS based false positives:
https://github.com/ooni/probe-engine/issues/770
* Categorizing bugs that affect data quality:
https://github.com/ooni/probe-engine/issues/892

### Working towards having our own ASN database for geolocating probes

We made some progress towards building our own ASN database:
https://github.com/ooni/probe-engine/issues/727

## Released OONI Probe Mobile 2.6.0

In September 2020, we released OONI Probe Mobile 2.6.0 for:

* Android: https://github.com/ooni/probe-android/releases/tag/v2.6.0
* iOS: https://github.com/ooni/probe-ios/releases/tag/v2.6.0

This release includes Countly
(https://github.com/ooni/probe/issues/1241) and includes the following
new features:

* Notifications to run OONI Probe tests when censorship events emerge
* Self-hosted platform for the collection of anonymized usage metrics
(which can help us improve the OONI Probe apps)

We also worked on the following:

* Displaying the messages of push notifications within a modal in the
OONI Probe Android app: https://github.com/ooni/probe/issues/1242
* Finished designing the new API used by the probe-engine
* Progress on re-adding daily automatic testing:
https://github.com/ooni/probe/issues/916
* Investigated (and addressed) the issues in the Android app (flagged by
the Play Store)
* Reviewed how we request for countly permissions:
https://github.com/ooni/probe/issues/1244
* Edited the network count after the deletion of test data (bug fix):
https://github.com/ooni/probe/issues/1240
* Added support for asking notification permission in the app:
https://github.com/ooni/probe/issues/1210
* Replaced the links that discuss false positives:
https://github.com/ooni/probe/issues/1095
* Discussed using multiple mobile deep link URL prefixes:
https://github.com/ooni/probe/issues/1246

## Improving the OONI Probe desktop app

As part of our ongoing efforts to improve upon the OONI Probe desktop
app, we worked on the following activities:

* Investigated performance issues related to the loading time
* Fixed the onboarding button alignment:
https://github.com/ooni/probe/issues/1138
* Fixed app crash that occurred when viewing interrupted performance
test results: https://github.com/ooni/probe/issues/1245
* Worked towards the renewal of the Windows code signing certificate for
OONI Probe Desktop

## Added support to use the new migrated API in OONI Explorer

As we worked towards migrating our infrastructure to servers in
Amsterdam (and building a new API to accommodate the migration), we also
had to make OONI Explorer rely on our new API and debug OONI
Explorer/API interactions.

This work involved activities that include:

* Ensuring that OONI Explorer uses the refactored API:
https://github.com/ooni/explorer/issues/489
* Removing tested_since from the OONI Explorer country page:
https://github.com/ooni/explorer/issues/482
* Ensuring that OONI Explorer pages render minimal content when
`test_keys` are missing: https://github.com/ooni/explorer/issues/416

## Improvements to OONI Explorer

As part of our ongoing improvements to OONI Explorer, we worked on
restoring the anomaly flag indicators for the OONI Probe instant
messaging tests in the search results:
https://github.com/ooni/explorer/issues/425

This means that OONI Explorer users can now visually and easily find all
anomalous measurements pertaining to the OONI Probe WhatsApp, Telegram,
and Facebook Messenger tests via the OONI Explorer search page.

## URL prioritisation frontend

We implemented a basic UI function (as an MVP) for the prioritized
testing of URLs. This can be viewed here:
https://github.com/ooni/ooni.org/issues/524#issuecomment-690707849

## Published report on censorship events in Belarus amid protests

We published a report documenting censorship events in Belarus amid
anti-government protests. We prepared this report in collaboration with
Human Constanta and the Digital Observers Community Belarus.

Our report is available here:
https://ooni.org/post/2020-belarus-internet-outages-website-censorship/

We detected the blocking of 86 websites in Belarus, which include news
media, political opposition, pro-democracy, and election related
websites, as well as communication and circumvention tool sites.

We observed a variance in blocking both in terms of which websites were
blocked across ISPs (i.e. different sites blocked on different
networks), as well as in terms of censorship techniques. In some cases,
when sites were hosted on HTTP, we saw that ISPs served a
blockpage. But when sites were hosted on encrypted HTTPS, we observed
interference during the TLS handshake (after the TCP connection and
before the HTTP request), resulting in a connection reset error.

As blocking appears to have been implemented during the TLS handshake,
this suggests that Deep Packet Inspection (DPI) technology was likely
used. It seems likely that the SNI was being used to decide whether to
block or let connections go through.

## Press coverage

CoinDesk wrote a story about OONI, which is available here:
https://www.coindesk.com/hacker-decentralized-network-censorship

This story features an interview with Arturo Filastò (OONI’s project
lead), as well as interviews with OONI community members from Malaysia
and Venezuela. The story also links to and discusses our Belarus report.

Our report on Belarus also received press coverage from an Italian media
outlet: https://ilmanifesto.it/la-bielorussia-ha-messo-il-bavaglio-al-web/

## Preparation for Internet Shutdown Measurement Training for Advocates

OONI is a lead partner on Internews’ OPTIMA Internet Shutdown
Measurement Training for Advocates in Sub-Saharan Africa.

Information about this training program (and its call for applications)
is available here:
https://internews.org/call-applications-internet-shutdown-measurements-advocates


As part of our preparation for the Internet Shutdown Measurement
Training for Advocates (a 6-week training program which started on 12th
October 2020), we carried out the following activities:

* Compiled relevant resources (reading materials and videos) for the
training participants
* Created homework assignments for the training participants
* Reviewed applications for the training program
* Reviewed the webinar videos of other mentors and shared feedback
* Created a survey (in collaboration with Internews) to collect
participant feedback prior to the start of the training program 

### Recorded webinar on Introduction to Network Measurement

OONI’s Maria recorded a 30-minute webinar which provides an introduction
to network measurement for human rights defenders. An assignment was
also prepared for this webinar.

Internews will publish this webinar on their OPTIMA YouTube channel.

## Community use of OONI data

### Blocking of social media in Azerbaijan

Azerbaijan Internet Watch published a report on the blocking of social
media in Azerbaijan amid clashes with Armenia.

Their report (which makes use of OONI data) is available here:
https://www.az-netwatch.org/news/country-wide-internet-disruptions-reported-in-azerbaijan/

### Reviewed OTF Fellow research report

Over the past year we have collaborated with an OTF Information Controls
Fellow who investigated internet censorship (through the use of OONI
Probe and OONI data) in Myanmar. We offered OONI data analysis support
(and relevant charts), and we reviewed their final report and shared
feedback.

## Community activities

### Arusha Women School of Internet Governance (AruWSIG) 2020

On 10th September 2020, Maria presented OONI (“Measuring internet
censorship with OONI Probe”) at the Arusha Women School of Internet
Governance (AruWSIG) -- an annual event in Tanzania which partly took
place online this year in light of the COVID-19 pandemic.

Information about the event is available here:
https://www.ksgen.or.tz/aruwsig-2020/

### Forum on Internet Freedom in Africa 2020 (FIFAfrica)

We attended the Forum on Internet Freedom in Africa 2020 (FIFAfrica)
conference remotely, and viewed some sessions in order to learn more
about Africa’s digital rights environment.

Information about this event is available here:
https://cipesa.org/fifafrica/

### SMEX resource including OONI Probe

SMEX published a post on “Website Blocking in the Arab Region:
Techniques, Monitoring, and Response”, which is available here:
https://smex.org/ar/%d8%ad%d8%ac%d8%a8-%d8%a7%d9%84%d9%88%d9%8a%d8%a8-%d8%a7%d9%84%d9%85%d9%86%d8%b7%d9%82%d8%a9-%d8%a7%d9%84%d8%b9%d8%b1%d8%a8%d9%8a%d8%a9-%d8%b1%d8%b5%d8%af/

This resource mentions and promotes OONI Probe as a tool for measuring
internet censorship in the Arab region.

## Userbase

In September 2020, 6,479,995 OONI Probe measurements were collected from
5,532 networks in 203 countries around the world.

This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/

~ The OONI team.

-- 
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E