Hello,
Below we share OONI's September 2020 status report. Sincere apologies for the huge delay (our team has been quite swamped these past few months)! Our October 2020 report will follow soon too.
# OONI Monthly Report: September 2020
Throughout September 2020, the OONI team worked on the following sprints:
* Sprint 21 - Steve Z. (31st August 2020 - 13th September 2020) * Sprint 22 - Ποσειδάων (14th September 2020 - 27th September 2020)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
## Migrating OONI infrastructure to Amsterdam
One of our top priorities in September 2020 was the migration of critical OONI infrastructure (hosting the OONI data processing pipeline) to servers in Amsterdam, and updating the OONI PostgreSQL metaDB to adapt it to the requirements of our new data processing pipeline.
We worked towards replacing the batch OONI data processing pipeline with our new fastpath pipeline, which analyzes and publishes OONI measurements from around the world in real-time (the measurements are published as soon as the report is closed).
To this end, we built a new PostgreSQL metaDB, which is powered by the latest version of PostgreSQL (11) available in debian stable, and which uses a different set of tables (based on our improved, fastpath pipeline).
More specifically, our work related to migrating OONI infrastructure involved:
* Implementing a fast-path based API to support OONI Explorer: https://github.com/ooni/backend/issues/442 * Refactoring the API codebase: https://github.com/ooni/api/pull/192 * Implementing some form of canning to use the current collector and/or the new API to receive measurements without relying on the batch pipeline: https://github.com/ooni/backend/issues/284 * Setting up a new database host in Amsterdam: https://github.com/ooni/backend/issues/400 * Implementing measurement_meta lookup: autoclaved, minican, fastpath host * Implementing measurement_uid * Implementing jsonl in the uploader * Updating the fastpath pipeline to support jsonl cans * Updating the fastpath pipeline to support API POST submissions * Implementing measurement upload into the API with new mini-canning * End-to-end testing of all probes * Investigating, implementing, testing Tor measurement submission * Adding software_name and software_version to the fastpath pipeline and rerunning it * Fixing the S3 bucket public permissions
All of the above work is tracked through the following master ticket: https://github.com/ooni/backend/issues/437
## Improving the OONI Probe engine
### Improved our golang probe-engine to support calling geolocate and measurement resubmission APIs
To improve the interaction between our probe-engine and the OONI Probe mobile app, we designed new Go based APIs. The core design choice was to expose Go objects directly to Java/ObjC code, rather than continuing to implement a set of JSON based wrappers.
This work is tracked here: https://github.com/ooni/probe-engine/issues/893
### Improving the data quality of OONI measurements
As part of our ongoing efforts to improve the data quality of OONI measurements, we worked on:
* Understanding DNS based false positives: https://github.com/ooni/probe-engine/issues/770 * Categorizing bugs that affect data quality: https://github.com/ooni/probe-engine/issues/892
### Working towards having our own ASN database for geolocating probes
We made some progress towards building our own ASN database: https://github.com/ooni/probe-engine/issues/727
## Released OONI Probe Mobile 2.6.0
In September 2020, we released OONI Probe Mobile 2.6.0 for:
* Android: https://github.com/ooni/probe-android/releases/tag/v2.6.0 * iOS: https://github.com/ooni/probe-ios/releases/tag/v2.6.0
This release includes Countly (https://github.com/ooni/probe/issues/1241) and includes the following new features:
* Notifications to run OONI Probe tests when censorship events emerge * Self-hosted platform for the collection of anonymized usage metrics (which can help us improve the OONI Probe apps)
We also worked on the following:
* Displaying the messages of push notifications within a modal in the OONI Probe Android app: https://github.com/ooni/probe/issues/1242 * Finished designing the new API used by the probe-engine * Progress on re-adding daily automatic testing: https://github.com/ooni/probe/issues/916 * Investigated (and addressed) the issues in the Android app (flagged by the Play Store) * Reviewed how we request for countly permissions: https://github.com/ooni/probe/issues/1244 * Edited the network count after the deletion of test data (bug fix): https://github.com/ooni/probe/issues/1240 * Added support for asking notification permission in the app: https://github.com/ooni/probe/issues/1210 * Replaced the links that discuss false positives: https://github.com/ooni/probe/issues/1095 * Discussed using multiple mobile deep link URL prefixes: https://github.com/ooni/probe/issues/1246
## Improving the OONI Probe desktop app
As part of our ongoing efforts to improve upon the OONI Probe desktop app, we worked on the following activities:
* Investigated performance issues related to the loading time * Fixed the onboarding button alignment: https://github.com/ooni/probe/issues/1138 * Fixed app crash that occurred when viewing interrupted performance test results: https://github.com/ooni/probe/issues/1245 * Worked towards the renewal of the Windows code signing certificate for OONI Probe Desktop
## Added support to use the new migrated API in OONI Explorer
As we worked towards migrating our infrastructure to servers in Amsterdam (and building a new API to accommodate the migration), we also had to make OONI Explorer rely on our new API and debug OONI Explorer/API interactions.
This work involved activities that include:
* Ensuring that OONI Explorer uses the refactored API: https://github.com/ooni/explorer/issues/489 * Removing tested_since from the OONI Explorer country page: https://github.com/ooni/explorer/issues/482 * Ensuring that OONI Explorer pages render minimal content when `test_keys` are missing: https://github.com/ooni/explorer/issues/416
## Improvements to OONI Explorer
As part of our ongoing improvements to OONI Explorer, we worked on restoring the anomaly flag indicators for the OONI Probe instant messaging tests in the search results: https://github.com/ooni/explorer/issues/425
This means that OONI Explorer users can now visually and easily find all anomalous measurements pertaining to the OONI Probe WhatsApp, Telegram, and Facebook Messenger tests via the OONI Explorer search page.
## URL prioritisation frontend
We implemented a basic UI function (as an MVP) for the prioritized testing of URLs. This can be viewed here: https://github.com/ooni/ooni.org/issues/524#issuecomment-690707849
## Published report on censorship events in Belarus amid protests
We published a report documenting censorship events in Belarus amid anti-government protests. We prepared this report in collaboration with Human Constanta and the Digital Observers Community Belarus.
Our report is available here: https://ooni.org/post/2020-belarus-internet-outages-website-censorship/
We detected the blocking of 86 websites in Belarus, which include news media, political opposition, pro-democracy, and election related websites, as well as communication and circumvention tool sites.
We observed a variance in blocking both in terms of which websites were blocked across ISPs (i.e. different sites blocked on different networks), as well as in terms of censorship techniques. In some cases, when sites were hosted on HTTP, we saw that ISPs served a blockpage. But when sites were hosted on encrypted HTTPS, we observed interference during the TLS handshake (after the TCP connection and before the HTTP request), resulting in a connection reset error.
As blocking appears to have been implemented during the TLS handshake, this suggests that Deep Packet Inspection (DPI) technology was likely used. It seems likely that the SNI was being used to decide whether to block or let connections go through.
## Press coverage
CoinDesk wrote a story about OONI, which is available here: https://www.coindesk.com/hacker-decentralized-network-censorship
This story features an interview with Arturo Filastò (OONI’s project lead), as well as interviews with OONI community members from Malaysia and Venezuela. The story also links to and discusses our Belarus report.
Our report on Belarus also received press coverage from an Italian media outlet: https://ilmanifesto.it/la-bielorussia-ha-messo-il-bavaglio-al-web/
## Preparation for Internet Shutdown Measurement Training for Advocates
OONI is a lead partner on Internews’ OPTIMA Internet Shutdown Measurement Training for Advocates in Sub-Saharan Africa.
Information about this training program (and its call for applications) is available here: https://internews.org/call-applications-internet-shutdown-measurements-advoc...
As part of our preparation for the Internet Shutdown Measurement Training for Advocates (a 6-week training program which started on 12th October 2020), we carried out the following activities:
* Compiled relevant resources (reading materials and videos) for the training participants * Created homework assignments for the training participants * Reviewed applications for the training program * Reviewed the webinar videos of other mentors and shared feedback * Created a survey (in collaboration with Internews) to collect participant feedback prior to the start of the training program
### Recorded webinar on Introduction to Network Measurement
OONI’s Maria recorded a 30-minute webinar which provides an introduction to network measurement for human rights defenders. An assignment was also prepared for this webinar.
Internews will publish this webinar on their OPTIMA YouTube channel.
## Community use of OONI data
### Blocking of social media in Azerbaijan
Azerbaijan Internet Watch published a report on the blocking of social media in Azerbaijan amid clashes with Armenia.
Their report (which makes use of OONI data) is available here: https://www.az-netwatch.org/news/country-wide-internet-disruptions-reported-...
### Reviewed OTF Fellow research report
Over the past year we have collaborated with an OTF Information Controls Fellow who investigated internet censorship (through the use of OONI Probe and OONI data) in Myanmar. We offered OONI data analysis support (and relevant charts), and we reviewed their final report and shared feedback.
## Community activities
### Arusha Women School of Internet Governance (AruWSIG) 2020
On 10th September 2020, Maria presented OONI (“Measuring internet censorship with OONI Probe”) at the Arusha Women School of Internet Governance (AruWSIG) -- an annual event in Tanzania which partly took place online this year in light of the COVID-19 pandemic.
Information about the event is available here: https://www.ksgen.or.tz/aruwsig-2020/
### Forum on Internet Freedom in Africa 2020 (FIFAfrica)
We attended the Forum on Internet Freedom in Africa 2020 (FIFAfrica) conference remotely, and viewed some sessions in order to learn more about Africa’s digital rights environment.
Information about this event is available here: https://cipesa.org/fifafrica/
### SMEX resource including OONI Probe
SMEX published a post on “Website Blocking in the Arab Region: Techniques, Monitoring, and Response”, which is available here: https://smex.org/ar/%d8%ad%d8%ac%d8%a8-%d8%a7%d9%84%d9%88%d9%8a%d8%a8-%d8%a7...
This resource mentions and promotes OONI Probe as a tool for measuring internet censorship in the Arab region.
## Userbase
In September 2020, 6,479,995 OONI Probe measurements were collected from 5,532 networks in 203 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/
~ The OONI team.