Hello,
Throughout June 2020, the OONI team worked on the following sprints:
* Sprint 14 - Ponyo (1st June 2020 - 7th June 2020) * Sprint 15 - Globster (8th June 2020 - 21st June 2020) * Sprint 16 - Neon (22nd June 2020 - 30th June 2020)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
## Released OONI Probe Mobile 2.5
We released OONI Probe Mobile 2.5!
Android: https://github.com/ooni/probe-android/releases/tag/v2.5.0
iOS: https://github.com/ooni/probe-ios/releases/tag/v2.5.0
Highlights from the latest release include:
1) Circumvention tool testing: The latest release includes OONI Probe tests for measuring the blocking of Tor & Psiphon!
2) 5 new languages: Thanks to the Localization Lab community, the OONI Probe mobile app is now also available in Indonesian, Thai, Romanian, Slovak, and Icelandic (in total, the OONI Probe mobile app is now available in 20 languages!)
As part of the iOS release, we also rewrote NDT to increase its reliability and measurement accuracy, and wrote the Telegram, NDT, and DASH tests in golang.
## OONI Probe desktop app
In June 2020, we made two OONI Probe desktop app releases:
* OONI Probe Desktop 3.0.2: https://github.com/ooni/probe-desktop/releases/tag/v3.0.2 * OONI Probe Desktop 3.0.3: https://github.com/ooni/probe-desktop/releases/tag/v3.0.3
These releases feature bug fixes and other improvements based on community feedback.
## Making the OONI Probe apps rely entirely on the golang engine
As part of our ongoing efforts to make the OONI Probe apps rely entirely on the golang engine, we:
* Used JCenter to fetch probe-engine on Android: https://github.com/ooni/probe/issues/1191 * Updated to the latest version of probe-cli/probe-engine: https://github.com/ooni/probe/issues/1178 * Made a series of updates: https://github.com/ooni/probe-engine/issues/689
## Adding support for configuring push notifications
Our goal is to be able to send push notifications to OONI Probe mobile app users through the use of Countly, which enables us to include links (such as OONI Run links for coordinated OONI Probe testing) in push notifications.
To this end, we evaluated how to manage the collection of app usage metrics, crash reports, and privacy settings, as documented through the following ticket: https://github.com/ooni/probe/issues/1182
We also made progress on implementing countly on Android: https://github.com/ooni/probe-android/pull/341
## Adding support in OONI Probe for availability testing of the circumvention tools Tor, obfs4proxy, and Psiphon
We have completed our work related to adding support in OONI Probe for availability testing of Tor, obfs4proxy, and Psiphon. The new Tor and Psiphon tests are available through the recently launched OONI Probe desktop app (https://ooni.org/install/desktop).
In June 2020, we also implemented the STUN server reachability experiment: https://github.com/ooni/probe-engine/issues/243
## Improving censorship circumvention tool methodologies by including performance metrics
We completed our (short-term) work on improving our methodologies for measuring censorship circumvention tools by measuring the performance of those tools as well.
As part of our final relevant activities, we:
* Recorded website accessibility performance metrics when using Psiphon: https://github.com/ooni/probe-engine/issues/404 * Improved the data format versioning: https://github.com/ooni/probe-engine/issues/423 * Allowed using an external tor binary to run ndt7, DASH, and urlgetter over it: https://github.com/ooni/probe-engine/issues/587
## Developing OONI Probe orchestration logic that is specific to circumvention tool testing
We completed our work related to developing OONI Probe orchestration logic that is specific to circumvention tool testing.
As part of our final relevant activities, we:
* Wrote an integration test for private Tor bridges: https://github.com/ooni/probe-engine/issues/721 * Stripped private bridge addresses: https://github.com/ooni/probe-engine/issues/643 * Included the country code in the Tor query string: https://github.com/ooni/probe-engine/issues/629
## Making OONI Probe’s reporting logic more resilient to censorship
We completed our (short-term) efforts related to making OONI Probe’s reporting logic more resilient to censorship.
As part of our final relevant activities, we:
* Discussed requirements for detecting the blocking of OONI services in the pipeline: https://github.com/ooni/backend/issues/417 * Created new domain names: https://github.com/ooni/backend/issues/424 * Implemented a prototype of Probe Services blocking detection: https://github.com/ooni/backend/issues/428 * Implemented failover between available OONI data centers: https://github.com/ooni/probe-engine/issues/621 * Used best available OONI data center for orchestration and other services: https://github.com/ooni/probe-engine/issues/651
We did an analysis (https://github.com/ooni/probe/issues/886), using the fact that we added support for probes to failover to cloudfront collectors, and we did not notice any probes failing over using this method.
We have not identified any cases of blocking of OONI infrastructure. That said, we will continue to do more investigations and eventually automate the process.
## Improving URL testing
### Implementing beta quality backend logic for prioritising URLs
As part of our work on implementing beta quality backend logic for the prioritization of URLs, we worked on refactoring and improving our URL prioritization backend (and on creating an internal Grafana dashboard). See: https://github.com/ooni/pipeline/pull/321
Moreover, we started adding an haproxy server in front of the URL prioritisation backend so that we are able to start using it in production for a percentage of the probes out there. See: https://github.com/ooni/sysadmin/pull/446.
### Implementing beta quality frontend to support management of the URL priorities
In order to implement a beta quality frontend to support the management of URL priorities, we started off by reviewing the relevant priorities.
As part of this process, we documented the next steps in the following tickets: https://github.com/ooni/backend/issues/429 https://github.com/ooni/ooni.org/issues/524
## Analyzing data to extract website metrics
As part of our ongoing data analysis work to extract website metrics, we published an aggregation API endpoint and started working on creating a page to try out the new aggregation API. The goal of this page is to query the API for different kinds of data and to render charts based on that data. Eventually, these charts will be integrated in the country pages of OONI Explorer. See: https://github.com/ooni/explorer/issues/463
Our initial experiment for trying out the new aggregation API is available here: https://codesandbox.io/s/nivobar-extra-layers-pbgg4?file=/src/index.js
## Presenting website measurements to account users
We are working towards enabling community members to have access to advanced and database-heavy views of website-centric measurements (through accounts that we will create). To this end, we started off by discussing and deciding what it means to have an account within the context of the various OONI platforms: https://github.com/ooni/ooni.org/issues/434
As part of this research, we wrote an internal document outlining our plans for implementing accounts and authentication to OONI services. We also documented the use cases (user stories) for these accounts, feedback and community requests pertaining to accounts (as communicated to us through a pull in a previous community meeting, as well as through other long-term interactions with community members), as well as implementation details and requirements.
We also worked on creating mock-ups for website-centric views: https://github.com/ooni/explorer/issues/472
## Improving our server infrastructure
As part of our ongoing work to improve our server infrastructure, we improved the performance of the counters table (https://github.com/ooni/backend/issues/411) and we made improvements to the monitoring of our infrastructure (https://github.com/ooni/backend/issues/427).
## Testing and quality assurance
We made progress on improving the quality of our software through better testing and quality assurance procedures. We performed data analysis based on specific measurements (https://github.com/ooni/probe-engine/issues/662) and implemented changes (described below in detail) to simplify subsequent data analysis attempts. We also further evaluated the code for selecting the closest OONI probe service API among the available data centers (HK, MIA, AMS), and extended this functionality to OONI orchestration.
Specifically, we:
* Performed experiments and data analysis based on measurements performed in India (which resulted in a blog post): https://github.com/ooni/probe-engine/issues/662 * Simplified OONI data analysis by ensuring that the resolver is added to the measurement: https://github.com/ooni/probe-engine/issues/642 * Simplified OONI data analysis by setting the DNS cache to null when empty: https://github.com/ooni/probe-engine/issues/658 * Improved urlgetter measurements: https://github.com/ooni/probe-engine/issues/656 * Used best data center providing the probe service API for orchestration and other services: https://github.com/ooni/probe-engine/issues/651 * Wrote an integration test for private Tor bridges: https://github.com/ooni/probe-engine/issues/721
Throughout June 2020, we focused quite a bit on improving the data quality of OONI measurements.
## Published report on the blocking of DNS over TLS in Iran
In June 2020, we published a research report, titled "DNS over TLS blocked in Iran", which is available here: https://ooni.org/post/2020-iran-dot/
We investigated whether DoT works in Iran by gathering a list of 31 well-known DoT endpoints and running experiments from four distinct Iranian mobile and fixed-line Internet Service Providers (ISPs): MCI, TCI, Irancell, and Shatel.
We discovered that:
* 57% of the endpoints are blocked on a least one ISP;
* the blocking is not implemented uniformly across ISPs;
* most blocking happens by interfering with the TLS handshake;
* in some cases TLS handshake blocking seems to depend on the SNI, while in other cases it seems to depend strictly on the TCP endpoint being used;
* forcing TLSv1.3 does not change the rate of successful TLS handshakes compared to letting the server choose a TLS version between v1.0 and v1.3.
In our report, we share details from our experiments and findings. This research is part of our ongoing efforts to expand and improve upon our measurement methodologies.
## Published report on TLS blocking in India
In collaboration with India’s Centre for Internet and Society (CIS), we co-published a research report investigating TLS blocking in India.
This report is available here: https://ooni.org/post/2020-tls-blocking-india/
This investigation sought to understand whether there were cases of TLS blocking that were not only caused by the value of the Server Name Indication (SNI) field in the ClientHello TLS message, but also by the destination IP address. This was part of our efforts to expand our SNI blocking methodology (discussed here: https://ooni.org/post/2020-iran-sni-blocking/).
To this end, we wrote and ran a series of experiments (that will eventually be integrated into the OONI Probe measurement engine) to measure the blocking of four domains (facebook.com, google.com, collegehumor.com, and pornhub.com) on three popular Indian ISPs: ACT Fibernet (fixed line), Bharti Airtel, and Reliance Jio (mobile).
We recorded SNI-based blocking on both Bharti Airtel and Reliance Jio. We also discovered that Reliance Jio blocks TLS traffic not just based on the SNI value, but also on the web server involved with the TLS handshake.
We also noticed that ACT Fibernet’s DNS resolver directs users towards servers owned by ACT Fibernet itself. Such servers caused the TLS handshake to fail, but the root cause of censorship was the DNS.
We also found that one of the tested endpoints (for collegehumor.com:443) does not allow establishing a TCP connection from several vantage points and control measurements. Yet, in Reliance Jio, we saw cases where the connections to such endpoints completed successfully and a timeout occurred during the TLS handshake. This is likely caused by some kind of proxy that terminates the TCP connection and performs the TLS handshake.
## Expanding OONI Probe measurement methodologies
Our aforementioned research reports on DNS over TLS blocking in Iran (https://ooni.org/post/2020-iran-dot/) and TLS blocking in India (https://ooni.org/post/2020-tls-blocking-india/) document our ongoing efforts to expand our measurement methodologies.
The experiments that we design and run as part of these research efforts will eventually be integrated into OONI Probe and shipped as part of our apps.
Some of our efforts on expanding our measurement methodologies are also documented through the following tickets:
https://github.com/ooni/probe-engine/issues/624 https://github.com/ooni/probe-engine/issues/576 https://github.com/ooni/probe-engine/issues/659
## Published report of OTF Information Controls Fellow
Over the last year, OONI has had the opportunity to serve as the host organization of OTF Information Controls Fellow, Chinmayi S K (https://www.opentech.fund/about/people/chinmayi-sk/).
In June 2020, we published Chinmayi's research report, titled "Those Unspoken Thoughts: A study of censorship and media freedom in Manipur, India".
Summary of report: https://ooni.org/post/2020-those-unspoken-thoughts-otf-fellow-report/
Full length report: https://ooni.org/documents/those-unspoken-thoughts-otf-fellow.pdf
As part of her fellowship, Chinmayi researched internet censorship in Manipur, India, and its effect on the womxn in the region. To investigate internet censorship, Chinmayi ran OONI Probe in various regions of India and analyzed relevant OONI measurements. To explore the impact of internet censorship on womxn in the region and to investigate restrictions to media freedom, she circulated a survey and carried out interviews.
Throughout her fellowship, OONI provided research, editing, and data analysis support. When we published Chinmayi’s report in June 2020, we also assisted with relevant outreach activities.
## Published statement in support of the OTF
Following the firing of OTF’s leadership, we published a statement in support of the OTF, explaining why the OTF is essential for internet freedom.
Our statement is available here: https://ooni.org/post/2020-06-19-save-internet-freedom-support-the-open-tech...
## Collaboration with Azerbaijan Internet Watch
### Published guest blog post to encourage more OONI Probe testing in Azerbaijan
To encourage more OONI Probe testing in Azerbaijan, we published a guest blog post (in Azerbaijani) written by Arzu Geybullayeva of Azerbaijan Internet Watch (with whom we partner).
This blog post is available here: https://ooni.org/post/2020-azerbaijan-ooni-probe-testing/
The post provides step-by-step instructions for OONI Probe testing, particularly with regards to the testing of media websites and circumvention tools, which present signs of blocking based on OONI data analysis.
### Data analysis
As part of our partnership with Azerbaijan Internet Watch, we analyzed all OONI measurements collected from Azerbaijan between 1st January 2020 to 30th June 2020 (https://github.com/ooni/ooni.org/issues/543), based on which we produced relevant charts and wrote a report.
## Collaboration with Netalitica
Netalitica researchers continued to do great work in reviewing and updating the Citizen Lab test lists (https://github.com/citizenlab/test-lists).
In June 2020, we reviewed URLs shared by Netalitica researchers and added them to the Turkish test list: https://github.com/citizenlab/test-lists/pull/642
We also made updates to the Azerbaijan (https://github.com/citizenlab/test-lists/pull/640) and Hong Kong (https://github.com/citizenlab/test-lists/pull/641) test lists, based on URLs shared by community members.
## Press coverage
RESET interviewed the OONI team and published an article about OONI.
This article was published in: * German: https://reset.org/blog/ooni-software-spuert-menschenrechtsverletzungen-im-in... * English: https://en.reset.org/blog/ooni-free-software-designed-detect-human-rights-vi...
DARK Reading also published an article that mentions OONI Probe, following an interview with OONI’s Arturo. This article is published here: https://www.darkreading.com/cloud/no-internet-access-amid-protests-heres-how...
## Community use of OONI data
### Psiphon Data Engine (PDE)
Over the last year, we have collaborated with Psiphon on integrating OONI measurements into a new platform, called Psiphon Data Engine (PDE). This platform displays OONI and M-Lab measurements, along with Psiphon metrics, through an interactive dashboard.
In June 2020, Psiphon launched the PDE, which is available here: https://psix.ca/
### Report on the blocking of Blogger in Lebanon
Through the use of OONI Probe and OONI data, SMEX examined the blocking of Blogger in Lebanon.
They published a report based on their findings, which is available here: https://smex.org/ar/%d8%ad%d8%ac%d8%a8-%d8%a8%d9%84%d9%88%d8%ba%d8%b1-%d9%81...
### Report on potentially blocked media in Belarus
In June 2020, Human Constanta published a report examining several cases of network interference in Belarus. As part of their study, they also used OONI Probe to examine the blocking of several media websites.
Their report (which includes OONI findings) is available here: https://humanconstanta.by/chto-proisxodilo-s-internetom-v-belarusi-19-iyunya...
### 2019 Annual Report on Digital Rights in Venezuela
Our Venezuelan partners, IPYS Venezuela, published their 2019 annual report on digital rights in Venezuela, which is available here: https://ipysvenezuela.org/2020/05/17/desconexion-y-censura-reporte-anual-der...
This report discusses censorship findings that they discovered through the use of OONI Probe and OONI data.
### Blocking of YouTube in Venezuela
Through the use of OONI Probe and OONI data, our local partners in Venezuela reported the blocking of YouTube: https://twitter.com/ipysvenezuela/status/1271196383463247872
## Community activities
### Internet Measurement Village 2020
Throughout June 2020, we organized and hosted the first Internet Measurement Village (IMV) 2020. Information about the event is available through our relevant blog post: https://ooni.org/post/2020-internet-measurement-village/
The IMV is an online community event -- dedicated to discussions around internet measurement -- that took place throughout June 2020, starting on 10th June 2020 and ending on 3rd July 2020.
During this 4-week period, we hosted one session almost every day (on weekdays), all of which were live-streamed on the OONI YouTube channel: https://www.youtube.com/c/OONIorg
In total, we hosted 18 live-streamed sessions, covering a wide range of internet measurement projects, as well as local censorship measurement projects, advocacy efforts against internet shutdowns, and censorship circumvention tool projects.
To encourage participation, we hosted a live chat on several platforms (YouTube chat, Slack, IRC) and, on average, each presenter received around 10 questions from the viewers. All questions discussed during each session are available through this pad: https://pad.riseup.net/p/imv2020-keep
Apart from hosting 18 live-streamed sessions, we also worked on promoting each session through the creation of relevant social media assets, as well as through outreach on multiple mailing lists and on all our social media platforms (Twitter, Mastodon, Facebook, Instagram).
The Internet Measurement Village 2020 also featured an online social event, the IMV2020 Distance Disco, hosted by Sandy Ordonez on the following platform: https://distancedisco.nl/
At the end of the Internet Measurement Village 2020, we shared a survey to collect community feedback and learn how we can host better events in the future. Our survey is available here: https://ooni.typeform.com/to/cPVT4ILB
As all the live-streamed sessions of the IMV will continue to live on the OONI YouTube channel, we hope that these recordings will serve as a valuable resource on internet measurement for the internet freedom community.
### OONI presentation at the Internet Measurement Village 2020
On 10th June 2020, as part of the Internet Measurement Village 2020 (that we organized and hosted throughout June 2020), we live-streamed a presentation about OONI.
The video recording of our presentation is available here: https://www.youtube.com/watch?v=SES8PAeQvvs
### Online discussion of findings of OTF Information Controls Fellow research report
Over the last year, OONI has served as the host organization for OTF Information Controls Fellow, Chinmayi SK, whose research report was published on the OONI website in mid-June 2020 (see: https://ooni.org/post/2020-those-unspoken-thoughts-otf-fellow-report/).
On 25th June 2020, we hosted a session on the OONI Slack channel (https://slack.ooni.org/) -- bridged with IRC -- to provide Chinmayi an opportunity to present and discuss her findings with the community.
As part of this session, Chinmayi discussed her research findings in depth, and received several questions from participants.
## Userbase
In June 2020, 7,657,872 OONI Probe measurements were collected from 5,942 networks in 208 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/
~ The OONI team.