[tor-project] OONI Monthly Report: April 2020
Maria Xynou
maria at openobservatory.org
Mon May 11 11:51:36 UTC 2020
Hello,
Throughout April 2020, the OONI team worked on the following sprints:
* Horseshoe Crab Sprint (30th March 2020 - 12th April 2020)
* Blue Bottles Sprint (13th April 2020 - 26th April 2020)
* Beluga Sprint (27th April 2020 - 3rd May 2020)
Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni
Details and highlights are available in this report.
## Public launch of new OONI Probe desktop app
We launched a new OONI Probe desktop app for Windows and macOS!
You can download and install the new app here:
https://ooni.org/install/desktop
To announce the launch of the new OONI Probe desktop app and share
relevant highlights, we published a blog post:
https://ooni.org/post/2020-ooni-probe-desktop-app/
### Localization
Thanks to support from the Localization Lab community, the new OONI
Probe desktop app was translated to 14 languages: Chinese, Arabic,
Farsi, Russian, Thai, Turkish, Spanish, French, Portuguese, Italian,
Greek, Slovak, Catalan, and German.
We launched the desktop app with 12 of these languages, excluding Arabic
and Farsi, which we will be included in future releases once we have
added RTL support.
### Press coverage
The launch of the new OONI Probe desktop app received press coverage
from Deutsche Welle (DW), who published an article about it:
https://www.dw.com/en/ooni-an-app-for-detecting-internet-censorship/a-53224221
The DW article was also published in the following languages:
* German:
https://www.dw.com/de/ooni-eine-app-um-internetzensur-aufzusp%C3%BCren/a-53198295
* Chinese:
https://www.dw.com/zh/ooni%E5%BA%94%E7%94%A8%E7%A8%8B%E5%BA%8F-%E8%AE%A9%E4%BA%92%E8%81%94%E7%BD%91%E5%AE%A1%E6%9F%A5%E6%97%A0%E6%89%80%E9%81%81%E5%BD%A2/a-53207287
* Farsi:
https://www.dw.com/fa-af/ooni-%D8%A7%D9%BE%D9%84%DB%8C%DA%A9%DB%8C%D8%B4%D9%86%DB%8C-%D8%A8%D8%B1%D8%A7%DB%8C-%D8%B1%D8%AF%DB%8C%D8%A7%D8%A8%DB%8C-%D8%B3%D8%A7%D9%86%D8%B3%D9%88%D8%B1-%D8%A7%DB%8C%D9%86%D8%AA%D8%B1%D9%86%D8%AA%DB%8C/a-53205844?maca=dar-rss-dar-all-1481-rdf
* Spanish:
https://www.dw.com/es/ooni-la-aplicaci%C3%B3n-para-detectar-la-censura-en-internet/a-53212192
## Adding support in OONI Probe for availability testing of the
circumvention tools Tor, obfs4proxy, and Psiphon
We released OONI Probe Desktop 3.0.0 (the new OONI Probe desktop app for
Windows and macOS) which includes support for running the new Tor and
the Psiphon circumvention tool tests.
Tor test: https://ooni.org/post/2020-ooni-probe-desktop-app/#tor
Psiphon test: https://ooni.org/post/2020-ooni-probe-desktop-app/#psiphon
## Analyzing collected censorship circumvention tool test results and
integrating them into OONI Explorer and the OONI API
Following the release of the new OONI Probe desktop app (which includes
the new Tor and Psiphon tests), we started to collect, analyze, and
publish OONI Probe measurements from the testing of Tor and Psiphon
around the world.
These measurements -- which are also available via the OONI API
(https://api.ooni.io/) -- can easily be searched and accessed through
OONI Explorer:
* Tor measurements:
https://explorer.ooni.org/search?until=2020-05-12&test_name=tor
* Psiphon measurements:
https://explorer.ooni.org/search?until=2020-05-12&test_name=psiphon
Measurements collected from around the world are published on OONI
Explorer in near real-time.
## Improving censorship circumvention tool methodologies to include
metrics that are also related to the performance of the tool
To improve our methodologies for measuring censorship circumvention
tools, we aim to measure the performance of those tools as well.
To this end, we:
* Added support for running the NDT speed test over Psiphon:
https://github.com/ooni/probe-engine/issues/506
* Switched to using the golang implementation of DASH, which will allow
us to run this performance test over the Psiphon network:
https://github.com/ooni/probe-engine/issues/501
* Made progress in refactoring netx to enable which more performance
testing in the testing clients:
https://github.com/ooni/probe-engine/issues/543
## Developing OONI Probe orchestration logic that is specific to
circumvention tool testing
As part of our work on developing OONI Probe orchestration logic that is
specific to circumvention tool testing, we discussed with Tor’s
anti-censorship team how we would integrate bridge testing into OONI
probe services.
See: https://github.com/ooni/backend/issues/396
## OONI Probe mobile app
Some of the main new changes that the next releases of the OONI Probe
mobile app will feature include:
* New/improved design for the Dashboard
* “Run” button (in the Dashboard) to run all OONI Probe tests at once
* Grouping the middlebox tests inside the Performance card
* Adding a new Circumvention card to release the new Tor & Psiphon tests
In preparation for the next releases, we worked on the following tickets:
* Implement new dashboard layout: https://github.com/ooni/probe/issues/1094
* Run All Tests button: https://github.com/ooni/probe/issues/826
* Implement Run All Tests button: https://github.com/ooni/probe/issues/1093
* Group the middlebox tests with the performance test group:
https://github.com/ooni/probe/issues/835
* Remove notification settings screens:
https://github.com/ooni/probe/issues/1090
* Verify network information settings:
https://github.com/ooni/probe/issues/1112
* Remove country code in the opt out settings of the app:
https://github.com/ooni/probe/issues/1105
* New navbar appearance on iOS: https://github.com/ooni/probe/issues/1108
* Create experimental flavour for the Android app:
https://github.com/ooni/probe/issues/1080
* Remove/update obsolete NDT fields:
https://github.com/ooni/probe/issues/1107
We also created a beta branch with support for automatic daily testing:
https://github.com/ooni/probe/issues/1059
## Push notifications
As part of our ongoing efforts to add support for sending out push
notifications, we deployed a production countly instance on Digital
Ocean: https://github.com/ooni/backend/issues/379
We also made a developer build of the OONI Probe mobile app which uses
countly to receive push notification and try it out:
https://github.com/ooni/probe/issues/1101
We also did some testing of the countly backend to ensure that it is not
a critical path and that downtime of the service will not affect our
mobile app user: https://github.com/ooni/backend/issues/393
Then we updated the developer OONI Probe mobile app build to also
support for receiving OONI Run links via push notification:
https://github.com/ooni/probe/issues/874
## Making the OONI Probe apps rely entirely on the OONI Probe golang engine
We continued to make progress on making the OONI Probe mobile and
desktop apps rely entirely on the golang based probe-engine.
To this end, we completed the work in the following tickets:
https://github.com/ooni/probe-engine/issues/125
https://github.com/ooni/probe-engine/issues/349
https://github.com/ooni/probe-engine/issues/352
https://github.com/ooni/probe-engine/issues/376
https://github.com/ooni/probe-engine/issues/396
https://github.com/ooni/probe-engine/issues/453
https://github.com/ooni/probe-engine/issues/482
https://github.com/ooni/probe-engine/issues/489
https://github.com/ooni/probe-engine/issues/501
https://github.com/ooni/probe-engine/issues/504
https://github.com/ooni/probe-engine/issues/538
https://github.com/ooni/probe-engine/issues/543
https://github.com/ooni/probe/issues/1017
https://github.com/ooni/probe/issues/1036
https://github.com/ooni/probe/issues/1040
As part of the process of switching entirely to the new golang
probe-engine, Measurement Kit
(https://github.com/measurement-kit/measurement-kit) has now been
deprecated.
Since several third-party projects rely on Measurement Kit, we published
an upgrade path for all the use cases that we know of. This
documentation is available here:
https://github.com/measurement-kit/measurement-kit/blob/master/README.md
As part of the process of deprecating Measurement Kit, we also worked on
the activities in the following tickets:
https://github.com/measurement-kit/measurement-kit/issues/1921
https://github.com/measurement-kit/measurement-kit/issues/1913
## Expanding OONI Probe measurement methodologies
As part of our ongoing efforts to expand our measurement methodologies, we:
* Integrated the Go implementation of the DASH test into probe-engine:
https://github.com/ooni/probe-engine/issues/501 (and implemented
ConnectTime in the DASH test:
https://github.com/ooni/probe-engine/issues/281)
* Made improvements in the specification of the data format to allow for
incrementally updating the data format without introducing breaking
changes: https://github.com/ooni/probe-engine/issues/423
* Refactored and carried out research related to netx:
https://github.com/ooni/probe-engine/issues/543 (i.e. netx is code used
by probe-engine as a low level library to collect network, DNS, and HTTP
events that occur during OONI measurements)
* Analyzed measurements and performed quality assurance for our SNI
blocking experiment: https://github.com/ooni/probe-engine/issues/547 &
https://github.com/ooni/probe-engine/issues/299
### Measuring SNI based filtering in Iran
To check whether our new SNI blocking experiment
(https://github.com/ooni/spec/blob/adf9537c43ef848b96792f8dea99fbd7f93fe608/nettests/ts-024-sni-blocking.md)
works as expected, we tested it in Iran (where this censorship technique
is known to be deployed) with the help of local community members.
Following our experiments in Iran, we concluded that the new OONI SNI
blocking experiment works as expected.
To document and share our experiments and findings, we published the
report “Measuring SNI based blocking in Iran”, which is available here:
https://ooni.org/post/2020-iran-sni-blocking/
## Prototype of system for probe URL selection
To improve URL testing, we created a prototype system for probe URL
selection which is now being deployed in production. Below we share details.
### Policy for URL prioritization
We started off by writing and publishing a document that describes the
new smart URL list system, why it’s needed, which problems it aims to
address, how community members can contribute to URL priorities, and how
we plan to roll this system out. This document is available here:
https://ooni.org/post/ooni-smart-url-list-system/
We also published a policy that determines the criteria for URL
prioritization:
https://ooni.org/post/policy-url-prioritization/ooni-policy-url-prioritization.pdf
### Adding weights to Citizen Lab categories
Creating a smart URL list system requires adding weights to the Citizen
Lab category codes (based on which tested URLs are categorized), to
ensure that certain categories of URLs receive more testing over others
(depending on the criteria outlined in the aforementioned policy for URL
prioritization).
Our first steps therefore involved adding weights to the Citizen Lab
category codes: https://github.com/ooni/ooni.org/issues/431
We will likely revise these weights as the system is rolled out and we
receive community feedback.
### Prototype system
The prototype is available through this ticket:
https://github.com/ooni/backend/issues/361
The prototype system for URL prioritization is implemented in the
backend through the following pull request:
https://github.com/ooni/pipeline/pull/312
## Analyzing data to extract website metrics
We completed, deployed, and merged a series of improvements to our
fast-path data processing pipeline:
https://github.com/ooni/backend/issues/233
To add data export capabilities for website-related metrics, we
implemented an API call to extract metrics from the counters table (in
JSON and CSV): https://github.com/ooni/backend/issues/321
## OONI Explorer
Once the smart URL list system has been rolled out, we aim to add
support to OONI Explorer (or, potentially, another web platform) for
user accounts that (among other things) would enable users to run
database-heavy queries for website-centric measurements.
To this end, we did some research to understand, in practice, how these
accounts could work across the various OONI products. See:
https://github.com/ooni/backend/issues/391 and
https://github.com/ooni/ooni.org/issues/434.
During the April 2020 OONI Community Meeting, we did a poll to collect
feedback and understand if OONI users would have issues with certain
authentication methods, if OONI accounts were created.
Our idea is to have an account that has interoperability across OONI
Probe, OONI Explorer, and OONI Run to support seamless integration
between the platforms.
## OONI backend
To improve our backend services, we:
* Implemented rate limiting on exposed services:
https://github.com/ooni/backend/issues/260
* Completed our work around investigating API slowness and DB issues:
https://github.com/ooni/backend/issues/381
* Improved DB query monitoring: https://github.com/ooni/backend/issues/382
* Decreased DB query load (throttling, caching, blocking bots):
https://github.com/ooni/backend/issues/383
* Implemented throttling of the API based on IP address:
https://github.com/ooni/backend/issues/394
* Investigated missing Psiphon measurements in the fast-path pipeline:
https://github.com/ooni/backend/issues/296
* Ensured that the API gives priority to fast-path table data:
https://github.com/ooni/backend/issues/283
* Investigated API query slowdown:
https://github.com/ooni/backend/issues/401
* Added rate limiting to the API:
https://github.com/ooni/backend/issues/405
* Added support for parallelising can processing in the fast-path
pipeline: https://github.com/ooni/backend/issues/392
## Improving OONI server infrastructure
As part of our ongoing efforts to improve our server infrastructure, we:
* Deployed workarounds for slow networking between servers:
https://github.com/ooni/backend/issues/384
* Ensured that the new OONI Explorer redirects to HTTPS by default:
https://github.com/ooni/backend/issues/252
* Made improvements to the fast-path pipeline:
https://github.com/ooni/backend/issues/233
* Installed a clickhouse test instance on EC2:
https://github.com/ooni/backend/issues/388
* Tested and benchmarked Greenhost object storage:
https://github.com/ooni/backend/issues/247
* Dropped UI components from OONI API that are duplicated in OONI
Explorer: https://github.com/ooni/backend/issues/139
## Improving the quality of OONI software
As part of our ongoing efforts to improve the quality of OONI software, we:
* Refactored and did research around netx:
https://github.com/ooni/probe-engine/issues/509
* Sanitized the netx error strings:
https://github.com/ooni/probe-engine/issues/546
* Discussed the react-native integration plan:
https://github.com/ooni/probe/issues/1058
## OONI Run usability study
To enable community members to more effectively use OONI Run
(https://run.ooni.io/), we aim to research its limitations and collect
structured community feedback through an OONI Run usability study.
In April 2020, we created a user research plan and decided to collect
feedback through a combination of a survey and a series of interviews
with community members. Throughout the month, we created and circulated
the survey, and we carried out interviews.
### Survey
We created a survey with the goal of collecting more quantitative
information on community needs, challenges, and feedback for the
improvement of OONI Run.
This survey is primarily meant to be completed by OONI Run users, as we
include detailed questions that require experience with the OONI Run
platform.
The survey is available here: https://ooni.typeform.com/to/r9c5ee
Throughout April 2020, we circulated the survey on social media and on
multiple mailing lists, as well as with our network of partners. We aim
to analyze the data collected through the survey in mid-May 2020.
### Interviews
Through interviews with community members, we aim to collect more
in-depth qualitative information to better understand their needs,
challenges, and feedback for the improvement of OONI Run. The interviews
also provide us an opportunity to observe how community members interact
with the OONI Run platform.
The interviews involve three user personas:
* Measurement campaign coordinators: Experienced OONI Probe users who
use OONI Run to generate links and coordinate censorship measurement
campaigns
* Measurement campaign volunteers: OONI Probe users who receive OONI Run
links to perform testing and participate in censorship measurement campaigns
* Internet freedom community members: People who care about internet
censorship, but who haven’t necessarily used OONI Run before
As each of these personas have different experiences with OONI tools, we
drafted separate interview questions for each persona.
During the last week of April 2020, we interviewed 7 community members.
We plan to continue and complete the interviews by mid-May 2020.
### Blog post
To promote the survey and encourage more community members to
participate in the interviews, we wrote and published a blog post that
shares the survey and information about the interviews.
This blog post is available here:
https://ooni.org/post/2020-ooni-run-survey-and-interviews/
## OONI website
Leading up to the public launch of the OONI Probe desktop app, we made a
series of improvements to the OONI website.
These include new Install pages (https://ooni.org/install/) for our
mobile (https://ooni.org/install/mobile) and desktop clients
(https://ooni.org/install/desktop). The new Install pages have a new
layout, design, and copy.
We also edited the copy for most of the other pages on the OONI website
to ensure that the information we provide is up to date and relevant. As
part of these copy edits, we updated the risks documentation
(https://ooni.org/about/risks/), we updated the test descriptions
(https://ooni.org/nettest/), and we improved the layout and copy of the
landing page (https://ooni.org/).
We also wrote and published OONI’s mission statement:
https://ooni.org/about/
## Updated OONI Data Policy
We published an updated version of the OONI Data Policy:
https://ooni.org/about/data-policy
The new version of the OONI Data Policy shares the principles that
govern our data practices, and we explain what data we collect, process,
store, and publish when you use OONI Probe. We explain how you can opt
out of the various forms of data collection and publication.
We also share which analytics we use for the OONI website, OONI
Explorer, and the OONI desktop app, and we explain how you can opt out
of analytics all-together.
## Test list updates
Throughout April 2020, we:
* Updated the Global test list:
https://github.com/citizenlab/test-lists/pull/600
* Updated multiple test lists with COVID-19 related sites shared by the
Citizen Lab: https://github.com/citizenlab/test-lists/pull/601
## Data analysis
In April 2020, we analyzed OONI measurements for the following:
* OTF Information Controls Fellow:
https://github.com/ooni/ooni.org/issues/453 (analysis of OONI
measurements collected from Myanmar)
* Community members in Myanmar:
https://github.com/ooni/ooni.org/issues/471 (longer-term analysis of
OONI measurements collected from Myanmar)
* Community members in Guinea:
https://github.com/ooni/ooni.org/issues/488 (to examine the blocking of
social media during Guinea’s March 2020 elections)
* Website monitoring project: https://github.com/ooni/ooni.org/issues/454
## GitHub sponsorship
Our application to receive donations through our GitHub repositories was
accepted!
OONI now has a GitHub sponsors profile: https://github.com/sponsors/ooni
We are therefore in the position to receive recurring donations.
## Community use of OONI data
### Blocking of sites hosting COVID-19 related information in Venezuela
Through the use of OONI Probe and OONI data, VEsinFiltro reported new
censorship cases in Venezuela involving the blocking of sites hosting
COVID-19 related information.
Many of these cases were shared on Twitter:
https://twitter.com/vesinfiltro/status/1250579061937770496,
https://twitter.com/vesinfiltro/status/1250571183579373568,
https://twitter.com/vesinfiltro/status/1253423468273471490,
https://twitter.com/vesinfiltro/status/1253122594976468993.
VEsinFiltro also published a report (which links to OONI measurements)
on the blocking of and the phishing attempts against the Heroes of
Health platform (created by the management of Juan Guiado):
https://vesinfiltro.com/noticias/2020-04-26-phishing_heroes_salud
### Blocking of websites in Egypt
Masaar published a report on the blocking of websites in Egypt over the
last months (“Internet Censorship in Time of Social Distancing”),
through the use of OONI Probe and OONI data.
Their report is available here:
https://masaar.net/en/internet-censorship-in-time-of-social-distancing/
## Community activities
### OONI Community Meeting
On 28th April 2020, we facilitated the monthly OONI Community Meeting on
our Slack channel (https://slack.ooni.org/), during which we discussed
the following topics:
1. Updates from the OONI team.
2. Adding user accounts to OONI Explorer.
3. Monitoring potential censorship events during the upcoming elections
in West Africa.
4. Challenging the blocking of social media during Guinea’s March 2020
elections in a strategic litigation case with OONI data.
5. Women on Web blocked in Spain: Technical analysis.
6. Reducing the latency of publication of measurements from custom URL
tests and experimental tests.
7. SNI-based blocking.
### IFF Glitter Meetup
On 30th April 2020, we were invited to attend the IFF Glitter Meetup as
featured guests to discuss the OONI Probe desktop app launch.
The notes from what was discussed are available here:
https://internetfreedomfestival.org/wiki/index.php/April_30_2020_GM#Conversation_Topic:_OONI
### Coordination of COVID-19 site testing in Mexico
Our Mexican partners, R3D, published a blog post where they encourage
OONI Probe testing of COVID-19 related websites.
Their post is available here:
https://r3d.mx/2020/03/27/ayudanos-a-descubrir-bloqueos-de-informacion-sobre-covid-19-en-internet/
## Userbase
In April 2020, 7,175,892 OONI Probe measurements were collected from
6,124 networks in 214 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/
~ The OONI team.
--
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20200511/b6bb239d/attachment-0001.sig>
More information about the tor-project
mailing list