[tor-project] OONI Monthly Report: May 2020
Maria Xynou
maria at openobservatory.org
Tue Jun 16 12:50:47 UTC 2020
Hello,
Throughout May 2020, the OONI team worked on the following sprints:
* Sprint 12 - Beluga (1st May 2020 - 10th May 2020)
* Sprint 13 - Orca (11th May 2020 - 24th May 2020)
* Sprint 14 - Ponyo (25th May 2020 - 31st May 2020)
Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni
Highlights are shared in this report below.
## OONI Probe desktop app
Following the launch of the new OONI Probe desktop app (for Windows and
macOS) in April 2020, we made improvements and released OONI Probe
Desktop 3.0.1 in May 2020:
https://github.com/ooni/probe-desktop/releases/tag/v3.0.1
## Released OONI Probe Mobile 2.4.0
We released OONI Probe Mobile 2.4.0!
* Android: https://github.com/ooni/probe-android/releases/tag/v2.4.0
* iOS: https://github.com/ooni/probe-ios/releases/tag/v2.4.0
Highlights from the latest OONI Probe mobile app release include:
* New Dashboard design
* New button to run all tests in one go
* Improved NDT speed test (i.e. NDT test re-written to increase
reliability and measurement accuracy)
* Middlebox tests are now grouped with the performance tests
* Telegram, NDT, and DASH tests re-written in golang
## Making the OONI Probe apps rely entirely on the golang engine
As part of our ongoing efforts to make the OONI Probe apps rely entirely
on our golang engine, we:
* Implemented the golang framework on iOS:
https://github.com/ooni/probe/issues/1166
* Made a series of updates:
https://github.com/ooni/probe-engine/issues/551 &
https://github.com/ooni/probe-engine/issues/598
## Add support for configuring push notifications
In May 2020, we created an experimental countly plugin for configuring
and sending push notifications to users by country
(https://github.com/ooni/backend/issues/378). We also deployed a
production instance of the countly instance
(https://github.com/ooni/backend/issues/379) using their digital ocean
droplet. This was tested with the OONI Probe mobile app.
Through these activities, it seems that Countly offers all of the
necessary features that we require to implement all of our planned
activities around push notifications.
## Adding support in OONI Probe for availability testing of the
circumvention tools Tor, obfs4proxy, and Psiphon
The new Tor and Psiphon tests are available through the recently
launched OONI Probe desktop app (https://ooni.org/install/desktop).
In May 2020, we worked towards integrating these tests into the OONI
Probe mobile app as well, in addition to expanding our methodologies.
To this end, we:
* Wrote the UI for the Tor test results on mobile:
https://github.com/ooni/probe/issues/885
* Wrote the UI for the Psiphon test results on mobile:
https://github.com/ooni/probe/issues/884
* Wrote a prototype for a STUN reachability test:
https://github.com/ooni/probe-engine/issues/626
* Saved Tor logs to a file: https://github.com/ooni/probe-engine/issues/625
## Developing OONI Probe orchestration logic that is specific to
circumvention tool testing
As part of our work on developing OONI Probe orchestration logic that is
specific to circumvention tool testing, we added backend support for
retrieving Tor bridges from bridges.tpo
(https://github.com/ooni/orchestra/pull/88). We also documented the next
steps towards deploying and merging the Tor bridgedb orchestra
integration in the following ticket:
https://github.com/ooni/backend/issues/426
## Making OONI Probe’s reporting logic more resilient to censorship
As part of our ongoing efforts to make OONI Probe’s reporting logic more
resilient to censorship, we added support for submitting measurements
through the use of circumvention tools.
The OONI Probe engine now supports setting an external SOCKS5 proxy,
which we tested for performing and submitting measurements. The
probe-engine also supports starting an on-demand Psiphon tunnel.
Relevant tickets:
https://github.com/ooni/probe/issues/888
https://github.com/ooni/probe-engine/issues/509
https://github.com/ooni/probe-engine/issues/543
We also added logic for detecting which circumvention strategy should be
used when speaking to backends (https://github.com/ooni/probe/issues/887).
More specifically, we:
* Added DoH as the primary DNS for the session, with the system resolver
as the fallback;
* Added fallback support to cloudfronting;
* Added support for persistent proxies, like Tor or Psiphon.
We also added failover support to probes
(https://github.com/ooni/probe-engine/issues/407), enabling us to
connect to different collectors and choose the one with the best
performance.
Finally, we added support for discovering when OONI backend
infrastructure is being blocked (https://github.com/ooni/probe/issues/886).
## Analyzing collected censorship circumvention tool test results and
integrating them into OONI Explorer and the OONI API
Following the release of the new OONI Probe desktop app (which includes
the new Tor and Psiphon tests), we have been collecting, analyzing, and
openly publishing censorship circumvention tool results on OONI Explorer
and the OONI API.
In May 2020, we began research on how to present country-level
information on OONI Explorer for the Tor test. As part of this activity,
we made a CSV dump of all the recent Tor test measurements
(https://gist.github.com/hellais/784a8115642d2b47072bfe4e177d1c59).
## Implement beta quality backend logic for prioritising URLs
We made progress on testing and developing the URL prioritization
backend. Specifically, we worked on improving the performance of the
counter tables. See: https://github.com/ooni/backend/issues/411
## Analyze data to extract website metrics
We made progress on backend work required to extract website metrics.
Specifically, we experimented a bit more with a different database
solution, called ClickHouse.
In order to better understand how performant it would be for our use
case, we worked on ingesting all OONI data from the metadb into
ClickHouse (https://github.com/ooni/backend/issues/410). We also had
several discussions on the requirements for the extraction and analysis
of website metrics (https://github.com/ooni/backend/issues/419).
We added support to the aggregation API (which builds per website
aggregate views of the data) to handle the category_code as an axis
(https://github.com/ooni/backend/issues/409).
This enables us to count the number of blocked sites per category or the
number of blocked categories in a given country.
We also deployed and further tested the CSV and JSON aggregate endpoints
(https://github.com/ooni/backend/issues/402).
## Reprocessing old measurements using the fast-path pipeline
Our new fast-path pipeline makes the near real-time analysis and
publication of measurements from around the world possible. However,
OONI measurements span all the way back to 2012, and many of the older
measurements have not been processed by our new fast-path pipeline yet.
As part of our ongoing efforts to reprocess older OONI measurements, we
added support for parallelising “can” processing in the fast-path
pipeline: https://github.com/ooni/backend/issues/392
## Managing the MaxMindDB license change
Over the years we have used the MaxMind GeoIP database
(https://www.maxmind.com/en/home), but since they changed their license,
we have been working towards creating our own ASN database.
This work is documented through the following tickets:
https://github.com/ooni/probe-engine/issues/620
https://github.com/ooni/probe-engine/issues/584
https://github.com/ooni/probe-engine/issues/507
## Published report on the OONI Run usability study findings
During May 2020, we completed the OONI Run usability study and collected
feedback as part of our research on the limitations of the current OONI
Run link format.
In May 2020, we carried out 9 interviews (out of 16 interviews overall),
through which we were able to collect detailed community feedback on the
challenges associated with using the OONI Run platform, as well as
suggestions for improvement and feature requests.
Following our analysis of information and feedback collected through our
survey (https://ooni.typeform.com/to/r9c5ee) and the 16 interviews, we
documented community feedback and shared it through a report.
This report is available here:
https://ooni.org/post/2020-06-09-ooni-run-usability-study-findings/
Our report outlines the objectives, personas, assumptions, and methods
of this study. It also shares, in detail, the challenges and suggestions
that survey and interview participants shared with us. Based on this
feedback, we aim to release an improved version of OONI Run by the end
of 2020.
## Improving our server infrastructure
As part of our ongoing efforts to improve our server infrastructure, we
continued to perform regular maintenance work on our servers, but we
also had to deal with a few incidents.
Specifically, we:
* Cleaned-up the autoclaved files from the data collector, as we were
running out of storage space: https://github.com/ooni/backend/issues/407
* Expanded disk on hkgmetadb to account for the growing dataset:
https://github.com/ooni/backend/issues/422
* Improved SSH connection handling in our fastpath pipeline, allowing us
to be more resilient in handling the retrieval of report files from the
collectors: https://github.com/ooni/backend/issues/404
## Testing and quality assurance
We made progress on improving the quality of our software through better
testing and quality assurance procedures.
Namely, we:
* Started using a random host header when not speaking with OONI:
https://github.com/ooni/probe-engine/issues/578
* Started using 127.0.0.2 as default resolver IP:
https://github.com/measurement-kit/measurement-kit/issues/1884
* Wrote several blog posts about the improvements to the testing engine
based on the analysis of measurements:
https://github.com/ooni/probe-engine/issues/619
## Expanding OONI Probe measurement methodologies
We published a report which evaluates OONI's new measurement engine
within the context of the blocking of Women on Web (www.womenonweb.org)
in Spain.
This report is available here:
https://ooni.org/post/2020-engine-evaluation-spain/
We have been working on expanding our measurement methodologies to
detect more forms of website censorship. To this end, we have been
working on experimental code that combines OONI's Web Connectivity test
with SNI blocking detection methodology (and other techniques).
Our aim is to eventually ship a new and improved Web Connectivity test
-- as part of the OONI Probe apps -- that also measures SNI based
filtering, as part of the broader set of website censorship checks.
To evaluate how these experiments work in practice (and if they work as
expected), we collaborated with Vasilis Ververis (Magma project) and
Spanish hacktivists to measure the blocking of www.womenonweb.org in
Spain through the use of these new experimental methodologies.
Our report (https://ooni.org/post/2020-engine-evaluation-spain/)
describes our experimental implementation for the www.womenonweb.org
case and discusses some next steps.
## Published report on the blocking of websites in Myanmar
In collaboration with OTF Fellow Phyu Kyaw, we published a report on the
recent blocking of 174 domains in Myanmar.
This report is available here:
https://ooni.org/post/2020-myanmar-blocks-websites-amid-covid19/
In March 2020, authorities in Myanmar directed local ISPs to block "fake
news" as part of efforts to tackle disinformation around COVID-19, but
an official blocklist hasn't been publicly disclosed.
OONI measurements confirm the DNS based blocking of 174 domains in
Myanmar, many of which include ethnic media websites.
We compared the blocked news outlets with the Myanmar Press Council's
"fake news" list, and found that many media sites that aren't in the
public "fake news" list are blocked anyway.
Our report received press coverage by Deutsche Welle:
https://www.dw.com/de/r%C3%BCckschritte-bei-pressefreiheit-in-myanmar/a-53582731
## Published report on the blocking of social media in Burundi
We published a report on the blocking of social media in Burundi amid
its 2020 general election.
This report is available here:
https://ooni.org/post/2020-burundi-blocks-social-media-amid-election/
OONI measurements collected from Burundi reveal the blocking of 18
social media platforms on 20th May 2020 (election day), including
skype.com, youtube.com, whatsapp.com, twitter.com, facebook.com,
instagram.com -- among many others.
These sites appear to be blocked -- on Econet (AS37336) and Viettel
Burundi (AS327799) -- through the use of some form of Deep Packet
Inspection (DPI) technology, as the blocking only occurs when we attempt
to establish an HTTPS connection (resulting in connection reset errors),
as opposed to the IPs being blocked or the DNS queries being interfered
with.
OONI data also shows the blocking of the WhatsApp and Telegram apps, and
suggests that access to Facebook Messenger was blocked too.
Interestingly, the blocking of Facebook Messenger on the Lacell Burundi
(AS327720) network was implemented on a DNS level by returning an IP
address in the DNS answer which is in the “reserved for future use”
240.0.0.0/8 netblock.
## Collaboration with Netalitica
Netalitica researchers continued to do great work in reviewing and
updating the Citizen Lab test lists
(https://github.com/citizenlab/test-lists).
In May 2020, we reviewed the updates (made by Netalitica researchers) to
the following test lists:
* Venezuela: https://github.com/citizenlab/test-lists/pull/634
* Qatar: https://github.com/citizenlab/test-lists/pull/633
We also made the following test list updates:
https://github.com/citizenlab/test-lists/pull/623
https://github.com/citizenlab/test-lists/pull/628
https://github.com/citizenlab/test-lists/pull/629
## Data analysis
As part of our partnership with Azerbaijan Internet Watch, we analyzed
all OONI measurements collected from Azerbaijan between 1st January 2020
to 31st May 2020 (https://github.com/ooni/ooni.org/issues/506), based on
which we produced relevant charts and wrote a report.
We also completed data analysis pertaining to a separate research
project examining the blocking of LGBTQI websites
(https://github.com/ooni/ooni.org/issues/437) in several countries
around the world.
## Google Summer of Code (GSoC) student
We are excited to have a Google Summer of Code (GSoC) student join our
team for the summer of 2020!
In May 2020, Krona Emmanuel started his GSoC internship with the OONI
team. As part of his 4-month internship, Krona will work on making
improvements related to social media sharing on OONI Explorer. Further
details about this project are available here:
https://community.torproject.org/gsoc/ooni-explorer-findings/
We started off by collaborating with Krona on defining user stories for
social media sharing: https://github.com/ooni/explorer/issues/462
## Mastodon
To better reach free software (and digital rights) communities, we
started using Mastodon.
The OONI Mastodon account is available here: https://mastodon.social/@ooni
## Community use of OONI data
### Blocking of Women on Web in Spain
The Magma project -- in collaboration with several Spanish hacktivists
-- published a research report documenting the blocking of reproduction
rights website womenonweb.org in Spain.
Their report (which was supported by the use of OONI Probe and OONI
data) is available here:
https://blog.magma.lavafeld.org/post/women-on-web-blocking/
### Blocking of news portal in Venezuela
Through the use of OONI Probe and OONI data, VEsinFiltro reported the
blocking of a news portal (runrun.es) in Venezuela. See their relevant
tweets: https://twitter.com/i/status/1261978668945018880
This was also reported by IPYS Venezuela (who also made use of OONI
Probe and OONI data):
https://twitter.com/ipysvenezuela/status/1262482417069285383
## Community activities
### Organizing the online Internet Measurement Village 2020
Before the IFF got canceled in light of the escalating global impact of
the COVID-19 pandemic, we had originally planned to organize and host an
Internet Measurement Village during the last 2 days of the IFF.
While we still hope to organize and host an in-person Internet
Measurement Village next year as part of IFF 2021, this year we decided
to organize and host a version of this village online.
To this end, we spent time throughout May 2020 planning the online
Internet Measurement Village 2020, coordinating with other measurement
projects and community members, creating the schedule, and trying out
various platforms (e.g. to check which streaming solutions to use).
The Internet Measurement Village 2020 will be hosted entirely online,
and will consist of presentations that are live-streamed on the OONI
YouTube channel (https://www.youtube.com/c/OONIorg), enabling anyone
from the internet to view the presentations, learn about the projects,
and participate in discussions through the live chat.
The Internet Measurement Village will take place mostly throughout June
2020, starting on 10th June 2020 and ending on 3rd July 2020.
We have published the schedule here:
https://ooni.org/post/2020-internet-measurement-village/
### Clean Insights Symposium Extraordinaire
On 12th May 2020, OONI’s Arturo was invited to participate in the Clean
Insights Symposium Extraordinaire (a podcast organized by the Guardian
Project) to discuss OONI.
As part of the podcast, Arturo was asked the following questions:
* OONI provides benefits but with risk. How do you communicate these to
the people doing the measuring or being measured? How does user
experience and consent play a role?
* Within an internet freedom and human rights context, are there
additional threats and risks to consider? Is measuring itself a crime in
some places?
* What do you do with the data that OONI collects? How long do you keep
it? Is it public or private? Can it cause harm?
* Who does OONI benefit and how? Is it just an advocacy tool, or does it
have technical impact in the design of other services and apps?
The podcast is available here:
https://guardianproject.info/podcast/2020/cleaninsights-panel-ooni.html
### OONI Community Meeting
On 26th May 2020, we facilitated the monthly OONI Community Meeting on
our Slack channel (https://slack.ooni.org/), during which we discussed
the following topics:
* Brief updates from the OONI team
* Update on OONI Run survey outcomes
* Improving the SEO of OONI.org
* Creating a Censorship Alert System
* Investigating DoS attacks against media websites
## Userbase
In May 2020, 8,430,917 OONI Probe measurements were collected from 6,199
networks in 211 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/
~ The OONI team.
--
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20200616/fafa87fb/attachment.sig>
More information about the tor-project
mailing list