Greetings everyone!
This week, the network team will release (likely on Friday) tor 0.3.5.18 (from
tor.git, not arti ;) which will also be the very last version of the LTS
0.3.5.x series.
On February 1st 2022, 0.3.5.x will be end-of-life and so this will close the
chapter of this LTS in favor of the 0.4.5.x series as the latest LTS.
You can find our release schedule here:
https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/CoreTorRele…
(Keep in mind that it is a best effort and not everything on that page is set
in stone.)
As a reminder, 0.4.7.x is still in alpha stage as we are finalizing network
congestion control which is quite large and complex. We will only release a
stable when we consider it ready for which we are very hopeful to be well
within the first half of 2022. Keep an eye out!
Upcoming version:
- 0.3.5.18
@network-team: It is _now_ a good time to start reviewing changes/ files:
https://gitlab.torproject.org/tpo/core/tor/-/tree/release-0.3.5/changes
Last, we've asked the directory authorities to recommend this new version some
minutes ago.
Cheers!
David
--
4+F4wjk7FBUF50LsbGBE3Mdsr5O7ugc/Fs02hg+sIf4=
Hey everyone!
Here are our meeting logs:
http://meetbot.debian.net/tor-meeting/2022/tor-meeting.2022-01-20-15.59.html
And our meeting pad:
Anti-censorship work meeting pad
--------------------------------
Next meeting: Thursday January 20th 16:00 UTC
Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC
(channel is logged while meetings are in progress)
== Goal of this meeting ==
Weekly checkin about the status of anti-censorship work at Tor.
Coordinate collaboration between people/teams on anti-censorship at Tor.
== Announcements ==
== Discussion ==
resume posting weekly meeting notes?
no strong feeling either way, but cohosh will resume because our
discussion notes are worth saving
Kazakhstan shutdown
shutdown ended 2022-01-11, and nothing new noticed since then
China "Anti-Fraud" Webpage Redirection Censorship
https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/iss…
Reply from censored planet:
https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/iss…
currently have no access to an affected vantage point
RIPE Atlas?
want to be able to test a large custom list of domains, to find out
(1) which are being blocked, and (2) whether the blocklist is the same
in different networks in China
snowflake bridge load balancing migration next week?
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
dcf can set up a server with adequate resources and the same keys
change DNS to use this staging server while we make configuration
changes on the main server
change DNS back when finished
eclips.is (Greenhost) is having issues, a VPS with similar
resources is $100–200 / month, not a problem if only used for staging
for a week or so
whom to contact about changing snowflake.torproject.net DNS records?
snowflake.bamsoftware.com and snowflake.freehaven.net are already
CNAMEs for snowflake.torproject.net, so we only need to change the one
https://gitlab.torproject.org/tpo/tpa/team/
good idea to set a short TTL before making the change
still don't now what to do about onion key rotation, short of
patching tor, or hacking LastRotatedOnionKey in the state file
Copy TLS keys in addition to Tor's keys
Reading group?
2022-02-03
== Actions ==
== Interesting links ==
https://fightcensorship.tech/ Icarus Project, an online repository
for documented Internet censorship circumvention techniques and methods
https://internews.org/resource/optimashutdownworkflow/ The OPTIMA
Internet Shutdowns Advocacy Workflow
== Reading group ==
We will discuss "Meteor: Cryptographically Secure Steganography for
Realistic Distributions" on 2022-02-03
https://dl.acm.org/doi/10.1145/3460120.3484550https://eprint.iacr.org/2021/686https://meteorfrom.space/
Questions to ask and goals to have:
What aspects of the paper are questionable?
Are there immediate actions we can take based on this work?
Are there long-term actions we can take based on this work?
Is there future work that we want to call out, in hopes that others
will pick it up?
== Updates ==
Name:
This week:
- What you worked on this week.
Next week:
- What you are planning to work on next week.
Help with:
- Something you need help with.
cecylia (cohosh): last updated 2022-01-20
Last week:
- commented on snowflake metrics churn
- lots of reviews
- look at obfs4 vulnerability
- documented reachability probe tests for shell
- gave shell access to probe site in china
- created MR to close NAT check peer connections snowflake-webext!25
This week:
- more reviews
- finish documenting and cleaning up shadow simulation scripts
- use these snowflake + shadow scripts to do more performance testing
Needs help with:
dcf: 2022-01-20
Last week:
- tried to reproduce the snowflake webextension disabling screen
lock
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- did brief code inspection of probetest
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- disabled our standalone snowflake proxies to free up more CPU
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- set up another load-balanced snowflake bridge, this time with the
same identity keys
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…https://forum.torproject.net/t/tor-relays-how-to-reduce-tor-cpu-load-on-a-s…
- noticed and posted about upstream obfs4 vulnerability fixes
https://lists.torproject.org/pipermail/anti-censorship-team/2022-January/00…https://lists.torproject.org/pipermail/anti-censorship-team/2022-January/00…
Next week:
- do the load balanced snowflake bridge installation again, this
time on a production-capable host
Help with:
agix:2021-01-13
Last week:
- Busy with work on Censored Planet
Next week:
- Continue work on gettor-twitter
Help with:
-
arlolra: 2022-01-20
Last week:
- [added 2022-01-20 by dcf] ALPN support for pion DTLS
https://github.com/pion/dtls/pull/415
Next week:
- Figure out where in pion/webrtc ALPN should be configured and used
- Maybe add Chacha20Poly1305 to pion/dtls
https://github.com/pion/dtls#planned-featureshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Help with:
-
maxb: 2021-09-23
Last week:
- Worked on
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
re: utls for broker negotiation
- Had conversation with someone about upstream utls http round
tripper https://github.com/refraction-networking/utls/pull/74
- Too busy with work :/
Next week:
- _Really_ want to get a PR for utls round tripper
meskio: 2022-1-20
Last week:
- metrics bridge distributor metrics for rdsys (rdsys#67)
- don't distribute private bridges in rdsys (rdsys#84)
- plan with TPA the test deployment of bridgedb (tpa/team#40581)
- add country list to circumnvetion API (rdsys#83)
- review snowlfake's event bus (snowflake!67)
- telegram bot weekly rotation of bridges
Next week:
- select the type of bridges distributed by each distributor (rdsys#63)
- test deployment for the new rdsys/bridgedb setup (rdsys#12)
Shelikhoo: 2022-01-20
Last Week:
- [Merge Request Awaiting] Add SOCKS5 forward proxy support to
snowflake(snowflake!64)
- [Merge Request Awaiting] Multi-arch docker image support with
Dockerfile refactor for probetest(probetest!1)
- [Merge Request Awaiting] Add callbacks in the client for key
events during snowflake connections(snowflake#40076)
- [Merge Request Pending] Privacy preserving stats in Snowflake
standalone proxy(snowflake#40079)
- [Discussion] Implement metrics to measure snowflake
churn(snowflake#34075)
- [Discussion] Proposal: Support for Dynamic IP obfs4 bridges
with unattended proxy information update(aka "Subscription")
- [Discussion] Proposal: Push Notification Based Signaling Channel
- [Investigate] Is there a better moat/snowflake SNI than
cdn.sstatic.net? (snowflake#40068)
- [Investigate] Multi-instance Load Balanced Tor - Snowflake
Deployment
- [Investigate] China "Anti-Fraud" Webpage Redirection
Censorship(censorship-analysis#40026)
Next Week:
- [Discussion] Designing the Container Layout for rdsys
- [Discussion] Implement metrics to measure snowflake
churn(snowflake#34075)
- [Discussion] Proposal: Push Notification Based Signaling Channel
- [Discussion] rdsys Deployment Environment
- [Merge Request] Add callbacks in the client for key events
during snowflake connections(snowflake#40076)
- [Discussion] Proposal: Centralized Probe Result Collector
HackerNCoder: 2021-12-16
This week:
Last/done:
Setup web mirror on tor.encryptionin.space
Next:
Get (new VPs with) new IP and setup new web mirror on new domain
hanneloresx: 2021-3-4
Last week:
- Submitted MR for bridgestrap issue #14
Next week:
- Finish bridgestrap #14
- Find new issue to work on
Help with:
-
Hello,
Here are our meeting logs:
http://meetbot.debian.net/tor-meeting/2022/tor-meeting.2022-01-10-14.00.log…
And our meeting pad:
## Tor Community Team meeting pad
Next meeting: Wednesday, January 19, 2022 - 1300 UTC
## Goal of this meeting
Weekly checkin about the status of Community Team work at Tor.
## Links to Useful documents
* Looking for volunteers:
https://forum.torproject.net/t/looking-for-volunteers-to-contribute-to-the-…
* Internet shutdown in Kazakhstan:
- Metrics:
https://metrics.torproject.org/userstats-relay-country.html?start=2021-10-1…
- https://metrics.torproject.org/userstats-bridge-combined.html?start=2021-10…
## Discussion
* Gamification project update:
- main ticket:
https://gitlab.torproject.org/tpo/community/relays/-/issues/26
- Survey status
- Next steps: survey report and scheduling interviews
* Internet shutdown in Kazakhstan:
- New Support guide:
https://forum.torproject.net/t/internet-shutdown-in-kazakhstan-how-to-circu…
- https://www.reddit.com/r/onions/comments/s04xvp/sos_from_kazakhstan_need_br…
* Tor user support in Telegram:
- What's the scope?
- Onboarding, testing phase, public launch
* Bridges campaign:
- Status - network growth:
- 2021-11-17: 1252 bridges
- 2022-01-10: 2289 bridges
- TODO:
- Write a blog post wrapping up the campaign
- Keep testing bridges till EOY
- Delivery the first reward on January
## Updates
Name:
Last week:
- What you worked on last week.
This week:
- What you are you planning to work on this week.
Help with:
- Something you need help with.
Gus:
This week:
- Writing Bridges campaign wrap up blog post
- Bridge Campaign: contacting bridge operators to confirm that
their bridges are working.
- Meeting about ECC and Onion-SRE positions
- Monitoring Telegram bridges reachability in Russia:
https://gitlab.torproject.org/tpo/community/support/-/issues/40054
- Russia: contacting operators to rotate their bridges' IP addresses. Working on a template.
- Sponsor9: writing up new project phase (June 2022 - June 2023).
- Setting up telegram support channel with nina
- Outreachy: Wrapping up relay operator survey with miko
Joydeep:
This week:
- Backlog of frontdesk tickets
- Gitlab to-do/reviews:
https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&state=opened&as…
Nina:
This week:
- Responding to user queries (from Russian and KZ);
- Translation if necessary;
- Adding new article on access to the onion services resources.
Hepl with:
- Help with resolving some tickets.
Miko:
This week:
- Starting data analysis on UX survey for relay operators
gamification project
https://gitlab.torproject.org/tpo/community/relays/-/issues/27
- Sharing a competitive audit report on Tor's competitors and
other volunteer-based gamification strategies active
- Setting up UX interviews with Tor Relay Operators who opted in
Help with:
- Looking for suggestions for good gamification examples to
include in the audit report
Next week:
- Gathering feedback on competitive audit report and UX survey
analysis
emmapeel:
This week:
- Come back from vacations, check on translations status
- move to weblate
- translations preview if possible with lavamind
--
The Tor Project
Community Team Lead
Hi everyone,
Given the holiday on Monday, we'll be moving the weekly Tor Browser
meeting to Tuesday the 18th (same time, same place).
Meeting pad: https://pad.riseup.net/p/tor-tbb-keep
best,
-Richard
Hello,
OONI's December 2021 status report is shared below.
*# OONI Monthly Report: December 2021*
Throughout December 2021, the OONI team worked on the following sprints:
* Sprint 53 (1st-5th December 2021)
* Sprint 54 (6th-19th December 2021)
Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni
Highlights are shared in this report below.
*## Published report on Tor blocking in Russia*
We published a report documenting the blocking of Tor in Russia based on
OONI data.
Read the report here: https://ooni.org/post/2021-russia-blocks-tor/
Our report shares OONI data and analysis on the blocking of the Tor network
and Tor Project website on some AS networks in Russia.
Wired also published a story on Tor blocking in Russia, sharing OONI data
and findings: https://www.wired.co.uk/article/russia-block-tor-censorship
*## Year in Review: OONI in 2021*
In December 2021, we published our annual end-of-year post, "Year in
Review: OONI in 2021", which shares:
* OONI highlights from 2021
* Exciting OONI activities by community members in 2021
* Upcoming OONI projects in 2022
You can read this post here: https://ooni.org/post/ooni-in-2021/
Warm thanks to our amazing community for making our work possible!
We thank every OONI Probe user out there, and we're excited for 2022!
*## Published blog post about OONI Probe integration into iThena*
iThena (https://root.ithena.net/) is a distributed computation and
measurement project based on the Berkeley Open Infrastructure for Network
Computing (BOINC) platform.
In December 2021, we published a blog post where we discussed how iThena
helped boost OONI censorship measurement coverage worldwide by integrating
OONI Probe into their platform.
This blog post is available here:
https://ooni.org/post/2021-ithena-boosts-ooni-measurement-coverage/
*## Published guest post by The Bachchao Project*
In October 2021, we participated in the Bachchao Project’s 2-day event
(“India, Let’s Build the List”) where we helped facilitate sessions for
communities in India to update the Citizen Lab’s test list for India.
In December 2021, we published the Bachchao Project’s guest blog post about
the event on the OONI website:
https://ooni.org/post/2021-event-india-test-list/
*## OONI Probe Mobile*
We released OONI Probe Mobile 3.5.0 for Android (
https://github.com/ooni/probe-android/releases/tag/v3.5.0) and iOS (
https://github.com/ooni/probe-ios/releases/tag/v3.5.0).
This release includes the following improvements:
* Automated testing for instant messaging app tests (WhatsApp, Facebook
Messenger, Signal, Telegram)
* Automated testing for circumvention tool tests (Tor, Psiphon, RiseupVPN)
* Automated testing requires at least 20% of battery level
* OONI Probe CLI updated to 3.13.0
* Bug fixes
Following this release, OONI Probe mobile app users can run all OONI Probe
tests (excluding the bandwidth-intensive performance tests) automatically
on a regular basis. We expect that this will not only help boost OONI
measurement coverage further, but it will also be helpful in ensuring that
social media apps and circumvention tools (both of which are frequently
blocked around the world) are more regularly tested.
We started making improvements to how we do testing of the OONI Probe apps
by doing integration testing of all components inside of a mono-repo (
https://github.com/ooni/probe/issues/1926). We also worked towards
addressing OONI Probe technical debt (
https://github.com/ooni/probe/issues/1808).
*## OONI Probe Web*
We now have a working prototype of OONI Probe Web (
https://github.com/ooni/probe-web) that implements the full functionality.
Features of OONI Probe Web currently include:
* Informed consent onboarding process (
https://github.com/ooni/probe-web/pull/1);
* Communication with the check-in API for receiving test-list targets;
* Submission of measurements to the testing collector (
https://github.com/ooni/probe/issues/1494).
Unlike the OONI Probe apps, OONI Probe Web is limited to website testing.
The prototype includes a feature that enables users to determine if they
would like to limit their testing to a specific number of URLs.
*## Integrating tor into OONI Probe*
We completed the integration of tor into OONI Probe Mobile and OONI Probe
Desktop (https://github.com/ooni/probe/issues/1867). It is now possible to
use go-libtor when it's available. We switch between using go-libtor or
searching for the tor binary in the path by using build constraints. The
branch that integrates this work is the following:
https://github.com/ooni/probe-cli/pull/614
We also integrated tor into OONI Probe Desktop (
https://github.com/ooni/probe-desktop/pull/261) and into OONI Probe CLI (
https://github.com/ooni/probe/issues/1872,
https://github.com/ooni/probe/issues/1866).
*## Expanding censorship measurement methodologies*
We released OONI Probe CLI 3.13 (
https://github.com/ooni/probe-cli/releases/tag/v3.13.0) through which users
can now run our new DNScheck experiment (
https://github.com/ooni/spec/blob/master/nettests/ts-028-dnscheck.md). This
release (along with our 3.13-beta and 3.14-alpha releases made in December
2021) includes a series of other improvements to our experiments and
measurement methodologies (https://github.com/ooni/probe-cli/releases), as
well as updated dependencies.
*## OONI Measurement Aggregation Toolkit (MAT)*
We continued to make improvements to the OONI Measurement Aggregation
Toolkit (MAT).
In particular, we improved the usability of the one-dimensional charts, we
fixed a tooltip rendering issue, we resized the panels for the charts and
table, we fixed issues with non-measurement-start-day based charts, we
fixed the sorting order of the table, and we worked on other UI
improvements (https://github.com/ooni/explorer/issues/624). We also added
support for displaying the various blocking types (DNS, TCP/IP,
HTTP-failure, HTTP-diff) as an axis value in the MAT charts. Improvements
to the MAT can be viewed through commits made in the following pull
request: https://github.com/ooni/explorer/pull/622
*## OONI backend*
In December 2021, we continued to make progress on migrating our backend
infrastructure to Clickhouse (https://github.com/ooni/pipeline/pull/371).
To this end, we worked on:
* Switching more tables over to Clickhouse and improving test coverage;
* Implementing Clickhouse in our fastpath data processing pipeline;
* Implementing the Clickhouse feeder;
* Updating JSONL table in Clickhouse from feeder;
* Updating CI to test Clickhouse;
* Splitting the platform in tiers;
* Adding a deletion flag on the uploader;
* Re-importing the JSONL table and rebuilding the fastpath table with the
domain column.
We also worked on the following backend activities:
* Implemented an integrated API/fastpath multi-host dashboard;
* Added backend support for Tor Snowflake experiment feature extraction (
https://github.com/ooni/pipeline/pull/372);
* Added backend support for extracting Tor Snowflake runtime metrics and
exposing them via a new API endpoint (https://github.com/ooni/api/pull/274);
* Handled upcoming VM reboots;
* Continued to monitor OONI measurement coverage from unattended runs on
OONI Probe Mobile and Desktop.
*## Established new partnership with Zaina Foundation*
In December 2021, we established a new partnership with Tanzania's Zaina
Foundation (https://zainafoundationtz.org/).
Over the past year, we had already been collaborating with Zaina
Foundation, who facilitated OONI training for women human rights defenders
in Tanzania, conducted usability studies for OONI Probe and OONI Explorer,
and translated OONI Probe to Swahili. We are therefore thrilled to have now
formalized our partnership with Zaina Foundation, and we’re excited for our
ongoing collaboration over the next few years.
To highlight Zaina Foundation’s important work, we published a new page on
the OONI Partners page: https://ooni.org/partners/zaina-foundation/
*## OONI website### New Donate page*
In December 2021, we published a new Donate page on the OONI website:
https://ooni.org/donate/
Through this page, we aim to encourage donations in support of OONI’s work.
*### Edited various OONI webpages*
We edited and updated the copy of several web pages on the OONI site. In
particular, we edited the About page (
https://github.com/ooni/ooni.org/pull/1041), we updated the Frequently
Asked Questions (FAQ) section (https://github.com/ooni/ooni.org/pull/1042),
and we restructured and edited the Nettests section (
https://github.com/ooni/ooni.org/pull/1043). We also
*## Hosting an OTF Information Controls Fellow*
As of December 2021, we started to serve as the host organization for Open
Technology Fund (OTF) Information Controls research fellow, Gurshabad
Grover (https://twitter.com/OpenObservatory/status/1471454369866719237).
During his 1-year research fellowship with us, Gurshabad aims to uncover
how ISPs, through their technical decisions, can exacerbate or minimize the
effects of state-ordered censorship.
*## Community activities### PiProbe*
Our Venezuelan partners, Venezuela Inteligente, created a new OONI Probe
distribution for Raspberry Pis, called “PiProbe” (
https://github.com/VEinteligente/PiProbe). On 15th December 2021, they
presented PiProbe to collect feedback from the OONI team and broader
community.
*## Userbase*
In December 2021, 39,609,318 OONI Probe measurements were collected from
2,785 AS networks in 165 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/
~ OONI team.
Hello,
OONI's November 2021 status report is shared below.
*# OONI Monthly Report: November 2021*
Throughout November 2021, the OONI team worked on the following sprints:
* Sprint 51 (1st - 7th November 2021)
* Sprint 52 (8th - 21st November 2021)
Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni
Highlights are shared in this report below.
*## Published research report investigating internet shutdowns through
Mozilla telemetry*
In collaboration with IODA, we published a new research report which
analyzes Mozilla telemetry to investigate internet shutdowns in Myanmar,
Uganda, Belarus, and Iran over the last year.
Our research report is available here:
https://ooni.org/post/2021-investigating-internet-shutdowns-mozilla-telemet…
Mozilla also published a blog post about our report:
https://blog.mozilla.org/data/2021/11/08/detecting-internet-outages-with-mo…
The goal of our research was to evaluate whether Mozilla telemetry could be
a valuable resource for the Internet freedom community to investigate
Internet shutdowns. To this end, we analyzed an aggregated dataset of
network activity based on Firefox web browser usage worldwide (access to
which was provided to us by Mozilla).
To evaluate how useful Mozilla telemetry is for researching Internet
shutdowns, we:
* Selected case studies (known shutdown events from January 2020 onwards)
* Analyzed Mozilla telemetry
* Compared Mozilla telemetry with other public datasets
We checked whether Mozilla telemetry provides signals for:
* Myanmar: Internet outages following February 2021 military coup
* Uganda: Nationwide internet outage amid 2021 elections
* Belarus: Internet outages amid 2020 elections
* Iran: Internet outages following 2020 elections
Our detailed findings are documented in our report. Overall, our analysis
demonstrates that Mozilla telemetry is a valuable resource for
investigating Internet shutdowns worldwide, providing novel insights
through geographical data granularity.
We encourage Mozilla to publish aggregated Mozilla telemetry as open data
to support research and advocacy efforts investigating Internet shutdowns
worldwide. We also share further recommendations in our report:
https://ooni.org/post/2021-investigating-internet-shutdowns-mozilla-telemet…
*## Published blog post on Myanmar research paper findings*
A few months ago, the OONI team collaborated with researchers at IODA,
Kentik, UC San Diego, and University of Michigan / Censored Planet on a
research paper examining censorship events in Myanmar (through diverse
measurement datasets) following the military coup.
The paper, titled "A multi-perspective view of Internet censorship in
Myanmar", was published by FOCI 2021:
https://dl.acm.org/doi/pdf/10.1145/3473604.3474562
In November 2021, we published a blog post which summarizes the research
paper findings:
https://ooni.org/post/2021-multiperspective-view-internet-censorship-myanma…
*## Blog post for Internet Society (ISOC)*
In November 2021, our partner, Internet Society (ISOC), published our guest
blog post where we discuss why collaboration and transparency is key to
internet measurement:
https://pulse.internetsociety.org/blog/internet-measurements-collaboration-…
This blog post was also cross-posted on the OONI blog:
https://ooni.org/post/2021-internet-measurements-collaboration-transparency/
*## OONI Probe Mobile*
We released OONI Probe Mobile 3.4.0 for Android (
https://github.com/ooni/probe-android/releases/tag/v3.4.0) and iOS (
https://github.com/ooni/probe-ios/releases/tag/v3.4.0).
This release includes the following new features and improvements:
* New label indicating when a VPN is being used
* Improved UI for requesting autorun permission (
https://github.com/ooni/probe/issues/1820)
* Logs are retained for 1 week
* Bug fix in RiseupVPN and Signal analysis logic
We also released OONI Probe Mobile 3.4.1 with further bug fixes and
improvements (https://github.com/ooni/probe-android/releases/tag/v3.4.1).
Following the release of OONI Probe Mobile 3.4.0, we started to observe an
increase in the volume of measurements contributed by automated runs. The
new label which indicates the use of a VPN will hopefully also help improve
OONI data quality.
We also worked on improvements to our build system for OONI Probe so that
it can be further automated through continuous integration. In particular,
we added support to build the Android and iOS dependencies as part of
continuous integration.
*### Interviewed mobile developer candidates*
In September 2021, we published a job opening to hire a new mobile
developer for OONI Probe (
https://ooni.org/post/2021-job-opening-ooni-mobile-developer/). We reviewed
applications and throughout November 2021, we interviewed candidates.
## OONI Run
Based on community feedback, we made improvements to the OONI Run (
https://run.ooni.io/) platform. Specifically, we rewrote the URL editor (
https://github.com/ooni/run/pull/74), we limited the OONI Run platform to
website testing (https://github.com/ooni/run/pull/79), and we deployed the
changes to production.
*## Integrating tor into OONI Probe*
We made progress on integrating tor into OONI Probe Mobile and OONI Probe
Desktop. In particular, we worked on adding support for building go-libtor
on Android, iOS, and Linux: https://github.com/ooni/probe/issues/1867
*## Expanding censorship measurement methodologies*
Two essential steps required for shipping our new Websteps experiment (for
measuring the blocking of websites) include:
* Comparing our new Websteps experiment with our current Web Connectivity
experiment;
* Deploying the new Websteps test helper in production.
In November 2021, we made considerable progress on both. To compare
Websteps with Web Connectivity, we developed test cases to measure accuracy
and to compare the two experiments in terms of bandwidth used, overall test
runtime, and accuracy at detecting blocking events (
https://github.com/ooni/probe/issues/1805). We also worked on polishing the
Websteps design documents and we shared them with community members who
have relevant expertise, in order to receive their feedback for the
improvement of our new experiment (https://github.com/ooni/probe/issues/1811).
We worked towards deploying and using the new Websteps test helper (
https://github.com/ooni/backend/issues/544). We started off with an
incremental rollout where only a few ASNs use the new test helper, and we
eventually plan to replace the old test helper.
We continued to investigate TLS parroting and interference (
https://github.com/ooni/probe/issues/1812), and we made several OONI Probe
CLI releases which include a series of improvements to our experiments
(improving the data quality of measurements), as documented through the
following releases:
https://github.com/ooni/probe-cli/releases/tag/v3.11.0-beta.2https://github.com/ooni/probe-cli/releases/tag/v3.12.0-alpha.3https://github.com/ooni/probe-cli/releases/tag/v3.12.0-alpha.2https://github.com/ooni/probe-cli/releases/tag/v3.11.0-beta.1https://github.com/ooni/probe-cli/releases/tag/v3.12.0-alpha.1
*## OONI Explorer*
We added support to the OONI Explorer Search page for filtering based on
failed measurements (https://github.com/ooni/explorer/pull/623). We also
worked towards making OONI Explorer easier to debug (to improve the quality
of OONI Explorer, and to make the MAT easier to debug) by improving how we
log and report errors in OONI Explorer.
*## OONI Measurement Aggregation Toolkit (MAT)*
We continued to make improvements to the OONI Measurement Aggregation
Toolkit (MAT). We measured the performance of the MAT UI components to
identify bottlenecks. We worked on improving the interactivity and
filtering of the table chart, we fixed the presentation of the horizontal
axis of the chart, we worked on improvements to one-dimensional charts, and
we worked towards making measurable performance improvements to the
two-dimensional (2-axis) chart pages with large datasets (
https://github.com/ooni/explorer/pull/622).
*## Building a web platform for test list updates*
We continued to make progress on the web platform (
https://test-lists.ooni.org/) that we’re building to enable community
contributions to the Citizen Lab test lists (
https://github.com/citizenlab/test-lists).
In particular, we made improvements to display the contributor and the date
of when a URL was last updated in the list view (
https://github.com/ooni/test-lists-ui/pull/15), we added a notifications
feature (https://github.com/ooni/test-lists-ui/pull/16) and a favicon (
https://github.com/ooni/test-lists-ui/pull/17), and we fixed a backend bug
affecting the platform’s submit button (
https://github.com/ooni/backend/issues/545).
*## OONI backend*
In November 2021, we started migrating our infrastructure over to
Clickhouse (an alternative database solution).
To this end, we reviewed the architecture by:
* Implementing Clickhouse in our fastpath data processing pipeline;
* Switching the fasthpath and API CI to bullseye;
* Implemented the experimental Measurement Aggregation Toolkit (MAT) using
Clickhouse, and added relevant tests;
* Implemented the experimental PG to the Clickhouse feeder;
* Updating the CI to test Clickhouse;
* Determining what to keep in PG (failure modes and recovery);
* Preparing an architecture diagram;
* Splitting the platform in tiers;
* Using raw SQL or SQLalchemy;
* Implementing an experimental PG to the Clickhouse feeder;
* Performing tests by using Clickhouse for the measurements API;
* Handling JSONL.
We also worked on the following backend activities:
* Reviewed the new monitoring host, and removed the old host;
* Handled unplanned and upcoming VM reboots;
* Investigated an issue affecting the timing of measurements (probe clock
issues);
* Fixed a bug resulting in probe failures (
https://github.com/ooni/probe/issues/1328);
* Started reviewing and cleaning up DNS records used by OONI;
* Tested switching some ASNs to the new test helper;
* Experimented with centralized logging management;
* Continued to monitor OONI measurement coverage from unattended runs on
OONI Probe Mobile and Desktop (particularly in light of the recent iThena
integration of OONI Probe).
*## Collaboration with Netalitica*
Netalitica researchers continued to do excellent work in reviewing and
updating the Citizen Lab test lists. In November 2021, we reviewed their
updates to the test lists for Nicaragua and Ecuador and shared feedback for
improvements.
*## Test list updates*
In November 2021, we contributed minor updates to the following Citizen Lab
test lists:
* Egypt: https://github.com/citizenlab/test-lists/pull/863
* Global: https://github.com/citizenlab/test-lists/pull/862
We also opened a pull request with updates to the test list for India (
https://github.com/citizenlab/test-lists/pull/864), contributed by
participants of the Bachchao Project’s 2-day test list event (
https://ooni.org/post/2021-event-india-test-list/).
*## Notable community use of OONI Probe and OONI data### OONI Probe
integrated into Murakami*
In November 2021, Measurement Lab (M-Lab) integrated OONI Probe into their
Murakami measurement tool (https://www.measurementlab.net/blog/murakami/).
This is documented through the following pull request:
https://github.com/m-lab/murakami/pull/103
We thank M-Lab for integrating OONI Probe into Murakami, as this will help
boost OONI measurement coverage!
*### VEsinFiltro report on censorship during 2021 regional elections in
Venezuela*
In November 2021, VEsinFiltro published a report documenting the blocking
of independent news media websites, political criticism sites, human rights
sites, multimedia dissemination sites, and circumvention tool sites during
the 2021 regional elections in Venezuela. Their report makes use of OONI
data, among other data sources.
VEsinFiltro’s report is available in:
* English: https://vesinfiltro.com/noticias/2021-election-blocks/
* Spanish: https://vesinfiltro.com/noticias/2021-bloqueos-elecciones/
*## Community activities### OONI presentation at ISOC Community Week 2021*
On 16th November 2021, OONI’s Maria presented OONI as part of the Internet
Society’s (ISOC) Community Week event.
Information about the ISOC Community Week 2021 is available here:
https://www.internetsociety.org/events/community-week/
*### OONI presentation at Tor’s State of the Onion 2021*
On 17th November 2021, OONI’s Arturo presented OONI as part of the Tor
Project’s annual State of the Onion 2021 event (
https://blog.torproject.org/state-of-the-onion-2021), during which he
shared OONI highlights from 2021, as well as upcoming OONI projects for
2022.
The event was live-streamed on YouTube:
https://www.youtube.com/watch?v=mNhIjtXuVzk
*## OONI training for civil society in the Gambia*
On 29th November 2021, OONI’s Maria facilitated an (online) OONI training
for human rights defenders in the Gambia in preparation for the country’s
2021 elections.
*### OONI Community Meeting*
On 30th November 2021, we hosted the monthly OONI Community Meeting on our
Slack channel (https://slack.ooni.org/), during which we discussed the
following topics:
1. OONI research reports published in November 2021.
2. Iranian government is using a new type of regional Internet censorship
during protests which blocks International internet access (except few
services) but allows local websites and services. It would be nice if an
OONI test could be developed to detect & log this scenario and submit the
result to OONI once the international connectivity is fully restored. This
is tracked through the following epic:
https://github.com/ooni/probe/issues/1600
*## Userbase*
In November 2021, 31,393,906 OONI Probe measurements were collected from
2,276 AS networks in 157 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/
~ OONI team.
Hello,
Happy new year! Hope you're all doing well.
These days we're primarily supported by the DRL, which is why we do
quarterly reporting. That said, we'd like to share monthly updates from the
OONI team with the community, hence our October 2021 report shared below.
I'll also follow-up to share our November and December 2021 reports.
*# OONI Monthly Report: October 2021*
Throughout October 2021, the OONI team worked on the following sprints:
* Sprint 49 - Humpback whale (1st - 10th October 2021)
* Sprint 50 - Amphipoda (11th - 24th October 2021)
Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni
Highlights are shared in this report below.
*## Published research report on the blocking of the Signal Private
Messenger App *
We published a research report which shares OONI data and analysis on the
blocking of the Signal Private Messenger App in Iran, China, Cuba, and
Uzbekistan.
This report is available here:
https://ooni.org/post/2021-how-signal-private-messenger-blocked-around-the-…
In April 2021, we released a new OONI Probe test for measuring the blocking
of the Signal Private Messenger App (https://ooni.org/nettest/signal).
Since then, OONI community members tested Signal worldwide every day,
contributing measurements as open data:
https://explorer.ooni.org/search?until=2021-10-23&since=2021-09-23&test_nam…
As part of this study, we analyzed all Signal measurements collected from
around the world between April 2021 to September 2021. During this period,
we found that the testing of Signal primarily presented signs of blocking
in Iran, China, Cuba, and Uzbekistan. Further information and findings are
available in our research report.
*## OONI Probe Mobile*
We released OONI Probe Mobile 3.3.1 for Android (
https://github.com/ooni/probe-android/releases/tag/v3.3.1) which includes
bug fixes and optimizations.
In particular, we:
* Fixed an Android bug related to our RiseupVPN test:
https://github.com/ooni/probe/issues/1787
* Fixed an Android bug related to automated runs:
https://github.com/ooni/probe/issues/1789
* Fixed an Android bug related to our use of Sentry for crash reports:
https://github.com/ooni/probe/issues/1788
We also worked on:
* Adding support for detecting if an OONI Probe user is behind a VPN:
https://github.com/ooni/probe/issues/954
* Adding support for a VPN label and modal:
https://github.com/ooni/probe-android/pull/453
* Deleting logs weekly (instead of immediately):
https://github.com/ooni/probe/issues/1800
*## OONI Probe Desktop*
We released OONI Probe Desktop 3.6.0:
https://github.com/ooni/probe-desktop/releases/tag/v3.6.0
This release includes our new Experimental card, which currently features
our STUN reachability test. We plan to release new experimental tests as
part of this new card.
OONI Probe Desktop 3.6.0 also includes support for 2 new languages: Swahili
and Dutch. We thank the Localization Lab community for ongoing OONI Probe
translations, and we are particularly grateful to Tanzania’s Zaina
Foundation for leading efforts related to translating OONI Probe to Swahili
(and increasing the reach of our app in East Africa).
We also released OONI Probe Desktop 3.6.1 (
https://github.com/ooni/probe-desktop/releases/tag/v3.6.1) with bug fixes
and further improvements.
Moreover, we worked on improvements towards a new OONI Probe CLI release (
https://github.com/ooni/probe/issues/1843).
*## Expanding censorship measurement methodologies*
Shipping our new Websteps experiment (
https://github.com/ooni/probe/issues/1733) requires deploying the new
Websteps test helper in production. To this end, we worked on comparing the
old with the new OONI Probe test helper (
https://github.com/ooni/probe/issues/1707). This required extensive testing
and investigation to ensure that no issues emerge when we deploy the new
test helper.
We now have some TLS parroting capabilities in our miniooni research tool (
https://github.com/ooni/probe-cli#miniooni), and we started investigating
TLS parroting and interference (https://github.com/ooni/probe/issues/1812).
*## OONI Explorer*
We deployed recently merged OONI Explorer pull requests to production, and
we ensured that anomaly search is included for all supported tests
(including the Signal and RiseupVPN tests). We also worked towards
addressing issues that result in failed measurements not being displayed
correctly (https://github.com/ooni/explorer/issues/611).
Moreover, we refactored OONI Explorer pages to remove the need for a custom
server (https://github.com/ooni/explorer/pull/619), and we upgraded core
dependencies (react, next) to the latest versions (
https://github.com/ooni/explorer/pull/620,
https://github.com/ooni/explorer/pull/621).
*## OONI Measurement Aggregation Toolkit (MAT)*
We made several improvements to the OONI Measurement Aggregation Toolkit
(MAT). Specifically, we improved the MAT table view by adjusting the
tooltip UI, sorting labels alphabetically, and deploying progress to
production.
To add interactivity features to the MAT table view, we made filtered table
rows the source for charts, and we fixed the link from the tooltip link to
the search page (https://github.com/ooni/explorer/pull/618), among several
other improvements. To improve the performance of the MAT, we switched to
the Clickhouse-based backend for the MAT pages (
https://github.com/ooni/explorer/pull/622).
*## Building a web platform for test list updates*
We continued to make progress on the web platform that we’re building to
enable community contributions to the Citizen Lab test lists (
https://github.com/citizenlab/test-lists).
In particular, we made a series of UI improvements based on initial
community feedback (from the testing of the private beta version of the
platform). We added an onboarding section to guide users through the
process of using the test list web platform (
https://github.com/ooni/test-lists-ui/issues/5,
https://github.com/ooni/test-lists-ui/pull/11). We reduced the ambiguity
between the “notes” and “comment” sections of the web platform (
https://github.com/ooni/test-lists-ui/issues/7,
https://github.com/ooni/test-lists-ui/pull/9). We also improved upon error
messages (https://github.com/ooni/test-lists-ui/pull/8) and added support
for displaying website category descriptions (
https://github.com/ooni/test-lists-ui/pull/10).
*## OONI backend*
In October 2021, we worked on the following OONI backend activities:
* Deployed the new monitoring host;
* Created a set of benchmark queries, and tested BigQuery (as a potential
alternative database) by running benchmark queries against the whole
dataset;
* Continued to investigate alternative database solutions to boost the
performance of our services by comparing Clickhouse, DuckDB, PostgreSQL 13,
and BigQuery;
* Implemented an experimental PG to Clickhouse feeder;
* Added backend support to the Measurement Aggregation Toolkit (MAT) using
the Clickhouse-based backend;
* Continued to monitor OONI measurement coverage from unattended runs on
OONI Probe Mobile and Desktop;
* Updated DNS handling in ansible and added graphs;
* Continued to investigate missing jsonl files from the s3 bucket (
https://github.com/ooni/backend/issues/538).
*## Partnership with Internet Society (ISOC)*
In July 2021, we established a new partnership with Internet Society (ISOC)
to collaborate on the ISOC Pulse project:
https://pulse.internetsociety.org/shutdowns
In particular, OONI is now a data partner (
https://pulse.internetsociety.org/partners) for ISOC’s Pulse project on
internet shutdowns. Our goal is to provide relevant OONI data and analysis
that can help support the project’s timeline on blocking events around the
world.
In October 2021, ISOC published a blog post which announced OONI as a new
data partner for their Pulse project, and which explained why OONI data
will support their project. Their blog post is available here:
https://pulse.internetsociety.org/blog/welcome-cloudflare-ooni-and-w3techs
We also updated the OONI Partners page on our website to feature ISOC as
one of our new partners and to highlight their work:
https://ooni.org/partners/internet-society
*## Collaboration with Netalitica*
Netalitica researchers continued to do excellent work in reviewing and
updating the Citizen Lab test lists. Based on our feedback, they made
further improvements to the test lists for India and Burundi, and we opened
a pull request for both test list updates:
https://github.com/citizenlab/test-lists/pull/829
*## Community use of OONI data### Report on social media blocks in Sudan*
Community members from Sudan published a report which documents the
blocking of social networking sites in Sudan based on OONI data (
https://twitter.com/ga800l/status/1449385940175265792).
Their report is available here:
https://www.beamreports.com/2021/10/16/%D9%85%D8%A7-%D8%AD%D9%82%D9%8A%D9%8…
*## Community activities### DIG Festival 2021*
On 3rd October 2021, OONI’s Arturo participated in a panel discussion at
the DIG Festival for investigative journalism in Modena, Italy (
https://dig-awards.org/en/dig-festival-2021-english-only/).
As part of this panel, Arturo discussed OONI’s censorship measurement
approach and algorithmic censorship.
*### Bachchao Project test list event*
On 9th and 10th October 2021, OONI’s Maria participated in the Bachchao
Project’s 2-day event (organized for communities in India), titled “India,
Let’s Build the List”.
As part of her participation, Maria gave a 1-hour presentation
(“Introduction to Internet censorship”) on Day 1 to introduce participants
to basic concepts around internet censorship. On Day 2 of the event, Maria
demoed OONI’s new web platform for updating test lists, and helped
facilitate the hands-on sessions for updating the Citizen Lab’s test list
for India.
Information about the 2-day event is available via the Bachchao Project’s
relevant blog post:
https://thebachchaoproject.org/event-report-india-lets-build-the-list/
*### OPTIMA 2.0 Network Measurement Training*
On 21st and 28th October 2021, OONI’s Maria attended Internews’ live
sessions for the “Introduction to Network Measurement” and “OONI” training
modules of the OPTIMA 2.0 training program, where she addressed participant
questions pertaining to the training modules.
Information about Internews’ OPTIMA Network Measurement Training program is
available here:
https://internews.org/resource/optima-network-measurement-training/
*### Swahilipot Hub event for HacktoberFest Mombasa*
On 22nd October 2021, OONI’s Maria facilitated an OONI workshop (titled
“How to measure Internet censorship”) for Kenyan civil society as part of
the Swahilipot Hub event for HacktoberFest Mombasa (
https://swahilipothub.co.ke/).
*### Ford Foundation Retreat Webinar*
On 28th October 2021, OONI’s Maria participated as a speaker on the Ford
Foundation’s Retreat webinar titled “Designing the Internet We Want”. As
part of her participation, Maria highlighted the importance of censorship
measurement for a free and open internet.
*### OONI Community Meeting*
On 26th October 2021, we hosted the monthly OONI Community Meeting on our
Slack channel (https://slack.ooni.org/), during which we discussed the
following topics:
1. OONI’s research report on the blocking of the Signal app in China, Cuba,
Iran, and Uzbekistan.
2. Improving the OONI Probe sharing button.
3. Featuring latest censorship events and issues (such as those tracked on
GitHub) through the OONI Probe app.
*## Userbase*
In October 2021, 24,910,212 OONI Probe measurements were collected from
2,410 AS networks in 165 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/
~ OONI team.