Hi! Below is my March’25 (Period: 2025-03-01 - 2025-03-31) report!
In March, I resolved 156 tickets:
* On Telegram (@TorProjectSupportBot) – 72 (68 from farsi-speaking
users, 4 from chinese-speaking users);
* On RT (frontdesk@tpo) – 84 (2 from farsi-speaking users, 82 from
chinese-speaking users);
* On WhatsApp (+447421000612) - 0;
* And on Signal (+17787431312) - 0;
I completed the documentation about using pluggable transports with
little-t tor[1].
Became familiar with various types of …
[View More]user-reported issues and learned
how to handle them effectively on RT and CDR.Link. Additionally,
translated some of user support templates to Farsi.
Learned how to reply to certain types of tickets from Chinese-speaking
users. Understood the importance of using tags in CDR tickets and made
an effort to apply them.
Tried to put the things I learned from user support overview
documentation into action by following it to my responses to the
tickets.
Getting the permission of moderator for Tor Forum and reviewing the
related documentation and learn more about Tor Forum.
Thanks,
Haidi
[1]
https://support.torproject.org/little-t-tor/tor-pluggable-transports/
[View Less]
Hey everyone!
Here are our meeting logs:
http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-04-03-16.00.html
And our meeting pad:
Anti-censorship work meeting pad
--------------------------------
Anti-censorship
--------------------------------
Next meeting: Thursday, April 10 16:00 UTC
Facilitator: shelikhoo
^^^(See Facilitator Queue at tail)
Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC
(channel is logged while meetings are in progress)
This week's …
[View More]Facilitator: onyinyang
== Goal of this meeting ==
Weekly check-in about the status of anti-censorship work at Tor.
Coordinate collaboration between people/teams on anti-censorship at the
Tor Project and Tor community.
== Links to Useful documents ==
* Our anti-censorship roadmap:
*
Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards
* The anti-censorship team's wiki page:
*
https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home
* Past meeting notes can be found at:
* https://lists.torproject.org/pipermail/tor-project/
* Tickets that need reviews: from projects, we are working on:
* All needs review tickets:
*
https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s…
* Project 158 <-- meskio working on it
*
https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na…
== Announcements ==
*
== Discussion ==
* Give @renovate-bot guest access to anti-censorship group for it
to use dependency proxy src shell
*
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
* shelikhoo did enable it on monday
* it did not produce MRs in unexpected repos
* Should we move amp library to ptutil?
* we discussed this internally but didn't come to a decision
* conjure will enable registration through amp cache as of this
MR:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj…
* Upstream changes to conjure station actually implement
ampcache logic, pointing to snowflake's amp library:
https://github.com/cohosh/conjure/pull/1/commits
* Leave it for now, build a signalling channel library that
includes it later
* start with ampcache
== Actions ==
== Interesting links ==
*
== Reading group ==
* We will discuss "Differential Degradation Vulnerabilities in
Censorship Circumvention Systems" on April 3
* https://arxiv.org/abs/2409.06247
* Questions to ask and goals to have:
* What aspects of the paper are questionable?
* Are there immediate actions we can take based on this work?
* Are there long-term actions we can take based on this work?
* Is there future work that we want to call out in hopes
that others will pick it up?
== Updates ==
Name:
This week:
- What you worked on this week.
Next week:
- What you are planning to work on next week.
Help with:
- Something you need help with.
cecylia (cohosh): 2025-04-03
Last week:
- fixed caching for snowflake shadow integration tests
(snowflake#40457)
- fixed rust installation bug in shadow CI tests (snowflake#40456)
- added option in conjure pt to switch between three supported
transports (conjure#10)
- set up a test conjure registration server
This week:
- support conjure work
- follow up on snowflake rendezvous failures
- reduce/remove use of mutexes for broker metrics
(snowflake#40458)
- take a look at potential snowflake orbot bug
- https://github.com/guardianproject/orbot-android/issues/1183
dcf: 2025-04-03
Last week:
- archived snowflake-webextension-0.9.3
https://archive.org/details/snowflake-webextension-0.9.3
Next week:
- open issue to have snowflake-client log whenever KCPInErrors
is nonzero
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- parent:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Help with:
meskio: 2024-04-03
Last week:
- create a container for bridgestrap
- investigate why email is not going out in gettor/email
distributor (tpa/team#42109)
- lyrebird: support dual stack IPv4/v6 in non-linux
(lyrebird#40023)
- update obfs4-bridge docker image (docker-obfs4-bridge#21)
Next week:
- steps towards a rdsys in containers (rdsys#219)
Shelikhoo: 2024-04-03
Last Week:
- [Testing] Unreliable+unordered WebRTC data channel transport
for Snowflake rev2 (cont.)(
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
) testing environment setup/research
- Snowflake Staging Server (cont.)
Discussion(https://gitlab.torproject.org/tpo/tpa/team/-/issues/42080)
- Snowfalke Staging Server Experiment
- Vantage maintaince
- Merge request reviews
Next Week/TODO:
- Merge request reviews
- [Testing] Unreliable+unordered WebRTC data channel transport
for Snowflake rev2 (cont.)(
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
) improvements
- Snowfalke Staging Server Experiment
onyinyang: 2025-04-03
Last week(s):
- HACS/Zkproofs/OSCW/RWC
- Testing ampcache functionality with test registration server
Next week:
- Begin work on decoy registration option
- review Tor browser Lox integration
https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests…
- add TTL cache to lox MR for duplicate responses:
https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305
As time allows:
- Work on outstanding milestone issues:
- key rotation automation
Later:
pending decision on abandoning lox wasm in favour of some kind
of FFI?
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096):
- add pref to handle timing for pubkey checks in Tor browser
- add trusted invitation logic to tor browser integration:
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974
- improve metrics collection/think about how to show Lox is
working/valuable
- sketch out Lox blog post/usage notes for forum
(long term things were discussed at the meeting!):
- brainstorming grouping strategies for Lox buckets (of
bridges) and gathering context on how types of bridges are
distributed/use in practice
Question: What makes a bridge usable for a given user, and
how can we encode that to best ensure we're getting the most appropriate
resources to people?
1. Are there some obvious grouping strategies that we
can already consider?
e.g., by PT, by bandwidth (lower bandwidth bridges
sacrificed to open-invitation buckets?), by locale (to be matched with a
requesting user's geoip or something?)
2. Does it make sense to group 3 bridges/bucket, so
trusted users have access to 3 bridges (and untrusted users have access
to 1)? More? Less?
theodorsm: 2025-03-06
Last weeks:
- Implementing key_share extension support in covert-dtls
- Implementing mimicking DTLS 1.3 extensions
- Debugging Tor Build with covert-dtls:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Next weeks:
- Update MR with new covert-dtls version.
- Write instructions on how to configure covert-dtls with
snowflake client
- Fix merge conflicts in MR
(https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…).
- Condensing thesis into paper (on hold)
Help with:
- Test stability of covert-dtls in snowflake
Facilitator Queue:
onyinyang shelikhoo meskio
1. First available staff in the Facilitator Queue will be the
facilitator for the meeting
2. After facilitating the meeting, the facilitator will be moved to the
tail of the queue
--
---
onyinyang
GPG Fingerprint 3CC3 F8CC E9D0 A92F A108 38EF 156A 6435 430C 2036
[View Less]
Hi,
Tails
=====
Project 165
-----------
- Worked on navigation improvements for the Tails website (#20766)
I have a draft branch for the following issues: (!2046)
* Add a secondary navigation for the Documentation section of
our website, with accordions that work without JavaScript. (#20393)
* Add a hamburger menu for this navigation on mobile. (#20065)
* Use a bigger font everywhere on our website. (#17665)
See screenshots in attachment.
- Evaluate GNOME …
[View More]Secrets as a replacement for KeePassXC. (#19136)
The UX is decent and it would solve several integration and
maintenance issues.
- Tested anonym's branch that runs Tor Browser as flatpak and would
solve several old usability issues. (!1933)
- Tested 17 Wi-Fi adapters to debug Wi-Fi issues. (#20828)
Misc
----
- Made a case to fund a backup feature based on prior user research.
https://gitlab.torproject.org/tpo/operations/proposals/-/issues/98
- Fixed the changes that point the Donate page to Tor's. (!1968)
Tor Browser
===========
Project 163
-----------
- Ideated on how to display Conflux (aka. Multi Path Circuits) in Tor
Browser together with felicia. (applications/tor-browser#41716)
--
sajolida
The Tor Project — UX Designer
[View Less]
Hello everyone,
Much of my work last month focussed on helping users in regions where
Tor is censored, which includes helping users with instructions to
download Tor Browser binaries from GetTor and/or official mirrors,
verifying Tor Browser's GPG signature, help with using censorship
circumvention methods that works best for them and overall
troubleshooting.
I wrote (mostly small modifications)[0][1] and reviewed some changes to
the user-facing documentation on the Support Portal and Tor …
[View More]Browser User
Manual.
Worked on a couple of post-campaign tasks as we wrapped up the WebTunnel
Bridges Campaign[2].
Following is a more detailed report about the tickets our user support
team worked on last month.
# Frontdesk (email user support channel)
* 596(↑) RT tickets created
* 646(↑) RT tickets resolved
Tickets by topics and numbers:
1. 411(↑) RT tickets: instructions to circumvent censorship for Chinese
speaking users.
2. 102(↓) RT tickets: circumventing censorship in Russian speaking countries.
3. 37(↑) RT tickets: help with troubleshooting existing Tor Browser install
on Desktop (Windows, macOS and Linux).
4. 22(↑) RT tickets: WebTunnel bridges campaign wrap-up.[2]
5. 10(↑) RT tickets: questions about how Tor and Tor Browser works - is
my IP visible when using Tor? what application level protections I get
when using Tor Browser? Why Tor Browser doesn't store my cookies,
passwords and browsing history? etc.
6. 6(↑) RT tickets: Unable to login on X.com (formerly Twitter) when using Tor Browser.[3]
7. 5(↑) RT tickets: installing Tor Browser on Linux.
8. 3(↓) RT tickets: reports of fake apps on iOS AppStore masquerading as official Tor Browser.
9. 3(↑) RT ticket: reports of websites blocking Tor connections.
10. 3(↑) RT tickets: Bookmarks are failing to load on Tor Browser Alpha.[4]
11. 2(↑) RT tickets: circumventing censorship with Tor in Farsi.
12. 2(↑) RT tickets: questions about what "onionize" means on about:tor in Tor
Browser[5]
13. 1(↑) RT ticket: installing Tor Browser on ChromeOS (Chromebooks).[6]
14. 1(-) RT ticket: questions about onion services and how to access them.
15. 1(↑) RT ticket: Websites display in other languages when using Tor Browser.[7]
16. 1(↑) RT ticket: Embedded snowflake client URL redirect is giving a 404[8] (Issue has been
resolved.)
17. 1(↑) RT ticket: Email bridge distributor is not responding[9]. (Issue has been resolved.)
18. 1(-) RT ticket: instructions to download Tor Browser 13.5 legacy for legacy operating
systems.
19. 1(-) RT ticket: static Captcha when requesting bridges in Tor Browser.[10]
20. 1(↓) RT ticket: help with installing Tor Browser on Windows.
21. 1(-) ticket: question about setting up a Tor relay.
22. 1(↑) ticket: Modify about:rights (on Tor Browser) with proper
descriptions that apply to Tor Browser.[11]
# Telegram, WhatsApp and Signal user support channels
* 751(↑) tickets resolved
Breakdown:
* 734(↑) tickets on Telegram
* 15(↑) tickets on WhatsApp
* 2(↓) tickets on Signal
Tickets by topics and numbers:
1. 511(↑) tickets: circumventing censorship in Russian speaking countries.
2. 56(↑) tickets: instructions to circumvent censorship for Chinese speaking users.
3. 17(↑) tickets: circumventing censorship with Tor in Farsi.
4. 17(↑) tickets: helping users on iOS, using Onion Browser or Orbot, to
use censorship circumvention methods.
5. 12(↓) tickets: help with troubleshooting Tor Browser Desktop on Windows, macOS
and Linux.
6. 10(↑) tickets: help with troubleshooting Tor Browser Android.
7. 7(↑) tickets: help with instructions to use bridges with Tails.
8. 6(↑) tickets: reports of our Telegram Tor bridges distributor
(@GetBridgesBot) not responding. (Issue has been resolved.)
9. 4(↑) tickets: questions about onion services and how to access them.
10. 4(↑) tickets: instructions on how to get Tor Browser binaries from GetTor.
11. 4(↑) tickets: users seeing a "proxy refused" error when visiting
websites on Tor Browser for Android using Samsung devices.[12]
12. 2(↑) tickets: help with using Snowflake with Tor to circumvent censorship.
13. 2(↑) tickets: instructions to download Tor Browser 13.5 legacy for legacy operating systems.
14. 2(↑) tickets: Unable to login on X.com (formerly Twitter) when using Tor Browser.[3]
15. 1(↑) ticket: report of a fake apps on iOS AppStore masquerading as official Tor Browser.
16. 1(↑) ticket: Bookmarks are failing to load on Tor Browser for Android Alpha.[4]
# Highlights from the Tor Forum
1. X.com (formerly Twitter) issues with Tor Browser.[3]
2. Older Tor Browsers are breaking in functionality and pose a security risk, please
update to the latest versions of Tor Browser.[13]
3. The Impact of Privacy Addons on Tor Browser Fingerprinting.[14]
Note: (↑), (↓) and (-) are indicating if the number of tickets we
received for these topics have been increasing, decreasing or have been
the same from the previous month respectively.
best,
e.
[0]: https://gitlab.torproject.org/tpo/web/support/-/commits/main?author=ebanam
[1]: https://gitlab.torproject.org/tpo/web/manual/-/commits/main?author=ebanam
[2]: https://blog.torproject.org/fighting-censorship-with-webtunnel/
[3]: https://forum.torproject.org/t/x-com-formerly-twitter-issues-with-tor-brows…
[4]: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43581
[5]: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43549
[6]: https://support.torproject.org/tbb/tbb-15/
[7]: https://support.torproject.org/tbb/tbb-43/
[8]: https://gitlab.torproject.org/tpo/web/snowflake/-/issues/20
[9]: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/260
[10]: https://forum.torproject.org/t/always-the-same-captcha-when-requesting-brig…
[11]: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43471
[12]: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42714
[13]: https://forum.torproject.org/t/older-tor-browsers-breaking-update-now/17885
[14]: https://forum.torproject.org/t/the-impact-of-privacy-addons-on-tor-browser-…
[View Less]
Summary: start upgrading servers during the Debian 13 ("trixie")
freeze, if it goes well, complete most of the fleet upgrade in around
June 2025, with full completion by the end of 2025, with a 2026 year
free of major upgrades entirely. Improve automation, retire old
container images.
Deadline: 2 weeks, 2025-04-01
# Background
Debian 13 ("trixie"), currently "testing", is going into freeze soon, which
means we should have a new Debian stable release in 2025. It has been
a long-standing …
[View More]tradition at TPA to collaborate in the Debian
development process and part of that process is to upgrade our servers
during the freeze. Upgrading during the freeze makes it easier for us
to fix bugs as we find them and contribute them to the community.
The [freeze dates announced by the debian.org release team][] are:
2025-03-15 - Milestone 1 - Transition and toolchain freeze
2025-04-15 - Milestone 2 - Soft Freeze
2025-05-15 - Milestone 3 - Hard Freeze - for key packages and
packages without autopkgtests
To be announced - Milestone 4 - Full Freeze
We have entered the "transition and toolchain freeze" which locks
changes on packages like compilers and interpreters unless
exceptions. See the [Debian freeze policy][] for an explanation of
each step.
Even though we've just completed the Debian 11 ("bullseye") and 12
("bookworm") upgrades in late 2024, we feel it's a good idea to start
*and* complete the Debian 13 upgrades in 2025. That way, we can hope of
having a year or two (2026-2027?) *without* any major upgrades.
This proposal is part of the [Debian 13 trixie upgrade milestone][],
itself part of the [2025 TPA roadmap][].
[freeze dates announced by the debian.org release team]: https://lists.debian.org/debian-devel-announce/2025/01/msg00004.html
[Debian freeze policy]: https://release.debian.org/testing/freeze_policy.html
[Debian 13 trixie upgrade milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/12
[2025 TPA roadmap]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2025
# Proposal
As usual, we perform the upgrades in three batches, in increasing
order of complexity, starting in 2025Q2, hoping to finish by the end
of 2025.
Note that, this year, this proposal also includes upgrading the Tails
infrastructure as well. To help with merging rotations in the two
teams, TPA staff will upgrade Tails machines, with Tails folks
assistance, and vice-versa.
## Affected users
All service admins are affected by this change. If you have shell
access on any TPA server, you want to read this announcement.
In the past, TPA has typically kept a page detailing notable changes
and a proposal like this one would link against the upstream release
notes. Unfortunately, at the time writing, upstream hasn't yet
produced release notes (as we're still in testing).
We're hoping the documentation will be refined by the time we're ready
to coordinate the second batch of updates, around May 2025, when we
will send reminders to affected teams.
We do expect the Debian 13 upgrade to be less disruptive than bookworm,
mainly because Python 2 is already retired.
## Notable changes
For now, here are some known changes that are already in Debian 13:
| Package | 12 (bookworm) | 13 (trixie) |
|--------------------|---------------|-------------|
| Ansible | 7.7 | 11.2 |
| Apache | 2.4.62 | 2.4.63 |
| Bash | 5.2.15 | 5.2.37 |
| Emacs | 28.2 | 30.1 |
| Fish | 3.6 | 4.0 |
| Git | 2.39 | 2.45 |
| GCC | 12.2 | 14.2 |
| Golang | 1.19 | 1.24 |
| Linux kernel image | 6.1 series | 6.12 series |
| LLVM | 14 | 19 |
| MariaDB | 10.11 | 11.4 |
| Nginx | 1.22 | 1.26 |
| OpenJDK | 17 | 21 |
| OpenLDAP | 2.5.13 | 2.6.9 |
| OpenSSL | 3.0 | 3.4 |
| PHP | 8.2 | 8.4 |
| Podman | 4.3 | 5.4 |
| PostgreSQL | 15 | 17 |
| Prometheus | 2.42 | 2.53 |
| Puppet | 7 | 8 |
| Python | 3.11 | 3.13 |
| Rustc | 1.63 | 1.85 |
| Vim | 9.0 | 9.1 |
Most of those, except "tool chains" (e.g. LLVM/GCC) can still change,
as we're not in the full freeze yet.
## Upgrade schedule
The upgrade is split in multiple batches:
- automation and installer changes
- low complexity: mostly TPA services and less critical Tails servers
- moderate complexity: TPA "service admins" machines and remaining
Tails physical servers and VMs running services from the official
Debian repositories only
- high complexity: Tails VMs running services not from the official
Debian repositories
- cleanup
The free time between the first two batches will also allow us to
cover for unplanned contingencies: upgrades that could drag on and
other work that will inevitably need to be performed.
The objective is to do the batches in collective "upgrade parties"
that should be "fun" for the team. This policy has proven to be
effective in the previous upgrades and we are eager to repeat it
again.
### Upgrade automation and installer changes
First, we tweak the installers to deploy Debian 13 by default to avoid
installing further "old" systems. This includes the bare-metal
installers but also and especially the virtual machine installers and
container images.
Concretely, we're planning on changing the `latest` container image
tag to point to `trixie` in early April. A full *year* later, the
`bookworm` container images will be retired. Note that we are already
planning the retirement of the "old stable" (`bullseye`) container
images, see [tpo/tpa/base-images#19][], for which you may have
already been contacted.
New `idle` canary servers will be setup in Debian 13 to test
integration with the rest of the infrastructure, and future new
machine installs will be done in Debian 13.
We also want to work on automating the upgrade procedure
further. We've had catastrophic errors in the PostgreSQL upgrade
procedure in the past, in particular, but the whole procedure is now
considered ripe for automation, see [tpo/tpa/team#41485][] for
details.
[tpo/tpa/base-images#19]: https://gitlab.torproject.org/tpo/tpa/base-images/-/issues/19
[tpo/tpa/team#41485]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41485
### Batch 1: low complexity
This is scheduled during two weeks: TPA boxes will be upgraded in
the last week of April, and Tails in the first week of May.
The idea is to start the upgrade long enough before the vacations to
give us plenty of time to recover, and some room to start the second
batch.
In April, Debian should also be in "soft freeze", not quite a fully
"stable" environment, but that should be good enough for simple
setups.
35 TPA machines:
```
archive-01.torproject.orgcdn-backend-sunet-02.torproject.orgchives.torproject.orgdal-rescue-01.torproject.orgdal-rescue-02.torproject.orggayi.torproject.orghetzner-hel1-02.torproject.orghetzner-hel1-03.torproject.orghetzner-nbg1-01.torproject.orghetzner-nbg1-02.torproject.orgidle-dal-02.torproject.orgidle-fsn-01.torproject.orglists-01.torproject.orgloghost01.torproject.orgmandos-01.torproject.orgmedia-01.torproject.orgminio-01.torproject.orgmta-dal-01.torproject.orgmx-dal-01.torproject.orgneriniflorum.torproject.orgns3.torproject.orgns5.torproject.orgpalmeri.torproject.orgperdulce.torproject.orgsrs-dal-01.torproject.orgssh-dal-01.torproject.orgstatic-gitlab-shim.torproject.orgstaticiforme.torproject.orgstatic-master-fsn.torproject.orgsubmit-01.torproject.orgvault-01.torproject.orgweb-dal-07.torproject.orgweb-dal-08.torproject.orgweb-fsn-01.torproject.orgweb-fsn-02.torproject.org
```
4 Tails machines:
```
ecours.tails.net
puppet.lizard
skink.tails.netstone.tails.net
```
In the [first batch of bookworm machines][], we ended up taking 20
minutes per machine, done in a single day, but warned that the second
batch took longer.
It's probably safe to estimate 20 hours (30 minutes per machine) for
this work, in a single week.
Feedback and coordination of this batch happens in [issue batch 1][].
[first batch of bookworm machines]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41251
[issue batch 1]: "https://gitlab.torproject.org/tpo/tpa/team/-/issues/42071"
### Batch 2: moderate complexity
This is scheduled for the last week of may for TPA machines, and the
first week of June for Tails.
At this point, Debian testing should be in "hard freeze", which should
be more stable.
40 TPA machines:
```
anonticket-01.torproject.orgbackup-storage-01.torproject.orgbacula-director-01.torproject.orgbtcpayserver-02.torproject.orgbungei.torproject.orgcarinatum.torproject.orgcheck-01.torproject.orgci-runner-x86-02.torproject.orgci-runner-x86-03.torproject.orgcolchicifolium.torproject.orgcollector-02.torproject.orgcrm-int-01.torproject.orgdangerzone-01.torproject.orgdonate-01.torproject.orgdonate-review-01.torproject.orgforum-01.torproject.orggitlab-02.torproject.orghenryi.torproject.orgmaterculae.torproject.orgmeronense.torproject.orgmetricsdb-01.torproject.orgmetricsdb-02.torproject.orgmetrics-store-01.torproject.orgonionbalance-02.torproject.orgonionoo-backend-03.torproject.orgpolyanthum.torproject.orgprobetelemetry-01.torproject.orgrdsys-frontend-01.torproject.orgrdsys-test-01.torproject.orgrelay-01.torproject.orgrude.torproject.orgsurvey-01.torproject.orgtbb-nightlies-master.torproject.orgtb-build-02.torproject.orgtb-build-03.torproject.orgtb-build-06.torproject.orgtb-pkgstage-01.torproject.orgtb-tester-01.torproject.orgtelegram-bot-01.torproject.orgweather-01.torproject.org
```
17 Tails machines:
```
apt-proxy.lizard
apt.lizard
bitcoin.lizard
bittorrent.lizard
bridge.lizard
dns.lizard
dragon.tails.net
gitlab-runner.iguana
iguana.tails.netlizard.tails.net
mail.lizard
misc.lizard
puppet-git.lizard
rsync.lizard
teels.tails.net
whisperback.lizard
www.lizard
```
The [second batch of bookworm upgrades][] took 33 hours for 31
machines, so about one hour per box. Here we have 57 machines, so it
will likely take us 60 hours (or two weeks) to complete the upgrade.
Feedback and coordination of this batch happens in [issue batch 2][].
[second batch of bookworm upgrades]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252
[issue batch 2]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42070
### Batch 3: high complexity
Those machines are harder to upgrade, or more critical. In the case of
TPA machines, we typically regroup the Ganeti servers and all the
"snowflake" servers that are not properly Puppetized and full of
legacy, namely the LDAP, DNS, and Puppet servers.
That said, we waited a long time to upgrade the Ganeti cluster for
bookworm, and it turned out to be trivial, so perhaps those could
eventually be made part of the second batch.
15 TPA machines:
```
- [ ] alberti.torproject.org
- [ ] dal-node-01.torproject.org
- [ ] dal-node-02.torproject.org
- [ ] dal-node-03.torproject.org
- [ ] fsn-node-01.torproject.org
- [ ] fsn-node-02.torproject.org
- [ ] fsn-node-03.torproject.org
- [ ] fsn-node-04.torproject.org
- [ ] fsn-node-05.torproject.org
- [ ] fsn-node-06.torproject.org
- [ ] fsn-node-07.torproject.org
- [ ] fsn-node-08.torproject.org
- [ ] nevii.torproject.org
- [ ] pauli.torproject.org
- [ ] puppetdb-01.torproject.org
```
It seems like the [bookworm Ganeti upgrade][] took roughly 10h of
work. We ballpark the rest of the upgrade to another 10h of work, so
possibly 20h.
11 Tails machines:
```
- [ ] isoworker1.dragon
- [ ] isoworker2.dragon
- [ ] isoworker3.dragon
- [ ] isoworker4.dragon
- [ ] isoworker5.dragon
- [ ] isoworker6.iguana
- [ ] isoworker7.iguana
- [ ] isoworker8.iguana
- [ ] jenkins.dragon
- [ ] survey.lizard
- [ ] translate.lizard
```
The challenge with Tails upgrades is the coordination with the Tails
team, in particular for the Jenkins upgrades.
Feedback and coordination of this batch happens in [issue batch 3][].
[bookworm Ganeti upgrade]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41254
[issue batch 3]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42069
### Cleanup work
Once the upgrade is completed and the entire fleet is again running a
single OS, it's time for cleanup. This involves updating configuration
files to the new versions and removing old compatibility code in
Puppet, removing old container images, and generally wrapping things
up.
This process has been historically neglected, but we're hoping to wrap
this up, worst case in 2026.
## Timeline
- 2025-Q2
- W14 (first week of April): default container image changed to
`trixie`, installer defaults changed and first tests in
production
- W18 (last week of April): Batch 1 upgrades, TPA machines
- W19 (first week of May): Batch 1 upgrades, Tails machines
- W22 (last week of May): Batch 2 upgrades, TPA machines
- W23 (first week of June): Batch 2 upgrades, Tails machines
- 2025-Q3 to Q4: Batch 3 upgrades
- 2026-Q2: bookworm container image retired
## Deadline
The community has until the beginning of the above timeline to
manifest concerns or objections.
Two weeks before performing the upgrades of each batch, a new
announcement will be sent with details of the changes and impacted
services.
# Alternatives considered
## Retirements or rebuilds
We do not plan any major upgrade or retirements in the third phase
this time.
In the future, we hope to decouple those as much as possible, as the
Icinga retirement and Mailman 3 became blockers that slowed down the
upgrade significantly for bookworm. In both cases, however, the
upgrades *were* challenging and had to be performed one way or
another, so it's unclear if we can optimize this any further.
We are clear, however, that we will not postpone an upgrade for a
server retirement. Dangerzone, for example, is scheduled for
retirement ([TPA-RFC-78][]) but is still planned as normal above.
[TPA-RFC-78]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-78-danger…
# Costs
| Task | Estimate | Certainty | Worst case |
|-------------------|----------|-----------|------------|
| Automation | 20h | extreme | 100h |
| Installer changes | 4h | low | 4.4h |
| Batch 1 | 20h | low | 22h |
| Batch 2 | 60h | medium | 90h |
| Batch 3 | 20h | high | 40h |
| Cleanup | 20h | medium | 30h |
| **Total** | 144h | ~high | ~286h |
The entire work here should consist of over 140 hours of work, or 18
days, or about 4 weeks full time. Worst case doubles that.
The above is done in "hours" because that's how we estimated batches
in the past, but here's an estimate that's based on the [Kaplan-Moss
estimation technique][].
[Kaplan-Moss estimation technique]: https://jacobian.org/2021/may/25/my-estimation-technique/
| Task | Estimate | Certainty | Worst case |
|-------------------|----------|-----------|------------|
| Automation | 3d | extreme | 15d |
| Installer changes | 1d | low | 1.1d |
| Batch 1 | 3d | low | 3.3d |
| Batch 2 | 10d | medium | 20d |
| Batch 3 | 3d | high | 6d |
| Cleanup | 3d | medium | 4.5d |
| **Total** | 23d | ~high | ~50d |
This is *roughly* equivalent, if a little higher (23 days instead of
18), for example.
It should be noted that automation is not expected to drastically
reduce the total time spent in batches (currently 16 days or 100
hours). The main goal of automation is more to reduce the likelihood
of catastrophic errors, and make it easier to share our upgrade
procedure with the world. We're still hoping to reduce the time spent
in batches, hopefully by 10-20%, which would bring the total number of
days across batches from 16 days to 14d, or from 100 h to 80 hours.
# Approvals required
This proposal needs approval from TPA team members, but service admins
can request additional delay if they are worried about their service
being affected by the upgrade.
Comments or feedback can be provided in issues linked above, or the
general process can be commented on in issue [tpo/tpa/team#41990][].
# References
* [Debian 13 trixie upgrade milestone][]
* [discussion ticket][tpo/tpa/team#41990]
[TPA bookworm upgrade procedure]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm
[tpo/tpa/team#41990]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41990
--
Antoine Beaupré
torproject.org system administration
[View Less]
Hi! Below is my March’25 report!
In March, I resolved 676 (↑109) tickets:
* On Telegram (@TorProjectSupportBot) - 571 (↑110);
* On RT (frontdesk@tpo) - 102 (↓4);
* On WhatsApp (+447421000612) - 3 (↑3);
* And on Signal (+17787431312) - 0 (0).
My main focus in March was helping Russian-speaking users to bypass
internet censorship, which includes:
- Sharing instructions,
- Gathering users' feedback,
- Help in troubleshooting any issues they were facing with Tor …
[View More]Browser.
I also worked on providing feedback for Google Play Store users’ reviews.
In March, I continued participating in translating the Tor Browser user
survey to Russian[1] and monitored censorship events in Russia[2],
Belarus [3], Kazakhstan [4], and in Türkiye. I also took part in
investigating the issue of bookmarks failing in TBA Alpha [5] and
collected user feedback on the issue with bridge distribution [6].
I created a new support template for Google Play reviews regarding the
bookmark issue[5] and updated existing templates in Russian.
*##**Google Play Reviews for Tor Browser**(TBA)**and Tor Browser
Alpha**for Android*
* Tor Browser for Android had a Google Play rating of *4.406*(↑0.009)
stars in March, which is higher than in February’25.
* Tor Browser for Android (TBA) got 694 (↑79) reviews out of 61,452
for the lifetime.
* Tor Browser for Android Alpha (TBA-Alpha) app had a rating of 4.208
(↓0.011) which is lower than in February’25.
* In March, Tor Browser for Android (TBA-Alpha) got 49 (↑7) reviews
out of 8,514 for the lifetime.
In March the most significant issue reported by users in the comments
was the bookmark issue in Tor Browser Alpha [5].
[1] https://gitlab.torproject.org/tpo/community/l10n/-/issues/40152
[2]
https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/iss…
[3]
https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/iss…
[4]
https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/iss…
[5]
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43581
[6] https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/156
[View Less]