Summary: start upgrading servers during the Debian 13 ("trixie")
freeze, if it goes well, complete most of the fleet upgrade in around
June 2025, with full completion by the end of 2025, with a 2026 year
free of major upgrades entirely. Improve automation, retire old
container images.
Deadline: 2 weeks, 2025-04-01
# Background
Debian 13 ("trixie"), currently "testing", is going into freeze soon, which
means we should have a new Debian stable release in 2025. It has been
a long-standing …
[View More]tradition at TPA to collaborate in the Debian
development process and part of that process is to upgrade our servers
during the freeze. Upgrading during the freeze makes it easier for us
to fix bugs as we find them and contribute them to the community.
The [freeze dates announced by the debian.org release team][] are:
2025-03-15 - Milestone 1 - Transition and toolchain freeze
2025-04-15 - Milestone 2 - Soft Freeze
2025-05-15 - Milestone 3 - Hard Freeze - for key packages and
packages without autopkgtests
To be announced - Milestone 4 - Full Freeze
We have entered the "transition and toolchain freeze" which locks
changes on packages like compilers and interpreters unless
exceptions. See the [Debian freeze policy][] for an explanation of
each step.
Even though we've just completed the Debian 11 ("bullseye") and 12
("bookworm") upgrades in late 2024, we feel it's a good idea to start
*and* complete the Debian 13 upgrades in 2025. That way, we can hope of
having a year or two (2026-2027?) *without* any major upgrades.
This proposal is part of the [Debian 13 trixie upgrade milestone][],
itself part of the [2025 TPA roadmap][].
[freeze dates announced by the debian.org release team]: https://lists.debian.org/debian-devel-announce/2025/01/msg00004.html
[Debian freeze policy]: https://release.debian.org/testing/freeze_policy.html
[Debian 13 trixie upgrade milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/12
[2025 TPA roadmap]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2025
# Proposal
As usual, we perform the upgrades in three batches, in increasing
order of complexity, starting in 2025Q2, hoping to finish by the end
of 2025.
Note that, this year, this proposal also includes upgrading the Tails
infrastructure as well. To help with merging rotations in the two
teams, TPA staff will upgrade Tails machines, with Tails folks
assistance, and vice-versa.
## Affected users
All service admins are affected by this change. If you have shell
access on any TPA server, you want to read this announcement.
In the past, TPA has typically kept a page detailing notable changes
and a proposal like this one would link against the upstream release
notes. Unfortunately, at the time writing, upstream hasn't yet
produced release notes (as we're still in testing).
We're hoping the documentation will be refined by the time we're ready
to coordinate the second batch of updates, around May 2025, when we
will send reminders to affected teams.
We do expect the Debian 13 upgrade to be less disruptive than bookworm,
mainly because Python 2 is already retired.
## Notable changes
For now, here are some known changes that are already in Debian 13:
| Package | 12 (bookworm) | 13 (trixie) |
|--------------------|---------------|-------------|
| Ansible | 7.7 | 11.2 |
| Apache | 2.4.62 | 2.4.63 |
| Bash | 5.2.15 | 5.2.37 |
| Emacs | 28.2 | 30.1 |
| Fish | 3.6 | 4.0 |
| Git | 2.39 | 2.45 |
| GCC | 12.2 | 14.2 |
| Golang | 1.19 | 1.24 |
| Linux kernel image | 6.1 series | 6.12 series |
| LLVM | 14 | 19 |
| MariaDB | 10.11 | 11.4 |
| Nginx | 1.22 | 1.26 |
| OpenJDK | 17 | 21 |
| OpenLDAP | 2.5.13 | 2.6.9 |
| OpenSSL | 3.0 | 3.4 |
| PHP | 8.2 | 8.4 |
| Podman | 4.3 | 5.4 |
| PostgreSQL | 15 | 17 |
| Prometheus | 2.42 | 2.53 |
| Puppet | 7 | 8 |
| Python | 3.11 | 3.13 |
| Rustc | 1.63 | 1.85 |
| Vim | 9.0 | 9.1 |
Most of those, except "tool chains" (e.g. LLVM/GCC) can still change,
as we're not in the full freeze yet.
## Upgrade schedule
The upgrade is split in multiple batches:
- automation and installer changes
- low complexity: mostly TPA services and less critical Tails servers
- moderate complexity: TPA "service admins" machines and remaining
Tails physical servers and VMs running services from the official
Debian repositories only
- high complexity: Tails VMs running services not from the official
Debian repositories
- cleanup
The free time between the first two batches will also allow us to
cover for unplanned contingencies: upgrades that could drag on and
other work that will inevitably need to be performed.
The objective is to do the batches in collective "upgrade parties"
that should be "fun" for the team. This policy has proven to be
effective in the previous upgrades and we are eager to repeat it
again.
### Upgrade automation and installer changes
First, we tweak the installers to deploy Debian 13 by default to avoid
installing further "old" systems. This includes the bare-metal
installers but also and especially the virtual machine installers and
container images.
Concretely, we're planning on changing the `latest` container image
tag to point to `trixie` in early April. A full *year* later, the
`bookworm` container images will be retired. Note that we are already
planning the retirement of the "old stable" (`bullseye`) container
images, see [tpo/tpa/base-images#19][], for which you may have
already been contacted.
New `idle` canary servers will be setup in Debian 13 to test
integration with the rest of the infrastructure, and future new
machine installs will be done in Debian 13.
We also want to work on automating the upgrade procedure
further. We've had catastrophic errors in the PostgreSQL upgrade
procedure in the past, in particular, but the whole procedure is now
considered ripe for automation, see [tpo/tpa/team#41485][] for
details.
[tpo/tpa/base-images#19]: https://gitlab.torproject.org/tpo/tpa/base-images/-/issues/19
[tpo/tpa/team#41485]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41485
### Batch 1: low complexity
This is scheduled during two weeks: TPA boxes will be upgraded in
the last week of April, and Tails in the first week of May.
The idea is to start the upgrade long enough before the vacations to
give us plenty of time to recover, and some room to start the second
batch.
In April, Debian should also be in "soft freeze", not quite a fully
"stable" environment, but that should be good enough for simple
setups.
35 TPA machines:
```
archive-01.torproject.orgcdn-backend-sunet-02.torproject.orgchives.torproject.orgdal-rescue-01.torproject.orgdal-rescue-02.torproject.orggayi.torproject.orghetzner-hel1-02.torproject.orghetzner-hel1-03.torproject.orghetzner-nbg1-01.torproject.orghetzner-nbg1-02.torproject.orgidle-dal-02.torproject.orgidle-fsn-01.torproject.orglists-01.torproject.orgloghost01.torproject.orgmandos-01.torproject.orgmedia-01.torproject.orgminio-01.torproject.orgmta-dal-01.torproject.orgmx-dal-01.torproject.orgneriniflorum.torproject.orgns3.torproject.orgns5.torproject.orgpalmeri.torproject.orgperdulce.torproject.orgsrs-dal-01.torproject.orgssh-dal-01.torproject.orgstatic-gitlab-shim.torproject.orgstaticiforme.torproject.orgstatic-master-fsn.torproject.orgsubmit-01.torproject.orgvault-01.torproject.orgweb-dal-07.torproject.orgweb-dal-08.torproject.orgweb-fsn-01.torproject.orgweb-fsn-02.torproject.org
```
4 Tails machines:
```
ecours.tails.net
puppet.lizard
skink.tails.netstone.tails.net
```
In the [first batch of bookworm machines][], we ended up taking 20
minutes per machine, done in a single day, but warned that the second
batch took longer.
It's probably safe to estimate 20 hours (30 minutes per machine) for
this work, in a single week.
Feedback and coordination of this batch happens in [issue batch 1][].
[first batch of bookworm machines]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41251
[issue batch 1]: "https://gitlab.torproject.org/tpo/tpa/team/-/issues/42071"
### Batch 2: moderate complexity
This is scheduled for the last week of may for TPA machines, and the
first week of June for Tails.
At this point, Debian testing should be in "hard freeze", which should
be more stable.
40 TPA machines:
```
anonticket-01.torproject.orgbackup-storage-01.torproject.orgbacula-director-01.torproject.orgbtcpayserver-02.torproject.orgbungei.torproject.orgcarinatum.torproject.orgcheck-01.torproject.orgci-runner-x86-02.torproject.orgci-runner-x86-03.torproject.orgcolchicifolium.torproject.orgcollector-02.torproject.orgcrm-int-01.torproject.orgdangerzone-01.torproject.orgdonate-01.torproject.orgdonate-review-01.torproject.orgforum-01.torproject.orggitlab-02.torproject.orghenryi.torproject.orgmaterculae.torproject.orgmeronense.torproject.orgmetricsdb-01.torproject.orgmetricsdb-02.torproject.orgmetrics-store-01.torproject.orgonionbalance-02.torproject.orgonionoo-backend-03.torproject.orgpolyanthum.torproject.orgprobetelemetry-01.torproject.orgrdsys-frontend-01.torproject.orgrdsys-test-01.torproject.orgrelay-01.torproject.orgrude.torproject.orgsurvey-01.torproject.orgtbb-nightlies-master.torproject.orgtb-build-02.torproject.orgtb-build-03.torproject.orgtb-build-06.torproject.orgtb-pkgstage-01.torproject.orgtb-tester-01.torproject.orgtelegram-bot-01.torproject.orgweather-01.torproject.org
```
17 Tails machines:
```
apt-proxy.lizard
apt.lizard
bitcoin.lizard
bittorrent.lizard
bridge.lizard
dns.lizard
dragon.tails.net
gitlab-runner.iguana
iguana.tails.netlizard.tails.net
mail.lizard
misc.lizard
puppet-git.lizard
rsync.lizard
teels.tails.net
whisperback.lizard
www.lizard
```
The [second batch of bookworm upgrades][] took 33 hours for 31
machines, so about one hour per box. Here we have 57 machines, so it
will likely take us 60 hours (or two weeks) to complete the upgrade.
Feedback and coordination of this batch happens in [issue batch 2][].
[second batch of bookworm upgrades]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252
[issue batch 2]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42070
### Batch 3: high complexity
Those machines are harder to upgrade, or more critical. In the case of
TPA machines, we typically regroup the Ganeti servers and all the
"snowflake" servers that are not properly Puppetized and full of
legacy, namely the LDAP, DNS, and Puppet servers.
That said, we waited a long time to upgrade the Ganeti cluster for
bookworm, and it turned out to be trivial, so perhaps those could
eventually be made part of the second batch.
15 TPA machines:
```
- [ ] alberti.torproject.org
- [ ] dal-node-01.torproject.org
- [ ] dal-node-02.torproject.org
- [ ] dal-node-03.torproject.org
- [ ] fsn-node-01.torproject.org
- [ ] fsn-node-02.torproject.org
- [ ] fsn-node-03.torproject.org
- [ ] fsn-node-04.torproject.org
- [ ] fsn-node-05.torproject.org
- [ ] fsn-node-06.torproject.org
- [ ] fsn-node-07.torproject.org
- [ ] fsn-node-08.torproject.org
- [ ] nevii.torproject.org
- [ ] pauli.torproject.org
- [ ] puppetdb-01.torproject.org
```
It seems like the [bookworm Ganeti upgrade][] took roughly 10h of
work. We ballpark the rest of the upgrade to another 10h of work, so
possibly 20h.
11 Tails machines:
```
- [ ] isoworker1.dragon
- [ ] isoworker2.dragon
- [ ] isoworker3.dragon
- [ ] isoworker4.dragon
- [ ] isoworker5.dragon
- [ ] isoworker6.iguana
- [ ] isoworker7.iguana
- [ ] isoworker8.iguana
- [ ] jenkins.dragon
- [ ] survey.lizard
- [ ] translate.lizard
```
The challenge with Tails upgrades is the coordination with the Tails
team, in particular for the Jenkins upgrades.
Feedback and coordination of this batch happens in [issue batch 3][].
[bookworm Ganeti upgrade]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41254
[issue batch 3]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42069
### Cleanup work
Once the upgrade is completed and the entire fleet is again running a
single OS, it's time for cleanup. This involves updating configuration
files to the new versions and removing old compatibility code in
Puppet, removing old container images, and generally wrapping things
up.
This process has been historically neglected, but we're hoping to wrap
this up, worst case in 2026.
## Timeline
- 2025-Q2
- W14 (first week of April): default container image changed to
`trixie`, installer defaults changed and first tests in
production
- W18 (last week of April): Batch 1 upgrades, TPA machines
- W19 (first week of May): Batch 1 upgrades, Tails machines
- W22 (last week of May): Batch 2 upgrades, TPA machines
- W23 (first week of June): Batch 2 upgrades, Tails machines
- 2025-Q3 to Q4: Batch 3 upgrades
- 2026-Q2: bookworm container image retired
## Deadline
The community has until the beginning of the above timeline to
manifest concerns or objections.
Two weeks before performing the upgrades of each batch, a new
announcement will be sent with details of the changes and impacted
services.
# Alternatives considered
## Retirements or rebuilds
We do not plan any major upgrade or retirements in the third phase
this time.
In the future, we hope to decouple those as much as possible, as the
Icinga retirement and Mailman 3 became blockers that slowed down the
upgrade significantly for bookworm. In both cases, however, the
upgrades *were* challenging and had to be performed one way or
another, so it's unclear if we can optimize this any further.
We are clear, however, that we will not postpone an upgrade for a
server retirement. Dangerzone, for example, is scheduled for
retirement ([TPA-RFC-78][]) but is still planned as normal above.
[TPA-RFC-78]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-78-danger…
# Costs
| Task | Estimate | Certainty | Worst case |
|-------------------|----------|-----------|------------|
| Automation | 20h | extreme | 100h |
| Installer changes | 4h | low | 4.4h |
| Batch 1 | 20h | low | 22h |
| Batch 2 | 60h | medium | 90h |
| Batch 3 | 20h | high | 40h |
| Cleanup | 20h | medium | 30h |
| **Total** | 144h | ~high | ~286h |
The entire work here should consist of over 140 hours of work, or 18
days, or about 4 weeks full time. Worst case doubles that.
The above is done in "hours" because that's how we estimated batches
in the past, but here's an estimate that's based on the [Kaplan-Moss
estimation technique][].
[Kaplan-Moss estimation technique]: https://jacobian.org/2021/may/25/my-estimation-technique/
| Task | Estimate | Certainty | Worst case |
|-------------------|----------|-----------|------------|
| Automation | 3d | extreme | 15d |
| Installer changes | 1d | low | 1.1d |
| Batch 1 | 3d | low | 3.3d |
| Batch 2 | 10d | medium | 20d |
| Batch 3 | 3d | high | 6d |
| Cleanup | 3d | medium | 4.5d |
| **Total** | 23d | ~high | ~50d |
This is *roughly* equivalent, if a little higher (23 days instead of
18), for example.
It should be noted that automation is not expected to drastically
reduce the total time spent in batches (currently 16 days or 100
hours). The main goal of automation is more to reduce the likelihood
of catastrophic errors, and make it easier to share our upgrade
procedure with the world. We're still hoping to reduce the time spent
in batches, hopefully by 10-20%, which would bring the total number of
days across batches from 16 days to 14d, or from 100 h to 80 hours.
# Approvals required
This proposal needs approval from TPA team members, but service admins
can request additional delay if they are worried about their service
being affected by the upgrade.
Comments or feedback can be provided in issues linked above, or the
general process can be commented on in issue [tpo/tpa/team#41990][].
# References
* [Debian 13 trixie upgrade milestone][]
* [discussion ticket][tpo/tpa/team#41990]
[TPA bookworm upgrade procedure]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm
[tpo/tpa/team#41990]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41990
--
Antoine Beaupré
torproject.org system administration
[View Less]
Hello tor-project@,
this month I have done the following things for the Tor project.
# Onion Service Reliability
I performed research on the reliability of onion services in various
setup environments, including arti to arti, ctor to ctor and the two
ones being mixed together. These findings resulted in the conclusion
that the arti implementation is well-done and has a similar failure of
its ctor pedant, which is around 0.5% or 1 in every 200 connection being
a failure. Those failures are …
[View More]flaky circuits which probably happen due
to simply having bad luck while choosing the circuit. Nonetheless,
those results are acceptable.
Besides, this research has led to an improvement within the arti
documentation, namely the addition of a method named
`is_fully_reachable` which checks whether the `State` of a
`tor-hsservice` indicates that it is reachable from the outside. This
change was necessary because there are right now two states which
indicate that the service is reachable, with only one of them sounding
like a success.
After all, this research led to the creation of the following tickets
and merge requests in the arti repository:
* arti#1887
* arti#1890
* arti!2850
# GPN23
I have opened a ticket (outreach#40103) for Tor activities at the GPN23
in Karlsruhe this June. I will be present at the event myself and I am
open for giving a Tor related talk there too. This is however still
early planning.
# oniux
This month, I continued working on oniux, namely statically integrating
onionmasq into the oniux binary, so oniux users no longer need to
provide a path to the onionmasq binary themselves; instead oniux is now
a beautiful self-container monolithic binary.
Besides all of this, I also did some research regarding the use of
`user_namespaces(7)` in oniux with the eventual goal of oniux being
capable to no longer need any `capabilities(7)`. This change is
non-trivial and would probably require a rewrite of the tool in a shell
scripting language, as a network stack such as [pasta](https://passt.top)
would be required for proper functioning. Right now, this is not a
priority but a nice to have. Our current use of capabilities is
probably not a security risk, as all capabilities are being dropped very
early on and no code originating from a malicious actor is executed with
capabilities at all.
Also, I have made two release tags for oniux this month: v0.1.0 and
v0.2.0.
Last but not least I have fixed a bug, namely that procfs was not
remounted in the PID namespace created by oniux leading to an incorrect
view when performing process related operations such as `ps aux`.
After all, the following tickets have been addressed:
* oniux#1 (for now)
* oniux#2
* oniux#3
# onionmasq
This month I gave onionmasq more attention again, this mainly included
fixing various RenovateBot related updates and keeping this up-to-date
in general.
Besides this, I have performed some clean-ups and fixed some clippy
warnings that occurred during the development.
Also, as outlined in oniux, I have moved code a little bit around,
meaning it is now possible to integrate onionmasq as a library into
existing Rust application like I did with oniux.
Last but not least I performed some testing with the onionmasq CI in
order to hopefully make it possible to run the integration tests outside
of privileged containers, there has not been much success there though.
After all, my work cumulated in the following tickets/MR's:
* onionmasq#135
* onionmasq#136
* onionmasq#137
* onionmasq#138
* onionmasq!404
* onionmasq!403
* onionmasq!401 (WIP)
* onionmasq!400
* onionmasq!398
* onionmasq!397 (WIP)
* onionmasq!393
# arti
Besides the addition of the `is_fully_reachable` method, I also support
and supported gabi@ in the release of the new arti version.
[View Less]
Hey everyone!
Here are our meeting logs:
http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-03-27-16.00.html
And our meeting pad:
Anti-censorship
--------------------------------
Next meeting: Thursday, April 3 16:00 UTC
Facilitator: onyinyang
^^^(See Facilitator Queue at tail)
Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC
(channel is logged while meetings are in progress)
This week's Facilitator: meskio
== Goal of this meeting ==
Weekly check-in about …
[View More]the status of anti-censorship work at Tor.
Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community.
== Links to Useful documents ==
* Our anti-censorship roadmap:
* Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards
* The anti-censorship team's wiki page:
* https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home
* Past meeting notes can be found at:
* https://lists.torproject.org/pipermail/tor-project/
* Tickets that need reviews: from projects, we are working on:
* All needs review tickets:
* https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s…
* Project 158 <-- meskio working on it
* https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na…
== Announcements ==
*
== Discussion ==
* Give @renovate-bot guest access to anti-censorship group for it to use dependency proxy src shell
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
* shelikhoo will give it rights on monday
== Actions ==
== Interesting links ==
*
== Reading group ==
* We will discuss "Differential Degradation Vulnerabilities in Censorship Circumvention Systems" on April 3
* https://arxiv.org/abs/2409.06247
* Questions to ask and goals to have:
* What aspects of the paper are questionable?
* Are there immediate actions we can take based on this work?
* Are there long-term actions we can take based on this work?
* Is there future work that we want to call out in hopes that others will pick it up?
== Updates ==
Name:
This week:
- What you worked on this week.
Next week:
- What you are planning to work on next week.
Help with:
- Something you need help with.
cecylia (cohosh): 2025-03-27
Last week:
- deployed a new snowflake broker with logging and client timeout fixes
- followed up on cleaning up broker log messages (snowflake#40454)
- snowflake!545
- wrote an update on snowflake sqs rendezvous channel
- looked more at rendezvous errors (snowflake#40447)
- tor safety board review
This week:
- support conjure work
- follow up on snowflake rendezvous failures
- reduce/remove use of mutexes for broker metrics (snowflake#40458)
- take a look at potential snowflake orbot bug
- https://github.com/guardianproject/orbot-android/issues/1183
- maybe do some lox work
dcf: 2025-03-20
Last week:
- archived a Kazakhstan URL blocklist https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/iss…
Next week:
- archive snowflake-webextension-0.9.3
- open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Help with:
meskio: 2024-03-27
Last week:
- crate a docker-compose file for rdsys (rdsys#219)
- try and fail to update obfs4-proxy docker image, arm* gives me timeouts in go get (docker-obfs4-bridge#21)
- deploy rdsys backend fixes to don't distribute dysfunctional bridges (team#156)
Next week:
- steps towards a rdsys in containers (rdsys#219)
Shelikhoo: 2024-03-27
Last Week:
- [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) testing environment setup/research
- Snowflake Staging Server (cont.) Discussion(https://gitlab.torproject.org/tpo/tpa/team/-/issues/42080)
- Snowfalke Staging Server Experiment
- Merge request reviews
Next Week/TODO:
- Merge request reviews
- [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) improvements
- Snowfalke Staging Server Experiment
onyinyang: 2025-03-13
Last week(s):
- continued work on ampcache registration method for conjure
- WIP MR: https://github.com/cohosh/conjure/pull/1
- https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj…
- lox issuer efficiency merged!
Next week:
- HACS/Zkproofs/OSCW/RWC
- finish up ampcache registration method
- Begin work on decoy registration option
- review Tor browser Lox integration
- https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests…
- add TTL cache to lox MR for duplicate responses:
https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305
As time allows:
- Work on outstanding milestone issues:
- key rotation automation
Later:
pending decision on abandoning lox wasm in favour of some kind of FFI? https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096):
- add pref to handle timing for pubkey checks in Tor browser
- add trusted invitation logic to tor browser integration:
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974
- improve metrics collection/think about how to show Lox is working/valuable
- sketch out Lox blog post/usage notes for forum
(long term things were discussed at the meeting!):
- brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice
Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people?
1. Are there some obvious grouping strategies that we can already consider?
e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?)
2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less?
theodorsm: 2025-03-06
Last weeks:
- Implementing key_share extension support in covert-dtls
- Implementing mimicking DTLS 1.3 extensions
- Debugging Tor Build with covert-dtls: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Next weeks:
- Update MR with new covert-dtls version.
- Write instructions on how to configure covert-dtls with snowflake client
- Fix merge conflicts in MR (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…).
- Condensing thesis into paper (on hold)
Help with:
- Test stability of covert-dtls in snowflake
Facilitator Queue:
onyinyang shelikhoo meskio
1. First available staff in the Facilitator Queue will be the facilitator for the meeting
2. After facilitating the meeting, the facilitator will be moved to the tail of the queue
--
meskio | https://meskio.net/
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
My contact info: https://meskio.net/crypto.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.
[View Less]
Hi,
So TPA has adopted this proposal, internally, to make yet another set of
emergency changes to our mail system, to respond to critical issues
affecting delivery and sustainability of our infrastructure.
I encourage you to read the "Affected users" section and "Timeline"
below. In particular, we will be experimenting with "sender rewriting"
soon, which will involve mangling emails we forward around to try and
fix deliverability on those.
The schleuder mailing list will also move servers.
…
[View More]Maintenance windows for those changes will be communicated separately.
Thank you for your attention!
PS: and no, we didn't submit this for adoption to everyone, because it
was felt it was mostly technical changes that didn't warrant outside
approval, let me know if that doesn't make sense, of course.
--
Antoine Beaupré
torproject.org system administration
---
title: TPA-RFC-71: Emergency email deployments, phase B
costs: staff
approval: TPA
affected users: all torproject.org email users
deadline: 5 days, 2024-10-01
status: draft
discussion: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41778
---
Summary: deploy a new sender-rewriting mail forwarder ASAP, migrate
mailing lists off the legacy server to a new machine, migrate the
remaining Schleuder list to the Tails server, upgrade `eugeni`.
Table of contents:
- Background
- Proposal
- Actual changes
- Mailman 3 upgrade
- New sender-rewriting mail exchanger
- Schleuder migration
- Upgrade legacy mail server
- Goals
- Must have
- Nice to have
- Non-Goals
- Scope
- Affected users
- Personas
- Timeline
- Optimistic timeline
- Worst case scenario
- Alternatives considered
- References
- History
- Personas descriptions
- Ariel, the fundraiser
- Blipblop, the bot
- Gary, the support guy
- John, the contractor
- Mallory, the director
- Nancy, the fancy sysadmin
- Orpheus, the developer
# Background
In [#41773][], we had yet another report of issues with mail delivery,
particularly with email forwards, that are plaguing Gmail-backed
aliases like grants@ and travel@.
[#41773]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41773
This is becoming critical. It has been impeding people's capacity of
using their email at work for a while, but it's been more acute since
google's recent changes in email validation (see [#41399][]) as now
hosts that have adopted the SPF/DKIM rules are bouncing.
[#41399]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41399
On top of that, we're way behind on our buster upgrade schedule. We
still have to upgrade our primary mail server, `eugeni`. The plan for
that ([TPA-RFC-45][], [#41009][]) was to basically re-architecture
everything. That won't happen fast enough for the LTS retirement which
we have crossed two months ago (in July 2024) already.
[TPA-RFC-45]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-45-mail-a…
[#41009]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41009
So, in essence, our main mail server is unsupported now, and we need
to fix this as soon as possible
Finally, we also have problems with certain servers (e.g. `state.gov`)
that seem to dislike our bespoke certificate authority (CA) which
makes *receiving* mails difficult for us.
# Proposal
So those are the main problems to fix:
- Email forwarding is broken
- Email reception is unreliable over TLS for some servers
- Mail server is out of date and hard to upgrade (mostly because of
Mailman)
## Actual changes
The proposed solution is:
- **Mailman 3 upgrade** ([#40471][])
- **New sender-rewriting mail exchanger** ([#40987][])
- **Schleuder migration**
- **Upgrade legacy mail server** ([#40694][])
[#40471]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40471
[#40987]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40987
[#40694]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40694
### Mailman 3 upgrade
Build a new mailing list server to host the upgraded Mailman 3
service. Move old lists over and convert them while retaining the old
archives available for posterity.
This includes lots of URL changes and user-visible disruption, little
can be done to work around that necessary change. We'll do our best to
come up with redirections and rewrite rules, but ultimately this is a
disruptive change.
This involves yet another authentication system being rolled out, as
Mailman 3 has its own user database, just like Mailman 2. At least
it's one user per site, instead of per list, so it's a slight
improvement.
This is issue [#40471][].
### New sender-rewriting mail exchanger
This step is carried over from [TPA-RFC-45][], mostly unchanged.
[Sender Rewriting Scheme]: https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme
[postsrsd]: https://github.com/roehling/postsrsd
[postforward]: https://github.com/zoni/postforward
Configure a new "mail exchanger" (MX) server with TLS certificates
signed by our normal public CA (Let's Encrypt). This replaces that
part of `eugeni`, will hopefully resolve issues with `state.gov` and
others ([#41073][], [#41287][], [#40202][], [#33413][]).
[#33413]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/33413
[#40202]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40202
[#41287]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41287
[#41073]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41073
This would handle forwarding mail to other services (e.g. mailing
lists) but also end-users.
To work around reputation problems with forwards ([#40632][],
[#41524][], [#41773][]), deploy a [Sender Rewriting Scheme][] (SRS)
with [postsrsd][] (packaged in Debian, but [not in the best shape][])
and [postforward][] (not packaged in Debian, but zero-dependency
Golang program).
It's possible deploying [ARC][] headers with [OpenARC][], Fastmail's
[authentication milter][] (which [apparently works better][]), or
[rspamd's arc module][] might be sufficient as well, to be tested.
[OpenARC]: https://tracker.debian.org/pkg/openarc
[not in the best shape]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017361
Having it on a separate mail exchanger will make it easier to swap in
and out of the infrastructure if problems would occur.
The mail exchangers should also sign outgoing mail with DKIM, and
*may* start doing better validation of incoming mail.
[authentication milter]: https://github.com/fastmail/authentication_milter
[apparently works better]: https://old.reddit.com/r/postfix/comments/17bbhd2/about_arc/k5iluvn/
[rspamd's arc module]: https://rspamd.com/doc/modules/arc.html
[#41524]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41524
[#40632]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40632
[ARC]: http://arc-spec.org/
### Schleuder migration
Migrate the remaining mailing list left (the Community Council) to the
Tails Shleuder server, retiring our Schleuder server entirely.
This requires configuring the Tails server to accept mail for
`(a)torproject.org`.
Note that this may require changing the addresses of the existing
Tails list to `(a)torproject.org` if Schleuder doesn't support virtual
hosting (which is likely).
### Upgrade legacy mail server
Once Mailman has been safely moved aside and is shown to be working
correctly, upgrade Eugeni using the normal procedures. This should be
a less disruptive upgrade, but is still risky because it's such an old
box with lots of legacy.
One key idea of this proposal is to keep the legacy mail server,
`eugeni`, in place. It will continue handling the "MTA" (Mail Transfer
Agent) work, which is to relay mail for other hosts, as a legacy
system.
The full eugeni replacement is seen as too complicated and unnecessary
at this stage. The legacy server will be isolated from the rewriting
forwarder so that outgoing mail is mostly unaffected by the forwarding
changes.
## Goals
This is not an exhaustive solution to all our email problems,
[TPA-RFC-45][] is that longer-term project.
### Must have
- Up to date, supported infrastructure.
- Functional legacy email forwarding.
### Nice to have
- Improve email forward deliverability to Gmail.
### Non-Goals
- **Clean email forwarding**: email forwards *may* be mangled and
rewritten to appear as coming from `(a)torproject.org` instead of the
original address. This will be figured out at the implementation
stage.
- **Mailbox storage**: out of scope, see [TPA-RFC-45][]. It is hoped,
however, that we *eventually* are able to provide such a service, as
the sender-rewriting stuff might be too disruptive in the long run.
- **Technical debt**: we keep the legacy mail server, `eugeni`.
- **Improved monitoring**: we won't have a better view in how well we
can deliver email.
- **High availability**: the new servers will not add additional
"single point of failures", but will not improve our availability
situation (issue [#40604][])
[#40604]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40604
## Scope
This proposal affects the all inbound and outbound email services
hosted under `torproject.org`. Services hosted under `torproject.net`
are *not* affected.
It also does *not* address directly phishing and scamming attacks
([#40596][]), but it is hoped the new mail exchanger will provide
a place where it is easier to make such improvements in the future.
[#40596]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40596
## Affected users
This affects all users which interact with `torproject.org` and its
subdomains over email. It particularly affects all "tor-internal"
users, users with LDAP accounts, or forwards under `(a)torproject.org`,
as their mails will get rewritten on the way out.
### Personas
Here we collect a few "personas" and try to see how the changes will
affect them, largely derived from [TPA-RFC-45][], but without the
alpha/beta/prod test groups.
For *all* users, a common impact is that emails will be rewritten by
the sender rewriting system. As mentioned above, the impact of this
still remains to be clarified, but at least the hidden `Return-Path`
header will be changed for bounces to go to our servers.
Actual personas are in the Reference section, see [Personas
descriptions][].
| Persona | Task | Impact |
|---------|-------------|--------------------------------------------------------------------------|
| Ariel | Fundraising | Improved incoming delivery |
| Blipbot | Bot | No change |
| Gary | Support | Improved incoming delivery, new moderator account on mailing list server |
| John | Contractor | Improved incoming delivery |
| Mallory | Director | Same as Ariel |
| Nancy | Sysadmin | No change in delivery, new moderator account on mailing list server |
| Orpheus | Developer | No change in delivery |
[Personas descriptions]: #personas-descriptions
## Timeline
### Optimistic timeline
- Late September (W39): issue raised again, proposal drafted (now)
- October:
- W40: proposal approved, installing new rewriting server
- W41: rewriting server deployment, new mailman 3 server
- W42: mailman 3 mailing list conversion tests, users required for testing
- W43: mailman 2 retirement, mailman 3 in production
- W44: Schleuder mailing list migration
- November:
- W45: `eugeni` upgrade
### Worst case scenario
- Late September (W39): issue raised again, proposal drafted (now)
- October:
- W40: proposal approved, installing new rewriting server
- W41-44: difficult rewriting server deployment
- November:
- W44-W48: difficult mailman 3 mailing list conversion and testing
- December:
- W49: Schleuder mailing list migration vetoed, Schleuder stays on
`eugeni`
- W50-W51: `eugeni` upgrade postponed to 2025
- January 2025:
- W3: `eugeni` upgrade
# Alternatives considered
We decided to not just run the sender-rewriting on the legacy mail
server because too many things are tangled up in that server. It is
just too risky.
We have also decided to not upgrade Mailman in place for the same
reason: it's seen as too risky as well, because we'd first need to
upgrade the Debian base system and if that fails, rolling back is too
hard.
# References
- [discussion issue][]
[discussion issue]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41778
## History
This is the the *fifth* proposal about our email services, here are
the previous ones:
* [TPA-RFC-15: Email services][] (rejected, replaced with TPA-RFC-31)
* [TPA-RFC-31: outsource email services][] (rejected, in favor of
TPA-RFC-44 and following)
* [TPA-RFC-44: Email emergency recovery, phase A][] (standard, and
mostly implemented except the sender-rewriting)
* [TPA-RFC-45: Mail architecture][] (still draft)
[TPA-RFC-15: Email services]: policy/tpa-rfc-15-email-services
[TPA-RFC-31: outsource email services]: policy/tpa-rfc-31-outsource-email
[TPA-RFC-44: Email emergency recovery, phase A]: policy/tpa-rfc-44-email-emergency-recovery
[TPA-RFC-45: Mail architecture]: policy/tpa-rfc-45-mail-architecture
## Personas descriptions
### Ariel, the fundraiser
Ariel does a lot of mailing. From talking to fundraisers through
their normal inbox to doing mass newsletters to thousands of people on
CiviCRM, they get a lot done and make sure we have bread on the table
at the end of the month. They're awesome and we want to make them
happy.
Email is absolutely mission critical for them. Sometimes email gets
lost and that's a major problem. They frequently tell partners their
personal Gmail account address to work around those problems. Sometimes
they send individual emails through CiviCRM because it doesn't work
through Gmail!
Their email forwards to Google Mail and they now have an LDAP account
to do email delivery.
### Blipblop, the bot
Blipblop is not a real human being, it's a program that receives
mails and acts on them. It can send you a list of bridges (bridgedb),
or a copy of the Tor program (gettor), when requested. It has a
brother bot called Nagios/Icinga who also sends unsolicited mail when
things fail.
There are also bots that sends email when commits get pushed to some
secret git repositories.
### Gary, the support guy
Gary is the ticket overlord. He eats tickets for breakfast, then
files 10 more before coffee. A hundred tickets is just a normal day at
the office. Tickets come in through email, RT, Discourse, Telegram,
Snapchat and soon, TikTok dances.
Email is absolutely mission critical, but some days he wishes there
could be slightly less of it. He deals with a lot of spam, and surely
something could be done about that.
His mail forwards to Riseup and he reads his mail over Thunderbird
and sometimes webmail. Some time after TPA-RFC_44, Gary managed to
finally get an OpenPGP key setup and TPA made him a LDAP account so he
can use the submission server. He has already abandoned the Riseup
webmail for TPO-related email, since it cannot relay mail through the
submission server.
### John, the contractor
John is a freelance contractor that's really into privacy. He runs his
own relays with some cools hacks on Amazon, automatically deployed
with Terraform. He typically run his own infra in the cloud, but
for email he just got tired of fighting and moved his stuff to
Microsoft's Office 365 and Outlook.
Email is important, but not absolutely mission critical. The
submission server doesn't currently work because Outlook doesn't allow
you to add just an SMTP server. John does have an LDAP account,
however.
### Mallory, the director
Mallory also does a lot of mailing. She's on about a dozen aliases
and mailing lists from accounting to HR and other unfathomable
things. She also deals with funders, job applicants, contractors,
volunteers, and staff.
Email is absolutely mission critical for her. She often fails to
contact funders and critical partners because `state.gov` blocks our
email -- or we block theirs! Sometimes, she gets told through LinkedIn
that a job application failed, because mail bounced at Gmail.
She has an LDAP account and it forwards to Gmail. She uses Apple Mail
to read their mail.
### Nancy, the fancy sysadmin
Nancy has all the elite skills in the world. She can configure a
Postfix server with her left hand while her right hand writes the
Puppet manifest for the Dovecot authentication backend. She browses
her mail through a UUCP over SSH tunnel using mutt. She runs her own
mail server in her basement since 1996.
Email is a pain in the back and she kind of hates it, but she still
believes entitled to run their own mail server.
Her email is, of course, hosted on her own mail server, and she has
an LDAP account. She has already reconfigured her Postfix server to
relay mail through the submission servers.
### Orpheus, the developer
Orpheus doesn't particular like or dislike email, but sometimes has
to use it to talk to people instead of compilers. They sometimes have
to talk to funders (`#grantlyfe`), external researchers, teammates or
other teams, and that often happens over email. Sometimes email is
used to get important things like ticket updates from GitLab or
security disclosures from third parties.
They have an LDAP account and it forwards to their self-hosted mail
server on a OVH virtual machine. They have already reconfigured their
mail server to relay mail over SSH through the jump host, to the
surprise of the TPA team.
Email is not mission critical, and it's kind of nice when it goes
down because they can get in the zone, but it should really be working
eventually.
--
Antoine Beaupré
torproject.org system administration
--
tpa-team mailing list
tpa-team(a)lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tpa-team
[View Less]
Hey everyone!
Here are our meeting logs:
http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-03-20-16.01.html
And our meeting pad:
Anti-censorship work meeting pad
--------------------------------
Anti-censorship
--------------------------------
Next meeting: Thursday,March, 27 16:00 UTC
Facilitator: meskio
^^^(See Facilitator Queue at tail)
Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC
(channel is logged while meetings are in progress)
This week's …
[View More]Facilitator: shelikhoo
== Goal of this meeting ==
Weekly check-in about the status of anti-censorship work at Tor.
Coordinate collaboration between people/teams on anti-censorship at the
Tor Project and Tor community.
== Links to Useful documents ==
* Our anti-censorship roadmap:
*
Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards
* The anti-censorship team's wiki page:
*
https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home
* Past meeting notes can be found at:
* https://lists.torproject.org/pipermail/tor-project/
* Tickets that need reviews: from projects, we are working on:
* All needs review tickets:
*
https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s…
* Project 158 <-- meskio working on it
*
https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na…
== Announcements ==
*
== Discussion ==
Mar 20 New:
* Snowflake Staging Server & rdsys containerization
* TPA supported system doesn't allow to have more than one
domain name per branch in a staging server
* snowflake needs two: broker and server
* shelikhoo is experiments with k8s + terraform
* shelikhoo will keep working on this since there is no overall
rejection to this design
== Actions ==
== Interesting links ==
*
https://github.com/doudoulong/Minecruft-PT/blob/main/README.md#tor-browser-…
== Reading group ==
* We will discuss "Differential Degradation Vulnerabilities in
Censorship Circumvention Systems" on April 3
* https://arxiv.org/abs/2409.06247
* Questions to ask and goals to have:
* What aspects of the paper are questionable?
* Are there immediate actions we can take based on this work?
* Are there long-term actions we can take based on this work?
* Is there future work that we want to call out in hopes
that others will pick it up?
== Updates ==
Name:
This week:
- What you worked on this week.
Next week:
- What you are planning to work on next week.
Help with:
- Something you need help with.
cecylia (cohosh): 2025-03-20
Last week:
- found and fixed cause of sqs rendezvous errors (snowflake#40363)
- implemented better timeouts at broker for client offers
(snowflake#40449)
- lower proxy answerTimeout for web-based proxies
(snowflake-webext#115)
- remove bad relay pattern log message and
default-relay-pattern option for broker (snowflake#40454)
- followed up on Snowflake rendezvous errors in general
(snowflake#40447)
- tagged and released snowflake proxy v2.11.0
- confirmed fixes for domain fronting issues on old android
versions (lyrebird#40012)
- opened issue to update PTs in Tor Browser
(tor-browser-build#41299)
- released snowflake-webext 0.9.3
This week:
- support conjure work
- follow up on snowflake rendezvous failures
- reduce/remove use of mutexes for broker metrics
(snowflake#40458)
- take a look at potential snowflake orbot bug
- https://github.com/guardianproject/orbot-android/issues/1183
- maybe do some lox work
dcf: 2025-03-20
Last week:
- archived a Kazakhstan URL blocklist
https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/iss…
Next week:
- archive snowflake-webextension-0.9.3
- open issue to have snowflake-client log whenever KCPInErrors
is nonzero
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- parent:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Help with:
meskio: 2024-03-20
Last week:
- build rdsys containers in the CI (rdsys#219)
- stop distributing dysfunctional bridges (team#156)
- downgrade lyrebird to use go 1.22 (lyrebird#40026)
- release lyrebird 0.6.0
- test covertdtls MR proxy (snowflake!448)
- debug dependency proxy for snowflake (snowflake!522)
Next week:
- steps towards a rdsys in containers (rdsys#219)
Shelikhoo: 2024-03-20
Last Week:
- [Testing] Unreliable+unordered WebRTC data channel transport
for Snowflake rev2 (cont.)(
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
) testing environment setup/research
- Snowflake Staging Server
Discussion(https://gitlab.torproject.org/tpo/tpa/team/-/issues/42080)
- Snowfalke Staging Server Experiment
- Merge request reviews
Next Week/TODO:
- Merge request reviews
- [Testing] Unreliable+unordered WebRTC data channel transport
for Snowflake rev2 (cont.)(
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
) improvements
- Snowfalke Staging Server Experiment
onyinyang: 2025-03-13
Last week(s):
- continued work on ampcache registration method for conjure
- WIP MR: https://github.com/cohosh/conjure/pull/1
-
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj…
- lox issuer efficiency merged!
Next week:
- HACS/Zkproofs/OSCW/RWC
- finish up ampcache registration method
- Begin work on decoy registration option
- review Tor browser Lox integration
-
https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests…
- add TTL cache to lox MR for duplicate responses:
https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305
As time allows:
- Work on outstanding milestone issues:
- key rotation automation
Later:
pending decision on abandoning lox wasm in favour of some kind
of FFI?
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096):
- add pref to handle timing for pubkey checks in Tor browser
- add trusted invitation logic to tor browser integration:
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974
- improve metrics collection/think about how to show Lox is
working/valuable
- sketch out Lox blog post/usage notes for forum
(long term things were discussed at the meeting!):
- brainstorming grouping strategies for Lox buckets (of
bridges) and gathering context on how types of bridges are
distributed/use in practice
Question: What makes a bridge usable for a given user, and
how can we encode that to best ensure we're getting the most appropriate
resources to people?
1. Are there some obvious grouping strategies that we
can already consider?
e.g., by PT, by bandwidth (lower bandwidth bridges
sacrificed to open-invitation buckets?), by locale (to be matched with a
requesting user's geoip or something?)
2. Does it make sense to group 3 bridges/bucket, so
trusted users have access to 3 bridges (and untrusted users have access
to 1)? More? Less?
theodorsm: 2025-03-06
Last weeks:
- Implementing key_share extension support in covert-dtls
- Implementing mimicking DTLS 1.3 extensions
- Debugging Tor Build with covert-dtls:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Next weeks:
- Update MR with new covert-dtls version.
- Write instructions on how to configure covert-dtls with
snowflake client
- Fix merge conflicts in MR
(https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…).
- Condensing thesis into paper (on hold)
Help with:
- Test stability of covert-dtls in snowflake
Facilitator Queue:
meskio onyinyang shelikhoo
1. First available staff in the Facilitator Queue will be the
facilitator for the meeting
2. After facilitating the meeting, the facilitator will be moved to the
tail of the queue
[View Less]
Summary: extend the retention limit for mail logs to 10 days
# Background
We currently rotate mail logs daily and keep them for 5 days. That's
great for privacy, but not so great for people having to report mail
trouble in time. In particular, when there are failures with mail sent
just before the weekend, it gives users a very short time frame to
report issues.
# Proposal
Extend the retention limit for mail (`postfix` and `rspamd`) logs to
10 days: one week, plus "flexible Friday", plus "…
[View More]weekend".
## Goals
Being able to debug mail issues when users notice and/or report them after five days.
## Tasks
Adjust logrotate configuration for syslog-ng and rspamd.
## Scope
All TPA servers.
## Affected users
Sysadmins and email users, which is pretty much everyone.
## Timeline
Logging policies will be changed on Wednesday March 19th.
# References
TPA has various log policies for various services, which we have been
meaning to document for a while. This policy proposal doesn't cover
for that, see [tpo/tpa/team#40960][] for followup on that more
general task.
[tpo/tpa/team#40960]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40960
See also the [discussion issue][].
[discussion issue]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42081
--
Antoine Beaupré
torproject.org system administration
[View Less]
Hey everyone!
Here are our meeting logs:
http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-03-13-16.00.html
And our meeting pad:
Anti-censorship work meeting pad
--------------------------------
Anti-censorship
--------------------------------
Next meeting: Thursday, March 20 16:00 UTC
Facilitator: shelikhoo
^^^(See Facilitator Queue at tail)
Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC
(channel is logged while meetings are in progress)
This week's …
[View More]Facilitator: onyinyang
== Goal of this meeting ==
Weekly check-in about the status of anti-censorship work at Tor.
Coordinate collaboration between people/teams on anti-censorship at the
Tor Project and Tor community.
== Links to Useful documents ==
* Our anti-censorship roadmap:
*
Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards
* The anti-censorship team's wiki page:
*
https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home
* Past meeting notes can be found at:
* https://lists.torproject.org/pipermail/tor-project/
* Tickets that need reviews: from projects, we are working on:
* All needs review tickets:
*
https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s…
* Project 158 <-- meskio working on it
*
https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na…
== Announcements ==
*
== Discussion ==
* snowflake dependencies need go 1.23
*
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
*
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
* should we update it?
* go 1.23 is already in debian backports
* Tor Browser is already using go 1.23?
* no, Tor Browser is staying on 1.21 on macOS until ESR
140, due June 2025
* we will hold off on the dependency updates (which contain
changes we believe do not affect us) until after ESR 140 and Tor Browser
updating to go 1.23
*
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
*
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
* should we change default relay pattern policy on snowflake broker
*
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
* we still have relays that have not updated since we added
support for multiple bridges
* The purpose of the allowed relay pattern check at the broker
is not to avoid surprising proxy operators by connecting to a new relay
destination; it's because old proxies that do not send a relay pattern
to the broker cannot connect to any other relay destination. They have
snowflake.torproject.net hardcoded and do not know how to connect to any
other relay. If a client were to request a connection to the
snowflake-02 relay, and was assigned such an old proxy, the proxy would
connect to snowflake-01 anyway, and leave the client with a Tor
fingerprint mismatch error.
* Therefore adjusting the presumedPatternForLegacyClient on the
broker to accept old proxies would not work. Old proxies (that don't
support relay patterns) would start getting clients again, but any
clients that request a relay other than snowflake-01 would get failed
connections.
* There's no facility on the broker to, say, match old proxies
only with those clients that happen to request snowflake-01. Once a
proxy is in the pool, it's treated equally to all other proxies. Such a
matching facility could conceivably be implemented, but none currently
exists. It may not be worth the complexity anyway.
* The intention therefore was that the broker simply reject old
proxies (that do not send a relay pattern). That is, in effect, what is
now happening, but it happens by a roundabout way: proxy polls and
doesn't set an allowed relay pattern → broker assigns a default allowed
relay pattern that never works → broker rejects the proxy poll because
of the relay pattern mismatch.
* The downsides of the status quo are that (1) the broker logs
are full of "rejected relay pattern from proxy" errors, and (2) the old
proxies just keep on uselessly polling.
* Consensus for dealing with the current situation:
* Make the broker code more straightforward with respect to
rejecting old proxies: do it at an early stage, don't even bother
assigning a default relay pattern to them (and remove the code that does).
* Update metrics/logging. Disable the "rejected relay
pattern from proxy" at the broker, and just keep it as a metrics counter.
* For the future, it would be handy if there were a feature
whereby the broker, by sending a specific kind of response to a proxy,
could cause the proxy to disable itself (say, stop polling for 24h) and
log a visible message to the operator. In effect, we could push the
"rejected relay pattern from proxy" error out to the proxy logs, which
would (1) stop proxies that only support an obsolete protocol from
uselessly polling, and (2) inform the operator and encourage them to
upgrade.
* In the absence of such a feature, it might be possible to
find a way to crash old proxies by sending a specific kind of broker
response. For example, an overlong HTTP body, or a JSON unmarshaling
error. Even if it is not possible to convey a reason why it is happening
to the proxy operator, it might get the operators attention, and at
least reduce unhelpful polling volume.
* One complication with this idea: some proxies are running
in a container, or similar, and are configured to auto-restart after
they exit. Proxies that work this way might paradoxically increase their
effective polling rate if remotely disabled, because there's no interval
rate limiting on the first poll.
* Even so, if we find such a "disable" broker response, we
could try having the broker send it to old proxies (those that don't
support relay patterns) for 1h. The ones that are configured to restart
will restart, but the one that are not will stop operating.
* Related issue: "Let the broker inform proxies how often
to poll"
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow….
The broker could send a time many hours in the future. This would help
with reducing the polling rate, even if it does not result in the proxy
log message.
* A standalone proxy auto-update mechanism could be an
alternative or addition to support for a broker "disable" signal.
== Actions ==
== Interesting links ==
*
== Reading group ==
* We will discuss "" on
*
* Questions to ask and goals to have:
* What aspects of the paper are questionable?
* Are there immediate actions we can take based on this work?
* Are there long-term actions we can take based on this work?
* Is there future work that we want to call out in hopes
that others will pick it up?
== Updates ==
Name:
This week:
- What you worked on this week.
Next week:
- What you are planning to work on next week.
Help with:
- Something you need help with.
cecylia (cohosh): 2025-03-13
Last week:
- wrote and deployed a CPU and goroutine profiling patch for
the snowflake broker (snowflake#40447)
- found a potential performance improvement related to
mutexes (snowflake#40458)
- confirmed that congestion isn't the cause of sqs
rendezvous errors (snowflake#4036)
- found source of "rejected relay pattern" log messages
(snowflake#40454)
- modified broker metrics to show client polls that timed out
(snowflake!530)
- reviewed merge requests
- helped with conjure pt work
This week:
- support conjure work
- follow up on snowflake rendezvous failures
- reduce/remove use of mutexes for broker metrics
(snowflake#40458)
- set absolute time limit on broker http responses
(snowflake40449)
- snowflake proxy release
- take a look at potential snowflake orbot bug
- https://github.com/guardianproject/orbot-android/issues/1183
- maybe do some lox work
dcf: 2025-03-13
Last week:
- snowflake VPS hosting bookkeeping
https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Snowflake-co…
Next week:
- open issue to have snowflake-client log whenever KCPInErrors
is nonzero
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- parent:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Help with:
meskio: 2024-03-13
Last week:
- set up snowflake proxies with coverdtls (snowflake!448)
- investigate why our email distributor is not handing out
emails (rdsys#260)
- debug gitlab container proxy (snowflake!522)
- investigate why users get bridges with local addresses (team#156)
Next week:
- steps towards a rdsys in containers (rdsys#219)
- investigate rdsys distributing local addressed bridges (team#156)
Shelikhoo: 2024-03-13
Last Week:
- [Testing] Unreliable+unordered WebRTC data channel transport
for Snowflake rev2 (cont.)(
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
) testing environment setup/research
- Vantage Point maintaince
- Merge request reviews
Next Week/TODO:
- Merge request reviews
- [Testing] Unreliable+unordered WebRTC data channel transport
for Snowflake rev2 (cont.)(
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
) improvements
- Snowflake Staging Server
Discussion(https://gitlab.torproject.org/tpo/tpa/team/-/issues/42080)
- Vantage Point maintaince
onyinyang: 2025-03-13
Last week(s):
- continued work on ampcache registration method for conjure
- WIP MR: https://github.com/cohosh/conjure/pull/1
-
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj…
- lox issuer efficiency merged!
Next week:
- HACS/Zkproofs/OSCW/RWC
- finish up ampcache registration method
- Begin work on decoy registration option
- review Tor browser Lox integration
-
https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests…
- add TTL cache to lox MR for duplicate responses:
https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305
As time allows:
- Work on outstanding milestone issues:
- key rotation automation
Later:
pending decision on abandoning lox wasm in favour of some kind
of FFI?
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096):
- add pref to handle timing for pubkey checks in Tor browser
- add trusted invitation logic to tor browser integration:
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974
- improve metrics collection/think about how to show Lox is
working/valuable
- sketch out Lox blog post/usage notes for forum
(long term things were discussed at the meeting!):
- brainstorming grouping strategies for Lox buckets (of
bridges) and gathering context on how types of bridges are
distributed/use in practice
Question: What makes a bridge usable for a given user, and
how can we encode that to best ensure we're getting the most appropriate
resources to people?
1. Are there some obvious grouping strategies that we
can already consider?
e.g., by PT, by bandwidth (lower bandwidth bridges
sacrificed to open-invitation buckets?), by locale (to be matched with a
requesting user's geoip or something?)
2. Does it make sense to group 3 bridges/bucket, so
trusted users have access to 3 bridges (and untrusted users have access
to 1)? More? Less?
theodorsm: 2025-03-06
Last weeks:
- Implementing key_share extension support in covert-dtls
- Implementing mimicking DTLS 1.3 extensions
- Debugging Tor Build with covert-dtls:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Next weeks:
- Update MR with new covert-dtls version.
- Write instructions on how to configure covert-dtls with
snowflake client
- Fix merge conflicts in MR
(https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…).
- Condensing thesis into paper (on hold)
Help with:
- Test stability of covert-dtls in snowflake
Facilitator Queue:
onyinyang shelikhoo meskio
1. First available staff in the Facilitator Queue will be the
facilitator for the meeting
2. After facilitating the meeting, the facilitator will be moved to the
tail of the queue
--
---
onyinyang
GPG Fingerprint 3CC3 F8CC E9D0 A92F A108 38EF 156A 6435 430C 2036
[View Less]
Hi! Below is my February’25 (Period: 2025-02-01 - 2025-02-28) report!
In February, I began working on tests and documentation, focusing on
integrating pluggable transports with little-t-tor on Windows. I then
collected data across different operating systems and contributed to the
corresponding GitLab documentation ticket.
Spent some time learning the workflow for merging and committing
documents in GitLab for the Tor Project.
*Add documentation about using pluggable …
[View More]transports with little-t-tor*
<https://gitlab.torproject.org/tpo/web/support/-/issues/378#top>
https://gitlab.torproject.org/tpo/web/support/-/issues/378
Thanks,
Haidi
[View Less]