tor-project March 2025

tor-project@lists.torproject.org

12 participants
18 discussions

TPA-RFC-80: Debian 13 ("trixie") upgrade schedule
by Antoine Beaupré 23 Apr '25

23 Apr '25

Summary: start upgrading servers during the Debian 13 ("trixie") freeze, if it goes well, complete most of the fleet upgrade in around June 2025, with full completion by the end of 2025, with a 2026 year free of major upgrades entirely. Improve automation, retire old container images. Deadline: 2 weeks, 2025-04-01 # Background Debian 13 ("trixie"), currently "testing", is going into freeze soon, which means we should have a new Debian stable release in 2025. It has been a long-standing tradition at TPA to collaborate in the Debian development process and part of that process is to upgrade our servers during the freeze. Upgrading during the freeze makes it easier for us to fix bugs as we find them and contribute them to the community. The [freeze dates announced by the debian.org release team][] are: 2025-03-15 - Milestone 1 - Transition and toolchain freeze 2025-04-15 - Milestone 2 - Soft Freeze 2025-05-15 - Milestone 3 - Hard Freeze - for key packages and packages without autopkgtests To be announced - Milestone 4 - Full Freeze We have entered the "transition and toolchain freeze" which locks changes on packages like compilers and interpreters unless exceptions. See the [Debian freeze policy][] for an explanation of each step. Even though we've just completed the Debian 11 ("bullseye") and 12 ("bookworm") upgrades in late 2024, we feel it's a good idea to start *and* complete the Debian 13 upgrades in 2025. That way, we can hope of having a year or two (2026-2027?) *without* any major upgrades. This proposal is part of the [Debian 13 trixie upgrade milestone][], itself part of the [2025 TPA roadmap][]. [freeze dates announced by the debian.org release team]: https://lists.debian.org/debian-devel-announce/2025/01/msg00004.html [Debian freeze policy]: https://release.debian.org/testing/freeze_policy.html [Debian 13 trixie upgrade milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/12 [2025 TPA roadmap]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2025 # Proposal As usual, we perform the upgrades in three batches, in increasing order of complexity, starting in 2025Q2, hoping to finish by the end of 2025. Note that, this year, this proposal also includes upgrading the Tails infrastructure as well. To help with merging rotations in the two teams, TPA staff will upgrade Tails machines, with Tails folks assistance, and vice-versa. ## Affected users All service admins are affected by this change. If you have shell access on any TPA server, you want to read this announcement. In the past, TPA has typically kept a page detailing notable changes and a proposal like this one would link against the upstream release notes. Unfortunately, at the time writing, upstream hasn't yet produced release notes (as we're still in testing). We're hoping the documentation will be refined by the time we're ready to coordinate the second batch of updates, around May 2025, when we will send reminders to affected teams. We do expect the Debian 13 upgrade to be less disruptive than bookworm, mainly because Python 2 is already retired. ## Notable changes For now, here are some known changes that are already in Debian 13: | Package | 12 (bookworm) | 13 (trixie) | |--------------------|---------------|-------------| | Ansible | 7.7 | 11.2 | | Apache | 2.4.62 | 2.4.63 | | Bash | 5.2.15 | 5.2.37 | | Emacs | 28.2 | 30.1 | | Fish | 3.6 | 4.0 | | Git | 2.39 | 2.45 | | GCC | 12.2 | 14.2 | | Golang | 1.19 | 1.24 | | Linux kernel image | 6.1 series | 6.12 series | | LLVM | 14 | 19 | | MariaDB | 10.11 | 11.4 | | Nginx | 1.22 | 1.26 | | OpenJDK | 17 | 21 | | OpenLDAP | 2.5.13 | 2.6.9 | | OpenSSL | 3.0 | 3.4 | | PHP | 8.2 | 8.4 | | Podman | 4.3 | 5.4 | | PostgreSQL | 15 | 17 | | Prometheus | 2.42 | 2.53 | | Puppet | 7 | 8 | | Python | 3.11 | 3.13 | | Rustc | 1.63 | 1.85 | | Vim | 9.0 | 9.1 | Most of those, except "tool chains" (e.g. LLVM/GCC) can still change, as we're not in the full freeze yet. ## Upgrade schedule The upgrade is split in multiple batches: - automation and installer changes - low complexity: mostly TPA services and less critical Tails servers - moderate complexity: TPA "service admins" machines and remaining Tails physical servers and VMs running services from the official Debian repositories only - high complexity: Tails VMs running services not from the official Debian repositories - cleanup The free time between the first two batches will also allow us to cover for unplanned contingencies: upgrades that could drag on and other work that will inevitably need to be performed. The objective is to do the batches in collective "upgrade parties" that should be "fun" for the team. This policy has proven to be effective in the previous upgrades and we are eager to repeat it again. ### Upgrade automation and installer changes First, we tweak the installers to deploy Debian 13 by default to avoid installing further "old" systems. This includes the bare-metal installers but also and especially the virtual machine installers and container images. Concretely, we're planning on changing the `latest` container image tag to point to `trixie` in early April. A full *year* later, the `bookworm` container images will be retired. Note that we are already planning the retirement of the "old stable" (`bullseye`) container images, see [tpo/tpa/base-images#19][], for which you may have already been contacted. New `idle` canary servers will be setup in Debian 13 to test integration with the rest of the infrastructure, and future new machine installs will be done in Debian 13. We also want to work on automating the upgrade procedure further. We've had catastrophic errors in the PostgreSQL upgrade procedure in the past, in particular, but the whole procedure is now considered ripe for automation, see [tpo/tpa/team#41485][] for details. [tpo/tpa/base-images#19]: https://gitlab.torproject.org/tpo/tpa/base-images/-/issues/19 [tpo/tpa/team#41485]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41485 ### Batch 1: low complexity This is scheduled during two weeks: TPA boxes will be upgraded in the last week of April, and Tails in the first week of May. The idea is to start the upgrade long enough before the vacations to give us plenty of time to recover, and some room to start the second batch. In April, Debian should also be in "soft freeze", not quite a fully "stable" environment, but that should be good enough for simple setups. 35 TPA machines: ``` archive-01.torproject.org cdn-backend-sunet-02.torproject.org chives.torproject.org dal-rescue-01.torproject.org dal-rescue-02.torproject.org gayi.torproject.org hetzner-hel1-02.torproject.org hetzner-hel1-03.torproject.org hetzner-nbg1-01.torproject.org hetzner-nbg1-02.torproject.org idle-dal-02.torproject.org idle-fsn-01.torproject.org lists-01.torproject.org loghost01.torproject.org mandos-01.torproject.org media-01.torproject.org minio-01.torproject.org mta-dal-01.torproject.org mx-dal-01.torproject.org neriniflorum.torproject.org ns3.torproject.org ns5.torproject.org palmeri.torproject.org perdulce.torproject.org srs-dal-01.torproject.org ssh-dal-01.torproject.org static-gitlab-shim.torproject.org staticiforme.torproject.org static-master-fsn.torproject.org submit-01.torproject.org vault-01.torproject.org web-dal-07.torproject.org web-dal-08.torproject.org web-fsn-01.torproject.org web-fsn-02.torproject.org ``` 4 Tails machines: ``` ecours.tails.net puppet.lizard skink.tails.net stone.tails.net ``` In the [first batch of bookworm machines][], we ended up taking 20 minutes per machine, done in a single day, but warned that the second batch took longer. It's probably safe to estimate 20 hours (30 minutes per machine) for this work, in a single week. Feedback and coordination of this batch happens in [issue batch 1][]. [first batch of bookworm machines]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41251 [issue batch 1]: "https://gitlab.torproject.org/tpo/tpa/team/-/issues/42071" ### Batch 2: moderate complexity This is scheduled for the last week of may for TPA machines, and the first week of June for Tails. At this point, Debian testing should be in "hard freeze", which should be more stable. 40 TPA machines: ``` anonticket-01.torproject.org backup-storage-01.torproject.org bacula-director-01.torproject.org btcpayserver-02.torproject.org bungei.torproject.org carinatum.torproject.org check-01.torproject.org ci-runner-x86-02.torproject.org ci-runner-x86-03.torproject.org colchicifolium.torproject.org collector-02.torproject.org crm-int-01.torproject.org dangerzone-01.torproject.org donate-01.torproject.org donate-review-01.torproject.org forum-01.torproject.org gitlab-02.torproject.org henryi.torproject.org materculae.torproject.org meronense.torproject.org metricsdb-01.torproject.org metricsdb-02.torproject.org metrics-store-01.torproject.org onionbalance-02.torproject.org onionoo-backend-03.torproject.org polyanthum.torproject.org probetelemetry-01.torproject.org rdsys-frontend-01.torproject.org rdsys-test-01.torproject.org relay-01.torproject.org rude.torproject.org survey-01.torproject.org tbb-nightlies-master.torproject.org tb-build-02.torproject.org tb-build-03.torproject.org tb-build-06.torproject.org tb-pkgstage-01.torproject.org tb-tester-01.torproject.org telegram-bot-01.torproject.org weather-01.torproject.org ``` 17 Tails machines: ``` apt-proxy.lizard apt.lizard bitcoin.lizard bittorrent.lizard bridge.lizard dns.lizard dragon.tails.net gitlab-runner.iguana iguana.tails.net lizard.tails.net mail.lizard misc.lizard puppet-git.lizard rsync.lizard teels.tails.net whisperback.lizard www.lizard ``` The [second batch of bookworm upgrades][] took 33 hours for 31 machines, so about one hour per box. Here we have 57 machines, so it will likely take us 60 hours (or two weeks) to complete the upgrade. Feedback and coordination of this batch happens in [issue batch 2][]. [second batch of bookworm upgrades]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252 [issue batch 2]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42070 ### Batch 3: high complexity Those machines are harder to upgrade, or more critical. In the case of TPA machines, we typically regroup the Ganeti servers and all the "snowflake" servers that are not properly Puppetized and full of legacy, namely the LDAP, DNS, and Puppet servers. That said, we waited a long time to upgrade the Ganeti cluster for bookworm, and it turned out to be trivial, so perhaps those could eventually be made part of the second batch. 15 TPA machines: ``` - [ ] alberti.torproject.org - [ ] dal-node-01.torproject.org - [ ] dal-node-02.torproject.org - [ ] dal-node-03.torproject.org - [ ] fsn-node-01.torproject.org - [ ] fsn-node-02.torproject.org - [ ] fsn-node-03.torproject.org - [ ] fsn-node-04.torproject.org - [ ] fsn-node-05.torproject.org - [ ] fsn-node-06.torproject.org - [ ] fsn-node-07.torproject.org - [ ] fsn-node-08.torproject.org - [ ] nevii.torproject.org - [ ] pauli.torproject.org - [ ] puppetdb-01.torproject.org ``` It seems like the [bookworm Ganeti upgrade][] took roughly 10h of work. We ballpark the rest of the upgrade to another 10h of work, so possibly 20h. 11 Tails machines: ``` - [ ] isoworker1.dragon - [ ] isoworker2.dragon - [ ] isoworker3.dragon - [ ] isoworker4.dragon - [ ] isoworker5.dragon - [ ] isoworker6.iguana - [ ] isoworker7.iguana - [ ] isoworker8.iguana - [ ] jenkins.dragon - [ ] survey.lizard - [ ] translate.lizard ``` The challenge with Tails upgrades is the coordination with the Tails team, in particular for the Jenkins upgrades. Feedback and coordination of this batch happens in [issue batch 3][]. [bookworm Ganeti upgrade]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41254 [issue batch 3]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42069 ### Cleanup work Once the upgrade is completed and the entire fleet is again running a single OS, it's time for cleanup. This involves updating configuration files to the new versions and removing old compatibility code in Puppet, removing old container images, and generally wrapping things up. This process has been historically neglected, but we're hoping to wrap this up, worst case in 2026. ## Timeline - 2025-Q2 - W14 (first week of April): default container image changed to `trixie`, installer defaults changed and first tests in production - W18 (last week of April): Batch 1 upgrades, TPA machines - W19 (first week of May): Batch 1 upgrades, Tails machines - W22 (last week of May): Batch 2 upgrades, TPA machines - W23 (first week of June): Batch 2 upgrades, Tails machines - 2025-Q3 to Q4: Batch 3 upgrades - 2026-Q2: bookworm container image retired ## Deadline The community has until the beginning of the above timeline to manifest concerns or objections. Two weeks before performing the upgrades of each batch, a new announcement will be sent with details of the changes and impacted services. # Alternatives considered ## Retirements or rebuilds We do not plan any major upgrade or retirements in the third phase this time. In the future, we hope to decouple those as much as possible, as the Icinga retirement and Mailman 3 became blockers that slowed down the upgrade significantly for bookworm. In both cases, however, the upgrades *were* challenging and had to be performed one way or another, so it's unclear if we can optimize this any further. We are clear, however, that we will not postpone an upgrade for a server retirement. Dangerzone, for example, is scheduled for retirement ([TPA-RFC-78][]) but is still planned as normal above. [TPA-RFC-78]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-78-danger… # Costs | Task | Estimate | Certainty | Worst case | |-------------------|----------|-----------|------------| | Automation | 20h | extreme | 100h | | Installer changes | 4h | low | 4.4h | | Batch 1 | 20h | low | 22h | | Batch 2 | 60h | medium | 90h | | Batch 3 | 20h | high | 40h | | Cleanup | 20h | medium | 30h | | **Total** | 144h | ~high | ~286h | The entire work here should consist of over 140 hours of work, or 18 days, or about 4 weeks full time. Worst case doubles that. The above is done in "hours" because that's how we estimated batches in the past, but here's an estimate that's based on the [Kaplan-Moss estimation technique][]. [Kaplan-Moss estimation technique]: https://jacobian.org/2021/may/25/my-estimation-technique/ | Task | Estimate | Certainty | Worst case | |-------------------|----------|-----------|------------| | Automation | 3d | extreme | 15d | | Installer changes | 1d | low | 1.1d | | Batch 1 | 3d | low | 3.3d | | Batch 2 | 10d | medium | 20d | | Batch 3 | 3d | high | 6d | | Cleanup | 3d | medium | 4.5d | | **Total** | 23d | ~high | ~50d | This is *roughly* equivalent, if a little higher (23 days instead of 18), for example. It should be noted that automation is not expected to drastically reduce the total time spent in batches (currently 16 days or 100 hours). The main goal of automation is more to reduce the likelihood of catastrophic errors, and make it easier to share our upgrade procedure with the world. We're still hoping to reduce the time spent in batches, hopefully by 10-20%, which would bring the total number of days across batches from 16 days to 14d, or from 100 h to 80 hours. # Approvals required This proposal needs approval from TPA team members, but service admins can request additional delay if they are worried about their service being affected by the upgrade. Comments or feedback can be provided in issues linked above, or the general process can be commented on in issue [tpo/tpa/team#41990][]. # References * [Debian 13 trixie upgrade milestone][] * [discussion ticket][tpo/tpa/team#41990] [TPA bookworm upgrade procedure]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm [tpo/tpa/team#41990]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41990 -- Antoine Beaupré torproject.org system administration

1 3

Felicia's Monthly Status Report, March 2025
by Priscila 31 Mar '25

31 Mar '25

Hi everyone, This is my status report for March 2025. It's a list of all tasks I've been involved with/responsible for. ## Design tasks 1. Allow authentication of Onion Services on Android: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/31672 2. Figure out how to display conflux circuits in Tor Browser’s UI: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/41716 3. Review error prompt shown when Tor exits: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43186 4. Update Connection Assist for Android designs: https://gitlab.torproject.org/tpo/ux/design/-/issues/136 ## UX support 1. TorSettings handles failures to apply Tor settings: https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… 2. Tor Browser User Experience survey 2025: https://gitlab.torproject.org/tpo/ux/research/-/issues/144 3. Bug 43473: change QuickStart to connect automatically in settings: https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… 4. Add user feedback for when Tor connects (Android): https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43349#n… 5. Draft: Bug_41188: Wire up stages to UI: https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… 6. Adjust preferences for contrast theme or forced colors: https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… 7. Proposal: make design happen in ux/design going forward: https://gitlab.torproject.org/tpo/ux/team/-/issues/105 Best, Priscila (aka Felicia) - UX Team

1 0

cve's monthly status report 03/2025
by Clara Engler 31 Mar '25

31 Mar '25

Hello tor-project@, this month I have done the following things for the Tor project. # Onion Service Reliability I performed research on the reliability of onion services in various setup environments, including arti to arti, ctor to ctor and the two ones being mixed together. These findings resulted in the conclusion that the arti implementation is well-done and has a similar failure of its ctor pedant, which is around 0.5% or 1 in every 200 connection being a failure. Those failures are flaky circuits which probably happen due to simply having bad luck while choosing the circuit. Nonetheless, those results are acceptable. Besides, this research has led to an improvement within the arti documentation, namely the addition of a method named `is_fully_reachable` which checks whether the `State` of a `tor-hsservice` indicates that it is reachable from the outside. This change was necessary because there are right now two states which indicate that the service is reachable, with only one of them sounding like a success. After all, this research led to the creation of the following tickets and merge requests in the arti repository: * arti#1887 * arti#1890 * arti!2850 # GPN23 I have opened a ticket (outreach#40103) for Tor activities at the GPN23 in Karlsruhe this June. I will be present at the event myself and I am open for giving a Tor related talk there too. This is however still early planning. # oniux This month, I continued working on oniux, namely statically integrating onionmasq into the oniux binary, so oniux users no longer need to provide a path to the onionmasq binary themselves; instead oniux is now a beautiful self-container monolithic binary. Besides all of this, I also did some research regarding the use of `user_namespaces(7)` in oniux with the eventual goal of oniux being capable to no longer need any `capabilities(7)`. This change is non-trivial and would probably require a rewrite of the tool in a shell scripting language, as a network stack such as [pasta](https://passt.top) would be required for proper functioning. Right now, this is not a priority but a nice to have. Our current use of capabilities is probably not a security risk, as all capabilities are being dropped very early on and no code originating from a malicious actor is executed with capabilities at all. Also, I have made two release tags for oniux this month: v0.1.0 and v0.2.0. Last but not least I have fixed a bug, namely that procfs was not remounted in the PID namespace created by oniux leading to an incorrect view when performing process related operations such as `ps aux`. After all, the following tickets have been addressed: * oniux#1 (for now) * oniux#2 * oniux#3 # onionmasq This month I gave onionmasq more attention again, this mainly included fixing various RenovateBot related updates and keeping this up-to-date in general. Besides this, I have performed some clean-ups and fixed some clippy warnings that occurred during the development. Also, as outlined in oniux, I have moved code a little bit around, meaning it is now possible to integrate onionmasq as a library into existing Rust application like I did with oniux. Last but not least I performed some testing with the onionmasq CI in order to hopefully make it possible to run the integration tests outside of privileged containers, there has not been much success there though. After all, my work cumulated in the following tickets/MR's: * onionmasq#135 * onionmasq#136 * onionmasq#137 * onionmasq#138 * onionmasq!404 * onionmasq!403 * onionmasq!401 (WIP) * onionmasq!400 * onionmasq!398 * onionmasq!397 (WIP) * onionmasq!393 # arti Besides the addition of the `is_fully_reachable` method, I also support and supported gabi@ in the release of the new arti version.

1 0

Anti-censorship team meeting notes, 2025-03-27
by meskio 27 Mar '25

27 Mar '25

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-03-27-16.00.html And our meeting pad: Anti-censorship -------------------------------- Next meeting: Thursday, April 3 16:00 UTC Facilitator: onyinyang ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: meskio == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from projects, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Project 158 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == * == Discussion == * Give @renovate-bot guest access to anti-censorship group for it to use dependency proxy src shell * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * shelikhoo will give it rights on monday == Actions == == Interesting links == * == Reading group == * We will discuss "Differential Degradation Vulnerabilities in Censorship Circumvention Systems" on April 3 * https://arxiv.org/abs/2409.06247 * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2025-03-27 Last week: - deployed a new snowflake broker with logging and client timeout fixes - followed up on cleaning up broker log messages (snowflake#40454) - snowflake!545 - wrote an update on snowflake sqs rendezvous channel - looked more at rendezvous errors (snowflake#40447) - tor safety board review This week: - support conjure work - follow up on snowflake rendezvous failures - reduce/remove use of mutexes for broker metrics (snowflake#40458) - take a look at potential snowflake orbot bug - https://github.com/guardianproject/orbot-android/issues/1183 - maybe do some lox work dcf: 2025-03-20 Last week: - archived a Kazakhstan URL blocklist https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/iss… Next week: - archive snowflake-webextension-0.9.3 - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Help with: meskio: 2024-03-27 Last week: - crate a docker-compose file for rdsys (rdsys#219) - try and fail to update obfs4-proxy docker image, arm* gives me timeouts in go get (docker-obfs4-bridge#21) - deploy rdsys backend fixes to don't distribute dysfunctional bridges (team#156) Next week: - steps towards a rdsys in containers (rdsys#219) Shelikhoo: 2024-03-27 Last Week: - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) testing environment setup/research - Snowflake Staging Server (cont.) Discussion(https://gitlab.torproject.org/tpo/tpa/team/-/issues/42080) - Snowfalke Staging Server Experiment - Merge request reviews Next Week/TODO: - Merge request reviews - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) improvements - Snowfalke Staging Server Experiment onyinyang: 2025-03-13 Last week(s): - continued work on ampcache registration method for conjure - WIP MR: https://github.com/cohosh/conjure/pull/1 - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj… - lox issuer efficiency merged! Next week: - HACS/Zkproofs/OSCW/RWC - finish up ampcache registration method - Begin work on decoy registration option - review Tor browser Lox integration - https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… - add TTL cache to lox MR for duplicate responses: https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305 As time allows: - Work on outstanding milestone issues: - key rotation automation Later: pending decision on abandoning lox wasm in favour of some kind of FFI? https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096): - add pref to handle timing for pubkey checks in Tor browser - add trusted invitation logic to tor browser integration: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974 - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2025-03-06 Last weeks: - Implementing key_share extension support in covert-dtls - Implementing mimicking DTLS 1.3 extensions - Debugging Tor Build with covert-dtls: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Next weeks: - Update MR with new covert-dtls version. - Write instructions on how to configure covert-dtls with snowflake client - Fix merge conflicts in MR (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…). - Condensing thesis into paper (on hold) Help with: - Test stability of covert-dtls in snowflake Facilitator Queue: onyinyang shelikhoo meskio 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue -- meskio | https://meskio.net/ -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- My contact info: https://meskio.net/crypto.txt -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Nos vamos a Croatan.

1 0

TPA-RFC-71: Emergency email deployments, phase B
by Antoine Beaupré 26 Mar '25

26 Mar '25

Hi, So TPA has adopted this proposal, internally, to make yet another set of emergency changes to our mail system, to respond to critical issues affecting delivery and sustainability of our infrastructure. I encourage you to read the "Affected users" section and "Timeline" below. In particular, we will be experimenting with "sender rewriting" soon, which will involve mangling emails we forward around to try and fix deliverability on those. The schleuder mailing list will also move servers. Maintenance windows for those changes will be communicated separately. Thank you for your attention! PS: and no, we didn't submit this for adoption to everyone, because it was felt it was mostly technical changes that didn't warrant outside approval, let me know if that doesn't make sense, of course. -- Antoine Beaupré torproject.org system administration --- title: TPA-RFC-71: Emergency email deployments, phase B costs: staff approval: TPA affected users: all torproject.org email users deadline: 5 days, 2024-10-01 status: draft discussion: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41778 --- Summary: deploy a new sender-rewriting mail forwarder ASAP, migrate mailing lists off the legacy server to a new machine, migrate the remaining Schleuder list to the Tails server, upgrade `eugeni`. Table of contents: - Background - Proposal - Actual changes - Mailman 3 upgrade - New sender-rewriting mail exchanger - Schleuder migration - Upgrade legacy mail server - Goals - Must have - Nice to have - Non-Goals - Scope - Affected users - Personas - Timeline - Optimistic timeline - Worst case scenario - Alternatives considered - References - History - Personas descriptions - Ariel, the fundraiser - Blipblop, the bot - Gary, the support guy - John, the contractor - Mallory, the director - Nancy, the fancy sysadmin - Orpheus, the developer # Background In [#41773][], we had yet another report of issues with mail delivery, particularly with email forwards, that are plaguing Gmail-backed aliases like grants@ and travel@. [#41773]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41773 This is becoming critical. It has been impeding people's capacity of using their email at work for a while, but it's been more acute since google's recent changes in email validation (see [#41399][]) as now hosts that have adopted the SPF/DKIM rules are bouncing. [#41399]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41399 On top of that, we're way behind on our buster upgrade schedule. We still have to upgrade our primary mail server, `eugeni`. The plan for that ([TPA-RFC-45][], [#41009][]) was to basically re-architecture everything. That won't happen fast enough for the LTS retirement which we have crossed two months ago (in July 2024) already. [TPA-RFC-45]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-45-mail-a… [#41009]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41009 So, in essence, our main mail server is unsupported now, and we need to fix this as soon as possible Finally, we also have problems with certain servers (e.g. `state.gov`) that seem to dislike our bespoke certificate authority (CA) which makes *receiving* mails difficult for us. # Proposal So those are the main problems to fix: - Email forwarding is broken - Email reception is unreliable over TLS for some servers - Mail server is out of date and hard to upgrade (mostly because of Mailman) ## Actual changes The proposed solution is: - **Mailman 3 upgrade** ([#40471][]) - **New sender-rewriting mail exchanger** ([#40987][]) - **Schleuder migration** - **Upgrade legacy mail server** ([#40694][]) [#40471]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40471 [#40987]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40987 [#40694]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40694 ### Mailman 3 upgrade Build a new mailing list server to host the upgraded Mailman 3 service. Move old lists over and convert them while retaining the old archives available for posterity. This includes lots of URL changes and user-visible disruption, little can be done to work around that necessary change. We'll do our best to come up with redirections and rewrite rules, but ultimately this is a disruptive change. This involves yet another authentication system being rolled out, as Mailman 3 has its own user database, just like Mailman 2. At least it's one user per site, instead of per list, so it's a slight improvement. This is issue [#40471][]. ### New sender-rewriting mail exchanger This step is carried over from [TPA-RFC-45][], mostly unchanged. [Sender Rewriting Scheme]: https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme [postsrsd]: https://github.com/roehling/postsrsd [postforward]: https://github.com/zoni/postforward Configure a new "mail exchanger" (MX) server with TLS certificates signed by our normal public CA (Let's Encrypt). This replaces that part of `eugeni`, will hopefully resolve issues with `state.gov` and others ([#41073][], [#41287][], [#40202][], [#33413][]). [#33413]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/33413 [#40202]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40202 [#41287]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41287 [#41073]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41073 This would handle forwarding mail to other services (e.g. mailing lists) but also end-users. To work around reputation problems with forwards ([#40632][], [#41524][], [#41773][]), deploy a [Sender Rewriting Scheme][] (SRS) with [postsrsd][] (packaged in Debian, but [not in the best shape][]) and [postforward][] (not packaged in Debian, but zero-dependency Golang program). It's possible deploying [ARC][] headers with [OpenARC][], Fastmail's [authentication milter][] (which [apparently works better][]), or [rspamd's arc module][] might be sufficient as well, to be tested. [OpenARC]: https://tracker.debian.org/pkg/openarc [not in the best shape]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017361 Having it on a separate mail exchanger will make it easier to swap in and out of the infrastructure if problems would occur. The mail exchangers should also sign outgoing mail with DKIM, and *may* start doing better validation of incoming mail. [authentication milter]: https://github.com/fastmail/authentication_milter [apparently works better]: https://old.reddit.com/r/postfix/comments/17bbhd2/about_arc/k5iluvn/ [rspamd's arc module]: https://rspamd.com/doc/modules/arc.html [#41524]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41524 [#40632]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40632 [ARC]: http://arc-spec.org/ ### Schleuder migration Migrate the remaining mailing list left (the Community Council) to the Tails Shleuder server, retiring our Schleuder server entirely. This requires configuring the Tails server to accept mail for `(a)torproject.org`. Note that this may require changing the addresses of the existing Tails list to `(a)torproject.org` if Schleuder doesn't support virtual hosting (which is likely). ### Upgrade legacy mail server Once Mailman has been safely moved aside and is shown to be working correctly, upgrade Eugeni using the normal procedures. This should be a less disruptive upgrade, but is still risky because it's such an old box with lots of legacy. One key idea of this proposal is to keep the legacy mail server, `eugeni`, in place. It will continue handling the "MTA" (Mail Transfer Agent) work, which is to relay mail for other hosts, as a legacy system. The full eugeni replacement is seen as too complicated and unnecessary at this stage. The legacy server will be isolated from the rewriting forwarder so that outgoing mail is mostly unaffected by the forwarding changes. ## Goals This is not an exhaustive solution to all our email problems, [TPA-RFC-45][] is that longer-term project. ### Must have - Up to date, supported infrastructure. - Functional legacy email forwarding. ### Nice to have - Improve email forward deliverability to Gmail. ### Non-Goals - **Clean email forwarding**: email forwards *may* be mangled and rewritten to appear as coming from `(a)torproject.org` instead of the original address. This will be figured out at the implementation stage. - **Mailbox storage**: out of scope, see [TPA-RFC-45][]. It is hoped, however, that we *eventually* are able to provide such a service, as the sender-rewriting stuff might be too disruptive in the long run. - **Technical debt**: we keep the legacy mail server, `eugeni`. - **Improved monitoring**: we won't have a better view in how well we can deliver email. - **High availability**: the new servers will not add additional "single point of failures", but will not improve our availability situation (issue [#40604][]) [#40604]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40604 ## Scope This proposal affects the all inbound and outbound email services hosted under `torproject.org`. Services hosted under `torproject.net` are *not* affected. It also does *not* address directly phishing and scamming attacks ([#40596][]), but it is hoped the new mail exchanger will provide a place where it is easier to make such improvements in the future. [#40596]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40596 ## Affected users This affects all users which interact with `torproject.org` and its subdomains over email. It particularly affects all "tor-internal" users, users with LDAP accounts, or forwards under `(a)torproject.org`, as their mails will get rewritten on the way out. ### Personas Here we collect a few "personas" and try to see how the changes will affect them, largely derived from [TPA-RFC-45][], but without the alpha/beta/prod test groups. For *all* users, a common impact is that emails will be rewritten by the sender rewriting system. As mentioned above, the impact of this still remains to be clarified, but at least the hidden `Return-Path` header will be changed for bounces to go to our servers. Actual personas are in the Reference section, see [Personas descriptions][]. | Persona | Task | Impact | |---------|-------------|--------------------------------------------------------------------------| | Ariel | Fundraising | Improved incoming delivery | | Blipbot | Bot | No change | | Gary | Support | Improved incoming delivery, new moderator account on mailing list server | | John | Contractor | Improved incoming delivery | | Mallory | Director | Same as Ariel | | Nancy | Sysadmin | No change in delivery, new moderator account on mailing list server | | Orpheus | Developer | No change in delivery | [Personas descriptions]: #personas-descriptions ## Timeline ### Optimistic timeline - Late September (W39): issue raised again, proposal drafted (now) - October: - W40: proposal approved, installing new rewriting server - W41: rewriting server deployment, new mailman 3 server - W42: mailman 3 mailing list conversion tests, users required for testing - W43: mailman 2 retirement, mailman 3 in production - W44: Schleuder mailing list migration - November: - W45: `eugeni` upgrade ### Worst case scenario - Late September (W39): issue raised again, proposal drafted (now) - October: - W40: proposal approved, installing new rewriting server - W41-44: difficult rewriting server deployment - November: - W44-W48: difficult mailman 3 mailing list conversion and testing - December: - W49: Schleuder mailing list migration vetoed, Schleuder stays on `eugeni` - W50-W51: `eugeni` upgrade postponed to 2025 - January 2025: - W3: `eugeni` upgrade # Alternatives considered We decided to not just run the sender-rewriting on the legacy mail server because too many things are tangled up in that server. It is just too risky. We have also decided to not upgrade Mailman in place for the same reason: it's seen as too risky as well, because we'd first need to upgrade the Debian base system and if that fails, rolling back is too hard. # References - [discussion issue][] [discussion issue]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41778 ## History This is the the *fifth* proposal about our email services, here are the previous ones: * [TPA-RFC-15: Email services][] (rejected, replaced with TPA-RFC-31) * [TPA-RFC-31: outsource email services][] (rejected, in favor of TPA-RFC-44 and following) * [TPA-RFC-44: Email emergency recovery, phase A][] (standard, and mostly implemented except the sender-rewriting) * [TPA-RFC-45: Mail architecture][] (still draft) [TPA-RFC-15: Email services]: policy/tpa-rfc-15-email-services [TPA-RFC-31: outsource email services]: policy/tpa-rfc-31-outsource-email [TPA-RFC-44: Email emergency recovery, phase A]: policy/tpa-rfc-44-email-emergency-recovery [TPA-RFC-45: Mail architecture]: policy/tpa-rfc-45-mail-architecture ## Personas descriptions ### Ariel, the fundraiser Ariel does a lot of mailing. From talking to fundraisers through their normal inbox to doing mass newsletters to thousands of people on CiviCRM, they get a lot done and make sure we have bread on the table at the end of the month. They're awesome and we want to make them happy. Email is absolutely mission critical for them. Sometimes email gets lost and that's a major problem. They frequently tell partners their personal Gmail account address to work around those problems. Sometimes they send individual emails through CiviCRM because it doesn't work through Gmail! Their email forwards to Google Mail and they now have an LDAP account to do email delivery. ### Blipblop, the bot Blipblop is not a real human being, it's a program that receives mails and acts on them. It can send you a list of bridges (bridgedb), or a copy of the Tor program (gettor), when requested. It has a brother bot called Nagios/Icinga who also sends unsolicited mail when things fail. There are also bots that sends email when commits get pushed to some secret git repositories. ### Gary, the support guy Gary is the ticket overlord. He eats tickets for breakfast, then files 10 more before coffee. A hundred tickets is just a normal day at the office. Tickets come in through email, RT, Discourse, Telegram, Snapchat and soon, TikTok dances. Email is absolutely mission critical, but some days he wishes there could be slightly less of it. He deals with a lot of spam, and surely something could be done about that. His mail forwards to Riseup and he reads his mail over Thunderbird and sometimes webmail. Some time after TPA-RFC_44, Gary managed to finally get an OpenPGP key setup and TPA made him a LDAP account so he can use the submission server. He has already abandoned the Riseup webmail for TPO-related email, since it cannot relay mail through the submission server. ### John, the contractor John is a freelance contractor that's really into privacy. He runs his own relays with some cools hacks on Amazon, automatically deployed with Terraform. He typically run his own infra in the cloud, but for email he just got tired of fighting and moved his stuff to Microsoft's Office 365 and Outlook. Email is important, but not absolutely mission critical. The submission server doesn't currently work because Outlook doesn't allow you to add just an SMTP server. John does have an LDAP account, however. ### Mallory, the director Mallory also does a lot of mailing. She's on about a dozen aliases and mailing lists from accounting to HR and other unfathomable things. She also deals with funders, job applicants, contractors, volunteers, and staff. Email is absolutely mission critical for her. She often fails to contact funders and critical partners because `state.gov` blocks our email -- or we block theirs! Sometimes, she gets told through LinkedIn that a job application failed, because mail bounced at Gmail. She has an LDAP account and it forwards to Gmail. She uses Apple Mail to read their mail. ### Nancy, the fancy sysadmin Nancy has all the elite skills in the world. She can configure a Postfix server with her left hand while her right hand writes the Puppet manifest for the Dovecot authentication backend. She browses her mail through a UUCP over SSH tunnel using mutt. She runs her own mail server in her basement since 1996. Email is a pain in the back and she kind of hates it, but she still believes entitled to run their own mail server. Her email is, of course, hosted on her own mail server, and she has an LDAP account. She has already reconfigured her Postfix server to relay mail through the submission servers. ### Orpheus, the developer Orpheus doesn't particular like or dislike email, but sometimes has to use it to talk to people instead of compilers. They sometimes have to talk to funders (`#grantlyfe`), external researchers, teammates or other teams, and that often happens over email. Sometimes email is used to get important things like ticket updates from GitLab or security disclosures from third parties. They have an LDAP account and it forwards to their self-hosted mail server on a OVH virtual machine. They have already reconfigured their mail server to relay mail over SSH through the jump host, to the surprise of the TPA team. Email is not mission critical, and it's kind of nice when it goes down because they can get in the zone, but it should really be working eventually. -- Antoine Beaupré torproject.org system administration -- tpa-team mailing list tpa-team(a)lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tpa-team

2 10

Anti-censorship team meeting notes, 2025-03-20
by Shelikhoo 20 Mar '25

20 Mar '25

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-03-20-16.01.html And our meeting pad: Anti-censorship work meeting pad -------------------------------- Anti-censorship -------------------------------- Next meeting: Thursday,March, 27 16:00 UTC Facilitator: meskio ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: shelikhoo == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from projects, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Project 158 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == * == Discussion == Mar 20 New: * Snowflake Staging Server & rdsys containerization * TPA supported system doesn't allow to have more than one domain name per branch in a staging server * snowflake needs two: broker and server * shelikhoo is experiments with k8s + terraform * shelikhoo will keep working on this since there is no overall rejection to this design == Actions == == Interesting links == * https://github.com/doudoulong/Minecruft-PT/blob/main/README.md#tor-browser-… == Reading group == * We will discuss "Differential Degradation Vulnerabilities in Censorship Circumvention Systems" on April 3 * https://arxiv.org/abs/2409.06247 * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2025-03-20 Last week: - found and fixed cause of sqs rendezvous errors (snowflake#40363) - implemented better timeouts at broker for client offers (snowflake#40449) - lower proxy answerTimeout for web-based proxies (snowflake-webext#115) - remove bad relay pattern log message and default-relay-pattern option for broker (snowflake#40454) - followed up on Snowflake rendezvous errors in general (snowflake#40447) - tagged and released snowflake proxy v2.11.0 - confirmed fixes for domain fronting issues on old android versions (lyrebird#40012) - opened issue to update PTs in Tor Browser (tor-browser-build#41299) - released snowflake-webext 0.9.3 This week: - support conjure work - follow up on snowflake rendezvous failures - reduce/remove use of mutexes for broker metrics (snowflake#40458) - take a look at potential snowflake orbot bug - https://github.com/guardianproject/orbot-android/issues/1183 - maybe do some lox work dcf: 2025-03-20 Last week: - archived a Kazakhstan URL blocklist https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/iss… Next week: - archive snowflake-webextension-0.9.3 - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Help with: meskio: 2024-03-20 Last week: - build rdsys containers in the CI (rdsys#219) - stop distributing dysfunctional bridges (team#156) - downgrade lyrebird to use go 1.22 (lyrebird#40026) - release lyrebird 0.6.0 - test covertdtls MR proxy (snowflake!448) - debug dependency proxy for snowflake (snowflake!522) Next week: - steps towards a rdsys in containers (rdsys#219) Shelikhoo: 2024-03-20 Last Week: - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) testing environment setup/research - Snowflake Staging Server Discussion(https://gitlab.torproject.org/tpo/tpa/team/-/issues/42080) - Snowfalke Staging Server Experiment - Merge request reviews Next Week/TODO: - Merge request reviews - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) improvements - Snowfalke Staging Server Experiment onyinyang: 2025-03-13 Last week(s): - continued work on ampcache registration method for conjure - WIP MR: https://github.com/cohosh/conjure/pull/1 - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj… - lox issuer efficiency merged! Next week: - HACS/Zkproofs/OSCW/RWC - finish up ampcache registration method - Begin work on decoy registration option - review Tor browser Lox integration - https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… - add TTL cache to lox MR for duplicate responses: https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305 As time allows: - Work on outstanding milestone issues: - key rotation automation Later: pending decision on abandoning lox wasm in favour of some kind of FFI? https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096): - add pref to handle timing for pubkey checks in Tor browser - add trusted invitation logic to tor browser integration: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974 - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2025-03-06 Last weeks: - Implementing key_share extension support in covert-dtls - Implementing mimicking DTLS 1.3 extensions - Debugging Tor Build with covert-dtls: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Next weeks: - Update MR with new covert-dtls version. - Write instructions on how to configure covert-dtls with snowflake client - Fix merge conflicts in MR (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…). - Condensing thesis into paper (on hold) Help with: - Test stability of covert-dtls in snowflake Facilitator Queue: meskio onyinyang shelikhoo 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue

1 0

TPA-RFC-83: Mail log retention
by Antoine Beaupré 18 Mar '25

18 Mar '25

Summary: extend the retention limit for mail logs to 10 days # Background We currently rotate mail logs daily and keep them for 5 days. That's great for privacy, but not so great for people having to report mail trouble in time. In particular, when there are failures with mail sent just before the weekend, it gives users a very short time frame to report issues. # Proposal Extend the retention limit for mail (`postfix` and `rspamd`) logs to 10 days: one week, plus "flexible Friday", plus "weekend". ## Goals Being able to debug mail issues when users notice and/or report them after five days. ## Tasks Adjust logrotate configuration for syslog-ng and rspamd. ## Scope All TPA servers. ## Affected users Sysadmins and email users, which is pretty much everyone. ## Timeline Logging policies will be changed on Wednesday March 19th. # References TPA has various log policies for various services, which we have been meaning to document for a while. This policy proposal doesn't cover for that, see [tpo/tpa/team#40960][] for followup on that more general task. [tpo/tpa/team#40960]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40960 See also the [discussion issue][]. [discussion issue]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42081 -- Antoine Beaupré torproject.org system administration

1 0

Anti-censorship team meeting notes, 2025-03-13
by onyinyang 13 Mar '25

13 Mar '25

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-03-13-16.00.html And our meeting pad: Anti-censorship work meeting pad -------------------------------- Anti-censorship -------------------------------- Next meeting: Thursday, March 20 16:00 UTC Facilitator: shelikhoo ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: onyinyang == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from projects, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Project 158 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == * == Discussion == * snowflake dependencies need go 1.23 * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * should we update it? * go 1.23 is already in debian backports * Tor Browser is already using go 1.23? * no, Tor Browser is staying on 1.21 on macOS until ESR 140, due June 2025 * we will hold off on the dependency updates (which contain changes we believe do not affect us) until after ESR 140 and Tor Browser updating to go 1.23 * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * should we change default relay pattern policy on snowflake broker * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * we still have relays that have not updated since we added support for multiple bridges * The purpose of the allowed relay pattern check at the broker is not to avoid surprising proxy operators by connecting to a new relay destination; it's because old proxies that do not send a relay pattern to the broker cannot connect to any other relay destination. They have snowflake.torproject.net hardcoded and do not know how to connect to any other relay. If a client were to request a connection to the snowflake-02 relay, and was assigned such an old proxy, the proxy would connect to snowflake-01 anyway, and leave the client with a Tor fingerprint mismatch error. * Therefore adjusting the presumedPatternForLegacyClient on the broker to accept old proxies would not work. Old proxies (that don't support relay patterns) would start getting clients again, but any clients that request a relay other than snowflake-01 would get failed connections. * There's no facility on the broker to, say, match old proxies only with those clients that happen to request snowflake-01. Once a proxy is in the pool, it's treated equally to all other proxies. Such a matching facility could conceivably be implemented, but none currently exists. It may not be worth the complexity anyway. * The intention therefore was that the broker simply reject old proxies (that do not send a relay pattern). That is, in effect, what is now happening, but it happens by a roundabout way: proxy polls and doesn't set an allowed relay pattern → broker assigns a default allowed relay pattern that never works → broker rejects the proxy poll because of the relay pattern mismatch. * The downsides of the status quo are that (1) the broker logs are full of "rejected relay pattern from proxy" errors, and (2) the old proxies just keep on uselessly polling. * Consensus for dealing with the current situation: * Make the broker code more straightforward with respect to rejecting old proxies: do it at an early stage, don't even bother assigning a default relay pattern to them (and remove the code that does). * Update metrics/logging. Disable the "rejected relay pattern from proxy" at the broker, and just keep it as a metrics counter. * For the future, it would be handy if there were a feature whereby the broker, by sending a specific kind of response to a proxy, could cause the proxy to disable itself (say, stop polling for 24h) and log a visible message to the operator. In effect, we could push the "rejected relay pattern from proxy" error out to the proxy logs, which would (1) stop proxies that only support an obsolete protocol from uselessly polling, and (2) inform the operator and encourage them to upgrade. * In the absence of such a feature, it might be possible to find a way to crash old proxies by sending a specific kind of broker response. For example, an overlong HTTP body, or a JSON unmarshaling error. Even if it is not possible to convey a reason why it is happening to the proxy operator, it might get the operators attention, and at least reduce unhelpful polling volume. * One complication with this idea: some proxies are running in a container, or similar, and are configured to auto-restart after they exit. Proxies that work this way might paradoxically increase their effective polling rate if remotely disabled, because there's no interval rate limiting on the first poll. * Even so, if we find such a "disable" broker response, we could try having the broker send it to old proxies (those that don't support relay patterns) for 1h. The ones that are configured to restart will restart, but the one that are not will stop operating. * Related issue: "Let the broker inform proxies how often to poll" https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…. The broker could send a time many hours in the future. This would help with reducing the polling rate, even if it does not result in the proxy log message. * A standalone proxy auto-update mechanism could be an alternative or addition to support for a broker "disable" signal. == Actions == == Interesting links == * == Reading group == * We will discuss "" on * * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2025-03-13 Last week: - wrote and deployed a CPU and goroutine profiling patch for the snowflake broker (snowflake#40447) - found a potential performance improvement related to mutexes (snowflake#40458) - confirmed that congestion isn't the cause of sqs rendezvous errors (snowflake#4036) - found source of "rejected relay pattern" log messages (snowflake#40454) - modified broker metrics to show client polls that timed out (snowflake!530) - reviewed merge requests - helped with conjure pt work This week: - support conjure work - follow up on snowflake rendezvous failures - reduce/remove use of mutexes for broker metrics (snowflake#40458) - set absolute time limit on broker http responses (snowflake40449) - snowflake proxy release - take a look at potential snowflake orbot bug - https://github.com/guardianproject/orbot-android/issues/1183 - maybe do some lox work dcf: 2025-03-13 Last week: - snowflake VPS hosting bookkeeping https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Snowflake-co… Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Help with: meskio: 2024-03-13 Last week: - set up snowflake proxies with coverdtls (snowflake!448) - investigate why our email distributor is not handing out emails (rdsys#260) - debug gitlab container proxy (snowflake!522) - investigate why users get bridges with local addresses (team#156) Next week: - steps towards a rdsys in containers (rdsys#219) - investigate rdsys distributing local addressed bridges (team#156) Shelikhoo: 2024-03-13 Last Week: - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) testing environment setup/research - Vantage Point maintaince - Merge request reviews Next Week/TODO: - Merge request reviews - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) improvements - Snowflake Staging Server Discussion(https://gitlab.torproject.org/tpo/tpa/team/-/issues/42080) - Vantage Point maintaince onyinyang: 2025-03-13 Last week(s): - continued work on ampcache registration method for conjure - WIP MR: https://github.com/cohosh/conjure/pull/1 - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj… - lox issuer efficiency merged! Next week: - HACS/Zkproofs/OSCW/RWC - finish up ampcache registration method - Begin work on decoy registration option - review Tor browser Lox integration - https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… - add TTL cache to lox MR for duplicate responses: https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305 As time allows: - Work on outstanding milestone issues: - key rotation automation Later: pending decision on abandoning lox wasm in favour of some kind of FFI? https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096): - add pref to handle timing for pubkey checks in Tor browser - add trusted invitation logic to tor browser integration: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974 - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2025-03-06 Last weeks: - Implementing key_share extension support in covert-dtls - Implementing mimicking DTLS 1.3 extensions - Debugging Tor Build with covert-dtls: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Next weeks: - Update MR with new covert-dtls version. - Write instructions on how to configure covert-dtls with snowflake client - Fix merge conflicts in MR (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…). - Condensing thesis into paper (on hold) Help with: - Test stability of covert-dtls in snowflake Facilitator Queue: onyinyang shelikhoo meskio 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue -- --- onyinyang GPG Fingerprint 3CC3 F8CC E9D0 A92F A108 38EF 156A 6435 430C 2036

1 0

Haidi February '25 Report
by Haidi 11 Mar '25

11 Mar '25

Hi! Below is my February’25 (Period: 2025-02-01 - 2025-02-28) report! In February, I began working on tests and documentation, focusing on integrating pluggable transports with little-t-tor on Windows. I then collected data across different operating systems and contributed to the corresponding GitLab documentation ticket. Spent some time learning the workflow for merging and committing documents in GitLab for the Tor Project. *Add documentation about using pluggable transports with little-t-tor* <https://gitlab.torproject.org/tpo/web/support/-/issues/378#top> https://gitlab.torproject.org/tpo/web/support/-/issues/378 Thanks, Haidi

1 0

minutes from the sysadmins
by Antoine Beaupré 10 Mar '25

10 Mar '25

Hi, We had a quick monthly meeting, here are the minutes. # Roll call: who's there and emergencies anarcat, groente, lavamind, lelutin and zen. lavamind and groente star. Tails pipelines are failing because of issues with the debian APT servers, zen and groente will look into it. # Checkin Normal per-user check-in: - <https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&…> - <https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&…> - <https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&…> - <https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&…> - <https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&…> General dashboards: - <https://gitlab.torproject.org/tpo/tpa/team/-/boards/117> - <https://gitlab.torproject.org/groups/tpo/web/-/boards> - <https://gitlab.torproject.org/groups/tpo/tpa/-/boards> # Roadmap review We reviewed the spreadsheet with plans for March. # Puppet merge broad plan Work is starting in March, there doesn't seem to be objections to the plan. We'll need volunteers to start work on TPA's side as well. anarcat will start haggling people near the end of March, hopefully. # Next meeting As usual. # Metrics of the month * hosts in Puppet: 89, LDAP: 89, Prometheus exporters: 583 * number of Apache servers monitored: 33, hits per second: 679 * number of self-hosted nameservers: 6, mail servers: 89 * pending upgrades: 0, reboots: 0 * average load: 1.57, memory available: 2.95 TiB/5.11 TiB, running processes: 187 * disk free/total: 58.40 TiB/143.90 TiB * bytes sent: 467.62 MB/s, received: 296.47 MB/s * [GitLab tickets][]: 246 tickets including... * open: 0 * icebox: 148 * future: 46 * needs information: 6 * backlog: 24 * next: 7 * doing: 9 * needs review: 7 * (closed: 3980) [Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards -- Antoine Beaupré torproject.org system administration

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

tor-project March 2025