- tor-project - lists.torproject.org

Anti-censorship team meeting notes, 2025-08-28
by Shelikhoo 28 Aug '25

28 Aug '25

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-08-28-16.00.html And our meeting pad: Anti-censorship work meeting pad -------------------------------- Anti-censorship -------------------------------- Next meeting: Thursday, Sep 04 16:00 UTC Facilitator: onyinyang ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: shelikoo == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from projects, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Project 158 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == == Discussion == * (Aug 28 New:) * WebTunnel container's setuid volume migration trade-off * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/webt… * The problem was initially reported by gus at https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/webt… * The issue is that changing the base of the container from Debian bullseye to bookworm changed the uid of the system debian-tor user from 101 to 100. The unpatched new bookworm container image assumes a uid of 100, so permissions are wrong. * Instructions for affected users have been published: * https://forum.torproject.org/t/tor-relays-bridge-operators-fix-required-for… * There has already been a merged change for the debian-tor UID to be pinned at 101 (but it can only be compatible with one or the other, not both) * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/webt… * Proposal in !33 is to add a setuid binary to the container (with attendant security risks) whose purpose is to chown the relevant files and force them to correct UID for debian-tor, whatever that UID may be. * How many WebTunnel bridges are affected? * This would tell us how important it is to take the step of installing a setuid binary. If only a small number of bridges are affected, than maybe it's not worth it. * Affected bridges won't even get to the bridgestrap step, so we won't have those logs. * We could get a quick approximate count (to see if the number of webtunnel bridges has declined and how much) with https://metrics.torproject.org/collector.html#bridgedb-metrics. * Never mind, actually what we need are the extra-info descriptors. * Rough counts from the tor-metrics tarballs, shows a decrease of about 400 or 14% compared to 7 days earlier: $ tar -O -xf bridge-extra-infos-2025-08.tar.xz | grep -B3 '^transport webtunnel' | grep '^published '|grep 2025-08-21|wc -l 3011 $ tar -O -xf bridge-extra-infos-2025-08.tar.xz | grep -B3 '^transport webtunnel' | grep '^published '|grep 2025-08-27|wc -l 2597 * We will take another sample from bridge metrics in 1 week, see how much the count has recovered, and then reevaluate the need for a setuid binary. == Actions == == Interesting links == * == Reading group == * We will discuss "IRBlock: A Large-Scale Measurement Study of the Great Firewall of Iran" on September 11 * https://www.petsymposium.org/foci/2025/foci-2025-0016.pdf * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2025-08-28 Last week: - reviewed lots of MRs - finished an updated of kcp-go and fixed some race conditions in the tests (snowflake#40483) - worked on tracking down reason for snowflake rendezvous failures (snowflake#40447) Next week: - follow up on snowflake rendezvous failures - take a look at potential snowflake orbot bug - https://github.com/guardianproject/orbot-android/issues/1183 dcf: 2025-08-28 Last week: - redeployed snowflake-server on snowflake-02 for golang.org/x/net security fix https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - opened issue to redeploy snowflake-server on snowflake-01 for golang.org/x/net security fix https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Help with: meskio: 2024-08-21 Last week: - restore /bridges in telegram distributor (team#146) - work with TPA to improve prometheus anti-censorship alerts - many merge reviews Next week: - AFK Shelikhoo: 2024-08-28 Last Week: - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) testing environment setup/research - [Done] Create new webtunnel release( (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/webt…) - Merge request reviews - [Done] Draft: Feature / SNI spoofing functionality for WebTunnel PT (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyre…) Next (working) Week/TODO: - Merge request reviews - Support the Testing of domain fronting sites ( https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/… ) (cont.) - [Deploy] Add Domain Fronting Test Support to probeobserver onyinyang: 2025-08-28 Last week(s): - Monitoring fix for: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/249 https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/merge_requests/562 - Created visualizations for flickering bridges - Fixing up translatable elements on bridges.torproject.org https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/merge_requests/57… Next week: -Finish up with !574 - Deploy rdsys!567 when rdsys!562 is confirmed to be fixed and remove extra logs from rdsys!562 - Look into why we are distributing malfunctioning webtunnel bridges https://gitlab.torproject.org/tpo/anti-censorship/bridgestrap/-/issues/47 - Lox still seems to be filling up the disk on the rdsys-test server despite changes made to delete old entries, look into what's going wrong Switch back to some of these: As time allows: Blog post for conjure: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj… - review Tor browser Lox integration https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… - add TTL cache to lox MR for duplicate responses: https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305 - Work on outstanding milestone issues: - key rotation automation Later: pending decision on abandoning lox wasm in favour of some kind of FFI? https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096) - add pref to handle timing for pubkey checks in Tor browser - add trusted invitation logic to tor browser integration: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974 - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2025-06-12 Last weeks: - Applying for funding from NLnet to implement DTLS 1.3 in Pion. Got through the first round. - Writing paper for FOCI: continuation of master thesis about reducing distinguishability of DTLS in Snowflake by implementing covert-dtls, further analysis of collected browser fingerprint and stability test of covert-dtls in snowflake proxies. Draft: https://theodorsm.net/FOCI25 - Key takeaways: * covert-dtls is stable when mimicking DTLS 1.2 handshakes, while the randomization approach— though more resistant to fingerprinting — tends to be less stable. * Chrome webextensions are more unstable than standalone proxies * covert-dtls should be integrated in Snowflake proxies as they produce the ClientHello messages during the DTLS handshake. * Chrome randomizes the order of extension list. * Firefox uses DTLS 1.3 by default in WebRTC. * A prompt adoption of DTLS 1.3 in both Snowflake and our fingerprint-resistant library is needed to keep up with browsers * The evolution of browsers’ fingerprints had no noticeable effect on Snowflake’s number of daily users over the last year. * Even with a sharp drop in the amount of proxies, it does not seem to affect the number of Snowflake users. * Browser extensions make Snowflake resistant to ClientHello fingerprinting. * Standalone proxies can serve more Snowflake clients per volunteer than webextensions. * We need metrics on which types of proxies are actually being matched and successfully used by clients. Next weeks: - Getting paper camera ready. - Fix merge conflicts in MR (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) Help with: - Should we do user testing of covert-dtls? Facilitator Queue: meskio onyinyang shelikhoo 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue

1 0

Anti-censorship team meeting notes, 2025-08-21
by Shelikhoo 21 Aug '25

21 Aug '25

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-08-21-16.00.html And our meeting pad: Anti-censorship work meeting pad -------------------------------- Anti-censorship -------------------------------- Next meeting: Thursday, Aug 28 16:00 UTC Facilitator: meskio ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: shelikoo == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from projects, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Project 158 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == == Discussion == * snowflake minimum go version updated to 1.23 * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * go1.23 means we can switch back to mainline uTLS, rather than using our current custom-patched uTLS * Doesn't interfere with Tor Browser build requirements. Tor Browser uses lyrebird, which uses snowflake as a library. "As long as the client library doesn't require go1.23 or higher, it will still build with the current version of snowflake with go1.22." * We can update the version of go used in lyrebird in September 2025, which is when Tor Browser based on Firefox 140 is scheduled. == Actions == == Interesting links == * Analysis of the GFW's Unconditional Port 443 Block on August 20, 2025 * https://gfw.report/blog/gfw_unconditional_rst_20250820/en/ * Can You Hear me? A First Study Of VoIP Censorship Techniques In Saudi Arabia And The UAE * https://www.eurosp2025.ieee-security.org/program.html#paper174 * Talks about STUN protocol fingerprinting * Not online anywhere yet as far as I can tell, PDF at https://share.riseup.net/#v0RjsGn_2EKLle6QxICj2g == Reading group == * We will discuss "IRBlock: A Large-Scale Measurement Study of the Great Firewall of Iran" on September 11 * https://www.petsymposium.org/foci/2025/foci-2025-0016.pdf * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2025-08-21 Last week: - lots of reviewing MRs - deployed a new version of the broker: - fixes a bug in prometheus snowflake_proxy_total stats - fixes a bug in snowflake metrics proxy unique IP counts - adds country stats to client-ampcache-ips - worked on data races in our safeprom library and snowflake broker - ptutil!6 - snowflake!617 - fixed some issues with the CI pipeline Next week: - follow up on snowflake rendezvous failures - take a look at potential snowflake orbot bug - https://github.com/guardianproject/orbot-android/issues/1183 dcf: 2025-08-21 Last week: - opened a refactoring merge request in the snowflake broker (thanks cohosh) https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - fixed a slight information leak in snowflake broker country stats (thanks cohosh) https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Help with: meskio: 2024-08-21 Last week: - restore /bridges in telegram distributor (team#146) - work with TPA to improve prometheus anti-censorship alerts - many merge reviews Next week: - AFK Shelikhoo: 2024-08-21 Last Week: - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) testing environment setup/research - [Merge Request] Create new webtunnel release(STALLED BY ISSUE BELOW)(https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transpor…) - Unexpected Gitlab Runner Failure from Rate Limiting (https://gitlab.torproject.org/tpo/tpa/team/-/issues/42245) - Merge request reviews - [Review Reworked] Draft: Feature / SNI spoofing functionality for WebTunnel PT (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyre…) Next (working) Week/TODO: - Merge request reviews - Support the Testing of domain fronting sites ( https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/… ) (cont.) - [Deploy] Add Domain Fronting Test Support to probeobserver - Create webtunnel release v0.0.3(stalled) - [Merge Request]Draft: Feature / SNI spoofing functionality for WebTunnel PT onyinyang: 2025-08-21 Last week(s): - Deployed fix for: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/249 https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/merge_requests/562 - Started looking into and implemented (partial?) fix for uneven bridge distribution issue: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/262 Next week: - Deploy !562 and look further into uneven bridge distribution issue: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/262#note_3… - Visualizations for flickering bridges - Lox still seems to be filling up the disk on the rdsys-test server despite changes made to delete old entries, look into what's going wrong Switch back to some of these: As time allows: Blog post for conjure: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj… - review Tor browser Lox integration https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… - add TTL cache to lox MR for duplicate responses: https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305 - Work on outstanding milestone issues: - key rotation automation Later: pending decision on abandoning lox wasm in favour of some kind of FFI? https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096) - add pref to handle timing for pubkey checks in Tor browser - add trusted invitation logic to tor browser integration: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974 - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2025-06-12 Last weeks: - Applying for funding from NLnet to implement DTLS 1.3 in Pion. Got through the first round. - Writing paper for FOCI: continuation of master thesis about reducing distinguishability of DTLS in Snowflake by implementing covert-dtls, further analysis of collected browser fingerprint and stability test of covert-dtls in snowflake proxies. Draft: https://theodorsm.net/FOCI25 - Key takeaways: * covert-dtls is stable when mimicking DTLS 1.2 handshakes, while the randomization approach— though more resistant to fingerprinting — tends to be less stable. * Chrome webextensions are more unstable than standalone proxies * covert-dtls should be integrated in Snowflake proxies as they produce the ClientHello messages during the DTLS handshake. * Chrome randomizes the order of extension list. * Firefox uses DTLS 1.3 by default in WebRTC. * A prompt adoption of DTLS 1.3 in both Snowflake and our fingerprint-resistant library is needed to keep up with browsers * The evolution of browsers’ fingerprints had no noticeable effect on Snowflake’s number of daily users over the last year. * Even with a sharp drop in the amount of proxies, it does not seem to affect the number of Snowflake users. * Browser extensions make Snowflake resistant to ClientHello fingerprinting. * Standalone proxies can serve more Snowflake clients per volunteer than webextensions. * We need metrics on which types of proxies are actually being matched and successfully used by clients. Next weeks: - Getting paper camera ready. - Fix merge conflicts in MR (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) Help with: - Should we do user testing of covert-dtls? Facilitator Queue: meskio onyinyang shelikhoo 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue

1 0

Anti-censorship team meeting notes, 2025-08-15
by onyinyang 15 Aug '25

15 Aug '25

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-08-14-16.00.html And our meeting pad: Anti-censorship work meeting pad -------------------------------- Anti-censorship -------------------------------- Next meeting: Thursday, Aug 21 16:00 UTC Facilitator: shelikhoo ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: onyinyang == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from projects, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Project 158 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == == Discussion == * SQS snowflake rendezvous price check? * cost for July 2025 is 22.63 USD https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Snowflake-co… * there is discussion about moving SQS to tpo's aws credits https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43842#n… * next steps: * see if tor will pay for sqs * if not, try to find other outside funding * otherwise, shut it down for not being sufficiently cost-efficient, and maybe look into how to make it more sustainable * is sqs rendezvous particularly important to any regions? * maybe china, per https://bridges.torproject.org/moat/circumvention/map and https://snowflake-broker.torproject.net/metrics * client-sqs-ips CN=61904,PL=31576,??=8728,RU=3000,US=2312,CA=744,GB=544,TH=248,IN=216,ID=208,JP=120,BG=80,NL=64,TW=64,IR=56,SA=56,DE=40,IE=40,BR=32,PK=32,UZ=24,ES=24,SG=16,SV=16,AU=16,IT=16,AZ=8,MO=8,VN=8,GQ=8,KR=8,IL=8,MA=8,TN=8,NZ=8,UA=8,HK=8 * Why was the snowflake broker restarted 2025-08-02? * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * OS uptime is since 2025-07-01, so it wasn't an OS restart. More likely a crash of the broker process and automatic restart by systemd. * There may be some trace in the snowflake-broker log. cohosh will look at it. * shelikhoo reports it was the OOM killer: * Aug 02 12:03:34 snowflake-broker-40349 kernel: Out of memory: Killed process 629 (broker) total-vm:7873300kB, anon-rss:5600452kB, file-rss:0kB, shmem-rss:0kB, UID:1003 pgtables:11224kB oom_score_adj:200 * cohosh will make an issue * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… == Actions == == Interesting links == * https://opencollective.com/censorship-circumvention/projects/snowflake-dail… * IRBlock: A Large-Scale Measurement Study of the Great Firewall of Iran * https://www.usenix.org/conference/usenixsecurity25/presentation/tai == Reading group == * We will discuss "Encrypted Client Hello (ECH) in Censorship Circumvention" on August 14 * https://www.petsymposium.org/foci/2025/foci-2025-0016.pdf * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? * We will discuss "IRBlock: A Large-Scale Measurement Study of the Great Firewall of Iran" on September 11 * https://www.petsymposium.org/foci/2025/foci-2025-0016.pdf * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2025-08-14 Last week: - looked into drop in snowflake proxy counts (snowflake#40472) - found a concurrency bug in ptutil/safeprom (ptutil#1) - fixups to pull client IP from SDP for AMP cache rendezvous (snowflake#40474) Next week: - deploy updates to Snowflake broker - continue looking into drop in proxy metrics - follow up on snowflake rendezvous failures - take a look at potential snowflake orbot bug - https://github.com/guardianproject/orbot-android/issues/1183 dcf: 2025-08-14 Last week (since 2025-07-24): - commented on a merge request to pull client IP address from SDP https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - made a sample webextension to test chrome.idle callbacks https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - helped investigate and respond to the snowflake-02 bridge outage https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - hacked on sqlite-ification of snowflake-graphs Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Help with: meskio: 2024-08-14 Last week: - mapping ideas for snowflake proxy integration in 3rd party apps - create a rdsys distributors dashboard (rdsys#228) - deal with alerts and work with TPA to improve them Next week: - Shelikhoo: 2024-08-14 Last Week: - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) testing environment setup/research - [Merge Request] Create new webtunnel release(STALLED BY ISSUE BELOW)(https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transpor…) - Unexpected Gitlab Runner Failure from Rate Limiting (https://gitlab.torproject.org/tpo/tpa/team/-/issues/42245) - [Merge Request Done] Add Domain Fronting Test Support to probeobserver ( https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/… ) - [Merge Request Awaiting] Remove s390x from container building targets (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) - Merge request reviews - [Review] Draft: Feature / SNI spoofing functionality for WebTunnel PT (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyre…) Next (working) Week/TODO: - Merge request reviews - Support the Testing of domain fronting sites ( https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/… ) (cont.) - Create webtunnel release v0.0.3(stalled) - support and dynamic bridge support - [Merge Request]Draft: Feature / SNI spoofing functionality for WebTunnel PT onyinyang: 2025-08-14 Last week(s): - Vacation - Continuing debugging for: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/249 Next week: - Next is looking into this mystery: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/262#note_3… Switch back to some of these: As time allows: Blog post for conjure: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj… - review Tor browser Lox integration https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… - add TTL cache to lox MR for duplicate responses: https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305 - Work on outstanding milestone issues: - key rotation automation Later: pending decision on abandoning lox wasm in favour of some kind of FFI? https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096) - add pref to handle timing for pubkey checks in Tor browser - add trusted invitation logic to tor browser integration: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974 - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2025-06-12 Last weeks: - Applying for funding from NLnet to implement DTLS 1.3 in Pion. Got through the first round. - Writing paper for FOCI: continuation of master thesis about reducing distinguishability of DTLS in Snowflake by implementing covert-dtls, further analysis of collected browser fingerprint and stability test of covert-dtls in snowflake proxies. Draft: https://theodorsm.net/FOCI25 - Key takeaways: * covert-dtls is stable when mimicking DTLS 1.2 handshakes, while the randomization approach— though more resistant to fingerprinting — tends to be less stable. * Chrome webextensions are more unstable than standalone proxies * covert-dtls should be integrated in Snowflake proxies as they produce the ClientHello messages during the DTLS handshake. * Chrome randomizes the order of extension list. * Firefox uses DTLS 1.3 by default in WebRTC. * A prompt adoption of DTLS 1.3 in both Snowflake and our fingerprint-resistant library is needed to keep up with browsers * The evolution of browsers’ fingerprints had no noticeable effect on Snowflake’s number of daily users over the last year. * Even with a sharp drop in the amount of proxies, it does not seem to affect the number of Snowflake users. * Browser extensions make Snowflake resistant to ClientHello fingerprinting. * Standalone proxies can serve more Snowflake clients per volunteer than webextensions. * We need metrics on which types of proxies are actually being matched and successfully used by clients. Next weeks: - Getting paper camera ready. - Fix merge conflicts in MR (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) Help with: - Should we do user testing of covert-dtls? Facilitator Queue: meskio onyinyang shelikhoo 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue -- --- onyinyang GPG Fingerprint 3CC3 F8CC E9D0 A92F A108 38EF 156A 6435 430C 2036

1 0

Anti-censorship team meeting notes, 2025-08-07
by meskio 07 Aug '25

07 Aug '25

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-08-07-16.00.html And our meeting pad: Anti-censorship -------------------------------- Next meeting: Thursday, Aug 7 16:00 UTC Facilitator: meskio ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator:shelikhoo == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from projects, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Project 158 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == == Discussion == * snowflake-02 is down * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * snowflake-02 is down since last friday * meskio is trying to contact the operators, no success yet * proxies are overloaded by requests failing to connect to it == Actions == == Interesting links == * Available implementation and report on quantum-safe fully encrypted protocols * https://lists.torproject.org/mailman3/hyperkitty/list/tor-dev@lists.torproj… * https://doi.org/10.3929/ethz-b-000746588 * https://github.com/marc-himmelberger/lyrebird-drivel * Exposing and Circumventing SNI-based QUIC Censorship of the Great Firewall of China * https://gfw.report/publications/usenixsecurity25/en/ == Reading group == * We will discuss "Encrypted Client Hello (ECH) in Censorship Circumvention" on August 14 * https://www.petsymposium.org/foci/2025/foci-2025-0016.pdf * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2025-07-31 Last week: - tried some broker nginx config modifications to see if it reduced the number of 5xx errors (it did not) - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Next week: - afk all week - follow up on snowflake rendezvous failures - take a look at potential snowflake orbot bug - https://github.com/guardianproject/orbot-android/issues/1183 dcf: 2025-07-24 Last week: Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Help with: meskio: 2024-08-07 Last week: - investigate snowflake proxies overload, snowflake-02 is down (snowflake#40475) - duplicate meek and meek-azure on moat APIs (rdsys#272) - redirect to frontdesk if no telegram bridges (rdsys#271) - make easier to copy bridgelines in telegram (rdsys#276) - wrap up rdsys staging (rdsys#219) Next week: - Shelikhoo: 2024-08-07 Last Week: - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) testing environment setup/research - [Merge Request] Create new webtunnel release(STALLED BY ISSUE BELOW)(https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transpor…) - Unexpected Gitlab Runner Failure from Rate Limiting (https://gitlab.torproject.org/tpo/tpa/team/-/issues/42245) - [Merge Request Awaiting] Add Domain Fronting Test Support to probeobserver ( https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/… ) - [Merge Request Done] Update snowflake proxy image to use most recent golang and geodb (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) - [Merge Request Done] Add certificate hash chain pinning support to webtunnel (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyre…) - [Merge Request] Remove s390x from container building targets (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) - Merge request reviews Next (working) Week/TODO: - Merge request reviews - Support the Testing of domain fronting sites ( https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/… ) (cont.) - Create webtunnel release v0.0.3 - invesgate webtunnel domain fronting support and dynamic bridge support onyinyang: 2025-07-24 Last week(s): - Finished up work on webtunnel button - Added some logs to help debug https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/249 Next week: - Next is looking into this mystery: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/262#note_3… - Vacation next 2 weeks Switch back to some of these: As time allows: Blog post for conjure: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj… - review Tor browser Lox integration https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… - add TTL cache to lox MR for duplicate responses: https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305 - Work on outstanding milestone issues: - key rotation automation Later: pending decision on abandoning lox wasm in favour of some kind of FFI? https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096) - add pref to handle timing for pubkey checks in Tor browser - add trusted invitation logic to tor browser integration: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974 - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2025-06-12 Last weeks: - Applying for funding from NLnet to implement DTLS 1.3 in Pion. Got through the first round. - Writing paper for FOCI: continuation of master thesis about reducing distinguishability of DTLS in Snowflake by implementing covert-dtls, further analysis of collected browser fingerprint and stability test of covert-dtls in snowflake proxies. Draft: https://theodorsm.net/FOCI25 - Key takeaways: * covert-dtls is stable when mimicking DTLS 1.2 handshakes, while the randomization approach— though more resistant to fingerprinting — tends to be less stable. * Chrome webextensions are more unstable than standalone proxies * covert-dtls should be integrated in Snowflake proxies as they produce the ClientHello messages during the DTLS handshake. * Chrome randomizes the order of extension list. * Firefox uses DTLS 1.3 by default in WebRTC. * A prompt adoption of DTLS 1.3 in both Snowflake and our fingerprint-resistant library is needed to keep up with browsers * The evolution of browsers’ fingerprints had no noticeable effect on Snowflake’s number of daily users over the last year. * Even with a sharp drop in the amount of proxies, it does not seem to affect the number of Snowflake users. * Browser extensions make Snowflake resistant to ClientHello fingerprinting. * Standalone proxies can serve more Snowflake clients per volunteer than webextensions. * We need metrics on which types of proxies are actually being matched and successfully used by clients. Next weeks: - Getting paper camera ready. - Fix merge conflicts in MR (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) Help with: - Should we do user testing of covert-dtls? Facilitator Queue: meskio onyinyang shelikhoo 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue -- meskio | https://meskio.net/ -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- My contact info: https://meskio.net/crypto.txt -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Nos vamos a Croatan.

1 0

User Support report for July 2025
by ebanam 06 Aug '25

06 Aug '25

Hello everyone, Following is the report from user support team for the month of July. Note: (↑), (↓) and (-) are indicating if the number of tickets we received for these topics have been increasing, decreasing or have been the same from the previous month respectively. - Summary of updates from user support - Farsi-speaking user support - Russian-speaking user support - General user support - Frontdesk (email user support channel) - Telegram, WhatsApp and Signal user support channels - Topics from the Tor Forum - Highlights from Google Play Store # Summary of updates from user support ## Farsi-speaking user support * 83 tickets in total (↓112 as compared to June) * 73 tickets on Telegram * 10 tickets on Email We published a guide on the Tor Forum for users trying to connect to Tor from Iran.[0] ## Russian-speaking user support * 2255 tickets in total (↑920 as compared to June) * 1914 tickets on Telegram * 336 tickets on Email * 3 tickets on WhatsApp * 2 tickets on Signal Since Nina began localized user support for Russian speaking users back in December 2021, July has been the month in which we have received and answered an unprecedented number of tickets. This exceeds the tickets that we received during key censorship events as when Tor was blocked in Russia (December 2021), the Internet shutdown in Kazakhstan (January 2022), and the Russia-Ukraine invasion (February 2022). We have never recorded so many tickets from users from a single region before. July followed the trend of recent months, but with the addition of blocking WebTunnel bridges by domain in Russia that has started in June. With WebTunnel bridges now available from our Telegram bridge distributor, we published a guide[1] on the Tor Forum with some instructions for users. The guide is localized in Russian as well as in Farsi. ## General user support * 721 tickets in total (↑20 as compared to June) * 551 tickets on Email * 155 tickets on Telegram * 7 tickets on WhatsApp * 8 tickets on Signal We published documentation about User-Agent and operating system spoofing in the Tor Browser User Manual.[2] That's it for the summary, following is a more detailed report about the tickets our user support team worked on last month. # Frontdesk (email user support channel) * 806(↑) RT tickets created * 903(↑) RT tickets resolved Tickets by topics and numbers: 1. 410(↑) tickets: instructions to circumvent censorship for Chinese speaking users. 2. 336(↑) tickets: circumventing censorship in Russian speaking countries. 3. 81(↑) tickets: help with troubleshooting existing Tor Browser install on Desktop (Windows, macOS and Linux). 4. 10(↓) tickets: circumventing censorship with Tor in Farsi. 5. 9(↑) tickets: reports of websites blocking Tor connections or not performing well in Tor Browser. 6. 5(↑) tickets: help with troubleshooting existing Tor Browser install on Android. 7. 5(↑) tickets: help with installing Tor Browser on Desktop on Linux. 8. 3(↑) tickets: reports of fake apps on iOS AppStore masquerading as official Tor Browser. 9. 2(↑) tickets: help with instructions to use bridges with Orbot. 10. 2(-) tickets: queries from Tor relay operators. 11. 2(↑) tickets: help with instructions to use bridges with Tails. 12. 1(↓) ticket: question about "Dark Mode" in Tor Browser.[3] 13. 1(↑) ticket: users seeing a "proxy refused" error when visiting websites on Tor Browser for Android.[4] 14. 1(↑) ticket: how operating system spoofing works on Tor Browser.[2] 15. 1(↑) ticket: question about Snowflake - how it works and how to use Snowflake in Tor Browser. 16. 1(↑) ticket: question about onion services and how to access them. # Telegram, WhatsApp and Signal user support channels * 2265(↑) tickets resolved Breakdown: * 2244(↑) tickets on Telegram * 11(↓) tickets on WhatsApp * 10(↓) tickets on Signal Tickets by topics and numbers: 1. 1919(↑) tickets: circumventing censorship in Russian speaking countries. 2. 73(↓) tickets: circumventing censorship with Tor in Farsi. 3. 43(↓) tickets: instructions to circumvent censorship for Chinese speaking users. 4. 17(↑) tickets: help with troubleshooting Tor Browser Desktop on Windows, macOS and Linux. 5. 14(↑) tickets: instructions on how to get Tor Browser binaries from GetTor. 6. 13(↓) tickets: helping users on iOS, using Onion Browser or Orbot, to use censorship circumvention methods. 7. 5(↑) tickets: questions about onion services and how to access them. 8. 4(↓) tickets: help with troubleshooting Tor Browser Android. 9. 4(-) tickets: help with instructions to use bridges with Tails. 10. 3(↑) tickets: help with instructions to use bridges with Orbot on Android. 11. 3(↑) ticket: instructions to download Tor Browser 13.5 legacy for legacy operating systems. 12. 3(↑) tickets: instructions to use WebTunnel bridges with Tor Browser. 13. 2(↓) tickets: help with using Snowflake with Tor to circumvent censorship. 14. 2(↓) tickets: help with instructions to use pluggable transports with little-t tor. 15. 2(-) tickets: users seeing a "proxy refused" error when visiting websites on Tor Browser for Android using Samsung devices. 16. 2(-) tickets: instructions to verify Tor Browser's GPG signature. 17. 2(↑) tickets: Tor Browser for Arm Linux.[5] 18. 1(↑) ticket: report of websites blocking Tor connections or not performing well in Tor Browser. # Topics from the Tor Forum * Iran: Internet disruptions and circumventing censorship with Tor.[0] * The Telegram bot @GetBridgesBot now supports WebTunnel bridges.[1] * Tails 6.18 with support for WebTunnel bridges! [6] # Highlights from Google Play Store * Tor Browser for Android had a Google Play rating of 4.383 (↓0.003) stars in July, lower as compared to in June. * In July, Tor Browser for Android (TBA) got 776 (↑51) new reviews. The total count of reviews for the app stands at 63,863. * Tor Browser for Android Alpha (TBA - Alpha) app had a rating of 4.178 (↓0.006) stars in July, lower as compared to in June. * In July, Tor Browser for Android (TBA - Alpha) got 32 (↑6) new reviews. The total count of reviews for the app stands at 8,622. Like last month, the major complaints we got about Tor Browser for Android were related to it's browsing speed. Additionally, in July, we received multiple comments from users in Russia who were experiencing issues with connecting to Tor. Thanks! -- ebanam [0]: https://forum.torproject.org/t/iran-internet-disruptions-and-circumventing-… [1]: https://forum.torproject.org/t/the-telegram-bot-getbridgesbot-now-supports-… [2]: https://tb-manual.torproject.org/anti-fingerprinting/#operating-system-spoo… [3]: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/40337 [4]: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42714 [5]: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/1… [6]: https://forum.torproject.org/t/new-release-tails-6-18/20067

1 0

sajolida's report for June 2025
by sajolida 04 Aug '25

04 Aug '25

Tails ===== Merge ----- - Did an inventory of the 200+ pages in our Contribute section and discussed how to integrate with the contributors doc at Tor. (#20768) Project P165 ------------ - Released the navigation and accessibility improvements to the website. Example: https://tails.net/doc/persistent_storage/additional_software Details: https://gitlab.tails.boum.org/tails/tails/-/ - Researched and discussed changes to Tails 7.0: * Replace GNOME Terminal with GNOME Console (#20161) * Consider adopting GNOME's design for Power Off (#20960) * unar is not installed on Trixie for special RAR archives (#20946) * Consider refreshing improve-window-list-styling.diff patch (#20903) - Fixed documentation issues identified through user support: * Document better the lack of support for Apple chips (#20807) https://tails.net/install/mac/ * Document the different tools to edit PDFs (#21021) https://tails.net/doc/sensitive_documents/pdf/ * Data is not securely erased when installing (#20749) https://gitlab.tails.boum.org/tails/tails/-/commit/24fddd86 * Document Microsoft Recall (#21000) https://gitlab.tails.boum.org/tails/tails/-/commit/edb47860 - Made translations easier by simplifying ~500 wordy bits of HTML/CSS in our documentation. (#21035) - Clarified how we update our list of "UX Debt". (#20779) https://gitlab.tails.boum.org/tails/team/-/wikis/task_management/#ux-debt -- sajolida The Tor Project — UX Designer

2 2

Anti-censorship team meeting notes, 2025-07-31
by Shelikhoo 31 Jul '25

31 Jul '25

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-07-31-16.00.html And our meeting pad: Anti-censorship work meeting pad -------------------------------- Anti-censorship -------------------------------- Next meeting: Thursday, Aug 7 16:00 UTC Facilitator: meskio ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator:shelikhoo == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from projects, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Project 158 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == * telegram distributor hands out webtunnel bridges == Discussion == * Remove s390x container image build for snowflake-proxy * not supported by the tpa base images * a rare VPS, probably the containers for s390x are not used * meek-azure being renamed to meek * https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43816 * https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/272#note_3… * we'll support both names for a while == Actions == == Interesting links == * == Reading group == * We will discuss "Encrypted Client Hello (ECH) in Censorship Circumvention" on August 14 * https://www.petsymposium.org/foci/2025/foci-2025-0016.pdf * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2025-07-31 Last week: - tried some broker nginx config modifications to see if it reduced the number of 5xx errors (it did not) - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Next week: - afk all week - follow up on snowflake rendezvous failures - take a look at potential snowflake orbot bug - https://github.com/guardianproject/orbot-android/issues/1183 dcf: 2025-07-24 Last week: Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Help with: meskio: 2024-07-31 Last week: - add alerts for snowflake poll drops (tpa/prometheus-alerts!73) - fix webtunnel support on telegram and deploy it (rdsys#269) - debug email distributor issues with gmail (rdsys#129) - brainstorm on renaming meek-azure to meek (rdsys#272) - brainstorm on gitlab status usage for issues (organization#419) - publish in the right place the rdsys metrics (rdsys!543) - publish the transport on telegram metrics (rdsys!545) - debug issues and fix them on the rdsys staging server Next week: - wrap up P158 (rdsys staging) Shelikhoo: 2024-07-31 Last Week: - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) testing environment setup/research - [Merge Request] Create new webtunnel release(STALLED BY ISSUE BELOW)(https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transpor…) - Unexpected Gitlab Runner Failure from Rate Limiting (https://gitlab.torproject.org/tpo/tpa/team/-/issues/42245) - [Merge Request] Add Domain Fronting Test Support to probeobserver ( https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/… ) - [Merge Request] Update snowflake proxy image to use most recent golang and geodb (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) - [Merge Request] Add certificate hash chain pinning support to webtunnel (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyre…) - Merge request reviews Next (working) Week/TODO: - Merge request reviews - Support the Testing of domain fronting sites ( https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/… ) (cont.) - Create webtunnel release v0.0.3 - invesgate webtunnel domain fronting support and dynamic bridge support onyinyang: 2025-07-24 Last week(s): - Finished up work on webtunnel button - Added some logs to help debug https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/249 Next week: - Next is looking into this mystery: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/262#note_3… - Vacation next 2 weeks Switch back to some of these: As time allows: Blog post for conjure: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj… - review Tor browser Lox integration https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… - add TTL cache to lox MR for duplicate responses: https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305 - Work on outstanding milestone issues: - key rotation automation Later: pending decision on abandoning lox wasm in favour of some kind of FFI? https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096) - add pref to handle timing for pubkey checks in Tor browser - add trusted invitation logic to tor browser integration: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974 - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2025-06-12 Last weeks: - Applying for funding from NLnet to implement DTLS 1.3 in Pion. Got through the first round. - Writing paper for FOCI: continuation of master thesis about reducing distinguishability of DTLS in Snowflake by implementing covert-dtls, further analysis of collected browser fingerprint and stability test of covert-dtls in snowflake proxies. Draft: https://theodorsm.net/FOCI25 - Key takeaways: * covert-dtls is stable when mimicking DTLS 1.2 handshakes, while the randomization approach— though more resistant to fingerprinting — tends to be less stable. * Chrome webextensions are more unstable than standalone proxies * covert-dtls should be integrated in Snowflake proxies as they produce the ClientHello messages during the DTLS handshake. * Chrome randomizes the order of extension list. * Firefox uses DTLS 1.3 by default in WebRTC. * A prompt adoption of DTLS 1.3 in both Snowflake and our fingerprint-resistant library is needed to keep up with browsers * The evolution of browsers’ fingerprints had no noticeable effect on Snowflake’s number of daily users over the last year. * Even with a sharp drop in the amount of proxies, it does not seem to affect the number of Snowflake users. * Browser extensions make Snowflake resistant to ClientHello fingerprinting. * Standalone proxies can serve more Snowflake clients per volunteer than webextensions. * We need metrics on which types of proxies are actually being matched and successfully used by clients. Next weeks: - Getting paper camera ready. - Fix merge conflicts in MR (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) Help with: - Should we do user testing of covert-dtls? Facilitator Queue: meskio onyinyang shelikhoo 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue

1 0

Anti-censorship team meeting notes, 2025-07-24
by onyinyang 25 Jul '25

25 Jul '25

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-07-24-16.00.html And our meeting pad: Anti-censorship work meeting pad -------------------------------- Anti-censorship -------------------------------- Next meeting: Thursday, July 31 16:00 UTC Facilitator: shelikhoo ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: onyinyang == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from projects, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Project 158 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == * == Discussion == * dan crashing to talk about Tor VPN app signing strategy and consequences * will be distributed over 3 mediums (tpo website, google play and fdroid) * sharing the key over the 3 mediums will make possible for users to switch upgrade channels if one gets blocked * google wants access to the private key * the plan is to share the key between fdroid and tpo, but have a different key for google * TorBrowser Android doesn't have that problem because is an old app and google don't require keys for old apps * meskio will coordinate with applications team to see how to integrate TorVPN in gettor == Actions == == Interesting links == * FOCI 2025.2 proceedings https://foci.community/foci25.html * "Fingerprint-resistant DTLS for usage in Snowflake" by theodorsm https://www.petsymposium.org/foci/2025/foci-2025-0006.pdf * "Encrypted Client Hello (ECH) in Censorship Circumvention" https://www.petsymposium.org/foci/2025/foci-2025-0016.pdf * PETS 2025 * "CenPush: Blocking-Resistant Control Channel Using Push Notifications" by cohosh et al. https://petsymposium.org/popets/2025/popets-2025-0153.pdf == Reading group == * We will discuss "Encrypted Client Hello (ECH) in Censorship Circumvention" on August 14 * https://www.petsymposium.org/foci/2025/foci-2025-0016.pdf * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2025-07-24 Last week: - dependency upgrades - release snowflake-webext 0.9.4 - opened issue for weird proxy count graphs (snowflake#40472) - finished adapting client poll metrics scripts for rendezvous method counts (snowflake#40464) - wrote a fix to get country metrics for ampcache polls (snowflake!591) - opened an issue about FIFO vs LIFO (snowflake#40473) - worked on investigating snowflake rendezvous errors (snowflake#40447) This week: - follow-up on new conjure bridge set up(conjure#46) - start work around snowflake enumeration attacks - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - follow up on snowflake rendezvous failures - take a look at potential snowflake orbot bug - https://github.com/guardianproject/orbot-android/issues/1183 dcf: 2025-07-24 Last week: Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Help with: meskio: 2024-07-24 Last week: - recommend snowflake in russia over circumventions settings (rdsys-admin!42) - add a prometheus alerts on a drop of snowflake client polls (team#161) - fix the rdsys staging test-authority deploy (tpa/team#41769) - reassign moat bridges (rdsys#233) - review the situation of Tor in Togo (censorship-analysis#40066) - some more report reviews Next week: - wrap up P158 (rdsys staging) Shelikhoo: 2024-07-24 Last Week: - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) testing environment setup/research - Create new webtunnel release(STALLED BY ISSUE BELOW) - Unexpected Gitlab Runner Failure from Rate Limiting (https://gitlab.torproject.org/tpo/tpa/team/-/issues/42245) - Add Domain Fronting Test Support to probeobserver ( https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/… ) - Vantge point maintaince - Merge request reviews Next (working) Week/TODO: - Merge request reviews - Support the Testing of domain fronting sites ( https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/… ) (cont.) - Create webtunnel release v0.0.3 - invesgate webtunnel domain fronting support and dynamic bridge support onyinyang: 2025-07-24 Last week(s): - Finished up work on webtunnel button - Added some logs to help debug https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/249 Next week: - Next is looking into this mystery: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/262#note_3… - Vacation next 2 weeks Switch back to some of these: As time allows: Blog post for conjure: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj… - review Tor browser Lox integration https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… - add TTL cache to lox MR for duplicate responses: https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305 - Work on outstanding milestone issues: - key rotation automation Later: pending decision on abandoning lox wasm in favour of some kind of FFI? https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096) - add pref to handle timing for pubkey checks in Tor browser - add trusted invitation logic to tor browser integration: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974 - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2025-06-12 Last weeks: - Applying for funding from NLnet to implement DTLS 1.3 in Pion. Got through the first round. - Writing paper for FOCI: continuation of master thesis about reducing distinguishability of DTLS in Snowflake by implementing covert-dtls, further analysis of collected browser fingerprint and stability test of covert-dtls in snowflake proxies. Draft: https://theodorsm.net/FOCI25 - Key takeaways: * covert-dtls is stable when mimicking DTLS 1.2 handshakes, while the randomization approach— though more resistant to fingerprinting — tends to be less stable. * Chrome webextensions are more unstable than standalone proxies * covert-dtls should be integrated in Snowflake proxies as they produce the ClientHello messages during the DTLS handshake. * Chrome randomizes the order of extension list. * Firefox uses DTLS 1.3 by default in WebRTC. * A prompt adoption of DTLS 1.3 in both Snowflake and our fingerprint-resistant library is needed to keep up with browsers * The evolution of browsers’ fingerprints had no noticeable effect on Snowflake’s number of daily users over the last year. * Even with a sharp drop in the amount of proxies, it does not seem to affect the number of Snowflake users. * Browser extensions make Snowflake resistant to ClientHello fingerprinting. * Standalone proxies can serve more Snowflake clients per volunteer than webextensions. * We need metrics on which types of proxies are actually being matched and successfully used by clients. Next weeks: - Getting paper camera ready. - Fix merge conflicts in MR (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) Help with: - Should we do user testing of covert-dtls? Facilitator Queue: onyinyang shelikhoo meskio 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue -- --- onyinyang GPG Fingerprint 3CC3 F8CC E9D0 A92F A108 38EF 156A 6435 430C 2036

1 0

Anti-censorship team meeting notes, 2025-07-17
by meskio 17 Jul '25

17 Jul '25

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2025/tor-meeting.2025-07-17-16.00.html And our meeting pad: Anti-censorship -------------------------------- Next meeting: Thursday, July 24 16:00 UTC Facilitator: onyinyang ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: meskio == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from projects, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Project 158 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == * == Discussion == == Actions == == Interesting links == == Reading group == * We will discuss "" on * * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2025-07-10 Last week: - found and fixed regression in snowflake prometheus stats (snowflake#40470) - met with new conjure bridge operator (conjure#46) - added June Snowflake SQS costs https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Snowflake-co… - added a clarification to the broker metrics spec (snowflake!577) - updated discussion on client rendezvous errors (snowflake#40447) - fixed up client-polls merge request in snowflake-graphs - https://gitlab.torproject.org/dcf/snowflake-graphs/-/merge_requests/1 - opened issue to follow up on reports of short-lived client connections (snowflake#40471) This week: - snowflake-webext version bump and deployment - follow-up on new conjure bridge set up(conjure#46) - start work around snowflake enumeration attacks - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - follow up on snowflake rendezvous failures - take a look at potential snowflake orbot bug - https://github.com/guardianproject/orbot-android/issues/1183 dcf: 2025-07-10 Last week: - further commenting on a FEP design https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyre… - snowflake VPS bookkeeping https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Snowflake-co… - readjusted the snowflake broker resource allocation downward to what it had been before the Iran shutdown https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - made graphs of snowflake usage during the Iran shutdown https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - did review of client-polls.csv MR in snowflake-graphs https://gitlab.torproject.org/dcf/snowflake-graphs/-/merge_requests/1 - opened a Dockerfile MR on behalf of an external user https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - speculated on snowflake proxy WebRTC connections preventing computer from sleeping https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - answered a WebTunnel failure report, weirdly the cause seems to be DNS https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/iss… Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… Help with: meskio: 2024-07-17 Last week: - catching up with russian censorship (censorship-analysis#40057) - review reports - catch up after vacation Next week: - wrap up P158 (rdsys staging) Shelikhoo: 2024-07-17 Last Week: - [Testing] Unreliable+unordered WebRTC data channel transport for Snowflake rev2 (cont.)( https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… ) testing environment setup/research - [Deployment] Meek Bridge Offline, we should consider switching to anther hosting provider(https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/16… - [Appoved Merge Request] Bug 41508: Switch built-in meek bridge to meek-unredacted (https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_re…) - Vantge point maintaince - Merge request reviews Next (working) Week/TODO: - Merge request reviews - Support the Testing of domain fronting sites ( https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/… ) (cont.) onyinyang: 2025-07-10 Last week(s): - Mid-year break - Finished up work on webtunnel distribution through telegram: - https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/158#note_32… - https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/merge_requests/534 - Looking into bridge distribution issues: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/267 - Related to rdsys-admin update on 06/26? - restarted rdsys to see if that helps - Next is looking into this mystery: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/262#note_3… Next week: Finish up webtunnel work on rdsys Blog post for conjure: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj… Switch back to some of these: As time allows: - review Tor browser Lox integration https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… - add TTL cache to lox MR for duplicate responses: https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/305 - Work on outstanding milestone issues: - key rotation automation Later: pending decision on abandoning lox wasm in favour of some kind of FFI? https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096) - add pref to handle timing for pubkey checks in Tor browser - add trusted invitation logic to tor browser integration: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974 - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2025-06-12 Last weeks: - Applying for funding from NLnet to implement DTLS 1.3 in Pion. Got through the first round. - Writing paper for FOCI: continuation of master thesis about reducing distinguishability of DTLS in Snowflake by implementing covert-dtls, further analysis of collected browser fingerprint and stability test of covert-dtls in snowflake proxies. Draft: https://theodorsm.net/FOCI25 - Key takeaways: * covert-dtls is stable when mimicking DTLS 1.2 handshakes, while the randomization approach— though more resistant to fingerprinting — tends to be less stable. * Chrome webextensions are more unstable than standalone proxies * covert-dtls should be integrated in Snowflake proxies as they produce the ClientHello messages during the DTLS handshake. * Chrome randomizes the order of extension list. * Firefox uses DTLS 1.3 by default in WebRTC. * A prompt adoption of DTLS 1.3 in both Snowflake and our fingerprint-resistant library is needed to keep up with browsers * The evolution of browsers’ fingerprints had no noticeable effect on Snowflake’s number of daily users over the last year. * Even with a sharp drop in the amount of proxies, it does not seem to affect the number of Snowflake users. * Browser extensions make Snowflake resistant to ClientHello fingerprinting. * Standalone proxies can serve more Snowflake clients per volunteer than webextensions. * We need metrics on which types of proxies are actually being matched and successfully used by clients. Next weeks: - Getting paper camera ready. - Fix merge conflicts in MR (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…) Help with: - Should we do user testing of covert-dtls? Facilitator Queue: onyinyang shelikhoo meskio 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue -- meskio | https://meskio.net/ -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- My contact info: https://meskio.net/crypto.txt -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Nos vamos a Croatan.

1 0

maintenance: Gitaly migration to improve GitLab performance (TPA-RFC-89)
by Antoine Beaupré 14 Jul '25

14 Jul '25

Summary: migrate all Git storage to the new `gitaly-01` back-end, each Git repository read-only during its migration, in the coming week. **Table of Contents** - Proposal - Affected projects - alpha phase, day one (2025-07-14) - beta phase, day two (2025-07-15) - production phase, day two or three (2025-07-15+) - Objections and exceptions - Impact - Projects read-only during migration - Additional complexity for TPA - Timeline - Hardware - Background - Alternatives considered - Full read-only backups - Partial or on-demand migration - References and discussions # Proposal Move all Git repositories to the new [Gitaly][] server during Week 28, progressively, which means it will be impossible to push new commits to a repository while it is migrated. This should be a series of short, scoped outage, as each repository is marked as read-only one at a time when it's migrated. The Gitaly migration procedure seems well test and robust, as each repository is checkedsummed before and after migration. We are hoping this will improve overall performance on the GitLab server, and is part of the design upstream GitLab suggests in scaling an installation of our size. ## Affected projects We plan on migrating the following name spaces in order: ### alpha phase, day one (2025-07-14) This is mostly dogfooding and automation: 1. `anarcat` (already done) 2. `tpo/tpa` 3. `tpo/web` ### beta phase, day two (2025-07-15) This is to include testers outside of TPA yet on projects that are less mission critical and could survive *some* issues with their Git repositories. 4. `tpo/community` 5. `tpo/onion-services` 6. `tpo/anti-censorship` 7. `tpo/network-health` ### production phase, day two or three (2025-07-15+) This is essentially all remaining projects: 8. `tpo/core` (includes c-tor and Arti!) 9. `tpo/applications` (includes Tor Browser and Mullvad Browser) 10. all remaining projects ### Objections and exceptions If you do not want any such disruption in your project, please let us know before the deadline (2025-07-15) so we can skip your project. But we would rather migrate *all* projects off of the server to simplify the architecture and better understand the impact of the change. We would like, in particular, to migrate all of `tpo/applications` repositories in the coming week. Inversely, if you want your project to be prioritized (it might mean a performance improvement!), let us know and you can jump the queue! ## Impact ### Projects read-only during migration While a project is migrated, it is "read-only", that is no change can be done to the Git repository. We believe that other features in projects (like issues and comments) should still work, but the [upstream documentation][] on this is not exactly clear: > To ensure data integrity, projects are put in a temporary read-only > state for the duration of the move. During this time, users receive > a The repository is temporarily read-only. Please try again > later. message if they try to push new commits. So far our test migrations have been so fast (a couple of seconds per project) that we have not really been able to test this properly. ### Additional complexity for TPA TPA will need to get familiar with this new service. [Installation documentation][] is available and all the code developed to deploy the service is visible in an [internal merge request][]. I understand this is a big change right before going on vacation, so any TPA member can veto this and switch to the alternative, a partial or on-demand migration. ## Timeline We plan on starting this work on July 15th, the coming Tuesday. ## Hardware Like the current git repositories on `gitlab-02` the git repositories on `gitaly-01` will be hosted on NVMe disks. # Background GitLab has been having performance problems for a long time now. And for almost as long, we've had the project to "scale GitLab to 2,000 users" ([tpo/tpa/team#40479][]). And while we believe bots (and now, in particular Large Language Models (LLM) bot nets) are responsible for a lot of that load, our [last performance incident][] concluded by observing that there seems to be a correlation between real usage and performance issues. Indeed, during the July break, GitLab's performance was stellar and, on Monday, as soon as Europe woke up from the break, GitLab's performance collapsed again. And while it's *possible* that bots are driven by the same schedule as Tor people, we now feel it's simply time to scale the resources associated with one of our most important services. Gitaly is GitLab's implementation of a Git server. It's basically a web interface to translate (GRPC) requests into Git. It's currently running on the same server as the main GitLab app, but a new server has been built. New servers could be built as needed as well. Anarcat performed [benchmarks][] showing equivalent or better performance of the new Gitaly server, even when influenced by the load of the current GitLab server. It is expected the new server should reduce the load on the main GitLab server, but it's not clear by how much just yet. We're hoping this new architecture will give us more flexibility to deploy new such backends in the future and isolate performance issues to improve diagnostics. It's part of the normal roadmap in scaling a large GitLab installation such as ours. # Alternatives considered ## Full read-only backups We have considered performing a full backup of the entire git repositories before the migration. Unfortunately, this would require setting a read-only mode on all of GitLab for the duration of the backup which, according to our test, could take anywhere from 20 to 60 minutes, which seemed like an unacceptable downtime. Note that we have nightly backups of the GitLab server of course, which is also backed by RAID-10 disk arrays on two different servers. We're only talking about a fully-consistent Git backup here, our normal backups (which, rarely, can be inconsistent and require manual work to reconnect some refs) are typically sufficient anyways. See [tpo/tpa/team#40518][] for a discussion on GitLab backups. ## Partial or on-demand migration We have also considered doing a more piecemeal approach and just migrating some repositories. We worry that this approach would lead to confusion about the real impact of the migration. Still, if any TPA member feels strongly enough about this to put a veto on this proposal, we can take this path and instead migrate a few repositories instead. We could, for example, migrate only the "alpha" targets and a few key repositories in the `tpo/applications` and `tpo/core` groups (since they're prime crawler targets), and leave the mass migration to a later time, with a longer test period. # References and discussions See the [discussion issue][] for comments and more background. [discussion issue]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42225 [Gitaly]: https://gitlab.com/gitlab-org/gitaly [upstream documentation]: https://docs.gitlab.com/api/project_repository_storage_moves/ [last performance incident]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42152 [Installation documentation]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab#gitaly [internal merge request]: https://gitlab.torproject.org/tpo/tpa/puppet-control/-/merge_requests/89 [tpo/tpa/team#40479]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40479 [benchmarks]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42225#note_3223245 [tpo/tpa/team#40518]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40518 -- Antoine Beaupré torproject.org system administration

2 1