March 2023 - tor-dev - lists.torproject.org

Two features that would help load-balanced bridges
by David Fifield 01 Jun '23

01 Jun '23

After the blocking of Tor in Russia in December 2022, the number of Snowflake users rapidly increased. Eventually the tor process became the limiting factor for performance, using all of one CPU core. In a thread on tor-relays, we worked out a design where we run multiple instances of tor on the same host, all with the same identity keys, in order to effectively use all the server's CPU resources. It's running on the live bridge now, and as a result the bridge's bandwidth use has roughly doubled. Design thread https://forum.torproject.net/t/tor-relays-how-to-reduce-tor-cpu-load-on-a-s… Installation instructions https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Gui… Two details came up that are awkward to deal with. We have workaround for them, but they could benefit from support from core tor. They are: 1. Provide a way to disable onion key rotation, or configure a custom onion key. 2. Provide a way to set a specific authentication cookie for ExtORPort SAFE_COOKIE authentication, or a new authentication type that doesn't require credentials that change whenever tor is restarted. I should mention that, apart from the load-balancing design we settled on, we have brainstormed some other options for scaling the Snowflake bridge or bridges. At this point, none of these ideas can immediately be put into practice, because there's no way to tell tor "connect to one of these bridges at random, but only one," or "connect to this bridge, but accept any of these fingerprints." https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfl… # Disable onion key rotation Multiple tor instances with the same identity keys will work fine for the first 5 weeks (onion-key-rotation-days + onion-key-grace-period-days), but after that time the instances will have independently rotated their onion keys, and clients will have connection failures unless the load balancer happens to connect them to the instance whose descriptor they have cached. This post investigates what the failure looks like: https://lists.torproject.org/pipermail/tor-relays/2022-January/020238.html Examples of what could work here are a torrc option to set onion-key-rotation-days to a large value, an option to disable onion key rotation, an option to set a certain named file as the onion key. What we are doing now is a bit of a nasty hack: we create a directory named secret_onion_key.old, so that a failed replace_file causes an early exit from rotate_onion_key. https://gitweb.torproject.org/tor.git/tree/src/feature/relay/router.c?h=tor… There are a few apparently benign side effects, like tor trying to rebuild its descriptor every hour, but it's effective at stopping onion key rotation. https://lists.torproject.org/pipermail/tor-relays/2022-January/020277.html # Stable ExtORPort authentication ExtORPort (extended ORPort) is a protocol that lets a pluggable transport attach transport and client IP metadata to a connection, for metrics purposes. In order to connect to the ExtORPort, the pluggable transport needs to authenticate using a scheme like ControlPort authentication. https://gitweb.torproject.org/torspec.git/tree/proposals/217-ext-orport-aut… tor generates a secret auth cookie and stores it in a file. When the pluggable transport process is managed by tor, tor tells the pluggable transport where to find the file by setting the TOR_PT_AUTH_COOKIE_FILE environment variable. In the load-balanced configuration, the pluggable transport server (snowflake-server) is not run and managed by tor. It is an independent daemon, so it doesn't have access to TOR_PT_AUTH_COOKIE_FILE (which anyway would be a different path for every tor instance). The bigger problem is that tor regenerates the auth cookie and rewrites the file on every restart. All the tor instances have different cookies, and snowflake-server does not know which it will get through the load balancer, so it doesn't know what cookie to use. Examples of what would work here are an option to use a certain file as the auth cookie, an option to leave the auth cookie file alone if it already exists, or a new ExtORPort authentication type that can use the same credentials across multiple instances. What we're doing now is using a shim program, extor-static-cookie, which presents an ExtORPort interface with a static auth cookie for snowflake-server to authenticate with, then re-authenticates to the ExtORPort of its respective instance of tor, using that instance's auth cookie. https://lists.torproject.org/pipermail/tor-relays/2022-January/020183.html

2 2

MSVC compile failure
by Gisle Vanem 21 Mar '23

21 Mar '23

Hello list. Anybody out there? Trying to compile Tor with 'cl' ver. 19.36.32323 from yesterdays 'git master', I got this error in 'src/feature/hs/hs_metrics_entry.c': hs/hs_metrics_entry.c(98,3): error C2099: initializer is not a constant { ^ hs/hs_metrics_entry.c(106,3): error C2099: initializer is not a constant { ^ ---------- This MSVC compiler is rather stupid; it fails to parse that 'static const size_t hs_metrics_circ_build_time_buckets_size' is truly 'const'. No problem with 'clang-cl'. But this patch works: --- a/feature/hs/hs_metrics_entry.c 2023-03-17 18:00:57 +++ b/feature/hs/hs_metrics_entry.c 2023-03-18 13:56:58 @@ -28,8 +28,7 @@ 60000 /* 60s */ }; -static const size_t hs_metrics_circ_build_time_buckets_size = - ARRAY_LENGTH(hs_metrics_circ_build_time_buckets); +#define hs_metrics_circ_build_time_buckets_size ARRAY_LENGTH(hs_metrics_circ_build_time_buckets) ----------------- Just letting you know here. -- --gv

2 3

Cross-user TLS traffic mixing in snowflake-server until 2023-03-13
by David Fifield 15 Mar '23

15 Mar '23

Between 2022-10-01 and 2023-03-13, there was a bug in the software deployed at Snowflake bridges that could cause parts of a user's stream to be overwritten by parts of other users' streams. Though the Snowflake team believes the privacy risks of the bug are minor, we are treating it as a security bug and therefore making this announcement. The fix is already deployed on the server, and no action is required by users. ## The cause The source of the bug was issue https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfl… and specifically commit https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…, which was intended as a performance optimization to reduce memory allocation during the large influx of Snowflake users in late September 2022. The analysis surrounding the performance improvement incorrectly concluded that certain memory buffers were not reused and therefore did not need to be copied. But in fact, the buffers were reused, with the effect that data meant for one user could be unpredictably overwritten by data for another user. ## The effect There is analysis at issue https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfl…. If our understanding of the bug is correct, it did not reveal plaintext data or IP addresses to other users. The Snowflake bridge is a Tor bridge, not an exit relay, so even a catastrophic failure could not leak users' plaintext data or what destinations they were accessing. A Snowflake connection consists of a layered set of protocols, and the bug was in a layer that does not have access to IP addresses or any other persistent identifiers. * IP * TCP * HTTPS (WebSocket) * session/retransmission ← The bug was here * Tor TLS * user data At most, users and proxies would see fragments of other users' Tor TLS streams. We believe the degree of information disclosure is less than what an eavesdropper listening at the Snowflake bridge's HTTPS port would see, which we already believe to be safe. The practical effect, more than a loss of privacy, was excessive retransmissions at the session/retransmission layer, and corrupted TLS data resulting in a terminated connection. ## Remediation We reverted the erroneous commit and added a test to check for the erroneous buffer reuse. We deployed the update to both Snowflake bridges on 2023-03-13. https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfl… ## Required action No action is required by users. The bug was in code that was only used by the Snowflake server, not proxies or clients. ## Thanks The problem was discovered and first investigated by Haz Æ 41 (https://gitlab.torproject.org/hazae41) and reported on HackerOne: https://hackerone.com/reports/1880610.

1 0

Is Arti expected to have better multi-CPU support than C-tor?
by David Fifield 08 Mar '23

08 Mar '23

Linus Nordberg and I are preparing a submission for FOCI about the special way we run tor on the Snowflake bridge. We run many tor processes with the same identity and onion keys, because otherwise tor being limited to one CPU would be the main bottleneck. I'm writing to fact-check a claim about Arti and how we hope the current complicated procedure will not be needed in the future: The first and most important bottleneck to overcome is the single-threaded nature of the Tor implementation.² A single Tor process is limited to one CPU core: once Tor hits 100% CPU, the performance of the bridge is capped, no matter the speed of the network connection or the number of CPU cores ²We expect that Arti, the in-progress reimplementation of Tor, will be natively multi-threaded, and remove this primary complication. Is this correct? Is a relay that uses a future version of Arti expected to be able to use all its CPU resources? Here is the a draft of the submission. If you have any comments, our submission deadline is 2023-03-15. https://www.bamsoftware.com/papers/pt-bridge-hiperf/pt-bridge-hiperf.202303… https://www.bamsoftware.com/papers/pt-bridge-hiperf/pt-bridge-hiperf.202303…

3 3