[I'm about to go off-line for some days, so I am sending my current suboptimally-organized reply, which I hope is better than waiting another week to respond :)]
On Thu, Dec 30, 2021 at 10:42:51PM -0700, David Fifield wrote:
Let's make a distinction between the "frontend" snowflake-server pluggable transport process, and the "backend" tor process. These don't necessarily have to be 1:1; either one could be run in multiple instances. Currently, the "backend" tor is the limiting factor, because it uses only 1 CPU core. The "frontend" snowflake-server can scale to multiple cores in a single process and is comparatively unrestrained.
Excellent point, and yes this simplifies. Great.
I believe that the "pinning" of a client session to particular tor instance will work automatically by the fact that snowflake-server keeps an outgoing connection alive (i.e., through the load balancer) as long as a KCP session exists. [...] But before starting the second instance the first time, copy keys from the first instance:
Hm. It looks promising! But we might still have a Tor-side problem remaining. I think it boils down to how long the KCP sessions last.
The details on how exactly these bridge instances will diverge over time:
The keys directory will start out the same, but after four weeks (DEFAULT_ONION_KEY_LIFETIME_DAYS, used to be one week but in Tor 0.3.1.1-alpha, proposal 274, we bumped it up to four weeks) each bridge will rotate its onion key (the one clients use for circuit-level crypto). That is, each instance will generate its own fresh onion key.
The two bridge instances actually haven't diverged completely at that point, since Tor remembers the previous onion key (i.e. the onion key from the previous period) and is willing to receive create cells that use it for one further week (DEFAULT_ONION_KEY_GRACE_PERIOD_DAYS). So it is after 5 weeks that the original (shared) onion key will no longer work.
Where this matters is (after this 5 weeks have passed) if the client connects to the bridge, fetches and caches the bridge descriptor of instance A, and then later it connects to the bridge again and gets passed to instance B. In this case, the create cell that the client generates will use the onion key for instance A, and instance B won't know how to decrypt it so it will send a destroy cell back.
If this is an issue, we can definitely work around it, by e.g. disabling the onion key rotation on the bridges, or setting up a periodic rsync+hup between the bridges, or teaching clients to use createfast cells in this situation (this type of circuit crypto doesn't use the onion key at all, and just relies on TLS for security -- which can only be done for the first hop of the circuit but that's the one we're talking about here).
But before we think about workarounds, maybe we don't need one: how long does "the KCP session" last?
Tor clients try to fetch a fresh bridge descriptor every three-ish hours, and once they fetch a bridge descriptor from their "current" bridge instance, they should know the onion key that it wants to use. So it is that up-to-three-hour window where I think things could go wrong. And that timeframe sounds promising.
(I also want to double-check that clients don't try to use the onion key from the current cached descriptor while fetching the updated descriptor. That could become an ugly bug in the wrong circumstances, and would be something we want to fix if it's happening.)
Here's how you can simulate a pair of bridge instances that have diverged after five weeks, so you can test how things would work with them:
Copy the keys directory as before, but "rm secret_onion_key*" in the keys directory on n-1 of the instances, before starting them.)
Thanks! --Roger