[anti-censorship-team] Turbo Tunnel in Snowflake

[David Fifield]

On Wed, Feb 05, 2020 at 02:46:58PM -0500, Cecylia Bocovich wrote:
> I've made some updates to snowbox to easily run all of the pieces needed
> for Turbo Tunnel :

Thanks for doing that.

> It took me a while to wrap my head around the fact that the
> QueuePacketConn is a single abstract connection that handles *all*
> incoming and outgoing traffic for *all* clients.  The result is a
> relatively clean interface with turbotunnel in the actual server code
> while there's a lot going on behind the scenes.

That's right. The analogy is with a UDP server, which is also a
PacketConn. You do ListenUDP once, and it does duty for all clients. In
fact, that's the typical way of using KCP, you do it over UDP. But the
kcp-go interface allows replacing the UDPConn with any other type
implementing the PacketConn interface.

> The behaviour I am still unsure about is which websocket connection the
> data from the server (data going from the server to the client) is
> written to. From what I can tell, each new websocket connection from a
> proxy will pull from the OutgoingQueue that corresponds to the clientID
> of the connection until the connection times out. This means that, since
> the server is not in charge of redial, there are potentially multiple
> connections pulling from this queue. If a connection is dropped and a
> new one redialed at the client, the server may write data out to the
> dropped connection instead of the newer redialed connection. Presumably
> KCP will take care of retransmitting the dropped packet, but I'm curious
> about the latency cost here. It's also a bit different from an earlier
> proposal to do connection migration similar to Mosh:
> https://github.com/net4people/bbs/issues/14

This is a good observation. Indeed, not only could there be an
about-to-die WebSocket connection pulling packets from OutgoingQueue,
losing them, and forcing another WebSocket connection to retransmit
them; there could be multiple simultaneous live WebSocket connections
with the same ClientID, all pulling from OutgoingQueue at once (as would
happen in a setup like https://bugs.torproject.org/25723). I haven't
tried it yet, but my guess is that it will all just work out. When a
WebSocket dies, it will lose some number of packets, but those will be
retransmitted after a timeout that I believe KCP dynamically calculates.
And if there are multiple, each will deliver packets as fast as it can
support, and the client will take care of reordering them. It may cause
some RTT estimation algorithm to freak out, but it's not really
different than TCP segments taking different routes within a connection.

I did the connection migration more straightforwardly in the obfs4proxy
implementation. The clientMap type in Snowflake is a descendant of
connMap in obfs4proxy, and in fact they store almost the same data:
In obfs4proxy:
	type connMapRecord struct {
		Addr     net.Addr
		LastSeen time.Time
		Conn     net.Conn
	}
In Snowflake:
	type clientRecord struct {
		Addr      net.Addr
		LastSeen  time.Time
		SendQueue chan []byte
	}
The difference is that Snowflake stores a (shared) SendQueue instead of
single Conn. The advantage is that many Conns can pull from the shared
queue at once. It's like a more general version of connection migration.
The obfs4proxy implementation also would have supported a limited
version of multiplexing across multiple connections simultaneously, but
it would be dependent on the client consciously striping its sends over
the connections, in order to coax the server into updating its Conn
mapping constantly.

> > But the situation in the turbotunnel branch is better than the status
> > quo, even without multiplexing, for two reasons. First, the connection
> > actually *can* recover after 30 seconds. Second, the smux layer sends
> > keepalives, which means that you won't discard a proxy merely because
> > you're temporarily idle, but only when it really stops working.
>
> Yes! This is really great work. We should talk at the next
> anti-censorship meeting perhaps on the steps we should take to get this
> merged and deployed.

I was going to propose deploying the backward-compatible turbotunnel
server to the public bridge, then making some Tor Browser builds that
our helpers on the issue tracker can try out.

> Minor note:
> - There is some old buffering code at the client side that could be
> rolled into the new RedialPacketConn:
> 
> https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/webrtc.go?h=turbotunnel#n82
> https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/webrtc.go?h=turbotunnel#n237

Thanks, I would not have noticed this myself. I guess the original idea
was that WebRTCPeer would be not a single WebRTC connection, but a
wrapper for a sequence of WebRTC connections? That's why you can Connect
it, Reset it, and Connect it again. I suppose we can eliminate all those
features and make WebRTCPeer less stateful by having it represent just
one WebRTC connection. RedialPacketConn takes care of sequencing
multiple connections.