[anti-censorship-team] Turbo Tunnel in Snowflake

[Cecylia Bocovich]

On 2020-01-31 9:24 p.m., David Fifield wrote:
> https://gitweb.torproject.org/user/dcf/snowflake.git/log/?h=turbotunnel&id=07495371d67f914d2c828bbd3d7facc455996bd2
>
> These are the elements of a Turbo Tunnel implementation for Snowflake.
> Turbo Tunnel is a name for overlaying an abstract, virtual session on
> top of concrete, physical network connections, such that the virtual
> session is not tied to any particular network connection. In Snowflake,
> it solves the problem of migrating a session across multiple WebRTC
> connections as temporary proxies come and go. This post is a walkthrough
> of the code changes and my design decisions.
This is good stuff, thanks for working on this!
> == How to try it ==
>
> Download the branch and build it:
> 	git remote add dcf https://git.torproject.org/user/dcf/snowflake.git
> 	git checkout -b turbotunnel --track dcf/turbotunnel
> 	for d in client server broker proxy-go; do (cd $d && go build); done
> Run the broker (not changed in this branch):
> 	broker/broker --disable-tls --addr 127.0.0.1:8000
> Run a proxy (not changed in this branch):
> 	proxy-go/proxy-go --broker http://127.0.0.1:8000/ --relay ws://127.0.0.1:8080/
> Run the server:
> 	tor -f torrc.server
> 	# contents of torrc.server:
> 	DataDirectory datadir-server
> 	SocksPort 0
> 	ORPort 9001
> 	ExtORPort auto
> 	BridgeRelay 1
> 	AssumeReachable 1
> 	PublishServerDescriptor 0
> 	ServerTransportListenAddr snowflake 0.0.0.0:8080
> 	ServerTransportPlugin snowflake exec server/server --disable-tls --log snowflake-server.log
> Run the client:
> 	tor -f torrc.client
> 	# contents of torrc.client:
> 	DataDirectory datadir-client
> 	UseBridges 1
> 	SocksPort 9250
> 	ClientTransportPlugin snowflake exec client/client --url http://127.0.0.1:8000/ --ice stun:stun.l.google.com:19302 --log snowflake-client.log
> 	Bridge snowflake 0.0.3.0:1

I've made some updates to snowbox to easily run all of the pieces needed
for Turbo Tunnel :

https://github.com/cohosh/snowbox

All you need to do is use the configuration to point the docker
container towards a snowflake repo with dcf's turbotunnel branch checked
out in it and run `$ build` in the docker container to compile it, and
run the components. Typing `$ run-client` will start a client process
that bootstraps through snowflake. The log files will all be in the home
directory.

> == Introduction to code changes ==
>
> Start by looking at the server changes:
> https://gitweb.torproject.org/user/dcf/snowflake.git/diff/server/server.go?h=turbotunnel&id=0cdb52611b6091dc02ce88e10abaa5cc0c70be9a
>
> The first thing to notice is a kind of "inversion" of control flow.
> Formerly, the ServeHTTP function accepted WebSocket connections and
> connected each one with the ORPort. There was no virtual session: each
> WebSocket connection corresponded to exactly one client session. Now,
> the main function, separately from starting the web server, starts a
> virtual listener (kcp.ServeConn) that calls into a chain of
> acceptSessions→acceptStreams→handleStream functions that ultimately
> connects a virtual stream with the ORPort. But this virtual listener
> doesn't actually open a network port, so what drives it? That's now the
> sole responsibility of the ServeHTTP function. It still accepts
> WebSocket connections, but it doesn't connect them directly to the
> ORPort—instead, it pulls out discrete packets (encoded into the stream
> using length prefixes) and feeds those packets to the virtual listener.
> The glue that links the virtual listener and the ServeHTTP function is
> QueuePacketConn, an abstract interface that allows the virtual listener
> to send and receive packets without knowing exactly how those I/O
> operations are implemented. (In this case, they're implemented by
> encoding packets into WebSocket streams.)

I like that this implementation is very tidy, in that it uses different
layers of abstraction (like KCP) to do a lot of the work required in
deciding which client each packet corresponds to. It took me a while to
wrap my head around the fact that the QueuePacketConn is a single
abstract connection that handles *all* incoming and outgoing traffic for
*all* clients.  The result is a relatively clean interface with
turbotunnel in the actual server code while there's a lot going on
behind the scenes.

The behaviour I am still unsure about is which websocket connection the
data from the server (data going from the server to the client) is
written to. From what I can tell, each new websocket connection from a
proxy will pull from the OutgoingQueue that corresponds to the clientID
of the connection until the connection times out. This means that, since
the server is not in charge of redial, there are potentially multiple
connections pulling from this queue. If a connection is dropped and a
new one redialed at the client, the server may write data out to the
dropped connection instead of the newer redialed connection. Presumably
KCP will take care of retransmitting the dropped packet, but I'm curious
about the latency cost here. It's also a bit different from an earlier
proposal to do connection migration similar to Mosh:
https://github.com/net4people/bbs/issues/14

> https://gitweb.torproject.org/user/dcf/snowflake.git/tree/common/turbotunnel/queuepacketconn.go?h=turbotunnel&id=07495371d67f914d2c828bbd3d7facc455996bd2
> https://gitweb.torproject.org/user/dcf/snowflake.git/tree/common/turbotunnel/clientmap.go?h=turbotunnel&id=07495371d67f914d2c828bbd3d7facc455996bd2
> QueuePacketConn and ClientMap are imported pretty much unchanged from
> the meek implementation (https://github.com/net4people/bbs/issues/21).
> Together these data structures manage queues of packets and allow you to
> send and receive them using custom code. In meek it was done over raw
> HTTP bodies; here it's done over WebSocket. These two interfaces are
> candidates for an eventual reusable Turbo Tunnel library.
+1 to this, I like the idea of QueuePacketConn and ClientMap as a part
of a reusable library.
> == Limitations ==
>
> I'm still using the same old logic for detecting a dead proxy, 30
> seconds without receiving any data. This is suboptimal for many reasons
> (https://bugs.torproject.org/25429), one of which is that when your
> proxy dies, you have to wait at least 30 seconds until the connection
> becomes useful again. That's why I had to use "--speed-time 60" in the
> curl command above; curl has a default idle timeout of 30 seconds, which
> would cause it to give up just as a new proxy was becoming available.
>
> I think we can ultimately do a lot better, and make better use of the
> available proxy capacity. I'm thinking of "striping" packets across
> multiple snowflake proxies simultaneously. This could be done in a
> round-robin fashion or in a more sophisticated way (weighted by measured
> per-proxy bandwidth, for example). That way, when a proxy dies, any
> packets sent to it would be detected as lost (unacknowledged) by the KCP
> layer, and retransmitted over a different proxy, much quicker than the
> 30-second timeout. The way to do this would be to replace
> RedialPacketConn—which uses once connection at a time—with a
> MultiplexingPacketConn, which manages a set of currently live
> connections and uses all of them. I don't think it would require any
> changes on the server.

This has been the subject of discussion on
https://trac.torproject.org/projects/tor/ticket/29206 as well. In fact,
one of the biggest usability challenges with Snowflake right now is that
if a user happens to get a bad snowflake the first time up, Tor
Browser's SOCKS connection to the PT will timeout before a circuit has
the chance to bootstrap a Tor connection through a new proxy (mostly
because it takes 30s to realize the snowflake is bad). If this happens,
the client is told that Snowflake is failing and is asked to reconfigure
their network settings. It is indistinguishable through the user
interface from the case in which Snowflake isn't working at all.

I agree that multiplexing is the way to go here. It is very neat that
the way you've implemented this doesn't require changes on the server
side to do it and mostly consists of swapping out RedialPacketConn. I'm
still curious about what I said above w.r.t. which proxy the server ends
up using for returning packets. There's some potential for some
server-side optimizations here if we want to go the route of using a
more  sophisticated method of choosing proxies in the other direction.

> But the situation in the turbotunnel branch is better than the status
> quo, even without multiplexing, for two reasons. First, the connection
> actually *can* recover after 30 seconds. Second, the smux layer sends
> keepalives, which means that you won't discard a proxy merely because
> you're temporarily idle, but only when it really stops working.
>
Yes! This is really great work. We should talk at the next
anti-censorship meeting perhaps on the steps we should take to get this
merged and deployed.

Minor note:
- There is some old buffering code at the client side that could be
rolled into the new RedialPacketConn:

https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/webrtc.go?h=turbotunnel#n82
https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/webrtc.go?h=turbotunnel#n237