[tor-bugs] #12857 [meek]: Use streaming downloads

[Tor Bug Tracker & Wiki]

#12857: Use streaming downloads
-------------------------+---------------------
 Reporter:  dcf          |          Owner:  dcf
     Type:  enhancement  |         Status:  new
 Priority:  major        |      Milestone:
Component:  meek         |        Version:
 Keywords:               |  Actual Points:
Parent ID:               |         Points:
-------------------------+---------------------
 How meek works now is, the client reads a small chunk of data from tor (up
 to 64 KB) and sends it in the body of a POST request. The server receives
 it, reads a small chunk of data from tor (up to 64 KB), and sends it back
 in the response. The client doesn't make another request until it has
 received the response to the first one, in order to keep all the chunks of
 data in order.

 Here's what would be better. The client sends a small chunk of data. The
 server sends a response header and any data it has pending, and leaves the
 response channel open. The server can use the chunked transfer-encoding to
 send a body of indeterminate length. The server sends downstream data over
 the existing streaming response channel until it receives another request.
 At that point, it closes the response channel and opens up a new one
 (which will be the response to the just-received request).

 The advantages are mainly about performance: there's no client polling;
 there's less HTTP header overhead (see #12778); and the client can send
 data (send a request) whenever it feels like it, without waiting for the
 server's most recent response, if the underlying HTTP library supports
 pipelining.

 Psiphon has already implemented something like this. There's a bit of
 difficulty in that the golang HTTP server
 [https://code.google.com/p/go/source/browse/src/pkg/net/http/server.go?r=238ff8c01b5b8749c5bffc00e1f6f5099359f1dd#1169
 doesn't notify you of new requests while you're still sending a response
 on the same keep-alive channel]. Their workaround (and I think it is a
 good one) is to put a timeout on the download streaming, so that a long
 response won't block upstream data forever.
  * https://bitbucket.org/psiphon/psiphon-circumvention-system/diff/go
 /meek-server/meek-
 server.go?diff1=0ca24558e14e772a5b0e3dfb7b166914682a8fc0&diff2=692f9187147c&at
 =meek-performance

 We'll need to overhaul the web browser extensions, because they currently
 assume requests and responses with sizes known in advance.

 This approach won't work with Google App Engine, because App Engine
 [https://developers.google.com/appengine/docs/go/requests#Go_Responses
 doesn't support streaming downloads]. But it should work with CloudFront.
 See #12428 for how to improve performance with App Engine.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/12857>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online