[tor-bugs] #22865 [Obfuscation/meek]: Explicitly set Content-Length to zero when there is no data to send
Tor Bug Tracker & Wiki
blackhole at torproject.org
Wed Jul 26 18:22:48 UTC 2017
#22865: Explicitly set Content-Length to zero when there is no data to send
------------------------------+-----------------------------
Reporter: twim | Owner: dcf
Type: defect | Status: merge_ready
Priority: Medium | Milestone:
Component: Obfuscation/meek | Version:
Severity: Normal | Resolution:
Keywords: | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
------------------------------+-----------------------------
Changes (by dcf):
* status: needs_review => merge_ready
Comment:
Thanks to your example code, I was finally able to reproduce the 411
"Length Required" error, and I think I have traced the source of the
problem. It comes down to http2 not treating body==http.NoBody the same as
body==nil in Go 1.8. Even though http.NewRequest correctly special-cases
*bytes.Reader, sets req.ContentLength = 0, and sets req.Body = http.NoBody
(as noted in comment:9), the http2 code doesn't recognize http.NoBody as
being special, and resets req.ContentLength = -1, which causes the
Content-Length header to be omitted when it should be included.
[https://github.com/golang/net/commit/973f3f3bbd50e92b13faa6c53ec16f49b45e851c
A change] to make body==http.NoBody and body==nil have the same meaning in
the http2 code is not due until Go 1.9.
The fix is attachment:0001-Explicitly-set-Content-Length-to-zero-when-
there-is-.patch, which sets body = nil. Setting body = http.NoBody, which
would seem to even more clearly signal an empty body, does ''not'' work,
for the reasons explained above.
The problem occurs only when:
* Using HTTP/2
* Sending a 0-length body
In the case of App Engine, you additionally have to be:
* Using the App Engine flexible environment (not the standard
environment)
* Domain fronting (as mentioned in comment:6)
Here is the go commit that added http.NoBody (it's the resolution of https
://go-review.googlesource.com/c/31445/ from the ticket description).
Notice how they expand `r.Body == nil` into `r.Body == nil || r.Body ==
NoBody`—but only for http code, not http2.
[https://github.com/golang/go/commit/b992c391d4aae64e147fc64c77ad41d61be8e2e7
net/http: add NoBody, don't return nil from NewRequest on zero bodies]
Here is a bug report about http2 not honoring http.NoBody, and the ensuing
code change (that is not in Go 1.8):
https://github.com/golang/go/issues/18891
[https://github.com/golang/net/commit/973f3f3bbd50e92b13faa6c53ec16f49b45e851c
#diff-afca25314e6df5ed75ab7a17d573b254 http2: make Transport treat
http.NoBody like it were nil]
The important diff is
[https://github.com/golang/net/commit/973f3f3bbd50e92b13faa6c53ec16f49b45e851c
#diff-afca25314e6df5ed75ab7a17d573b254 here], which causes
[https://github.com/golang/net/blob/973f3f3bbd50e92b13faa6c53ec16f49b45e851c/http2/transport.go#L693
actualContentLength] to return 0 instead of -1 for http.NoBody. The value
of 0 in turn causes
[https://github.com/golang/net/blob/973f3f3bbd50e92b13faa6c53ec16f49b45e851c/http2/transport.go#L1168
shouldSendReqContentLength] to return true instead of false.
I made a [[attachment:demo.go|demo program]] that demonstrates all the
possibilities. It allows you to represent an empty body as a 0-length
*strings.Reader (the default), a nil pointer (`-nil` option), or
http.NoBody (`-nobody` option). It additionally allows you to set a front
domain with the `-front` option.
* example.com (only nil or non-empty body works)
{{{
$ ./demo https://example.com/ ""
HTTP/2.0 411 Length Required
$ ./demo -nil https://example.com/ ""
HTTP/2.0 200 OK
$ ./demo -nobody https://example.com/ ""
HTTP/2.0 411 Length Required
$ ./demo https://example.com/ "content"
HTTP/2.0 200 OK
}}}
* App Engine flexible environment without front domain (everything works)
{{{
$ ./demo https://xxx.appspot.com/ ""
HTTP/2.0 200 OK
$ ./demo -nil https://xxx.appspot.com/ ""
HTTP/2.0 200 OK
$ ./demo -nobody https://xxx.appspot.com/ ""
HTTP/2.0 200 OK
$ ./demo https://xxx.appspot.com/ "content"
HTTP/2.0 200 OK
}}}
* App Engine flexible environment with front domain (only nil or non-
empty body works)
{{{
$ ./demo -front www.google.com https://xxx.appspot.com/ ""
HTTP/2.0 411 Length Required
$ ./demo -front www.google.com -nil https://xxx.appspot.com/ ""
HTTP/2.0 200 OK
$ ./demo -front www.google.com -nobody https://xxx.appspot.com/ ""
HTTP/2.0 411 Length Required
$ ./demo -front www.google.com https://xxx.appspot.com/ "content"
HTTP/2.0 200 OK
}}}
You can get a lot of HTTP/2 debugging info by setting
`GODEBUG=http2debug=1` in the environment. Here we can see that the header
`"content-length" = "0"` is not encoded when using a 0-length Reader, but
it when using nil:
{{{
$ GODEBUG=http2debug=1 ./demo https://example.com/ ""
2017/07/26 11:13:53 http2: Transport failed to get client conn for
example.com:443: http2: no cached connection was available
2017/07/26 11:13:54 http2: Transport creating client conn 0xc4200016c0 to
93.184.216.34:443
2017/07/26 11:13:54 http2: Transport encoding header ":authority" =
"example.com"
2017/07/26 11:13:54 http2: Transport encoding header ":method" = "POST"
2017/07/26 11:13:54 http2: Transport encoding header ":path" = "/"
2017/07/26 11:13:54 http2: Transport encoding header ":scheme" = "https"
2017/07/26 11:13:54 http2: Transport received SETTINGS len=30, settings:
HEADER_TABLE_SIZE=4096, MAX_CONCURRENT_STREAMS=100,
INITIAL_WINDOW_SIZE=1048576, MAX_FRAME_SIZE=16384,
MAX_HEADER_LIST_SIZE=16384
2017/07/26 11:13:54 http2: Transport encoding header "accept-encoding" =
"gzip"
2017/07/26 11:13:54 http2: Transport encoding header "user-agent" = "Go-
http-client/2.0"
2017/07/26 11:13:54 Unhandled Setting: [HEADER_TABLE_SIZE = 4096]
2017/07/26 11:13:54 Unhandled Setting: [MAX_HEADER_LIST_SIZE = 16384]
2017/07/26 11:13:54 http2: Transport received WINDOW_UPDATE len=4 (conn)
incr=983041
2017/07/26 11:13:54 http2: Transport received SETTINGS flags=ACK len=0
2017/07/26 11:13:54 http2: Transport received HEADERS flags=END_HEADERS
stream=1 len=19
HTTP/2.0 411 Length Required
}}}
{{{
$ GODEBUG=http2debug=1 ./demo -nil https://example.com/ ""
2017/07/26 11:14:02 http2: Transport failed to get client conn for
example.com:443: http2: no cached connection was available
2017/07/26 11:14:03 http2: Transport creating client conn 0xc4200016c0 to
93.184.216.34:443
2017/07/26 11:14:03 http2: Transport encoding header ":authority" =
"example.com"
2017/07/26 11:14:03 http2: Transport encoding header ":method" = "POST"
2017/07/26 11:14:03 http2: Transport encoding header ":path" = "/"
2017/07/26 11:14:03 http2: Transport encoding header ":scheme" = "https"
2017/07/26 11:14:03 http2: Transport encoding header "content-length" =
"0"
2017/07/26 11:14:03 http2: Transport encoding header "accept-encoding" =
"gzip"
2017/07/26 11:14:03 http2: Transport encoding header "user-agent" = "Go-
http-client/2.0"
2017/07/26 11:14:03 http2: Transport received SETTINGS len=30, settings:
HEADER_TABLE_SIZE=4096, MAX_CONCURRENT_STREAMS=100,
INITIAL_WINDOW_SIZE=1048576, MAX_FRAME_SIZE=16384,
MAX_HEADER_LIST_SIZE=16384
2017/07/26 11:14:03 Unhandled Setting: [HEADER_TABLE_SIZE = 4096]
2017/07/26 11:14:03 Unhandled Setting: [MAX_HEADER_LIST_SIZE = 16384]
2017/07/26 11:14:03 http2: Transport received WINDOW_UPDATE len=4 (conn)
incr=983041
2017/07/26 11:14:03 http2: Transport received SETTINGS flags=ACK len=0
2017/07/26 11:14:03 http2: Transport received HEADERS flags=END_HEADERS
stream=1 len=157
HTTP/2.0 200 OK
}}}
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/22865#comment:12>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
More information about the tor-bugs
mailing list