[tor-bugs] #22233 [Core Tor/Tor]: Reconsider behavior on .z URLs with Accept-Encoding header

[Tor Bug Tracker & Wiki]

#22233: Reconsider behavior on .z URLs  with Accept-Encoding header
--------------------------+------------------------------------
 Reporter:  nickm         |          Owner:
     Type:  defect        |         Status:  new
 Priority:  Medium        |      Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor  |        Version:
 Severity:  Normal        |     Resolution:
 Keywords:                |  Actual Points:
Parent ID:                |         Points:
 Reviewer:                |        Sponsor:  Sponsor4
--------------------------+------------------------------------

Comment (by yawning):

 Replying to [comment:3 arma]:
 > FYI, my wget didn't send any accept-encoding header. Neither did
 Sebastian's. Maybe Yawning's did? You can tell it to *add* an accept-
 encoding header, but then what do you expect.

 `wget http://example.com` on my system does this:

 {{{
 GET / HTTP/1.1
 User-Agent: Wget/1.19.1 (linux-gnu)
 Accept: */*
 Accept-Encoding: identity
 Host: example.com
 Connection: Keep-Alive
 }}}

 Python's HTTP client also includes the header with `identity`.

 > I think the issue here is more that there are two ways to indicate you
 want compression -- adding a .z to the url, and saying so in the accept-
 encoding header -- and we should build the two by two decision matrix and
 do the smart thing for all four cases.

 Yes.  The existing code tries to treat `.z` as `Accept-Encoding: deflate`,
 which is a shortcut, and not always correct.  Assuming we do not want to
 double compress, what I would consider working behavior looks like:

 || File         || Accept-Encoding     || Action
 ||
 || `foo`        || N/A                 || `foo`
 ||
 || `foo`        || `identity`          || `Content-Encoding: identity`,
 `foo`          ||
 || `foo`        || `deflate`           || `Content-Encoding: deflate`,
 `deflate(foo)` ||
 || `foo`        || `identity, deflate` || `Content-Encoding: deflate`,
 `deflate(foo)` ||
 || `foo`        || `identity, gzip`    || `Content-Encoding: gzip`,
 `gzip(foo)`    ||
 || `foo`        || `gzip`              || `Content-Encoding: gzip`,
 `gzip(foo)`    ||
 || `foo`        || `deflate, gzip`     || `Content-Encoding: gzip`,
 `gzip(foo)`    ||
 || `foo.z`      || N/A                 || `deflate(foo)`
 ||
 || `foo.z`      || `identity`          || `Content-Encoding: identity`,
 `deflate(foo)` ||
 || `foo.z`      || `deflate`           || `406 Not Acceptable`
 ||
 || `foo.z`      || `identity, deflate` || `Content-Encoding: identity`,
 `deflate(foo)` ||
 || `foo.z`      || `identity, gzip`    || `Content-Encoding: identity`,
 `deflate(foo)` ||
 || `foo.z`      || `gzip`              || `406 Not Acceptable`
 ||
 || `foo.z`      || `deflate, gzip`     || `406 Not Acceptable`
 ||

 (`gzip` used as a placeholder algorithm for "Something that is supported
 that is not `deflate`)

 The current code mishandles the cases in the table that should either
 double compress or return `406`.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/22233#comment:4>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online