[tbb-bugs] #23731 [Applications/Tor Browser]: some websites block requests by HTTP User-Agent

[Tor Bug Tracker & Wiki]

#23731: some websites block requests by HTTP User-Agent
------------------------------------------+--------------------------------
     Reporter:  cypherpunks               |      Owner:  tbb-team
         Type:  defect                    |     Status:  new
     Priority:  Medium                    |  Milestone:
    Component:  Applications/Tor Browser  |    Version:
     Severity:  Normal                    |   Keywords:  User-Agent,
                                          |  blocking
Actual Points:                            |  Parent ID:
       Points:                            |   Reviewer:
      Sponsor:                            |
------------------------------------------+--------------------------------
 Some websites will use the HTTP User-Agent field to determine whether the
 browser is allowed to visit.  Apparently, this is done in the name of
 "security," with the assumption that "insecure" browsers should not be
 allowed to visit the site.  (Probably, we should not assume that this has
 anything to do with security per se; perhaps it is really about
 correctness.)

 The approach is neither necessary nor sufficient to achieve the objectives
 of the site operators.  It is unnecessary because web standards define how
 browsers ought to behave, and any correctness should be determined by
 adherence to the standards, not by whether the name of the browser in
 question happens to be on some list.  It is insufficient because
 circumventing the filter is trivial and can be done simply by changing the
 HTTP User-Agent, which users of Tor Browser can edit by editing
 {{{general.useragent.override}}} on the {{{about:config}}} page.

 The default User-Agent that ships with Tor Browser appears to be:

 {{{
 Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0
 }}}

 This seems to work well if we want to appear to be using Firefox.
 However, sometimes Firefox is not on the approved list for websites such
 as those described above.  (At least one website approves Safari and
 Chrome while rejecting IE and Firefox.)

 [http://www.browser-info.net/useragents Browser-Info] provides a list of
 popular HTTP User-Agents, and choosing from this list we can configure Tor
 Browser to appear to be Safari by changing
 {{{general.useragent.override}}} to:

 {{{
 Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko)
 Chrome/13.0.782.112 Safari/535.1
 }}}

 Web users who do not value privacy may indeed have the option,
 inconvenient as it may be, to switch to a browser that satisfies the
 requirements of the site.  Tor users do not have such an option, because
 there is only one Tor Browser (it happens to be based on Firefox).

 We need to make it easier for everyday Tor users to circumvent filtering
 of this variety.  Some possible suggestions:

 1. Maintain a list of popular User-Agents and provide an option in the
 drop-down onion menu on Tor Browser to choose which one to be for this
 site.
 1. Establish a Wiki page that allows users to report websites that block
 specific browsers by User-Agent, along with examples of User-Agent
 strings, if any, that work.
 1. Where appropriate, liaise with the websites in question, particularly
 if they are popular ones, to make sure that Tor Browser is on the list of
 suitable browsers.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/23731>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online