[tor-bugs] #33046 [Applications/Tor Browser]: Test ETP enabled in Tor Browser

[Tor Bug Tracker & Wiki]

#33046: Test ETP enabled in Tor Browser
------------------------------------------+-----------------------------
     Reporter:  sysrqb                    |      Owner:  sysrqb
         Type:  defect                    |     Status:  assigned
     Priority:  Medium                    |  Milestone:
    Component:  Applications/Tor Browser  |    Version:
     Severity:  Normal                    |   Keywords:  tbb-performance
Actual Points:                            |  Parent ID:
       Points:                            |   Reviewer:
      Sponsor:                            |
------------------------------------------+-----------------------------
 Last year I ran some experiments comparing the loading of pages with
 Enhanced Tracking Protection (ETP) disabled and then with it enabled. The
 results from these experiments were not statistically significant, but the
 preliminary results showed that ETP noticeably reduces the best and
 average case timing, but the variance is still problematic.

 On the bright side, this is a good starting point. From an email I sent:
 {{{
 I started measuring Tor Browser's
 speed while loading various webpages. This was with the goal of
 comparing Tor Browser with the measurements Mozilla made two years ago
 [0] when they released Quantum (and compared Firefox and Chrome). It
 uses benchmarks provided by Firefox's Javascript API (using the
 Navigation Timing API).

 It seems like Tracking Protection does has a noticable impact on
 page-load time. I still have the feeling that investigating ad and
 tracker blockers is a worthwhile initiative (especially if we want to
 remain relevant). There are some interesting results that require
 further analysis because I can't explain them right now.

 In the future, it would be very helpful if we extended these
 measurements by including tor controller events related to circuit
 build and stream sentconnect/connected events. We should also run these
 tests from different geographic areas (particularly locations with low
 throughput connections).

 My takeaway from this is if (ideally) pages should load with 6 seconds
 (as was Mozilla's target [0]), then I believe this is achievable as the
 other network improvements (correcting weighting, solving bottlenecks,
 etc.) are rolled out. In addition, as usual, this will require some
 advocacy for helping promote tor-friendly websites.

 ----

 Mozilla's measurements used Selenium for automating the tests. Luckily,
 Kushal already made a Tor Browser Selenium driver[1], so adapting
 Mozilla's test[2] was relatively simple.

 These measurements used a slightly non-standard Tor Browser
 configuration. The Navigation Timing API is a fingerprinting vector, so
 it is disabled by default. |privacy.resistFingerprinting| was disabled
 for these tests. Similarly, NoScript's XSS protection required manual
 input while pages loaded, so the XSS protection was disabled for these
 tests.

 Two sets of tests were run. The first test did not change any other Tor
 Browser settings. The second test enabled Tracking Protection. Each test
 loaded 20 webpages[3] (19 webpages in that file plus
 http://newsweek.com/). The list of webpages is shuffled before each run,
 and Tor Browser is restarted between each run.

 [0]
 https://hacks.mozilla.org/2017/11/comparing-browser-page-load-time-an-
 introduction-to-methodology/
 [1] https://github.com/webfp/tor-browser-selenium
 [2] https://github.com/onkeltom/browser_pageloadspeed
 [3]
 https://github.com/onkeltom/browser_pageloadspeed/blob/master/news.txt
 [4] https://w3c.github.io/navigation-timing/#processing-model
 }}}

 #30939 is a consequence of this work.

 I described in another email how the tests were run:
 {{{
 I pushed a branch page_load_timing on
 https://github.com/sysrqb/tor-browser-selenium/

 It requires the same setup configuration as described in the README. I
 installed the dependencies with --user.
   - pip install --user tbselenium
   - pip install --user -r tor-browser-selenium/requirements-dev.txt
   - Downloaded geckodriver from the Github repo

 I didn't use xvfb (simply for convenience), so I ran the page-load test
 directly with:

 $ NO_XVFB=1 TBB_PATH=~/tor-browser_en-US/ py.test tor-browser-
 selenium/tbselenium/test/test_pageload.py

 Change TBB_PATH and path/to/test_pageload.py, as needed. Don't be
 surprised if the browser doesn't launch immediately (it take 5-10
 seconds on slower computers).

 And, in case you're not aware, tor-browser-selenium currently only works
 on Linux (the README says Debian/Ubuntu and I successfully used it on
 Fedora). You'll need a system tor installed, too (or at least an
 instance of tor running already listening on port 9050). I'd like to add
 support for letting Tor Browser bootstrap and control its own tor in the
 future.
 }}}

 See #32976 for a better way we get geckodriver.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/33046>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online