[tor-bugs] #27105 [Applications/Quality Assurance and Testing]: Fix Tor Browser testsuite

[Tor Bug Tracker & Wiki]

#27105: Fix Tor Browser testsuite
-------------------------------------------------+-------------------------
 Reporter:  boklm                                |          Owner:  acat
     Type:  task                                 |         Status:
                                                 |  assigned
 Priority:  Medium                               |      Milestone:
Component:  Applications/Quality Assurance and   |        Version:
  Testing                                        |
 Severity:  Normal                               |     Resolution:
 Keywords:  tbb-testsuite,                       |  Actual Points:
  TorBrowserTeam202004, ReleaseTrainMigration    |
Parent ID:  #33654                               |         Points:
 Reviewer:                                       |        Sponsor:
                                                 |  Sponsor58-must
-------------------------------------------------+-------------------------

Comment (by acat):

 I think it might be better to do all reviewing/followup here instead of
 all the N different children, but I'm not sure.

 For now, here is a branch https://github.com/acatarineu/tor-browser-
 bundle-testsuite/commits/27105 fixing several tests, mostly targeting a
 `33533+5` build (based on mozilla-beta 77). I only tested locally
 (linux64) via `./tbb-testsuite --no-xdummy` command. Note: the fix for
 #34250 is needed for several tests to pass. This will need to be ongoing
 work, in parallel to #33533, as we still have to do more rebases until
 esr78.

 There are a couple of tests (`noscript`, `https-everywhere`) which I
 enabled and partially fixed but do not work completely yet. For
 `noscript`, I suspect this might be a regression of `33533`, since the
 test passes with a `tor-browser-linux64-9.0.10_en-US.tar.xz` build (with
 the #34250 fix). I'll create a ticket to investigate that. For `https-
 everywhere` test, it seems to be flaky, as it usually works when tests are
 run with `./tbb-testsuite --no-xdummy --enable-tests tor_bootstrap,https-
 everywhere`, but usually doesn't when the full testsuite is run. I could
 not find out yet why `https-everywhere` is not redirecting to `https` for
 the test page in some cases, but I guess further investigation can be done
 later in #30431.

 Apart from this I think this should also be tested in the other platforms
 we support, as I only tried in linux64. I could try locally for Windows,
 but not sure about macosx. Do we still have a test server that we could
 use for this?

 I also think it would make sense to start moving some of the tests to
 `tor-browser.git` repo, at least the `marionette` ones. We currently can't
 test the full browser there, but we could try to make changes so that
 something close enough to "full" tor-browser is built from `tor-
 browser.git` to allow tests to pass (and be helpful). For me, I can see
 at least two reasons why this would be helpful.

 A couple of questions wrt the changes I made (or not made):

 * Added `TorBrowser/Tor/PluggableTransports/snowflake-client` to
 exceptions in `Fix readelf_RELRO test` and `Fix readelf_stack_canary
 test`, is this expected to be needed?

 * Added `updater` to exceptions in `Fix readelf_no_rpath test`, is this
 fine, or should we do something about it?

 * I did not try to enable `fpcentral`, as I'm not sure if we are going to
 maintain that?

 * I did not try to enable `acid3`, not sure if it's helpful as it is now,
 but maybe we could "pin" it's state via a screenshot to track changes
 between versions?

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/27105#comment:21>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online