[tor-bugs] #28021 [Applications/Quality Assurance and Testing]: Run the Tor Browser test suite as part of the Tails test suite

[Tor Bug Tracker & Wiki]

#28021: Run the Tor Browser test suite as part of the Tails test suite
-------------------------------------------------+-------------------------
     Reporter:  intrigeri                        |      Owner:  cypherpunks
         Type:  defect                           |     Status:  new
     Priority:  Medium                           |  Milestone:
    Component:  Applications/Quality Assurance   |    Version:
  and Testing                                    |   Keywords:
     Severity:  Normal                           |  AffectsTails
Actual Points:                                   |  Parent ID:
       Points:                                   |   Reviewer:
      Sponsor:                                   |
-------------------------------------------------+-------------------------
 Tails has a quite extensive integration test suite, which includes a
 number of Tor Browser tests, but we don't run the Tor Browser's own test
 suite.

 I see a few reasons why Tails CI should run the Tor Browser test suite in
 Tails:

 * it cheaply increases test coverage of the Tor Browser as shipped in
 Tails, hence lowering the chances that we break stuff;
 * it gives us an incentive to continuously build Tails ISO images on our
 CI with Tor Browser alpha/nightly, which in turn would help identify
 problems (be it actual Tor Browser bugs or changes that break Tails' use
 case) earlier in the dev process;
 * it gives us more options to implement Tor Browser tests: I bet that in
 some cases, we'd better add new tests to the Tor Browser test suite, so
 they benefit everyone, instead of in the Tails test suite;
 * it widens the collaboration area between Tails and TB teams, by
 providing more reasons to communicate and keep each other informed of what
 we're up to :)

 A few years ago (already!) boklm and I did some initial experiments and
 built a proof-of-concept that run great parts of the Tor Browser test
 suite as part of the Tails test suite
 (https://labs.riseup.net/code/issues/7066). But this was in the Mozmill +
 Selenium days, but since #16009 happened and all the tests were ported to
 Marionette).

 Since then we did not invest the time required to push this forward. We
 discussed this a bit with Georg at the Tails summit this year and he
 encouraged me to file a ticket here so we can discuss & plan this. To be
 clear, we'll have very little ressources at Tails to invest into this
 project but if the conclusion is that boklm will do most of the work, I'll
 happily introduce him to our test suite framework and be his primary point
 of contact on Tails side.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/28021>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online