[tbb-bugs] #28176 [Applications/Tor Browser]: Cleanup and add the testsuite to tor-browser-build.git

[Tor Bug Tracker & Wiki]

#28176: Cleanup and add the testsuite to tor-browser-build.git
-------------------------------------------------+-------------------------
 Reporter:  boklm                                |          Owner:  boklm
     Type:  task                                 |         Status:
                                                 |  assigned
 Priority:  Medium                               |      Milestone:
Component:  Applications/Tor Browser             |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:  tbb-rbm, tbb-qa,                     |  Actual Points:
  TorBrowserTeam201810                           |
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Comment (by gk):

 Could you elaborate a bit what made the previous version too complex? How
 would minimal testsuite tools look like compared to what we have today?

 I am not really convinced yet that we should put the whole testsuite stuff
 into our build stuff. My first reaction was "What do the tests we have to
 do with producing builds and configs/scripts defining those?". It does not
 speak anything against `make run-testsuite-testbuild` with a different
 repo, for instance. One could just clone the testsuite repo somewhere as a
 dependency during and then use that for this `make` target (like we
 already do with other dependencies). I think having such a `make` target
 is a good idea anyway, but for the cases where one needs to run the tests
 in addition to a build having an external repo should be working as well.
 That's for your third argument.

 I am not overly worried about the scenario in the second one: it's easy to
 point folks writing patches to that repo or to explain how to run the
 testsuite somehwere. Devs should be used to that kind of thing.

 Regarding the first one: I at least have not forgotten about the problem
 of making corresponding changes. We currently just don't have a policy of
 enforcing the rule of making changes to the code and, if needed, to the
 testsuite. I have the feeling most of the changes are in non-`tor-browser-
 build` repos, so I am not so sure that your proposed change is good for
 your first reason. At any rate, I think the right fix for that problem is
 to a) document exactly what we are testing right now and then enforce the
 policy that all new changes need to pass our tests, i.e. that tests need
 to get adapted when landing changes where needed. But for the latter to be
 effective we need to start from some baseline, meaning a running
 testsuite. :)

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/28176#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online