[tbb-dev] Running xpcshell tests

[Nicolas Vigier]

On Mon, 14 Jul 2014, Georg Koppen wrote:

> Nicolas Vigier:
> > Hello,
> > 
> > I am currently investigating how we can run the Mozilla xpcshell tests
> > on our version of firefox, and the other unit tests.
> > 
> > The xpcshell tests are included on the firefox source tree, and require
> > a complete build tree, built without the '--disable-tests' option. So
> > we cannot run those tests on the version that is shipped in the bundle.
> > 
> > To run those tests, I can see different options:
> > 
> > 1. Run the tests as part of the gitian build process and save the output
> >    in a file that can be analyzed later.
> > 
> >    The bundle build process is already very long, and we probably don't
> >    want to make it even longer. Running the xpcshell tests on my laptop
> >    (i5-2520M CPU @ 2.50GHz) took 2.5 hours. So this is probably not a
> >    good idea.
> > 
> > 
> > 2. Save the build tree in a tarball at the end of the build, and make it
> >    available with the bundle files. We can then use it to run the tests
> >    as part of our test suite.
> > 
> >    The main problem with this is that the obj directory at the end of
> >    the build is taking 6GB. It's possible to reduce it to a 888MB tar.xz
> >    file, but it takes 50 minutes to create it on my laptop.
> > 
> > 
> > 3. Build the browser separatly, in the VM that we use for running the
> >    tests. We enough test VMs to be able to do that.
> > 
> >    This also allow us to run those tests on any commit.
> > 
> >    The main problem with this is that we are not running the tests on
> >    the same binaries that are shipped in the bundle.
> > 
> > 
> > So I'm thinking about doing 3. Does anybody have any other idea or
> > comment about this ?
> 
> Assuming all commits make it into nightlies first I think I am a fan of
> having the tests as part of our regular nightly builds. I am even in
> favor of having the latter as debug builds by default which would help
> us debugging test failures more easily, too. But that is an orthogonal
> issue. Anyway, both could easily be achieved by a special .mozconfig
> file and some minor patching of our browser descriptors (like "take the
> nightly .mozconfig iff we are about to build a Tor Browser for
> versions.nightly")

I have started integrating xpcshell tests into our test suite by doing
[3].

I've been running it on two commits, 155c4c79792c997b8d70 (mozilla
commit), and c660ca4cb98351bbce6c (tor-browser-24.7.0esr-3.x-1 branch):
https://people.torproject.org/~boklm/tmp/tests/r/browser-155c4c79792c997b8d70/browser-155c4c79792c997b8d70.html
https://people.torproject.org/~boklm/tmp/tests/r/browser-c660ca4cb98351bbce6c/browser-c660ca4cb98351bbce6c.html

My plan now is to to add a diff of test failures between current and
previous commit, and launch it on all of our commits, to find which
commit is responsible for which test failure.

Then I will be looking at doing [2], to run the tests on our nightly
build binaries.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tbb-dev/attachments/20140718/4b0f27c8/attachment.sig>