On Mon, 09 Dec 2013, Georg Koppen wrote:

Hi Nicolas,

some remarks are below.

Hi Georg,

Thanks for your remarks.

Nicolas Vigier:

...

In order to help me doing that, I'm very interested to receive from developers of any tor components :

• a description or ticket number of bugs that you wish could have been detected earlier with automated tests

https://trac.torproject.org/projects/tor/ticket/8143 comes to mind here.

Ok. We should soon be able to push tor browser to Mozilla's Try servers, which will run all the unit tests, so we should be able to detect bugs like #8143.

https://wiki.mozilla.org/ReleaseEngineering/TryServer

...

• any wish or specific needs that you may have

You write: "Tor Browser includes some patches to make the build reproducible. We could have a test that check the reproducibility of the build by building the browser twice."

While this is indeed a good idea, it won't be enough as we had bugs in the past that were only visible when builds on different machines got compared. So, what I'd like to have (in addition to running browser builds twice? on different machines?) are tests that cover specific bugs we avoided (see: the "Remaining Build Reproducibility Issues" in Mike's blog post covering the technical details of the Gitian build) or tracked down. See: https://trac.torproject.org/projects/tor/ticket/10159 for an example for the latter. (There, one could write a test that automates the creation of the browser.manifest which would eventually (i.e. if run a couple of times) show whether this bug still exists or whether not).)

Ok, we can have a test rebuilding several times the files that are most likely to become non deterministic. However it would be better if we can find some way to trigger those non-deterministic builds with only two builds. Maybe we can try something like this :

- a library that we put in LD_PRELOAD as a wrapper on readdir to return directory entries in random order instead of inodes number, so it's more likely to be different on each build.

- a python library that we put in PYTHONPATH, to override dict.iterkeys to return keys in random order. If I understand ticket #10159 correctly, it was caused by iterkeys returning keys in the same order most of the time, but not all the times.

...

• anything else that you think might be useful for me to know

You write: "We can produce some packages for Tor Browser, to make testing of the browser easier."

In which regard is it easier to test the browser if you have packages?

Hmm yes, it does not really help in that regard, I've removed this. What can help is packages nightly builds, as it makes it easy for someone to update their tor browser every day to act as a beta tester.

And how should that look like outside of the Tor Browser Bundle? I fear we create extra bugs if we move the browser outside the environment it will be used in (i.e. the TBB) just for testing purposes. Even if we won't create extra bugs this way we might miss some. To be clear, running the tests you call "Usablility tests" and "Reproducible build test" outside the bundle, seems fine to me, while my concerns apply to the fingerprinting and privacy tests.

What it would look like is similar to what they use in tails : https://tails.boum.org/contribute/release_process/iceweasel/

But you are right that we can miss some bugs if we only test it outside TBB, so we should also run the tests in TBB.

The test helpers are a good idea. You might want to rename "Cookies tool" to something like "Identifier tool" matching the Tor Browser specification more closely (especially as you actually mean "Identifier tool" as "Later versions could be extended to also use other techniques for storing informations in the browser" indicates).

Ok, updated.

An other thing that can be done is keeping an archive of all builds for 3 months (or more depending on space available), so that git bisect can be done more quickly. Would that be helpful ?

On Tue, 10 Dec 2013, Lunar wrote:

Nicolas Vigier:

...

Ok, we can have a test rebuilding several times the files that are most likely to become non deterministic. However it would be better if we can find some way to trigger those non-deterministic builds with only two builds. Maybe we can try something like this :

• a library that we put in LD_PRELOAD as a wrapper on readdir to return directory entries in random order instead of inodes number, so it's more likely to be different on each build.

• a python library that we put in PYTHONPATH, to override dict.iterkeys to return keys in random order. If I understand ticket #10159 correctly, it was caused by iterkeys returning keys in the same order most of the time, but not all the times.

I would suggest using reverse order instead of random order. Random order has the probability of being just like the original order.

Indeed, reverse order is better.

For readdir(3), have a look at: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=719845#39

This looks like exactly what we need. Thanks !

Nicolas Vigier:

In addition to this, an other thing that we could do automatically is rebasing the tor-browser patches onto the unreleased next mozilla esr from http://hg.mozilla.org/releases/mozilla-esr24/ in the default branch. This allows to see in advance if rebasing the patches on the next ESR release will require some work, and if some new Mozilla patches break the build or some tests.

That is definitely a thing high on my wishlist for a while now. If we could get this going again (there was a time Mike rebased everything for a new Firefox release) I am all for it. While I am pretty sure rebasing the patches against the next ESR will require some work (provided we don't get a substantial portion of them merged upstream meanhwile) having more time fixing new Firefox issues alone is worth the effort IMO.

Georg