Tor project automation work
Hello, If you don't know me yet, I am the new person working on automation : https://www.torproject.org/about/jobs-lead-automation.html.en To get started working on tor automation, I am currently looking at all components that are part of tor project, to find what is the current status regarding build, packaging, testing, and what should be done. I'm going to write a summary of this, with some proposals and a general plan for the work to be done during the coming months, and it will be available at the following URLs : https://people.torproject.org/~boklm/automation/tor-automation-review.html https://people.torproject.org/~boklm/automation/tor-automation-review.pdf (you can find an early version there) In order to help me doing that, I'm very interested to receive from developers of any tor components : - a description or ticket number of bugs that you wish could have been detected earlier with automated tests - any wish or specific needs that you may have - anything else that you think might be useful for me to know If you want to discuss this on IRC, I'm 'boklm' on #tor-dev. Thanks! Nicolas
Hi Nicolas, some remarks are below. Nicolas Vigier:
In order to help me doing that, I'm very interested to receive from developers of any tor components :
- a description or ticket number of bugs that you wish could have been detected earlier with automated tests
https://trac.torproject.org/projects/tor/ticket/8143 comes to mind here.
- any wish or specific needs that you may have
You write: "Tor Browser includes some patches to make the build reproducible. We could have a test that check the reproducibility of the build by building the browser twice." While this is indeed a good idea, it won't be enough as we had bugs in the past that were only visible when builds on different machines got compared. So, what I'd like to have (in addition to running browser builds twice? on different machines?) are tests that cover specific bugs we avoided (see: the "Remaining Build Reproducibility Issues" in Mike's blog post covering the technical details of the Gitian build) or tracked down. See: https://trac.torproject.org/projects/tor/ticket/10159 for an example for the latter. (There, one could write a test that automates the creation of the browser.manifest which would eventually (i.e. if run a couple of times) show whether this bug still exists or whether not).)
- anything else that you think might be useful for me to know
You write: "We can produce some packages for Tor Browser, to make testing of the browser easier." In which regard is it easier to test the browser if you have packages? And how should that look like outside of the Tor Browser Bundle? I fear we create extra bugs if we move the browser outside the environment it will be used in (i.e. the TBB) just for testing purposes. Even if we won't create extra bugs this way we might miss some. To be clear, running the tests you call "Usablility tests" and "Reproducible build test" outside the bundle, seems fine to me, while my concerns apply to the fingerprinting and privacy tests. The test helpers are a good idea. You might want to rename "Cookies tool" to something like "Identifier tool" matching the Tor Browser specification more closely (especially as you actually mean "Identifier tool" as "Later versions could be extended to also use other techniques for storing informations in the browser" indicates). Georg
On Mon, 09 Dec 2013, Georg Koppen wrote:
Hi Nicolas,
some remarks are below.
Hi Georg, Thanks for your remarks.
Nicolas Vigier:
In order to help me doing that, I'm very interested to receive from developers of any tor components :
- a description or ticket number of bugs that you wish could have been detected earlier with automated tests
https://trac.torproject.org/projects/tor/ticket/8143 comes to mind here.
Ok. We should soon be able to push tor browser to Mozilla's Try servers, which will run all the unit tests, so we should be able to detect bugs like #8143. https://wiki.mozilla.org/ReleaseEngineering/TryServer
- any wish or specific needs that you may have
You write: "Tor Browser includes some patches to make the build reproducible. We could have a test that check the reproducibility of the build by building the browser twice."
While this is indeed a good idea, it won't be enough as we had bugs in the past that were only visible when builds on different machines got compared. So, what I'd like to have (in addition to running browser builds twice? on different machines?) are tests that cover specific bugs we avoided (see: the "Remaining Build Reproducibility Issues" in Mike's blog post covering the technical details of the Gitian build) or tracked down. See: https://trac.torproject.org/projects/tor/ticket/10159 for an example for the latter. (There, one could write a test that automates the creation of the browser.manifest which would eventually (i.e. if run a couple of times) show whether this bug still exists or whether not).)
Ok, we can have a test rebuilding several times the files that are most likely to become non deterministic. However it would be better if we can find some way to trigger those non-deterministic builds with only two builds. Maybe we can try something like this : - a library that we put in LD_PRELOAD as a wrapper on readdir to return directory entries in random order instead of inodes number, so it's more likely to be different on each build. - a python library that we put in PYTHONPATH, to override dict.iterkeys to return keys in random order. If I understand ticket #10159 correctly, it was caused by iterkeys returning keys in the same order most of the time, but not all the times.
- anything else that you think might be useful for me to know
You write: "We can produce some packages for Tor Browser, to make testing of the browser easier."
In which regard is it easier to test the browser if you have packages?
Hmm yes, it does not really help in that regard, I've removed this. What can help is packages nightly builds, as it makes it easy for someone to update their tor browser every day to act as a beta tester.
And how should that look like outside of the Tor Browser Bundle? I fear we create extra bugs if we move the browser outside the environment it will be used in (i.e. the TBB) just for testing purposes. Even if we won't create extra bugs this way we might miss some. To be clear, running the tests you call "Usablility tests" and "Reproducible build test" outside the bundle, seems fine to me, while my concerns apply to the fingerprinting and privacy tests.
What it would look like is similar to what they use in tails : https://tails.boum.org/contribute/release_process/iceweasel/ But you are right that we can miss some bugs if we only test it outside TBB, so we should also run the tests in TBB.
The test helpers are a good idea. You might want to rename "Cookies tool" to something like "Identifier tool" matching the Tor Browser specification more closely (especially as you actually mean "Identifier tool" as "Later versions could be extended to also use other techniques for storing informations in the browser" indicates).
Ok, updated. An other thing that can be done is keeping an archive of all builds for 3 months (or more depending on space available), so that git bisect can be done more quickly. Would that be helpful ?
Nicolas Vigier:
Ok, we can have a test rebuilding several times the files that are most likely to become non deterministic. However it would be better if we can find some way to trigger those non-deterministic builds with only two builds. Maybe we can try something like this :
- a library that we put in LD_PRELOAD as a wrapper on readdir to return directory entries in random order instead of inodes number, so it's more likely to be different on each build.
- a python library that we put in PYTHONPATH, to override dict.iterkeys to return keys in random order. If I understand ticket #10159 correctly, it was caused by iterkeys returning keys in the same order most of the time, but not all the times.
I would suggest using reverse order instead of random order. Random order has the probability of being just like the original order. For readdir(3), have a look at: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=719845#39 -- Lunar <lunar@torproject.org>
On Tue, 10 Dec 2013, Lunar wrote:
Nicolas Vigier:
Ok, we can have a test rebuilding several times the files that are most likely to become non deterministic. However it would be better if we can find some way to trigger those non-deterministic builds with only two builds. Maybe we can try something like this :
- a library that we put in LD_PRELOAD as a wrapper on readdir to return directory entries in random order instead of inodes number, so it's more likely to be different on each build.
- a python library that we put in PYTHONPATH, to override dict.iterkeys to return keys in random order. If I understand ticket #10159 correctly, it was caused by iterkeys returning keys in the same order most of the time, but not all the times.
I would suggest using reverse order instead of random order. Random order has the probability of being just like the original order.
Indeed, reverse order is better.
For readdir(3), have a look at: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=719845#39
This looks like exactly what we need. Thanks !
On Tue, 10 Dec 2013, Nicolas Vigier wrote:
In order to help me doing that, I'm very interested to receive from developers of any tor components :
- a description or ticket number of bugs that you wish could have been detected earlier with automated tests
https://trac.torproject.org/projects/tor/ticket/8143 comes to mind here.
Ok. We should soon be able to push tor browser to Mozilla's Try servers, which will run all the unit tests, so we should be able to detect bugs like #8143.
Regarding this, we will be testing the version that is in tor-browser.git, when a branch receive new commits. In addition to this, an other thing that we could do automatically is rebasing the tor-browser patches onto the unreleased next mozilla esr from http://hg.mozilla.org/releases/mozilla-esr24/ in the default branch. This allows to see in advance if rebasing the patches on the next ESR release will require some work, and if some new Mozilla patches break the build or some tests.
Nicolas Vigier:
In addition to this, an other thing that we could do automatically is rebasing the tor-browser patches onto the unreleased next mozilla esr from http://hg.mozilla.org/releases/mozilla-esr24/ in the default branch. This allows to see in advance if rebasing the patches on the next ESR release will require some work, and if some new Mozilla patches break the build or some tests.
That is definitely a thing high on my wishlist for a while now. If we could get this going again (there was a time Mike rebased everything for a new Firefox release) I am all for it. While I am pretty sure rebasing the patches against the next ESR will require some work (provided we don't get a substantial portion of them merged upstream meanhwile) having more time fixing new Firefox issues alone is worth the effort IMO. Georg
On Fri, 06 Dec 2013, Nicolas Vigier wrote:
- a description or ticket number of bugs that you wish could have been detected earlier with automated tests
Mike and Georg started tagging tickets that are possible test cases with keyword tbb-testcase: https://trac.torproject.org/projects/tor/query?keywords=~tbb-testcase If you know more tickets like this, you can tag them too and I will look at them.
participants (3)
-
Georg Koppen -
Lunar -
Nicolas Vigier