Hi,
I have been rebasing the tor browser patches split repo on 38.6.0esr: https://github.com/boklm/gecko-dev/branches/all?utf8=%E2%9C%93&query=38....
After submitting all branches to Try Server and ignoring the tests that already failed on 38.5.0esr, we have the following tests which failed: https://people.torproject.org/~boklm/try-results/2016-02-08_tor-browser-38.6...
The main things we can see is that we have a few tests that did not fail when they were run on 38.6.0esr without any other patch: - tbb-tests/test_tor_bug2875.html - tbb-tests/browser_tor_bug2950.js - tbb-tests/test_tor_bug4755.html - tbb-tests/test_tor_bug5856.html
The other failures seems to be intermitent (some of them have been run twice and didn't fail the 2nd time, I will try to see if it's possible to ignore them automatically in that case), on a new branch (Bug-16620), or in the tor-browser branch (which includes the pref change patches, causing many test failures).
The config used to filter results: https://people.torproject.org/~boklm/try-results/2016-02-08_tor-browser-38.6...
The full list of test results: https://people.torproject.org/~boklm/try-results/2016-02-08_tor-browser-38.6...
Nicolas
On Mon, Feb 8, 2016 at 10:57 AM, Nicolas Vigier boklm@mars-attacks.org wrote: [...]
Thanks for running these tests, Nicolas! Super useful.
The main things we can see is that we have a few tests that did not fail when they were run on 38.6.0esr without any other patch:
- tbb-tests/test_tor_bug2875.html
- tbb-tests/browser_tor_bug2950.js
- tbb-tests/test_tor_bug4755.html
- tbb-tests/test_tor_bug5856.html
I made a branch with these tests only and, at least on my Mac, they all fail as desired. Here's the branch I used: https://github.com/arthuredelstein/tor-browser/commits/check_tests_fail Note I also included the "Bug #12620: TorBrowser regression tests folder" patch, because this is required for the tests to run. Maybe that is the difference here?
As Nicolas noted before, there is a problem with the regression test for #2950. It seems to only work on OS X and not Linux. https://trac.torproject.org/projects/tor/ticket/16888
Arthur
On Tue, 09 Feb 2016, Arthur D. Edelstein wrote:
On Mon, Feb 8, 2016 at 10:57 AM, Nicolas Vigier boklm@mars-attacks.org wrote: [...]
Thanks for running these tests, Nicolas! Super useful.
The main things we can see is that we have a few tests that did not fail when they were run on 38.6.0esr without any other patch:
- tbb-tests/test_tor_bug2875.html
- tbb-tests/browser_tor_bug2950.js
- tbb-tests/test_tor_bug4755.html
- tbb-tests/test_tor_bug5856.html
I made a branch with these tests only and, at least on my Mac, they all fail as desired. Here's the branch I used: https://github.com/arthuredelstein/tor-browser/commits/check_tests_fail Note I also included the "Bug #12620: TorBrowser regression tests folder" patch, because this is required for the tests to run. Maybe that is the difference here?
I tried to push your check_tests_fail branch to Try Server, but the tests don't fail either: https://treeherder.mozilla.org/#/jobs?repo=try&revision=46dbd789fc1e&...
When looking at the logs in https://archive.mozilla.org/pub/firefox/try-builds/boklm@torproject.org-46db... I can't find the tbb-tests/test_tor_bug*.html tests, so I am not sure they have been run.
Nicolas Vigier:
[snip]
The other failures seems to be intermitent (some of them have been run twice and didn't fail the 2nd time, I will try to see if it's possible to ignore them automatically in that case), on a new branch (Bug-16620), or in the tor-browser branch (which includes the pref change patches, causing many test failures).
I wonder if tools like
http://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html
could help shining some light on issues like these.
Georg
-
Arthur D. Edelstein -
Georg Koppen -
Nicolas Vigier