[tbb-bugs] #33791 [Applications/Tor Browser]: Evaluate Firefox tests
Tor Bug Tracker & Wiki
blackhole at torproject.org
Mon Jun 1 17:52:48 UTC 2020
#33791: Evaluate Firefox tests
--------------------------------------+--------------------------------
Reporter: sysrqb | Owner: tbb-team
Type: defect | Status: new
Priority: Medium | Milestone:
Component: Applications/Tor Browser | Version:
Severity: Normal | Resolution:
Keywords: | Actual Points:
Parent ID: #33654 | Points:
Reviewer: | Sponsor: Sponsor58-must
--------------------------------------+--------------------------------
Comment (by acat):
I've been experimenting with running our tbb-tests via try. Not sure if
this strictly belongs here (I only tried to make our tests pass, not
Firefox ones), but I think there's some intersection, so I'll write it
here instead of creating a new ticket.
[https://github.com/acatarineu/tor-browser/commits/33791 Here] is a branch
with some changes that I needed to have something working. This is based
on 33533+5 from #33533, plus fixes from #30832 cherry-picked. The actual
changes for this ticket start with
`440d1bf4b9f401c91bdc4a10d7c85ad9566db046`.
For pushing to try I use a git setup similar to the one in
https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-
Gecko-development. There's also https://firefox-source-
docs.mozilla.org/mobile/android/geckoview/contributor/mc-quick-start.html,
but for me having mozilla-unified as a single remote instead of individual
ones for every `mozilla-*` has worked well.
----
Here's the list of changes that were required:
git-cinnabar doesn't like submodules, so there are issues with torbutton
if you do a try push. For that, I made a simple script that squashes all
Tor Browser patches into a single commit, and then converts the torbutton
module into a regular folder (and commits all the files). I guess
squashing is not strictly necessary, but it seemed easier as the script
doesn't need to check for each commit whether the torbutton submodule has
been modified.
I added a few build configs that are common for Tor Browser in
`build/mozconfig.common.override`, which should be picked by taskcluster
tasks for building. `--disable-warnings-as-errors` was needed for the
`Disable NTLM and Negotiate HTTP Auth`, as it complained about unused
functions and variables. However, adding this flag now makes the `gradle-
dependencies` task fail, though, so we have to find a better way of
dealing with this.
It was also necessary to whitelist some duplicates in `browser/installer
/allowed-dupes.mn`, mostly images and icons.
I had initially put `--with-tor-browser-version` and `--disable-tor-
launcher` in `build/mozconfig.common.override`, but there were tasks that
did not pick those and failed, such as `minidump-stackwalk`. I ended up
modifying the configure scripts and setting defaults --with-tor-browser-
version=dev-build and --disable-tor-launcher=true, avoiding the need for
these configs to be defined.
As found in #30832, some prefs were needed in order for marionette tests
to run. I put those in `testing/profiles/common/user.js`. I also checked
that enabling these prefs does not prevent mochitests from passing in a
beta branch:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=32540461eb07f249c0c65ab374442b10084d5701
(expected, as those are the browser defaults).
Finally, Windows and OSX builds were failing when `TOR_BROWSER_UPDATE =
false`, due to 'MAR_NSS' not being present in `DEFINES`, in
`modules/libmar/tool/moz.build`. So I replaced that check with
`CONFIG['TOR_BROWSER_UPDATE']` directly.
Currently, all the "*-shippable" builds seem to fail in taskcluster with
our patches (see
https://treeherder.mozilla.org/#/jobs?repo=try&revision=82a0e15dfca6cf0c5c7abdf6a583b607f38768b).
For Linux and Windows, there are other tasks we can use, but for OSX I
only see a -shippable one, so we'd have to fix that one if we want to run
our tests in that platform. I did not investigate these failures, but they
seem to be caused by missing artifacts that the task cannot download.
----
In order to push to try and just run our tests I had to use the [https
://firefox-source-docs.mozilla.org/tools/try/selectors/fuzzy.html fuzzy
selector]:
`./mach try fuzzy tbb-tests docshell/test/mochitest/test_tor_bug16620.html
-q "\!asan \!shippable \!qr \!tsan \!debug linux"` (the query can be
changed, this one builds a single linux64 platform)
See:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8096c5be7308998c35b69c372300268e9ae20342.
The other try selectors did not work for me: even though I passed the same
input paths, our tests would not run.
That fuzzy selector command generates and commits a `try_task_config.json`
file like:
{{{
{
"env": {
"MOZHARNESS_TEST_PATHS": "{\"mochitest-browser-chrome\": [\"tbb-
tests\"], \"mochitest-plain\": [\"tbb-tests\",
\"docshell/test/mochitest/test_tor_bug16620.html\"]}",
"TRY_SELECTOR": "fuzzy"
},
"tasks": [
"test-linux1804-64/opt-mochitest-browser-chrome-e10s-1",
"test-linux1804-64/opt-mochitest-browser-chrome-fis-e10s-1",
"test-linux1804-64/opt-mochitest-plain-e10s-1",
"test-linux1804-64/opt-mochitest-plain-fis-e10s-1",
"test-linux1804-64/opt-mochitest-plain-gpu-e10s",
"test-linux1804-64/opt-mochitest-plain-headless-e10s-1",
"test-linux1804-64/opt-mochitest-plain-headless-fis-e10s-1"
],
"version": 1
}
}}}
Perhaps in the future we could have our own scripts to generate these,
especially if we have to customize them with non-standard build/testing
configs.
----
I can think of several TODOs for this (roughly in descending priority):
- Create a whitelist of Firefox tests so that we can run as many green
tests as possible.
- For example, a script that pushes to try with a fixed
`try_task_config.json` and `MOZHARNESS_TEST_PATHS`.
- Move marionette tests (not requiring tor_bootstrap) from testsuite to
tbb-tests, and make them pass in try.
- Fix relevant Firefox tests that are currently not green.
- Fix OSX try builds for testing.
- Have tasks to build something close to Tor Browser in taskcluster
- For example, a script that download artifacts (https-everywhere,
noscript, tor, pluggable-transports, ...) and bundles everything together
with Firefox output.
- Also investigate whether the mozconfigs used in taskcluster are close
enough to tor-browser-build ones.
- Move marionette tests (requiring tor bootstrap) from testsuite and make
them pass in try.
- Not sure if downloading Tor binaries and running them in taskcluster
would be fine for Mozilla.
- Maybe port the rest of testsuite tests to run in taskcluster.
Do these (and the priorities) make sense?
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/33791#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
More information about the tbb-bugs
mailing list