[tbb-dev] Nightly rebase experiment proposal

Mon Dec 4 13:45:00 UTC 2017

Arthur D. Edelstein:
> Hi everyone,
> 
> Here's a plan for experimenting with moving Tor Browser from ESRs to
> mozilla-central. Why try this? There are several reasons I can think
> of:
> 
> * There is no mobile Firefox ESR. In order to keep mobile Tor Browser
> secure, we would either need to constantly monitor for security fixes
> and apply backports. It will be simpler to follow the Mozilla release
> cycle.
> * Tor Browser will receive all security patches and security features
> from Mozilla, not simply those deemed important enough to backport to
> the last ESR.
> * Closer collaboration between Mozilla and the Tor Browser team will
> be possible, because everyone will be focused on the same codebase.
> For example, non-urgent patches can land directly on mozilla-central,
> bypassing tor-browser.git altogether, and those patches will be
> deployed to Tor Browser faster than they would have been on an
> ESR-only schedule. Mozilla will provide expert review of patches.
> Also, some non-urgent patches by Tor Browser may be written by Mozilla
> engineers.
> * The Tor Browser development process will be smoother and more
> gradual. Instead of rebasing and patch-fixing sprints ahead of each
> ESR, we can automatically rebase patches every night, and fix any
> patch whenever its rebase fails. (The total rebase effort should be
> approximately the same as before.)
> * Patches that frequently break because they live on "hot code" can be
> identified and uplifted.
> * We'll need to continuously monitor the mozilla codebase for new
> features that could harm privacy. That's potentially difficult, but by
> focusing on Mozilla's current work, we can hopefully stop patches that
> affect privacy before they land.

Here are some cons that we should take into account (without looking at
the process you outline below)

1) I expect the release engineering overhead will increase considerably.
Let's just look at the candidate builds for builds since Firefox 52 got
out (Firefox 52 sounds like a good place to start as it captures the
time when a new ESR release gets split off as well).

Firefox 52 2              Firefox 45.8ESR 2
Firefox 52.0.1 1
Firefox 52.0.2 1

Firefox 53 4              Firefox 45.9ESR 1
Firefox 53.0.2 1
Firefox 53.0.3 1

Firefox 54 3              Firefox 52.2ESR 1
Firefox 54.0.1            Firefox 52.2.1ESR 1

Firefox 55 3              Firefox 52.3ESR 2
Firefox 55.0.1 2
Firefox 55.0.2 1
Firefox 55.0.3 2

Firefox 56 6              Firefox 52.4ESR 2
Firefox 56.0.1 2          Firefox 52.4.1ESR 1
Firefox 56.0.2 1

Firefox 57 4              Firefox 52.5ESR 2
Firefox 57.0.1 2          Firefox 52.5.1ESR 1

We use those candidate builds to start our builds early on and while we
don't need to start a new build on our own once a new candidate build
shows up, a 37:13 ratio makes it very likely that quite a bunch of
additional rebuilds on our side would follow out of that, before we release.

That's not only problematic because of additional distractions.
Moreover, this is worrysome as the QA period *we* have will shrink
significantly as some of the rebuilds at least are due to last minute
patch landings on mozilla-release meaning Friday PST while the release
being schedule for the Tuesday thereafter. Additionally, don't let us
forget our downstream projects like Tails that need to get a Tor Browser
early as possible to get their release properly tested and out on that
Tuesday as well.

But even worse, this does not only mean additional candidate builds but
additional releases too for us and downstream projects. Looking just at
the security advisories, we would have needed additional releases to
include 52.0.1, 53.0.2 and 57.0.1. And we would probably have picked up
52.0.2 (fixes startup crash on Linux), 56.0.1 (increased crashes on
Windows with Intel graphics drivers), and 56.0.2 (fixing video related
crashes on Windows 7 among other things).

Not all of those releases would have been picked up by downstream
projects but it would have meant about 6 additional releases for us.

2) While we would get all the security fixes and security features if
switching, we would get all the newly introduced sec-high and sec-crit
bugs, too. Not only those that needed get fixed in the above mentioned
three out-of-the-order point releases, but generally those as well that
get fixed in newer code under the whole lifetime of an ESR release
series. Maybe one could make an argument that just more critical bugs
get found in mozilla-release than in mozilla-esr because no one looks at
the latter but everyone looks at the former. Maybe, maybe not but I am
not convinced.

> Here's how I envision the new rebase process:
> 
> * An automated script rebases all tor browser patches every night
> against mozilla-central, mozilla-beta, mozilla-stable and mozilla-esr.
> An email alert will be sent if the rebase of any patch fails. (I have
> a prototype script that does the basic rebasing -- it found that
> ~40/140 patches from our ESR-52 branch could be auto-rebased onto
> mozilla-central without any intervention.)
> * Any failed patches will need to be manually fixed up promptly so
> that the automated rebase can resume.
> * Diagnostic Tor Browser bundles will be built against the latest
> rebase branches. We can also run regression tests.

Sounds good to me.

> So how do we ramp up this experiment?
> 
> 1. 2018-1-1 to 2017-2-22: Rebase tor-browser.git patches manually on
> top of Firefox 59-beta. (We have to do this anyway.)
> 2. 2017-2-22 to 2018-3-13: Activate a script that rebases Tor Browser
> patches nightly onto mozilla-(central, beta, stable, esr).
> 3. 2018-03-13 to 2018-05-07: Create nightly Tor Browser bundle builds
> based on nightly rebases.

I am fine with that.

> At this point, we can evaluate whether we would like to transition to
> releasing Tor Browser alphas based on the mozilla-beta branch.
> Switching Tor Browser stable to mozilla-stable can happen sometime
> after that.

I am not convinced we should do that for all of our platforms we support
at this point. Here is an alternative plan. For me the main reason to
switch to builds based on mozilla-release at the moment is the mobile
Firefox not getting security updates on the ESR branch. As we are
working on our mobile Tor Browser anyway it seems to me to be quite
natural to test this idea for just platform alone. That could be part of
a normal alpha workflow: understanding what we need on the release
engineer, the patch work, the review side etc. to make this a smooth
stable experience at the end.

This I think would mean we won't switch during the ESR 59 cycle to this
new model for our desktop Tor Browser which seems worthwhile both
stability- and reliability-wise. We can then reconsider this once
preparations for ESR 66 are about to start. If we think the experiences
we made and the lessons we learnt during the experiment with the mobile
Tor Browser are sufficient for us to swicth away from ESR 52 on all
platforms, then let's do it. If not then let's stay on ESR.

That longer timeframe would allow us as well to get the remaining
patches landed that we think are necessary to switch to a non-ESR cycle.
The proxy bypass framework
(https://bugzilla.mozilla.org/show_bug.cgi?id=1314793) comes to mind,
for instance.

Georg

> How does this sound? Interested to hear what you think.
> 
> Thanks,
> Arthur
> _______________________________________________
> tbb-dev mailing list
> tbb-dev at lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tbb-dev
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tbb-dev/attachments/20171204/d78dee76/attachment.sig>