Nightly rebase experiment proposal
Hi everyone, Here's a plan for experimenting with moving Tor Browser from ESRs to mozilla-central. Why try this? There are several reasons I can think of: * There is no mobile Firefox ESR. In order to keep mobile Tor Browser secure, we would either need to constantly monitor for security fixes and apply backports. It will be simpler to follow the Mozilla release cycle. * Tor Browser will receive all security patches and security features from Mozilla, not simply those deemed important enough to backport to the last ESR. * Closer collaboration between Mozilla and the Tor Browser team will be possible, because everyone will be focused on the same codebase. For example, non-urgent patches can land directly on mozilla-central, bypassing tor-browser.git altogether, and those patches will be deployed to Tor Browser faster than they would have been on an ESR-only schedule. Mozilla will provide expert review of patches. Also, some non-urgent patches by Tor Browser may be written by Mozilla engineers. * The Tor Browser development process will be smoother and more gradual. Instead of rebasing and patch-fixing sprints ahead of each ESR, we can automatically rebase patches every night, and fix any patch whenever its rebase fails. (The total rebase effort should be approximately the same as before.) * Patches that frequently break because they live on "hot code" can be identified and uplifted. * We'll need to continuously monitor the mozilla codebase for new features that could harm privacy. That's potentially difficult, but by focusing on Mozilla's current work, we can hopefully stop patches that affect privacy before they land. Here's how I envision the new rebase process: * An automated script rebases all tor browser patches every night against mozilla-central, mozilla-beta, mozilla-stable and mozilla-esr. An email alert will be sent if the rebase of any patch fails. (I have a prototype script that does the basic rebasing -- it found that ~40/140 patches from our ESR-52 branch could be auto-rebased onto mozilla-central without any intervention.) * Any failed patches will need to be manually fixed up promptly so that the automated rebase can resume. * Diagnostic Tor Browser bundles will be built against the latest rebase branches. We can also run regression tests. So how do we ramp up this experiment? 1. 2018-1-1 to 2017-2-22: Rebase tor-browser.git patches manually on top of Firefox 59-beta. (We have to do this anyway.) 2. 2017-2-22 to 2018-3-13: Activate a script that rebases Tor Browser patches nightly onto mozilla-(central, beta, stable, esr). 3. 2018-03-13 to 2018-05-07: Create nightly Tor Browser bundle builds based on nightly rebases. At this point, we can evaluate whether we would like to transition to releasing Tor Browser alphas based on the mozilla-beta branch. Switching Tor Browser stable to mozilla-stable can happen sometime after that. How does this sound? Interested to hear what you think. Thanks, Arthur
On Tue, 21 Nov 2017, Arthur D. Edelstein wrote:
* We'll need to continuously monitor the mozilla codebase for new features that could harm privacy. That's potentially difficult, but by focusing on Mozilla's current work, we can hopefully stop patches that affect privacy before they land.
From your list it seems to me the most challenging part will be to continuously monitor the mozilla changes and be able to find workarounds for all issues in time before each release. But maybe the closer collaboration with Mozilla can help with this.
Here's how I envision the new rebase process:
* An automated script rebases all tor browser patches every night against mozilla-central, mozilla-beta, mozilla-stable and mozilla-esr. An email alert will be sent if the rebase of any patch fails. (I have a prototype script that does the basic rebasing -- it found that ~40/140 patches from our ESR-52 branch could be auto-rebased onto mozilla-central without any intervention.) * Any failed patches will need to be manually fixed up promptly so that the automated rebase can resume. * Diagnostic Tor Browser bundles will be built against the latest rebase branches. We can also run regression tests.
So how do we ramp up this experiment?
1. 2018-1-1 to 2017-2-22: Rebase tor-browser.git patches manually on top of Firefox 59-beta. (We have to do this anyway.) 2. 2017-2-22 to 2018-3-13: Activate a script that rebases Tor Browser patches nightly onto mozilla-(central, beta, stable, esr). 3. 2018-03-13 to 2018-05-07: Create nightly Tor Browser bundle builds based on nightly rebases.
At this point, we can evaluate whether we would like to transition to releasing Tor Browser alphas based on the mozilla-beta branch. Switching Tor Browser stable to mozilla-stable can happen sometime after that.
How does this sound? Interested to hear what you think.
This looks like a good plan to me. If we go with that plan I think we should also check whether a git branch is still the best way to maintain our patches. I have not been thinking about that a lot yet, but maybe an alternative could be to use quilt [1] or some similar tool to manage patch series. So we could store a quilt patch series as a directory somewhere into tor-browser-build.git, which should make it easier to apply our patches on different firefox trees. [1]: https://en.wikipedia.org/wiki/Quilt_(software) Nicolas
Hi, FWIW, as someone who has to manage such patch series elsewhere: Nicolas Vigier:
If we go with that plan I think we should also check whether a git branch is still the best way to maintain our patches. I have not been thinking about that a lot yet, but maybe an alternative could be to use quilt [1] or some similar tool to manage patch series.
For the record, quilt is heavily used in Debian packaging, but these days many Debian people are trying to use Git to manage their quilt patch series (e.g. with "gbp pq"), and now use quilt only as a standard format to publish and share patch series, never to actually manage it.
So we could store a quilt patch series as a directory somewhere into tor-browser-build.git,
In my experience, tracking patches in Git is painful due to the need to read diffs of diffs, which is error-prone. Cheers, -- intrigeri
On Tue, Nov 21, 2017 at 12:44 PM, Nicolas Vigier <boklm@mars-attacks.org> wrote:
From your list it seems to me the most challenging part will be to continuously monitor the mozilla changes and be able to find workarounds for all issues in time before each release. But maybe the closer collaboration with Mozilla can help with this.
Yes -- I guess one problem is, when a patch lands in Mozilla central, it will be in a release between 7 and 14 weeks later.
If we go with that plan I think we should also check whether a git branch is still the best way to maintain our patches. I have not been thinking about that a lot yet, but maybe an alternative could be to use quilt [1] or some similar tool to manage patch series. So we could store a quilt patch series as a directory somewhere into tor-browser-build.git, which should make it easier to apply our patches on different firefox trees.
I think that's a very interesting idea.
On Tue, 21 Nov 2017, Nicolas Vigier wrote:
This looks like a good plan to me.
If we go with that plan I think we should also check whether a git branch is still the best way to maintain our patches. I have not been thinking about that a lot yet, but maybe an alternative could be to use quilt [1] or some similar tool to manage patch series. So we could store a quilt patch series as a directory somewhere into tor-browser-build.git, which should make it easier to apply our patches on different firefox trees.
On his blog, Greg Kroah-Hartman described his workflow: http://www.kroah.com/log/blog/2019/08/14/patch-workflow-with-mutt-2019/ In the section "Stable patch review and apply" he describes how he uses Quilt for maintaining the stable kernel tree: The process of reviewing patches for the stable tree is much like the development patch process, but it differs in that I never use ‘git am’ for applying anything. The stable kernel tree, while under development, is kept as a series of patches that need to be applied to the previous release. This series of patches is maintained by using a tool called (quilt). Quilt is very powerful and handles sets of patches that need to be applied on top of a moving base very easily. The tool was based on a crazy set of shell scripts written by Andrew Morton a long time ago, and is currently maintained by Jean Delvare and has been rewritten in perl to make them more maintainable. It handles thousands of patches easily and quickly and is used by many developers to handle kernel patches for distributions as well as other projects. I highly recommend it as it allows you to reorder, drop, add in the middle of the series, and manipulate patches in all sorts of ways, as well as create new patches directly. I do this for the stable tree as lots of times we end up dropping patches from the middle of the series when reviewers say they should not be applied, adding new patches where needed as prerequisites of existing patches, and other changes that with git, would require lots of rebasing. Rebasing a git does not work for when you have developers working “down” from your tree. We usually have the rule with kernel development that if you have a public tree, it never gets rebased otherwise no one can use it for development. Anyway, the stable patches are kept in a quilt series in a repository that is kept under version control in git (complex, yeah, sorry.) That queue can always be found (here[1]). [1]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/ I think doing something similar might work for us, and would make it easier to apply our patches on different Mozilla branches. It would also make it easier to look at the history of our patches. Currently we create a new branch for each Mozilla release, where we rebase our patches, squashing or removing some of them, so looking at the history of a patch is difficult as you have to look at many branches. Nicolas
On Thu, Oct 03, 2019 at 12:39:59PM +0200, Nicolas Vigier wrote:
On Tue, 21 Nov 2017, Nicolas Vigier wrote:
This looks like a good plan to me.
If we go with that plan I think we should also check whether a git branch is still the best way to maintain our patches. I have not been thinking about that a lot yet, but maybe an alternative could be to use quilt [1] or some similar tool to manage patch series. So we could store a quilt patch series as a directory somewhere into tor-browser-build.git, which should make it easier to apply our patches on different firefox trees.
On his blog, Greg Kroah-Hartman described his workflow: http://www.kroah.com/log/blog/2019/08/14/patch-workflow-with-mutt-2019/
In the section "Stable patch review and apply" he describes how he uses Quilt for maintaining the stable kernel tree:
[snip]
The stable kernel tree, while under development, is kept as a series of patches that need to be applied to the previous release. This series of patches is maintained by using a tool called (quilt). Quilt is very powerful and handles sets of patches that need to be applied on top of a moving base very easily. The tool was based on a crazy set of shell scripts written by Andrew Morton a long time ago, and is currently maintained by Jean Delvare and has been rewritten in perl to make them more maintainable. It handles thousands of patches easily and quickly and is used by many developers to handle kernel patches for distributions as well as other projects.
[snip]
I think doing something similar might work for us, and would make it easier to apply our patches on different Mozilla branches.
It would also make it easier to look at the history of our patches. Currently we create a new branch for each Mozilla release, where we rebase our patches, squashing or removing some of them, so looking at the history of a patch is difficult as you have to look at many branches.
I like the idea of using quilt. I've thought about it (periodically) over the last few months, but I stuggled with the development workflow. I only read Greg's blog post today (unforunately), and his description of `linux-stable-rc` is exactly the problem I imagined. However, maybe this continual-rebasing branch is the trade-off we accept in this process. I don't want to make development more difficult by requiring someone apply all of our patches on top of mozilla-{central,beta,release} before they begin any actual development. On the other hand, I don't want to maintain two "official" sources of "Tor Browser" (as in, maintain a set of quilt-maintained patches, and a git repo). I would prefer only using quilt, if that was really an option. I think the idea of frequently rebasing the git repo and force-pushing to the public git mirror(s) is a reasonable compromise (while the patch sets are the official sources). gbp pq looks like a nice improvement around quilt, and I think we should experiment with it, too (thanks intrigeri). - Matt
On Tue, 21 Nov 2017 19:00:53 +0000 "Arthur D. Edelstein" <arthuredelstein@gmail.com> wrote:
Here's a plan for experimenting with moving Tor Browser from ESRs to mozilla-central. Why try this? There are several reasons I can think of:
[snip]
At this point, we can evaluate whether we would like to transition to releasing Tor Browser alphas based on the mozilla-beta branch. Switching Tor Browser stable to mozilla-stable can happen sometime after that.
How does this sound? Interested to hear what you think.
While I plan on doing this sooner rather than later anyway, depending on the amount of churn that exists, this would likely make me terminate the sandboxed-tor-browser maintenance. Regards, -- Yawning Angel
On Nov 21, 2017, at 2:00 PM, Arthur D. Edelstein <arthuredelstein@gmail.com> wrote:
* An automated script rebases all tor browser patches every night against mozilla-central, mozilla-beta, mozilla-stable and mozilla-esr.
Is it sufficient that there're merely no line conflicts?
On Wed, Nov 22, 2017 at 9:06 AM, Arlo Breault <arlo@torproject.org> wrote:
Is it sufficient that there're merely no line conflicts?
It's not sufficient, of course! :) As much as possible we should be attempting to run automated tests on our nightly builds. And the most ideal thing is that our patches are upstreamed.
Arthur D. Edelstein:
Hi everyone,
Here's a plan for experimenting with moving Tor Browser from ESRs to mozilla-central. Why try this? There are several reasons I can think of:
* There is no mobile Firefox ESR. In order to keep mobile Tor Browser secure, we would either need to constantly monitor for security fixes and apply backports. It will be simpler to follow the Mozilla release cycle. * Tor Browser will receive all security patches and security features from Mozilla, not simply those deemed important enough to backport to the last ESR. * Closer collaboration between Mozilla and the Tor Browser team will be possible, because everyone will be focused on the same codebase. For example, non-urgent patches can land directly on mozilla-central, bypassing tor-browser.git altogether, and those patches will be deployed to Tor Browser faster than they would have been on an ESR-only schedule. Mozilla will provide expert review of patches. Also, some non-urgent patches by Tor Browser may be written by Mozilla engineers. * The Tor Browser development process will be smoother and more gradual. Instead of rebasing and patch-fixing sprints ahead of each ESR, we can automatically rebase patches every night, and fix any patch whenever its rebase fails. (The total rebase effort should be approximately the same as before.) * Patches that frequently break because they live on "hot code" can be identified and uplifted. * We'll need to continuously monitor the mozilla codebase for new features that could harm privacy. That's potentially difficult, but by focusing on Mozilla's current work, we can hopefully stop patches that affect privacy before they land.
Here are some cons that we should take into account (without looking at the process you outline below) 1) I expect the release engineering overhead will increase considerably. Let's just look at the candidate builds for builds since Firefox 52 got out (Firefox 52 sounds like a good place to start as it captures the time when a new ESR release gets split off as well). Firefox 52 2 Firefox 45.8ESR 2 Firefox 52.0.1 1 Firefox 52.0.2 1 Firefox 53 4 Firefox 45.9ESR 1 Firefox 53.0.2 1 Firefox 53.0.3 1 Firefox 54 3 Firefox 52.2ESR 1 Firefox 54.0.1 Firefox 52.2.1ESR 1 Firefox 55 3 Firefox 52.3ESR 2 Firefox 55.0.1 2 Firefox 55.0.2 1 Firefox 55.0.3 2 Firefox 56 6 Firefox 52.4ESR 2 Firefox 56.0.1 2 Firefox 52.4.1ESR 1 Firefox 56.0.2 1 Firefox 57 4 Firefox 52.5ESR 2 Firefox 57.0.1 2 Firefox 52.5.1ESR 1 We use those candidate builds to start our builds early on and while we don't need to start a new build on our own once a new candidate build shows up, a 37:13 ratio makes it very likely that quite a bunch of additional rebuilds on our side would follow out of that, before we release. That's not only problematic because of additional distractions. Moreover, this is worrysome as the QA period *we* have will shrink significantly as some of the rebuilds at least are due to last minute patch landings on mozilla-release meaning Friday PST while the release being schedule for the Tuesday thereafter. Additionally, don't let us forget our downstream projects like Tails that need to get a Tor Browser early as possible to get their release properly tested and out on that Tuesday as well. But even worse, this does not only mean additional candidate builds but additional releases too for us and downstream projects. Looking just at the security advisories, we would have needed additional releases to include 52.0.1, 53.0.2 and 57.0.1. And we would probably have picked up 52.0.2 (fixes startup crash on Linux), 56.0.1 (increased crashes on Windows with Intel graphics drivers), and 56.0.2 (fixing video related crashes on Windows 7 among other things). Not all of those releases would have been picked up by downstream projects but it would have meant about 6 additional releases for us. 2) While we would get all the security fixes and security features if switching, we would get all the newly introduced sec-high and sec-crit bugs, too. Not only those that needed get fixed in the above mentioned three out-of-the-order point releases, but generally those as well that get fixed in newer code under the whole lifetime of an ESR release series. Maybe one could make an argument that just more critical bugs get found in mozilla-release than in mozilla-esr because no one looks at the latter but everyone looks at the former. Maybe, maybe not but I am not convinced.
Here's how I envision the new rebase process:
* An automated script rebases all tor browser patches every night against mozilla-central, mozilla-beta, mozilla-stable and mozilla-esr. An email alert will be sent if the rebase of any patch fails. (I have a prototype script that does the basic rebasing -- it found that ~40/140 patches from our ESR-52 branch could be auto-rebased onto mozilla-central without any intervention.) * Any failed patches will need to be manually fixed up promptly so that the automated rebase can resume. * Diagnostic Tor Browser bundles will be built against the latest rebase branches. We can also run regression tests.
Sounds good to me.
So how do we ramp up this experiment?
1. 2018-1-1 to 2017-2-22: Rebase tor-browser.git patches manually on top of Firefox 59-beta. (We have to do this anyway.) 2. 2017-2-22 to 2018-3-13: Activate a script that rebases Tor Browser patches nightly onto mozilla-(central, beta, stable, esr). 3. 2018-03-13 to 2018-05-07: Create nightly Tor Browser bundle builds based on nightly rebases.
I am fine with that.
At this point, we can evaluate whether we would like to transition to releasing Tor Browser alphas based on the mozilla-beta branch. Switching Tor Browser stable to mozilla-stable can happen sometime after that.
I am not convinced we should do that for all of our platforms we support at this point. Here is an alternative plan. For me the main reason to switch to builds based on mozilla-release at the moment is the mobile Firefox not getting security updates on the ESR branch. As we are working on our mobile Tor Browser anyway it seems to me to be quite natural to test this idea for just platform alone. That could be part of a normal alpha workflow: understanding what we need on the release engineer, the patch work, the review side etc. to make this a smooth stable experience at the end. This I think would mean we won't switch during the ESR 59 cycle to this new model for our desktop Tor Browser which seems worthwhile both stability- and reliability-wise. We can then reconsider this once preparations for ESR 66 are about to start. If we think the experiences we made and the lessons we learnt during the experiment with the mobile Tor Browser are sufficient for us to swicth away from ESR 52 on all platforms, then let's do it. If not then let's stay on ESR. That longer timeframe would allow us as well to get the remaining patches landed that we think are necessary to switch to a non-ESR cycle. The proxy bypass framework (https://bugzilla.mozilla.org/show_bug.cgi?id=1314793) comes to mind, for instance. Georg
How does this sound? Interested to hear what you think.
Thanks, Arthur _______________________________________________ tbb-dev mailing list tbb-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tbb-dev
On Mon, Dec 4, 2017 at 1:45 PM, Georg Koppen <gk@torproject.org> wrote:
I am not convinced we should do that for all of our platforms we support at this point. Here is an alternative plan. For me the main reason to switch to builds based on mozilla-release at the moment is the mobile Firefox not getting security updates on the ESR branch. As we are working on our mobile Tor Browser anyway it seems to me to be quite natural to test this idea for just platform alone. That could be part of a normal alpha workflow: understanding what we need on the release engineer, the patch work, the review side etc. to make this a smooth stable experience at the end.
This I think would mean we won't switch during the ESR 59 cycle to this new model for our desktop Tor Browser which seems worthwhile both stability- and reliability-wise. We can then reconsider this once preparations for ESR 66 are about to start. If we think the experiences we made and the lessons we learnt during the experiment with the mobile Tor Browser are sufficient for us to swicth away from ESR 52 on all platforms, then let's do it. If not then let's stay on ESR.
That longer timeframe would allow us as well to get the remaining patches landed that we think are necessary to switch to a non-ESR cycle. The proxy bypass framework (https://bugzilla.mozilla.org/show_bug.cgi?id=1314793) comes to mind, for instance.
I like this suggestion a lot. Testing with mobile first and waiting for ESR66 to decide on desktop makes a lot of sense to me. So I will go ahead with setting up a nightly rebase and we can integrate it with the mobile processes as needed.
Arthur D. Edelstein:
On Mon, Dec 4, 2017 at 1:45 PM, Georg Koppen <gk@torproject.org> wrote:
I am not convinced we should do that for all of our platforms we support at this point. Here is an alternative plan. For me the main reason to switch to builds based on mozilla-release at the moment is the mobile Firefox not getting security updates on the ESR branch. As we are working on our mobile Tor Browser anyway it seems to me to be quite natural to test this idea for just platform alone. That could be part of a normal alpha workflow: understanding what we need on the release engineer, the patch work, the review side etc. to make this a smooth stable experience at the end.
This I think would mean we won't switch during the ESR 59 cycle to this new model for our desktop Tor Browser which seems worthwhile both stability- and reliability-wise. We can then reconsider this once preparations for ESR 66 are about to start. If we think the experiences we made and the lessons we learnt during the experiment with the mobile Tor Browser are sufficient for us to swicth away from ESR 52 on all platforms, then let's do it. If not then let's stay on ESR.
I like this suggestion a lot. Testing with mobile first and waiting for ESR66 to decide on desktop makes a lot of sense to me. So I will go ahead with setting up a nightly rebase and we can integrate it with the mobile processes as needed.
This sounds like a very sensible approach to me. Thanks Georg! Cheers, -- intrigeri
Hi, Georg Koppen:
Additionally, don't let us forget our downstream projects like Tails that need to get a Tor Browser early as possible to get their release properly tested and out on that Tuesday as well.
Thank you so much for caring! I see this as a trade-off: Tails would also benefit from the advantages we envision from a switch to mozilla-release; I'm confident we can find a way to adjust the Tails release process to cope with the additional builds and delays (e.g. thanks to reproducible build properties). So let's keep this in mind and I'm happy to stay in the loop, but let's not block on Tails :) Cheers, -- intrigeri
participants (7)
-
Arlo Breault -
Arthur D. Edelstein -
Georg Koppen -
intrigeri -
Matthew Finkel -
Nicolas Vigier -
Yawning Angel