October 2015 - tbb-dev - lists.torproject.org

Using topic branches for Tor Browser patches
by Nicolas Vigier 15 Dec '15

15 Dec '15

Hello, Currently on tor-browser.git, each Tor Browser release has a single branch that includes all our patches. This is simple, however it makes it difficult to test each patch individually, and see which patch is responsible for which test failure. For instance, the patch to add preference overrides causes many tests to fail, probably because many tests don't expect the default preferences to change: 31.7.0esr pass all tests: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5bbe47119ac0 31.7.0esr + preference overrides patch fails many tests: https://treeherder.mozilla.org/#/jobs?repo=try&revision=fdc349fbbf5f After discussing about this quickly with intrigeri last week, I have been thinking about a different way to organize the patches: maybe we could use one branch for each topic, and merge all those branches in a single branch when we prepare the release. The naming of the branches could be: - ${firefox_version}/base: the commit for the firefox version we are using. - ${firefox_version}/${topic_name}: A branch containing patch(es) on a specific topic. $topic_name can be a bug number and/or a short description. We can submit those branches to Mozilla Try to see if they break some unit tests. - ${firefox_version}/${tbb_version}/${topic_name}: The same as the previous one, but for a topic that we only want to include in a specific Tor Browser version. - tor-browser-${firefox_version}-${tbb_version}-1: The branch that we use to build Tor Browser. This branch is based on ${firefox_version}/base, and includes merges of all branches matching ${firefox_version}/[^/]* or ${firefox_version}/${tbb_version}/[^/]*. As an exemple, I have converted a few patches from tor-browser-31.7.0esr-4.5-1 to this new naming in this git repository: https://github.com/boklm/gecko-dev/branches/all With all topic branches merged to this tor-browser-31.7.0esr-4.5-1 branch: https://github.com/boklm/gecko-dev/commits/tor-browser-31.7.0esr-4.5-1 I also started submitting some of them to Mozilla Try: 31.7.0esr/2874-Block-Components.interfaces-from-content: https://treeherder.mozilla.org/#/jobs?repo=try&revision=91cd13cf0ac7 (some mochitest tests failing) 31.7.0esr/2950-Make-Permissions-Manager-memory-only: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3f8cb0c1984e (tbb-tests/browser_tor_bug2950.js failing) 31.7.0esr/2949-Make-Intermediate-Cert-Store-memory-only: https://treeherder.mozilla.org/#/jobs?repo=try&revision=139538724d04 (no test failing) 31.7.0esr/3547-Block-all-plugins-except-flash: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c75fb4685994 (some xpcshell, mochitest, mochitest-chrome, reftest failing) 31.7.0esr/3229-Make-content-pref-service-memory-only-clearable: https://treeherder.mozilla.org/#/jobs?repo=try&revision=344873ae1513 (1 xpcshell test failing) 31.7.0esr/TB2-Provide-an-observer-event-to-close-persistent-connections: https://treeherder.mozilla.org/#/jobs?repo=try&revision=93af45d5111a (no test failing) 31.7.0esr/2875-Limit-device-and-system-specific-CSS-Media-Queries: https://treeherder.mozilla.org/#/jobs?repo=try&revision=32244f089736 (still running at the moment) 31.7.0esr/2872-Limit-the-number-of-fonts-per-document: https://treeherder.mozilla.org/#/jobs?repo=try&revision=4abfb45bba86 (still running at the moment) We could also have some helpers scripts to make management of all those branches easier: - a script to merge / list all topic branches that have not yet been merged into their tor-browser-${firefox_version}-${tbb_version}-1 branch. - a script to push / list all branches that need to be pushed for a firefox or tbb version to a remote git repository. - a script to rebase all patches from a previous firefox version to a new firefox version. What do you think about this idea ? Nicolas

1 7

Memory Allocator #10281 write-up
by Tom Ritter 02 Nov '15

02 Nov '15

Firefox has a feature where you can build a memory allocator library and shim it into Firefox - this is called building a memory replacement library. It requires an additional build-time configuration option. Our first investigation was with PartitionAlloc being built as a memory replacement library. This mostly worked, except we could not replicate the implementation of memalign, which required editing the core internals of PartitionAlloc. It's also important to note that our use of PartitionAlloc did not actually partition the allocations based on any intelligent design. This is an ongoing problem. Currently, when Firefox performs an allocation, it will snake its way through the layers of abstraction until it arrives at our replacement library. The only information that has been passed into our replacement library is the size of the allocation. We do not have any context about what the allocation is for, and therefore which partition we would like to place it in. (We theorized about walking the call stack in assembly and binning allocation based on that but this was an untested idea.) The work with PartitionAlloc paused. After some time, Firefox began investigating partitioning allocations using jemalloc3's partitioning capabilities. jemalloc3 is a new version of the currently used allocator, jemalloc. Mozilla's migration process is first to move to jemalloc3, and then to start partitioning allocations. Currently, their move to jemalloc3 is paused because <actually I'm not entirely sure why. I think "performance" and "mobile".> However the code for jemalloc3 is present, and making use of it is only a configuration change away. jemalloc3 solves one of the problems of PartitionAlloc - the allocator actually, completely works. Mostly. There is at least one issue which I believe is <an outstanding bug in jemalloc causing mismatched VirtualAlloc/Free calls - but this may be fixed now>. We did some performance comparisons of jemalloc vs jemalloc3 and it was reasonable: http://imgur.com/a/A6TV9 To actually partition allocations we can implement a replace library that makes use of jemalloc3's partitioning capabilities. But we run into the same problem as before - we don't have any context to know what partition to put an allocation into. One thought was that we could use memory tags, but after investigation, that did not seem feasible. We _can_ go through the Firefox codebase and change the allocation call to include an arena constant, but this patch will be difficult to maintain, we are likely to miss some allocation situations, and eventually Mozilla will do this themselves. References: https://lists.torproject.org/pipermail/tbb-dev/2015-May/000269.html was my most-recent message on this topic. It references a github issue that is now closed. The patch referenced below should not be needed if this patch solves Mozilla's issue. Does it? Was jemalloc's fix merged? Questions I don't know the answer to. https://lists.torproject.org/pipermail/tbb-dev/2015-May/000272.html Was my followup. Future Directions and Open Questions: - Do we have a clear picture on why mozilla isn't shipping jemalloc3? - Is the jemalloc3 code in ESR38 considered bug-free, or did the known github issue not get patched in it? - Is there concern over switching TBB's allocator over to jemalloc3? - Is there concern over enabling the replace-malloc switch in TBB? - If both of those are done, any TBB distribution can, at start-time, choose to use a randomizing allocator in the form of a replacement library. The beginnings of one is attached. The library does not need to be used by default, but can be included in every install.

2 1

New meeting time: 19:00 UTC on Mondays
by Georg Koppen 30 Oct '15

30 Oct '15

Hi, due to the summer time being gone next week we'll adjust our meeting time. From next Monday on we'll meet 19:00 UTC in #tor-dev. Georg

1 0

Moving Tor Browser meeting tomorrow
by Georg Koppen 12 Oct '15

12 Oct '15

Hi, due to another US federal holiday being on Monday we'll move the Tor Browser meeting tomorrow to Tuesday, October 13. We'll get started 18:00 UTC in #tor-project. Sorry for the short notice, Georg

1 0

Continuous builds
by Linus Nordberg 08 Oct '15

08 Oct '15

Hi, I'd like to contribute by building Tor Browser on a fast machine. My understanding is that -nightly was broken last week. Is this a good place to learn when it's unbroken so that I can start building? Thanks, Linus

3 2

Missing IRC meeting today
by Mike Perry 05 Oct '15

05 Oct '15

I will be missing the weekly IRC meeting today due to travel. Sorry for the late notice. Georg should be able to run the meeting this week again. -- Mike Perry

2 1