Gitian-based Deterministic Build System for TBB (Need MacOS Help!)
Over the past couple weeks I've been redoing the TBB build system to use Gitian to produce alpha TBBs using Tor Launcher instead of Vidalia. I have succeeded in producing deterministic, localized builds of TBB for Linux and Windows. This means that independent people all over the world can now easily produce their own bundles for these platforms fresh from sources, and have their bundles exactly match the bundles the Tor Project releases, down to the SHA256 hash. If we leverage this property wisely, it will allow us to defend against targeted attacks against our bundlers and their build machines, and even ultimately ensure the integrity of our bundles in the event of key compromise of the gpg keys used to sign the bundles. My plan for this is for there to be between 2-3 official signers for each bundle, where each person produces their build independently, and signs the (identical) result files. To further protect against targeted attack, in addition to these 2-3 official signers, we need some people to be "secret verifiers". Ideally these people would not be publicly affiliated with the Tor Project, but would still produce their own bundles anyway. If their SHA256 ever fails to match the signed bundles, that person should anonymously open a trac ticket (using the cypherpunks account) and attach the bundle files that differ for analysis. The differing files can be found easily enough with 'diff -r'. To ensure the existence of these "secret verifiers", I believe that the official signers should occasionally conspire to conduct "Fire Drills", where they all agree to alter the bundle in some innocuous way (such as adding whitespace to a config file or a Firefox JS file), and ensure that a verifiers anonymously report the verification failure. In future versions of Tor, we should probably add a consensus field consisting of a url to a file that lists the current recommended bundle hashes and versions, along with the current SHA256 of that file, to anchor the bundle authentication the Tor's current trust root (the 9 dirauth keys). To try out the new build system, please see the README, and let me know where the system could use clarification or improvement to make it easier to use: https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... The build system has some quirks that are worth mentioning: 1. It requires you run it from either an Ubuntu 12.04 or above host with KVM support, *or* you run it from an Ubuntu 12.04 or above chroot/VM. The bundle scripts try to detect your current situation and suggest that you "export USE_LXC=1" from your shell if you need to, to cause the system to use LXC instead of KVM (so that you can build from an Ubuntu VM or on a machine that does not otherwise support KVM). 2. We currently have no MacOS support. To support MacOS, we need to create cross-compilers for it so that we can produce builds from the Gitian VMs (which again are Ubuntu). A few people have done this. I have sent them mail asking for instructions on how to reproduce their compiler packages: http://www.tarnyko.net/en/?q=node/9 https://launchpad.net/~flosoft/+archive/cross-apple/+packages http://wiki.freepascal.org/Cross_compiling_OSX_on_Linux Unfortunately, at least one of those URLs say that to produce a cross-compiler, you need access to an OSX SDK. Since I do not have a Mac that is currently supported by recent OSX SDKs, and since we *really* want to be sure that the cross-compilers we produce use code from a fresh known-good SDK install, I won't be doing this. Please let me know if you'd like to help tackle this problem. In the meantime, I am going to work on the rest of the "Short Term" TODO items, and produce official alpha bundles for Linux and Windows, so we can test Tor Launcher in an official alpha release. Here's the TODO file: https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... Happy building! -- Mike Perry
On 2013-05-15 08:49 , Mike Perry wrote:
Over the past couple weeks I've been redoing the TBB build system to use Gitian to produce alpha TBBs using Tor Launcher instead of Vidalia. I have succeeded in producing deterministic, localized builds of TBB for Linux and Windows.
This means that independent people all over the world can now easily produce their own bundles for these platforms fresh from sources, and have their bundles exactly match the bundles the Tor Project releases, down to the SHA256 hash.
Awesome! [..]
Unfortunately, at least one of those URLs say that to produce a cross-compiler, you need access to an OSX SDK. Since I do not have a Mac that is currently supported by recent OSX SDKs, and since we *really* want to be sure that the cross-compilers we produce use code from a fresh known-good SDK install, I won't be doing this. Please let me know if you'd like to help tackle this problem.
XCode is free-to-download/use fortunately, one can just not redistribute it according to their terms.... See https://developer.apple.com/xcode/ I think you can get away with making a free/toss-away account for downloading it. At the moment it ends up at: http://adcdownload.apple.com/Developer_Tools/xcode_4.6.2/xcode4620419895a.dm... but that requires the cookies from the login, which does not even work under Chrome, but does work in Safari (thus you might have luck with Firefox) It is a hefty 1.61 GiB download though as it contains everything in OSX and IOS along with the full Xcode suite. Afaik there is no 'headers only' package available. The tutorials you provide all use OpenDarwin's cctools, which is a modified/"improved" version of Apple's opensource cctools. Most of that stuff is pretty outdated though, as OSX is at 10.8, not 10.5 from about 6 years ago. It seems though most people (reading what is being done on the web ;) are using the flosoft stuff which is at least 10.6 and embeds the OSX SDK. Some other 'older' details on all of it can be found at: http://www.sandroid.org/imcross/ which contains quite detailed version on downloading the SDK and getting cross-compilers up and running. Not a fun thing to do, as not a lot of people do it this way, the typical way is to "just" (yes, ahum etc) have a OSX box somewhere (physical as virtualization of them is also a lot of similar fun) and then let Jenkins or similar SSH into it and compile it that way. Greets, Jeroen
Thus spake Jeroen Massar (jeroen@massar.ch):
On 2013-05-15 08:49 , Mike Perry wrote:
Unfortunately, at least one of those URLs say that to produce a cross-compiler, you need access to an OSX SDK. Since I do not have a Mac that is currently supported by recent OSX SDKs, and since we *really* want to be sure that the cross-compilers we produce use code from a fresh known-good SDK install, I won't be doing this. Please let me know if you'd like to help tackle this problem.
The tutorials you provide all use OpenDarwin's cctools, which is a modified/"improved" version of Apple's opensource cctools. Most of that stuff is pretty outdated though, as OSX is at 10.8, not 10.5 from about 6 years ago. It seems though most people (reading what is being done on the web ;) are using the flosoft stuff which is at least 10.6 and embeds the OSX SDK.
Ok, I guess I will just go the flosoft route then. I will work on writing gitian descriptors using these binary packages. However, it looks like right now this means I have to use "Maverick" (11.04) as the Gitian VM's base install. Since "Maverick" is unsupported for security updates, this might be unwise for production.. Is anyone able to work on rebuilding flosofts's .deb packages for either lucid (10.04 LTS) or precise (12.04 LTS) while I work on getting initial Gitian descriptors using Maverick for now? https://launchpad.net/~flosoft/+archive/cross-apple/+packages -- Mike Perry
Mike Perry:
Thus spake Jeroen Massar (jeroen@massar.ch):
On 2013-05-15 08:49 , Mike Perry wrote:
Unfortunately, at least one of those URLs say that to produce a cross-compiler, you need access to an OSX SDK. Since I do not have a Mac that is currently supported by recent OSX SDKs, and since we *really* want to be sure that the cross-compilers we produce use code from a fresh known-good SDK install, I won't be doing this. Please let me know if you'd like to help tackle this problem.
The tutorials you provide all use OpenDarwin's cctools, which is a modified/"improved" version of Apple's opensource cctools. Most of that stuff is pretty outdated though, as OSX is at 10.8, not 10.5 from about 6 years ago. It seems though most people (reading what is being done on the web ;) are using the flosoft stuff which is at least 10.6 and embeds the OSX SDK.
Ok, I guess I will just go the flosoft route then. I will work on writing gitian descriptors using these binary packages. However, it looks like right now this means I have to use "Maverick" (11.04) as the Gitian VM's base install. Since "Maverick" is unsupported for security updates, this might be unwise for production..
Is anyone able to work on rebuilding flosofts's .deb packages for either lucid (10.04 LTS) or precise (12.04 LTS) while I work on getting initial Gitian descriptors using Maverick for now? https://launchpad.net/~flosoft/+archive/cross-apple/+packages
Good news, everyone! I ended up hitting an Internal Compiler Error in flosoft's GCC while trying to build Firefox with them. After a long and painful saga of trying to rebuild a newer version of GCC, I eventually discovered the excellent "toolchain4" cross-compilation project maintained by Ray Donnelly: https://github.com/mingwandroid/toolchain4 https://mingw-and-ndk.googlecode.com/files/multiarch-darwin11-cctools127.2-g... Despite using the same base GCC version as flosoft (4.2.1), his compiler does not hit the ICE with Firefox. Moreover, with Ray's help, I was able to use his binaries and the flosoft OSX 10.6 SDK to successfully build a working TBB for Mac from Gitian. These descriptors are now in the repository, and the Makefile should now build bundles for all three platforms by default. There still are some branding and localization bugs, and I have not yet redone the whole build to verify determinism for MacOS yet, but I am optimistic! -- Mike Perry
On Wed, May 22, 2013 at 6:40 PM, Mike Perry <mikeperry@torproject.org> wrote:
Mike Perry:
Thus spake Jeroen Massar (jeroen@massar.ch):
On 2013-05-15 08:49 , Mike Perry wrote:
Unfortunately, at least one of those URLs say that to produce a cross-compiler, you need access to an OSX SDK. Since I do not have a Mac that is currently supported by recent OSX SDKs, and since we *really* want to be sure that the cross-compilers we produce use code from a fresh known-good SDK install, I won't be doing this. Please let me know if you'd like to help tackle this problem.
The tutorials you provide all use OpenDarwin's cctools, which is a modified/"improved" version of Apple's opensource cctools. Most of that stuff is pretty outdated though, as OSX is at 10.8, not 10.5 from about 6 years ago. It seems though most people (reading what is being done on the web ;) are using the flosoft stuff which is at least 10.6 and embeds the OSX SDK.
Ok, I guess I will just go the flosoft route then. I will work on writing gitian descriptors using these binary packages. However, it looks like right now this means I have to use "Maverick" (11.04) as the Gitian VM's base install. Since "Maverick" is unsupported for security updates, this might be unwise for production..
Is anyone able to work on rebuilding flosofts's .deb packages for either lucid (10.04 LTS) or precise (12.04 LTS) while I work on getting initial Gitian descriptors using Maverick for now? https://launchpad.net/~flosoft/+archive/cross-apple/+packages
Good news, everyone!
I ended up hitting an Internal Compiler Error in flosoft's GCC while trying to build Firefox with them. After a long and painful saga of trying to rebuild a newer version of GCC, I eventually discovered the excellent "toolchain4" cross-compilation project maintained by Ray Donnelly: https://github.com/mingwandroid/toolchain4 https://mingw-and-ndk.googlecode.com/files/multiarch-darwin11-cctools127.2-g...
Despite using the same base GCC version as flosoft (4.2.1), his compiler does not hit the ICE with Firefox.
Moreover, with Ray's help, I was able to use his binaries and the flosoft OSX 10.6 SDK to successfully build a working TBB for Mac from Gitian. These descriptors are now in the repository, and the Makefile should now build bundles for all three platforms by default.
There still are some branding and localization bugs, and I have not yet redone the whole build to verify determinism for MacOS yet, but I am optimistic!
-- Mike Perry
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Would this work on an Ubuntu that's built from source, or are we dependent on their binaries?
AK:
On Wed, May 22, 2013 at 6:40 PM, Mike Perry <mikeperry@torproject.org> wrote:
Mike Perry:
Thus spake Jeroen Massar (jeroen@massar.ch):
On 2013-05-15 08:49 , Mike Perry wrote:
Unfortunately, at least one of those URLs say that to produce a cross-compiler, you need access to an OSX SDK. Since I do not have a Mac that is currently supported by recent OSX SDKs, and since we *really* want to be sure that the cross-compilers we produce use code from a fresh known-good SDK install, I won't be doing this. Please let me know if you'd like to help tackle this problem.
The tutorials you provide all use OpenDarwin's cctools, which is a modified/"improved" version of Apple's opensource cctools. Most of that stuff is pretty outdated though, as OSX is at 10.8, not 10.5 from about 6 years ago. It seems though most people (reading what is being done on the web ;) are using the flosoft stuff which is at least 10.6 and embeds the OSX SDK.
Ok, I guess I will just go the flosoft route then. I will work on writing gitian descriptors using these binary packages. However, it looks like right now this means I have to use "Maverick" (11.04) as the Gitian VM's base install. Since "Maverick" is unsupported for security updates, this might be unwise for production..
Is anyone able to work on rebuilding flosofts's .deb packages for either lucid (10.04 LTS) or precise (12.04 LTS) while I work on getting initial Gitian descriptors using Maverick for now? https://launchpad.net/~flosoft/+archive/cross-apple/+packages
Good news, everyone!
I ended up hitting an Internal Compiler Error in flosoft's GCC while trying to build Firefox with them. After a long and painful saga of trying to rebuild a newer version of GCC, I eventually discovered the excellent "toolchain4" cross-compilation project maintained by Ray Donnelly: https://github.com/mingwandroid/toolchain4 https://mingw-and-ndk.googlecode.com/files/multiarch-darwin11-cctools127.2-g...
Despite using the same base GCC version as flosoft (4.2.1), his compiler does not hit the ICE with Firefox.
Moreover, with Ray's help, I was able to use his binaries and the flosoft OSX 10.6 SDK to successfully build a working TBB for Mac from Gitian. These descriptors are now in the repository, and the Makefile should now build bundles for all three platforms by default.
There still are some branding and localization bugs, and I have not yet redone the whole build to verify determinism for MacOS yet, but I am optimistic!
Would this work on an Ubuntu that's built from source, or are we dependent on their binaries?
Unclear. Unfortunately, I've run into a series of differences that arise between different *hardware*, even though the underlying Ubuntu versions and VMs are the same.. Most of those are file reordering differences inside zip files that probably arise from filesystem/blockdevice-level differences, but there is some really spooky shit happening inside of Firefox's libxul.so too (no joke). I am now working on eliminating those differences, so that at least two different computers can produce the same binaries. I have to deal with that setback first before we could expect a full source rebuild to work, I think. :/ However, I think the better way to achieve what you're getting at is to recompile the Ubuntu toolchain (gcc/g++/binutils) itself from source as a Gitian descriptor. Once we get that piece, we *should* be able to run all the descriptors on different Linux distribution images and still get the same binaries, allowing us to require compromise of *all* Linux distributions that reproduce our builds (including say Gentoo). This idea is documented in the "Long-Term" section of the TODO file, but it would happen much faster if someone helped to make a simple Gitian descriptor to recompile the Ubuntu toolchain (which we consider known-good because it is what Mozilla uses). https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... We'd need to do the same with the mingw-w64 toolchain and Ray's 'toolchain4', too. Note: The toolchains themselves probably don't have to be compiled determistically (except for the statically linked bits like libgcc and the crt stuff). -- Mike Perry
On Sun, Jun 9, 2013 at 5:03 PM, Mike Perry <mikeperry@torproject.org> wrote:
AK:
On Wed, May 22, 2013 at 6:40 PM, Mike Perry <mikeperry@torproject.org> wrote:
Mike Perry:
Thus spake Jeroen Massar (jeroen@massar.ch):
On 2013-05-15 08:49 , Mike Perry wrote:
Unfortunately, at least one of those URLs say that to produce a cross-compiler, you need access to an OSX SDK. Since I do not have a Mac that is currently supported by recent OSX SDKs, and since we *really* want to be sure that the cross-compilers we produce use code from a fresh known-good SDK install, I won't be doing this. Please let me know if you'd like to help tackle this problem.
The tutorials you provide all use OpenDarwin's cctools, which is a modified/"improved" version of Apple's opensource cctools. Most of that stuff is pretty outdated though, as OSX is at 10.8, not 10.5 from about 6 years ago. It seems though most people (reading what is being done on the web ;) are using the flosoft stuff which is at least 10.6 and embeds the OSX SDK.
Ok, I guess I will just go the flosoft route then. I will work on writing gitian descriptors using these binary packages. However, it looks like right now this means I have to use "Maverick" (11.04) as the Gitian VM's base install. Since "Maverick" is unsupported for security updates, this might be unwise for production..
Is anyone able to work on rebuilding flosofts's .deb packages for either lucid (10.04 LTS) or precise (12.04 LTS) while I work on getting initial Gitian descriptors using Maverick for now? https://launchpad.net/~flosoft/+archive/cross-apple/+packages
Good news, everyone!
I ended up hitting an Internal Compiler Error in flosoft's GCC while trying to build Firefox with them. After a long and painful saga of trying to rebuild a newer version of GCC, I eventually discovered the excellent "toolchain4" cross-compilation project maintained by Ray Donnelly: https://github.com/mingwandroid/toolchain4 https://mingw-and-ndk.googlecode.com/files/multiarch-darwin11-cctools127.2-g...
Despite using the same base GCC version as flosoft (4.2.1), his compiler does not hit the ICE with Firefox.
Moreover, with Ray's help, I was able to use his binaries and the flosoft OSX 10.6 SDK to successfully build a working TBB for Mac from Gitian. These descriptors are now in the repository, and the Makefile should now build bundles for all three platforms by default.
There still are some branding and localization bugs, and I have not yet redone the whole build to verify determinism for MacOS yet, but I am optimistic!
Would this work on an Ubuntu that's built from source, or are we dependent on their binaries?
Unclear. Unfortunately, I've run into a series of differences that arise between different *hardware*, even though the underlying Ubuntu versions and VMs are the same.. Most of those are file reordering differences inside zip files that probably arise from filesystem/blockdevice-level differences, but there is some really spooky shit happening inside of Firefox's libxul.so too (no joke).
I am now working on eliminating those differences, so that at least two different computers can produce the same binaries. I have to deal with that setback first before we could expect a full source rebuild to work, I think. :/
However, I think the better way to achieve what you're getting at is to recompile the Ubuntu toolchain (gcc/g++/binutils) itself from source as a Gitian descriptor. Once we get that piece, we *should* be able to run all the descriptors on different Linux distribution images and still get the same binaries, allowing us to require compromise of *all* Linux distributions that reproduce our builds (including say Gentoo).
This idea is documented in the "Long-Term" section of the TODO file, but it would happen much faster if someone helped to make a simple Gitian descriptor to recompile the Ubuntu toolchain (which we consider known-good because it is what Mozilla uses). https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git...
We'd need to do the same with the mingw-w64 toolchain and Ray's 'toolchain4', too.
Note: The toolchains themselves probably don't have to be compiled determistically (except for the statically linked bits like libgcc and the crt stuff).
-- Mike Perry
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Thanks for the reply. Its good to know that these issues are being looked at. The build process was one of the main problems I had with the TBB, and this definitely seems like an improvement.
participants (3)
-
AK -
Jeroen Massar -
Mike Perry