brizental pushed to branch main at The Tor Project / Applications / tor-browser-build

Commits: 8fbb192b by Beatriz Rizental at 2025-04-02T15:01:36+02:00 Bug 41393 - Create special clang packaged for Linux

Building Firefox requires compiler-rt wasm32 library to be packaged alongside clang.

We were doing this step inside the firefox and geckoview projects, but that meant one can't simply download the clang toolchain and use it for dev builds.

MacOS and Windows already have separate projects to package clang with other libraries, so a step to include the wasm32 compiler-rt lib was added to each of them.

- - - - -

17 changed files:

- + projects/clang-linux/README.md - + projects/clang-linux/build - + projects/clang-linux/config - projects/clang/README.md - projects/firefox/build - projects/firefox/config - projects/geckoview/build - projects/geckoview/config - projects/macosx-toolchain/build - projects/macosx-toolchain/config - projects/mingw-w64-clang/build - projects/mingw-w64-clang/config - projects/wasi-sysroot/README.md - projects/wasi-sysroot/build - + projects/wasi-sysroot/build_common - + projects/wasi-sysroot/build_compiler_rt - projects/wasi-sysroot/config

Changes:

===================================== projects/clang-linux/README.md ===================================== @@ -0,0 +1,3 @@ +This projects packages extra libraries and tools alongside clang for Firefox developement +on Linux and Android. It is the default compiler for 'firefox' and 'geckoview' projects +on Linux and Android.

===================================== projects/clang-linux/build ===================================== @@ -0,0 +1,21 @@ +#!/bin/bash +[% c("var/set_default_env") -%] +distdir=/var/tmp/dist/[% project %] +mkdir -p /var/tmp/dist + +tar -C /var/tmp/dist -xf [% c('input_files_by_name/clang') %] +tar -C /var/tmp/dist -xf [% c('input_files_by_name/wasi-compiler-rt') %] + +[% IF c("var/android") -%] + tar -C /var/tmp/dist -xf [% c('input_files_by_name/llvm-runtimes') %] + cp -r /var/tmp/dist/llvm-runtimes/* /var/tmp/dist/clang +[% END -%] + +mkdir $distdir +mv /var/tmp/dist/clang/* $distdir/ + +cd /var/tmp/dist +[% c('tar', { + tar_src => [ project ], + tar_args => '-caf ' _ dest_dir _ '/' _ c('filename'), + }) %]

===================================== projects/clang-linux/config ===================================== @@ -0,0 +1,16 @@ +# vim: filetype=yaml sw=2 +version: '[% c("var/llvm_version") %]' +filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.[% c("compress_tar") %]' +container: + use_container: 1 + +input_files: + - project: container-image + - name: clang + project: clang + - project: wasi-sysroot + name: wasi-compiler-rt + pkg_type: build_compiler_rt + - project: llvm-runtimes + name: llvm-runtimes + enable: '[% c("var/android") %]'

===================================== projects/clang/README.md ===================================== @@ -1,7 +1,9 @@ This is the Clang compiler.

-For Linux and Android, we use it only to build Firefox and GeckoView. -For Windows and macOS, we use it to build everything, but we do not use this +For Linux and Android, we use it only to build Firefox and GeckoView, but we do +not use this project directly. Instead, we use `clang-linux`, +that adds platform-specific libraries and tools. +For Windows and macOS, we use it to build everything, but we also do not use this project directly. Instead, we use `mingw-w64-clang` and `macosx_toolchain`, that also add platform-specific libraries and tools.

===================================== projects/firefox/build ===================================== @@ -37,7 +37,7 @@ export PATH="/var/tmp/dist/rust/bin:/var/tmp/dist/cbindgen:/var/tmp/dist/node/bi [% END -%] # Use clang for everything on Linux now if we don't build with ASan. [% IF ! c("var/asan") -%] - export PATH="/var/tmp/dist/clang/bin:$PATH" + export PATH="/var/tmp/dist/clang-linux/bin:$PATH" [% END -%] [% IF c("var/linux-i686") -%] # Exporting `PKG_CONFIG_PATH` in the mozconfig file is causing build @@ -51,20 +51,7 @@ export PATH="/var/tmp/dist/rust/bin:/var/tmp/dist/cbindgen:/var/tmp/dist/node/bi

[% IF c("var/rlbox") -%] tar -C /var/tmp/dist -xf [% c('input_files_by_name/wasi-sysroot') %] - # XXX: We need the libclang_rt.builtins-wasm32.a in our clang lib directory. - # Copy it over. - # https://searchfox.org/mozilla-central/source/build/build-clang/build-clang.p..., - # include it directly in our clang - [% IF c("var/macos") -%] - rtdir=/var/tmp/dist/macosx-toolchain/clang/lib/clang/[% pc("llvm-project", "var/llvm_major") %]/lib/wasi - [% ELSIF c("var/windows") -%] - rtdir=/var/tmp/dist/mingw-w64-clang/lib/clang/[% pc("llvm-project", "var/llvm_major") %]/lib/wasi - [% ELSE -%] - rtdir=/var/tmp/dist/clang/lib/clang/[% pc("llvm-project", "var/llvm_major") %]/lib/wasi - [% END -%] - mkdir -p $rtdir - cp /var/tmp/dist/wasi-sysroot/lib/clang/*/lib/wasi/libclang_rt.builtins-wasm32.a $rtdir - export WASI_SYSROOT=/var/tmp/dist/wasi-sysroot/share/wasi-sysroot + export WASI_SYSROOT=/var/tmp/dist/wasi-sysroot [% END -%]

tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.[% c('compress_tar') %]

===================================== projects/firefox/config ===================================== @@ -206,7 +206,7 @@ input_files: - project: python name: python enable: '[% c("var/linux") %]' - - project: clang + - project: clang-linux name: clang enable: '[% c("var/linux") %]' - project: fxc2

===================================== projects/geckoview/build ===================================== @@ -5,21 +5,12 @@ tar -C /var/tmp/dist -xf [% c('input_files_by_name/rust') %] tar -C /var/tmp/dist -xf [% c('input_files_by_name/cbindgen') %] tar -C /var/tmp/dist -xf [% c('input_files_by_name/nasm') %] tar -C /var/tmp/dist -xf [% c('input_files_by_name/clang') %] -export LLVM_CONFIG="/var/tmp/dist/clang/bin/llvm-config" -tar -C /var/tmp/dist -xf [% c('input_files_by_name/llvm-runtimes') %] -cp -r /var/tmp/dist/llvm-runtimes/* /var/tmp/dist/clang/ -export PATH="/var/tmp/dist/rust/bin:/var/tmp/dist/cbindgen:/var/tmp/dist/nasm/bin:/var/tmp/dist/clang/bin:$PATH" +export LLVM_CONFIG="/var/tmp/dist/clang-linux/bin/llvm-config" +export PATH="/var/tmp/dist/rust/bin:/var/tmp/dist/cbindgen:/var/tmp/dist/nasm/bin:/var/tmp/dist/clang-linux/bin:$PATH"

[% IF c("var/rlbox") -%] tar -C /var/tmp/dist -xf [% c('input_files_by_name/wasi-sysroot') %] - # XXX: We need the libclang_rt.builtins-wasm32.a in our clang lib directory. - # Copy it over. - # https://searchfox.org/mozilla-central/source/build/build-clang/build-clang.p..., - # include it directly in our clang - rtdir="/var/tmp/dist/clang/lib/clang/[% pc('llvm-project', 'var/llvm_major') %]/lib/wasi" - mkdir -p "$rtdir" - cp /var/tmp/dist/wasi-sysroot/lib/clang/*/lib/wasi/libclang_rt.builtins-wasm32.a "$rtdir/" - export WASI_SYSROOT=/var/tmp/dist/wasi-sysroot/share/wasi-sysroot + export WASI_SYSROOT=/var/tmp/dist/wasi-sysroot/ [% END -%]

cd /var/tmp/build/[% project %]-[% c("version") %]

===================================== projects/geckoview/config ===================================== @@ -142,10 +142,8 @@ input_files: name: node - project: nasm name: nasm - - project: clang + - project: clang-linux name: clang - - project: llvm-runtimes - name: llvm-runtimes - project: wasi-sysroot name: wasi-sysroot enable: '[% c("var/rlbox") %]'

===================================== projects/macosx-toolchain/build ===================================== @@ -7,6 +7,7 @@ mkdir -p "$distdir" tar -C /var/tmp/dist -xf [% c('input_files_by_name/cmake') %] tar -C /var/tmp/dist -xf [% c('input_files_by_name/ninja') %] tar -C $distdir -xf [% c('input_files_by_name/clang') %] +tar -C $distdir -xf $rootdir/[% c('input_files_by_name/wasi-compiler-rt') %] tar -C $distdir -xf [% c('input_files_by_name/cctools') %] tar -C $builddir -xf [% c('input_files_by_name/llvm-project') %]

===================================== projects/macosx-toolchain/config ===================================== @@ -32,6 +32,9 @@ input_files: sha512sum: a4e05d2a051027620c86f72694da126a2ceed59b8740270426c4ca3adb38e16fe981b536a373a7f0cdcc84bbe9d9149608ccd03967dbd94262548ccadbfa0f5d - name: llvm-project project: llvm-project + - project: wasi-sysroot + name: wasi-compiler-rt + pkg_type: build_compiler_rt - filename: macpkg.py - filename: unpack-sdk.py - filename: tools.diff

===================================== projects/mingw-w64-clang/build ===================================== @@ -6,6 +6,7 @@ mkdir -p $distdir $builddir

cd /var/tmp/dist tar -xf $rootdir/[% c('input_files_by_name/clang') %] +tar -xf $rootdir/[% c('input_files_by_name/wasi-compiler-rt') %] mv clang/* mingw-w64-clang/ tar -xf $rootdir/[% c('input_files_by_name/cmake') %]

===================================== projects/mingw-w64-clang/config ===================================== @@ -20,6 +20,9 @@ input_files: name: cmake - project: llvm-project name: llvm-project + - project: wasi-sysroot + name: wasi-compiler-rt + pkg_type: build_compiler_rt # Wrapper from github.com/mstorsjo/llvm-mingw@b34e1cd82b9eff3dcedfac152f25e0ccb6da33aa. # This is the last commit at which windres-wrapper.c was modified. # Mozilla uses 9f8e5cebd6dbbb7546e8917e6c59284699a48d26 (see

===================================== projects/wasi-sysroot/README.md ===================================== @@ -30,13 +30,14 @@ We stick to the same version used by Firefox, that can be found in the usual

# Mozilla's build script

-Our build script is an adaptation of Firefox's -`taskcluster/scripts/misc/build-sysroot-wasi.sh`. -After ESR updates, we should check if that script was updated. - -The main difference is that Mozilla builds `libclang_rt.builtins-wasm32.a` with -Clang, whereas we build it here and inject it only in the `firefox`/`geckoview` -projects. +Similar to Firefox's build system, this project has two different build targets. +The default `build` target builds the wasi-sysroot, while the `build_compiler_rt` +target builds the wasm32 compiler-rt which gets injected into clang by the `clang-linux`, +`mingw-w64-clang` and `macosx-toolchain` projects. + +These build scripts are adaptations of Firefox's `taskcluster/scripts/misc/build-sysroot-wasi.sh` +and `taskcluster/scripts/misc/build-compiler-rt-wasi.sh`. **After ESR updates, we should +check if that script was updated.**

# Different build ids, same outputs

===================================== projects/wasi-sysroot/build ===================================== @@ -1,85 +1,17 @@ #!/bin/bash -[% c("var/set_default_env") -%] -distdir=/var/tmp/dist/[% project %] -builddir=/var/tmp/build/[% project %]-[% c("abbrev") %] -mkdir -p $distdir -tar -C /var/tmp/dist -xf [% c('input_files_by_name/cmake') %] -tar -C /var/tmp/dist -xf [% c('input_files_by_name/ninja') %] -export PATH="/var/tmp/dist/ninja:/var/tmp/dist/cmake/bin:$PATH" +[% INCLUDE 'build_common' %]

-[% IF c("var/linux") -%] - [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] - [% pc('python', 'var/setup', { python_tarfile => c('input_files_by_name/python') }) %] -[% END -%] +# We already packed compiler-rt alongside clang. +# Just let the build system know it's there. +touch build/compiler-rt.BUILT

-mkdir -p /var/tmp/build -tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.[% c('compress_tar') %] - -cd /var/tmp/build -tar -xf $rootdir/[% project %]-[% c('version') %].tar.[% c('compress_tar') %] -cd $builddir - -pushd src -rmdir config llvm-project wasi-libc -tar -xf $rootdir/[% c('input_files_by_name/wasi-config') %] -mv wasi-config-* config -tar -xf $rootdir/[% c('input_files_by_name/clang-source') %] -mv clang-source llvm-project -export LLVM_HOME=$(pwd)/llvm-project -tar -xf $rootdir/[% c('input_files_by_name/wasi-libc') %] -mv wasi-libc-* wasi-libc -popd - -# What follows has been copied from Firefox's -# taskcluster/scripts/misc/build-sysroot-wasi.sh and adapted to -# tor-browser-build. - -patch -p1 < $rootdir/wasi-sdk.patch - -mkdir -p build/install/wasi -# The wasi-sdk build system wants to build clang itself. We trick it into -# thinking it did, and put our own clang where it would have built its own. -tar -C build -xf $rootdir/[% c('input_files_by_name/clang') %] -mv build/clang build/llvm -touch build/llvm.BUILT - -# The wasi-sdk build system wants a clang and an ar binary in -# build/install/$PREFIX/bin -ln -s $(pwd)/build/llvm/bin build/install/wasi/bin -ln -s llvm-ar build/install/wasi/bin/ar - -# tor-browser-build: we have no .git in our tarball but the script version.sh -# used by the Makefile depends on that. We replace the script. -cat > 'version.sh' << EOF -#!/bin/sh -echo '[% c("abbrev") %]' -EOF - -# Build wasi-libc, and re-pack it, to make sure we get a deterministic output. -make PREFIX=/wasi build/wasi-libc.BUILT - -# We need to compile compiler-rt before building libc++ and libc++abi, because -# we need to inject it to Clang. -make \ - LLVM_PROJ_DIR=$LLVM_HOME \ - PREFIX=/wasi \ - build/compiler-rt.BUILT \ - NINJA_FLAGS='-j[% c("num_procs") %]' - -# Does Mozilla maybe do this when compiling Clang? -rtdir="build/llvm/lib/clang/$(ls build/llvm/lib/clang)/lib/wasi" -mkdir -p "$rtdir" -cp build/compiler-rt/lib/wasi/libclang_rt.builtins-wasm32.a "$rtdir/" - -# Now we can finally build libc++ and libc++abi. make \ LLVM_PROJ_DIR=$LLVM_HOME \ PREFIX=/wasi \ build/libcxx.BUILT \ NINJA_FLAGS='-j[% c("num_procs") %]'

-mv build/install/wasi/share $distdir/ -mv build/install/wasi/lib $distdir/ +mv build/install/wasi/share/wasi-sysroot/* $distdir/

cd /var/tmp/dist [% c('tar', {

===================================== projects/wasi-sysroot/build_common ===================================== @@ -0,0 +1,74 @@ +#!/bin/bash +[% c("var/set_default_env") -%] +is_compiler_rt=[% c("var/is_compiler_rt") %] +[% IF c("var/is_compiler_rt") -%] + distdir=/var/tmp/dist/wasi-compiler-rt +[% ELSE -%] + distdir=/var/tmp/dist/[% project %] +[% END -%] +builddir=/var/tmp/build/[% project %]-[% c("abbrev") %] +mkdir -p $distdir +tar -C /var/tmp/dist -xf [% c('input_files_by_name/cmake') %] +tar -C /var/tmp/dist -xf [% c('input_files_by_name/ninja') %] +export PATH="/var/tmp/dist/ninja:/var/tmp/dist/cmake/bin:$PATH" + +[% IF c("var/linux") -%] + [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] + [% pc('python', 'var/setup', { python_tarfile => c('input_files_by_name/python') }) %] +[% END -%] + +mkdir -p /var/tmp/build +tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.[% c('compress_tar') %] + +cd /var/tmp/build +tar -xf $rootdir/[% project %]-[% c('version') %].tar.[% c('compress_tar') %] +cd $builddir + +pushd src +rmdir config llvm-project wasi-libc +tar -xf $rootdir/[% c('input_files_by_name/wasi-config') %] +mv wasi-config-* config +tar -xf $rootdir/[% c('input_files_by_name/clang-source') %] +mv clang-source llvm-project +export LLVM_HOME=$(pwd)/llvm-project +tar -xf $rootdir/[% c('input_files_by_name/wasi-libc') %] +mv wasi-libc-* wasi-libc +popd + +# What follows has been copied from Firefox's +# taskcluster/scripts/misc/build-sysroot-wasi.sh and adapted to +# tor-browser-build. + +patch -p1 < $rootdir/wasi-sdk.patch + +mkdir -p build/install/wasi + +tar -C build -xf $rootdir/[% c('input_files_by_name/clang') %] +[% IF c("var/is_compiler_rt") -%] + mv build/clang build/llvm +[% ELSE -%] + mv build/[% c("var/clang-project") %] build/llvm +[% END -%] +# The wasi-sdk build system wants to build clang itself. We trick it into +# thinking it did, and put our own clang where it would have built its own. +touch build/llvm.BUILT + +# The wasi-sdk build system wants a clang and an ar binary in +# build/install/$PREFIX/bin +[% IF !c("var/is_compiler_rt") && c("var/macos") -%] + ln -s $(pwd)/build/llvm/clang/bin build/install/wasi/bin +[% ELSE -%] + ln -s $(pwd)/build/llvm/bin build/install/wasi/bin +[% END -%] + +ln -s llvm-ar build/install/wasi/bin/ar + +# tor-browser-build: we have no .git in our tarball but the script version.sh +# used by the Makefile depends on that. We replace the script. +cat > 'version.sh' << EOF +#!/bin/sh +echo '[% c("abbrev") %]' +EOF + +# Build wasi-libc, and re-pack it, to make sure we get a deterministic output. +make PREFIX=/wasi build/wasi-libc.BUILT

===================================== projects/wasi-sysroot/build_compiler_rt ===================================== @@ -0,0 +1,20 @@ +#!/bin/bash +[% INCLUDE 'build_common' %] + +make \ + LLVM_PROJ_DIR=$LLVM_HOME \ + PREFIX=/wasi \ + build/compiler-rt.BUILT \ + NINJA_FLAGS='-j[% c("num_procs") %]' + +# We are putting this file already in the folder it should be when moved into clang. +# This way, consumers can just untar this project into a clang folder and be done. +rtdir=/var/tmp/dist/clang/lib/clang/[% pc("llvm-project", "var/llvm_major") %]/lib/wasi +mkdir -p $rtdir +mv build/compiler-rt/lib/wasi/* $rtdir/ + +cd /var/tmp/dist +[% c('tar', { + tar_src => [ 'clang' ], + tar_args => '-caf ' _ dest_dir _ '/' _ c('filename'), + }) %]

===================================== projects/wasi-sysroot/config ===================================== @@ -15,27 +15,67 @@ container: var: no_crosscompile: 1

+targets: + linux: + var: + clang-project: clang-linux + android: + var: + clang-project: clang-linux + macos: + var: + clang-project: macosx-toolchain + windows: + var: + clang-project: mingw-w64-clang + +steps: + build_compiler_rt: + filename: 'wasi-compiler-rt-[% c("version") %]-[% c("var/build_id") %].tar.[% c("compress_tar") %]' + build_compiler_rt: '[% INCLUDE build_compiler_rt %]' + var: + is_compiler_rt: 1 + input_files: - project: container-image + pkg_type: build # We do not use GCC, but we need its libsdc++ to run Clang - name: '[% c("var/compiler") %]' project: '[% c("var/compiler") %]' enable: '[% c("var/linux") %]' + pkg_type: build - name: clang project: clang + enable: '[% c("var/is_compiler_rt") %]' + pkg_type: build + - name: clang + project: clang-linux + enable: '[% !c("var/is_compiler_rt") && (c("var/linux") || c("var/android")) %]' + - name: clang + project: macosx-toolchain + enable: '[% !c("var/is_compiler_rt") && c("var/macos") %]' + - name: clang + project: mingw-w64-clang + enable: '[% !c("var/is_compiler_rt") && c("var/windows") %]' - project: llvm-project name: clang-source + pkg_type: build - name: cmake project: cmake + pkg_type: build - name: ninja project: ninja + pkg_type: build - name: python project: python enable: '[% c("var/linux") %]' + pkg_type: build - name: wasi-config project: wasi-config + pkg_type: build - name: wasi-libc project: wasi-libc + pkg_type: build # taskcluster/scripts/misc/wasi-sdk.patch from Firefox's source code. # Taken from gecko-dev@823106ac4295f822717f5a953047dc89074d632b - filename: wasi-sdk.patch

View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/commit/8f...