Pier Angelo Vendrame pushed to branch main at The Tor Project / Applications / tor-browser-build Commits: 98e9c031 by Pier Angelo Vendrame at 2026-01-26T13:02:25+01:00 Bug 41684: Move gradle setup back to projects/gradle/config. - - - - - 5ea37387 by Pier Angelo Vendrame at 2026-01-26T13:02:28+01:00 Bug 41684: Minor cleanup in project configs. While working on this issue, I found that there were several minor changes that we could do to improve consistency with the rest of config files, therefore I thought of doing them in a separate commit first. - - - - - 88ad5449 by Pier Angelo Vendrame at 2026-01-26T15:00:03+01:00 Bug 41684: Use the NDK as Android compiler. We used to use a package with the full Android SDK+NDK. However, we have several projects that only need the NDK. Passing the SDK to them involved unneeded rebuilds for every SDK change, and longer iteration times for toolchain update as a consequence. Projects still needing the Android SDK will need to include both. - - - - - 22 changed files: - + projects/android-ndk/README.md - + projects/android-ndk/build - + projects/android-ndk/config - projects/android-toolchain/README.md → projects/android-sdk/README.md - projects/android-toolchain/build → projects/android-sdk/build - projects/android-toolchain/config → projects/android-sdk/config - projects/android-toolchain/platform-36.1-package.xml → projects/android-sdk/platform-36.1-package.xml - projects/application-services/build - projects/application-services/config - projects/application-services/list_toolchain_updates_checks - projects/browser/build.android - projects/browser/config - projects/geckoview/build_ac_fenix - projects/geckoview/build_common - projects/geckoview/config - projects/geckoview/list_toolchain_updates_checks - projects/glean/build - projects/glean/config - projects/gradle/config - projects/oss-licenses-plugin/build - rbm.conf - tools/signing/machines-setup/upload-tbb-to-signing-machine Changes: ===================================== projects/android-ndk/README.md ===================================== @@ -0,0 +1,8 @@ +This projects downloads and repacks the Android NDK, which we use as the +default compiler for Android. + +We used to set the `android-toolchain` project as `var/compiler`, but most of +the projects only need the NDK, so any error in `android-toolchain` would +trigger unneeded rebuilds, making iteration times longer. + +We keep the NDK version in sync with the version used by Mozilla. ===================================== projects/android-ndk/build ===================================== @@ -0,0 +1,9 @@ +#!/bin/bash +[% c("var/set_default_env") -%] + +unzip -qq $rootdir/[% c("input_files_by_name/ndk")%] +mv android-ndk-r[% c("var/release") %] [% project %] +[% c('tar', { + tar_src => [ project ], + tar_args => '-caf ' _ dest_dir _ '/' _ c('filename'), + }) %] ===================================== projects/android-ndk/config ===================================== @@ -0,0 +1,18 @@ +# vim: filetype=yaml sw=2 +filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.[% c("compress_tar") %]' +version: 29.0.14206865 + +var: + release: 29 + setup: | + mkdir -p /var/tmp/dist + tar -C /var/tmp/dist -xf $rootdir/[% c("compiler_tarfile") %] + export ANDROID_NDK_HOME=/var/tmp/dist/android-ndk/ + export ANDROID_NDK_ROOT=$ANDROID_NDK_HOME + export NDK_HOST_TAG=linux-x86_64 + export PATH=$PATH:$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/bin + +input_files: + - URL: 'https://dl.google.com/android/repository/android-ndk-r[% c("var/release") %]-linux.zip' + name: ndk + sha256sum: 4abbbcdc842f3d4879206e9695d52709603e52dd68d3c1fff04b3b5e7a308ecf ===================================== projects/android-toolchain/README.md → projects/android-sdk/README.md ===================================== @@ -1,14 +1,14 @@ -This project is configured as `var/compiler` for Android. - -It contains a few versions of the Android SDK, build tools and the NDK. +This project contains some packages that would normally be installed with +`sdkmanager`, such as the build tools, platform tools and platforms. It defines several environment variables that various build systems look for, such as `ANDROID_HOME`. -Optionally, it also prepares gradle. -The output artifact includes only one version of the NDK, because of its huge -size, but the setup commands can use a custom version of the NDK, since some -projects need different versions (e.g., GeckoView and Application Services). +We used to have a single package for the SDK and the NDK. +However, most of native binaries need only the NDK. +Therefore, we decided to split the project, to possibly reduce the iteration +time when updating the toolchains, as it is much easier to get the NDK right at +the first attempt. # Known issues ===================================== projects/android-toolchain/build → projects/android-sdk/build ===================================== @@ -3,20 +3,13 @@ distdir=$rootdir/[% project %] mkdir -p $distdir - -mkdir $distdir/android-sdk-linux -cd $distdir/android-sdk-linux +cd $distdir # Command line tools mkdir -p cmdline-tools unzip -qq $rootdir/[% c("input_files_by_name/android_commandlinetools") %] -d cmdline-tools mv cmdline-tools/cmdline-tools cmdline-tools/[% c("var/commandlinetools_version_string") %] -# NDK -mkdir ndk -unzip -qq $rootdir/[% c("input_files_by_name/android_ndk_compiler")%] -d ndk -ln -s android-ndk-r[% c("var/android_ndk_release_name") %] ndk/[% c("var/android_ndk_revision") %] - # Build tools mkdir build-tools unzip -qq $rootdir/[% c("input_files_by_name/build_tools") %] -d build-tools @@ -38,6 +31,11 @@ mv $rootdir/platform-36.1-package.xml platforms/android-36.1/package.xml # Platform tools unzip -qq $rootdir/[% c("input_files_by_name/platform_tools") %] +# Placeholder for the NDK, as some projects expect the NDK to be in +# $ANDROID_HOME/ndk/<ndk-version>. +# We will add a symlink to the actual NDK in the setup when available. +mkdir ndk + # That's a quirk required by GeckoView but let's have it here instead of in the # project's build script mkdir emulator ===================================== projects/android-toolchain/config → projects/android-sdk/config ===================================== @@ -5,33 +5,12 @@ version: 36.1.0 var: setup: | mkdir -p /var/tmp/dist - tar -C /var/tmp/dist -xf $rootdir/[% c("compiler_tarfile") %] - export ANDROID_HOME=/var/tmp/dist/[% project %]/android-sdk-linux + tar -C /var/tmp/dist -xf $rootdir/[% c("sdk_tarfile") %] + export ANDROID_HOME=/var/tmp/dist/[% project %] + [[ -n "${ANDROID_NDK_HOME}" ]] && ln -s $ANDROID_NDK_HOME $ANDROID_HOME/ndk/[% pc("android-ndk", "version") %] export PATH=$PATH:$ANDROID_HOME/build-tools/[% c("version") %] - # By default we use the old NDK which is the one GeckoView is using. - # However, we can overwrite that per project if needed (e.g. - # application-services is currently on a newer NDK). - # But an NDK is more than 1.5GB, so it does not make sense to copy one that - # is not even used in all containers. Instead, we download the additional - # NDK only in application-services, and adjust the other variables here. - [% IF c("ndk_release") -%] - export ANDROID_NDK_HOME=$ANDROID_HOME/ndk/android-ndk-r[% c("ndk_release") %] - [% ELSE -%] - export ANDROID_NDK_HOME=$ANDROID_HOME/ndk/android-ndk-r[% c("var/android_ndk_release_name") %] - [% END -%] - export ANDROID_NDK_ROOT=$ANDROID_NDK_HOME - export NDK_HOST_TAG=linux-x86_64 - export PATH=$PATH:$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/bin - [% IF c("gradle_tarfile") -%] - tar -C /var/tmp/dist/[% project %] -xf $rootdir/[% c("gradle_tarfile") %] - export GRADLE_HOME=/var/tmp/dist/[% project %]/gradle - export GRADLE_USER_HOME=$GRADLE_HOME - export PATH=$PATH:$GRADLE_HOME/bin - [% END -%] google_repo: https://dl.google.com/android/repository android_release_dir: android-16 - android_ndk_release_name: '29' - android_ndk_revision: 29.0.14206865 # We need the following two variables for get_build_tools, used by signing # scripts. build_tools_version: 36.1 @@ -45,9 +24,6 @@ input_files: - URL: '[% c("var/google_repo") %]/commandlinetools-linux-[% c("var/commandlinetools_version") %]_latest.zip' name: android_commandlinetools sha256sum: 7ec965280a073311c339e571cd5de778b9975026cfcbe79f2b1cdcb1e15317ee - - URL: '[% c("var/google_repo") %]/android-ndk-r[% c("var/android_ndk_release_name") %]-linux.zip' - name: android_ndk_compiler - sha256sum: 4abbbcdc842f3d4879206e9695d52709603e52dd68d3c1fff04b3b5e7a308ecf - URL: '[% c("var/google_repo") %]/[% c("var/build_tools_filename") %]' name: build_tools sha256sum: '[% c("var/build_tools_sha256sum") %]' ===================================== projects/android-toolchain/platform-36.1-package.xml → projects/android-sdk/platform-36.1-package.xml ===================================== ===================================== projects/application-services/build ===================================== @@ -3,9 +3,10 @@ [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')), - gradle_tarfile => c("input_files_by_name/gradle"), }) %] -pushd /var/tmp/dist/android-toolchain/android-sdk-linux/build-tools/ +[% pc('android-sdk', 'var/setup', { sdk_tarfile => c("input_files_by_name/android-sdk") }) %] +[% pc('gradle', 'var/setup', { gradle_tarfile => c("input_files_by_name/gradle") }) %] +pushd /var/tmp/dist/android-sdk/build-tools/ unzip $rootdir/'[% c("input_files_by_name/build_tools_35") %]' mv android-15 35.0 popd ===================================== projects/application-services/config ===================================== @@ -48,25 +48,20 @@ steps: - gyp input_files: - project: container-image - pkg_type: build - project: '[% c("var/compiler") %]' name: '[% c("var/compiler") %]' - pkg_type: build + - project: android-sdk + name: android-sdk - project: gradle name: gradle - pkg_type: build - project: rust name: rust - pkg_type: build - project: ninja name: ninja - pkg_type: build - project: uniffi-rs name: uniffi-rs - pkg_type: build - project: glean name: glean - pkg_type: build enable: '[% !c("var/generate_gradle_dependencies_list") %]' # Only Application Services currently requires build tools 35. # So, download them only here, rather than adding them to the shared ===================================== projects/application-services/list_toolchain_updates_checks ===================================== @@ -8,7 +8,7 @@ if (m/^\\s*compileSdkVersion:\\s([^"]*),/) { } EOF needed=$(cat build.gradle | perl -ne "$p") -current='[% pc("android-toolchain", "var/android_api_level") %]' +current=$(echo [% pc("android-sdk", "version") %] | cut -f 1 -d '.') check_update_needed compileSdkVersion "$needed" "$current" ===================================== projects/browser/build.android ===================================== @@ -1,6 +1,7 @@ #!/bin/bash [% c("var/set_default_env") -%] -[% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] + +[% pc('android-sdk', 'var/setup', { sdk_tarfile => c("input_files_by_name/android-sdk") }) %] [% IF c("var/android-x86_64") -%] tar -C /var/tmp/dist -xf $rootdir/[% c('input_files_by_name/node') %] ===================================== projects/browser/config ===================================== @@ -69,9 +69,12 @@ targets: input_files: - project: container-image + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + enable: '[% c("var/macos") %]' - project: firefox name: firefox - enable: '[% ! c("var/android") %]' + enable: '[% !c("var/android") %]' - project: firefox name: firefox-aarch64 enable: '[% c("var/macos_universal") %]' @@ -81,30 +84,11 @@ input_files: name: src-firefox pkg_type: src-tarballs enable: '[% c("var/macos") %]' - - project: geckoview - name: fenix - pkg_type: build_apk - enable: '[% c("var/android") && !c("var/android_single_arch") %]' - - project: geckoview - name: fenix - pkg_type: build - enable: '[% c("var/android") && c("var/android_single_arch") %]' - - project: tor-expert-bundle - name: tor-expert-bundle - enable: '[% c("var/tor-browser") %]' - - project: tor-expert-bundle - name: tor-expert-bundle-aarch64 - enable: '[% c("var/macos_universal") && c("var/tor-browser") %]' - target_replace: - '^torbrowser-macos.*': torbrowser-macos-aarch64 - - project: moat-settings - name: moat-settings - enable: '[% c("var/tor-browser") %]' - project: fonts name: fonts - enable: '[% ! c("var/android") %]' + enable: '[% !c("var/android") %]' - filename: Bundle-Data - enable: '[% ! c("var/android") %]' + enable: '[% !c("var/android") %]' - filename: dmg-root enable: '[% c("var/macos") %]' - URL: https://dist.torproject.org/torbrowser/noscript/noscript-13.5.1.90101984.xpi @@ -118,6 +102,20 @@ input_files: name: mullvad-extension sha256sum: 4597ee6fff6a2f19cbf78a6149d38d12ad8a9d6029b7e36f6c639dc3e07dd2c3 enable: '[% c("var/mullvad-browser") %]' + - project: manual + name: manual + enable: '[% !c("var/android") && c("var/tor-browser") %]' + - project: tor-expert-bundle + name: tor-expert-bundle + enable: '[% c("var/tor-browser") %]' + - project: tor-expert-bundle + name: tor-expert-bundle-aarch64 + enable: '[% c("var/macos_universal") && c("var/tor-browser") %]' + target_replace: + '^torbrowser-macos.*': torbrowser-macos-aarch64 + - project: moat-settings + name: moat-settings + enable: '[% c("var/tor-browser") %]' - filename: 'gtk3-settings.ini' enable: '[% c("var/linux") %]' - project: hfsplus-tools @@ -145,15 +143,20 @@ input_files: name: translation-mullvad-browser pkg_type: mullvad-browser enable: '[% c("var/mullvad-browser") && c("var/windows") && !c("var/testbuild") %]' + - project: geckoview + name: fenix + pkg_type: build_apk + enable: '[% c("var/android") && !c("var/android_single_arch") %]' + - project: geckoview + name: fenix + pkg_type: build + enable: '[% c("var/android") && c("var/android_single_arch") %]' # To generate a new keystore, see how-to-generate-keystore.txt - filename: android-qa.keystore enable: '[% c("var/android") %]' - - name: '[% c("var/compiler") %]' - project: '[% c("var/compiler") %]' - enable: '[% c("var/android") || c("var/macos") %]' - - project: manual - name: manual - enable: '[% ! c("var/android") && c("var/tor-browser") %]' + - project: android-sdk + name: android-sdk + enable: '[% c("var/android") %]' # tor-browser-build#40920 - filename: sort-baseline.py enable: '[% c("var/android") %]' ===================================== projects/geckoview/build_ac_fenix ===================================== @@ -12,7 +12,7 @@ gradle $GRADLE_FLAGS assembleGecko -x lint [% IF c('var/generate_gradle_dependen popd # The build might fail with "file exists" otherwise. -rm -rf /var/tmp/dist/android-toolchain/gradle/glean/pythonenv +rm -rf $GRADLE_HOME/glean/pythonenv echo "Building Fenix $(date)" pushd mobile/android/fenix ===================================== projects/geckoview/build_common ===================================== @@ -1,8 +1,9 @@ [% c("var/set_default_env") -%] [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')), - gradle_tarfile => c("input_files_by_name/gradle"), }) %] +[% pc('android-sdk', 'var/setup', { sdk_tarfile => c("input_files_by_name/android-sdk") }) %] +[% pc('gradle', 'var/setup', { gradle_tarfile => c("input_files_by_name/gradle") }) %] distdir=/var/tmp/dist builddir=/var/tmp/build outdir="[% dest_dir _ '/' _ c('filename') -%]" ===================================== projects/geckoview/config ===================================== @@ -78,6 +78,9 @@ steps: - name: '[% c("var/compiler") %]' project: '[% c("var/compiler") %]' pkg_type: build + - project: android-sdk + name: android-sdk + pkg_type: build - project: gradle name: gradle pkg_type: build @@ -156,6 +159,8 @@ input_files: refresh_input: 1 - name: '[% c("var/compiler") %]' project: '[% c("var/compiler") %]' + - project: android-sdk + name: android-sdk - project: gradle name: gradle - project: rust ===================================== projects/geckoview/list_toolchain_updates_checks ===================================== @@ -10,7 +10,7 @@ if (m/^\\s*NDK_VERSION\\s*=\\s*"(.+)"/) { } EOF needed=$(cat python/mozboot/mozboot/android.py | perl -ne "$p") -current='r[% pc("android-toolchain", "var/android_ndk_release_name") %]' +current='r[% pc("android-ndk", "var/release") %]' check_update_needed ndk_release_name "$needed" "$current" @@ -22,7 +22,7 @@ if (m/build-tools;(.+)$/) { } EOF needed=$(cat python/mozboot/mozboot/android-packages.txt | perl -ne "$p") -current='[% pc("android-toolchain", "version") %]' +current='[% pc("android-sdk", "version") %]' check_update_needed build_tools "$needed" "$current" @@ -43,7 +43,7 @@ while (<>) { } EOF needed=$(cat python/mozboot/mozboot/android.py | perl -e "$p") -current='[% pc("android-toolchain", "var/commandlinetools_version_string") %]-[% pc("android-toolchain", "var/commandlinetools_version") %]' +current='[% pc("android-sdk", "var/commandlinetools_version_string") %]-[% pc("android-sdk", "var/commandlinetools_version") %]' check_update_needed cmdline-tools "$needed" "$current" ===================================== projects/glean/build ===================================== @@ -3,10 +3,11 @@ [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')), - gradle_tarfile => c("input_files_by_name/gradle"), }) %] +[% pc('android-sdk', 'var/setup', { sdk_tarfile => c("input_files_by_name/android-sdk") }) %] +[% pc('gradle', 'var/setup', { gradle_tarfile => c("input_files_by_name/gradle") }) %] -pushd /var/tmp/dist/android-toolchain/android-sdk-linux/ +pushd /var/tmp/dist/android-sdk cd build-tools unzip $rootdir/'[% c("input_files_by_name/build_tools_35") %]' mv android-15 35.0.0 ===================================== projects/glean/config ===================================== @@ -23,9 +23,11 @@ steps: - project: container-image - project: '[% c("var/compiler") %]' name: '[% c("var/compiler") %]' + - project: android-sdk + name: android-sdk # These versions of build tools and platform are not the same version needed # by our other projects, therefore we download it only here instead of modifying - # the android-toolchain project. + # the android-sdk project. - URL: 'https://dl.google.com/android/repository/build-tools_r35_linux.zip' name: build_tools_35 sha256sum: bd3a4966912eb8b30ed0d00b0cda6b6543b949d5ffe00bea54c04c81e1561d88 ===================================== projects/gradle/config ===================================== @@ -5,6 +5,11 @@ container: use_container: 1 var: + setup: | + tar -C /var/tmp/dist -xf $rootdir/[% c("gradle_tarfile") %] + export GRADLE_HOME=/var/tmp/dist/gradle + export GRADLE_USER_HOME=$GRADLE_HOME + export PATH=$PATH:$GRADLE_HOME/bin gradle_sha256sum: 8.14.3: bd71102213493060956ec229d946beee57158dbd89d0e62b91bca0fa2c5f3531 ===================================== projects/oss-licenses-plugin/build ===================================== @@ -6,8 +6,7 @@ builddir=/var/tmp/build/[% project %]-[% c('version') %]/[% project %] mkdir -p $distdir $builddir tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.[% c('compress_tar') %] -tar -C /var/tmp/dist -xf [% c('input_files_by_name/gradle') %] -export PATH=/var/tmp/dist/gradle/bin:$PATH +[% pc('gradle', 'var/setup', { gradle_tarfile => c("input_files_by_name/gradle") }) %] gradle_repo=/var/tmp/dist/gradle-dependencies mv $rootdir/[% c('input_files_by_name/gradle-dependencies') %] $gradle_repo ===================================== rbm.conf ===================================== @@ -473,7 +473,7 @@ targets: var: android: 1 platform: android - compiler: android-toolchain + compiler: android-ndk android_min_api: 26 CC: '[% c("var/cross_prefix") %][% c("var/android_min_api") %]-clang' CFLAGS: '-Oz -fvisibility=hidden' ===================================== tools/signing/machines-setup/upload-tbb-to-signing-machine ===================================== @@ -37,9 +37,9 @@ if ! test -f "./out/yubihsm-shell/$yubihsm_filename"; then echo "Fetched $yubihsm_filename" fi -android_build_tools_filename=$(./rbm/rbm showconf --step get_build_tools android-toolchain filename) -if ! test -f "./out/android-toolchain/$android_build_tools_filename"; then - ./rbm/rbm build --step get_build_tools android-toolchain +android_build_tools_filename=$(./rbm/rbm showconf --step get_build_tools android-sdk filename) +if ! test -f "./out/android-sdk/$android_build_tools_filename"; then + ./rbm/rbm build --step get_build_tools android-sdk echo "Fetched $android_build_tools_filename" fi @@ -63,8 +63,8 @@ chmod go+r "./out/yubihsm-shell/$yubihsm_filename" rsync -v "./out/yubihsm-shell/$yubihsm_filename" "$setup_user@$signing_machine:$signing_dir/$yubihsm_filename" echo "Uploading $android_build_tools_filename" -chmod go+r "./out/android-toolchain/$android_build_tools_filename" -rsync -v "./out/android-toolchain/$android_build_tools_filename" "$setup_user@$signing_machine:$signing_dir/$android_build_tools_filename" +chmod go+r "./out/android-sdk/$android_build_tools_filename" +rsync -v "./out/android-sdk/$android_build_tools_filename" "$setup_user@$signing_machine:$signing_dir/$android_build_tools_filename" echo "Extracting $android_build_tools_filename" ssh "$setup_user@$signing_machine" mkdir -p $signing_dir/android-build-tools ssh "$setup_user@$signing_machine" unzip -qo -d $signing_dir/android-build-tools "$signing_dir/$android_build_tools_filename" View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/compare/6... -- View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/compare/6... You're receiving this email because of your account on gitlab.torproject.org.