boklm pushed to branch main at The Tor Project / Applications / tor-browser-build Commits: 7f96b8ad by Nicolas Vigier at 2026-03-09T20:40:48+01:00 Bug 41727: Rewrite the list_toolchain_updates_checks scripts The following changes have been done in this commit: - The various small scripts extracting informations from the firefox and application-services sources tree in the various `list_toolchain_updates_*` files have been put in the scripts `extract-firefox-versions-infos` and `extract-application-services-versions-infos`, in the directory `tools/toolchain-updates`. Some have been deduplicated (the ones extracting some information from the same file). Those two scripts output a json file containing the informations extracted from the source tree, and the corresponding rbm project, option and targets to query the current version in tor-browser-build of each of those components. - The `check-versions` scripts takes the json output from the `extract-*-versions-infos` scripts and check the current version in tor-browser-build of each of the components, and lists the one that are up-to-date, the ones that need an update and the ones that caused an error (either extracting the information from the source tree or from rbm config). - The `list-toolchain-updates` script calls the `extract-*-versions-infos` scripts from the source tree, in the `git_clones` directory after doing a checkout of the commit defined in the `list_toolchain_updates` target and saves the json outputs. It then calls the `check-versions` script on those json files. The `list-toolchain-updates` script can be run with `make list_toolchain_updates`. - The old list_toolchain_updates scripts have been removed. This rewrites provides the following improvements: - Running `make list_toolchain_updates` is much faster than before. We are now running the script to extract version information directly from `git_clones` (after a checkout of the correct commit), which avoids creating/extracting a tarball of the sources (which was the part taking the most time). - It should be much easier to modify/debug the scripts to extract version information (`extract-*-versions-infos`) since they can now be run from a source tree with no modification (this was not the case with the previous scripts since we had a single script generated from a template to extract version informations and compare them with current used version). - Once rbm#40091 is implemented (adding some easy way to update a value from rbm config), it should be easy to update the `check-versions` script to generate a script to apply the expected toolchain changes. - - - - - 20 changed files: - Makefile - doc/MAKEFILE.txt - doc/how-to-update-glean-parser.txt - projects/android-ndk/config - projects/application-services/config - − projects/application-services/list_toolchain_updates_checks - − projects/common/list_toolchain_updates - − projects/common/list_toolchain_updates-common-firefox-geckoview - projects/firefox/config - − projects/firefox/list_toolchain_updates_checks - projects/geckoview/config - − projects/geckoview/list_toolchain_updates_checks - projects/python/config - rbm.conf - − tools/list_toolchain_updates - + tools/toolchain-updates/ExtractVersionsInfos.pm - + tools/toolchain-updates/check-versions - + tools/toolchain-updates/extract-application-services-versions-infos - + tools/toolchain-updates/extract-firefox-versions-infos - + tools/toolchain-updates/list-toolchain-updates Changes: ===================================== Makefile ===================================== @@ -432,22 +432,7 @@ list_translation_updates-alpha: $(rbm) showconf --target alpha --step list_updates translation list_updates list_toolchain_updates: - tools/list_toolchain_updates - -list_toolchain_updates-firefox-linux: submodule-update - $(rbm) build firefox --step list_toolchain_updates --target nightly --target torbrowser-linux-x86_64 - -list_toolchain_updates-firefox-windows: submodule-update - $(rbm) build firefox --step list_toolchain_updates --target nightly --target torbrowser-windows-x86_64 - -list_toolchain_updates-firefox-macos: submodule-update - $(rbm) build firefox --step list_toolchain_updates --target nightly --target torbrowser-macos - -list_toolchain_updates-application-services: submodule-update - $(rbm) build application-services --step list_toolchain_updates --target nightly --target torbrowser-android-armv7 - -list_toolchain_updates-geckoview: submodule-update - $(rbm) build geckoview --step list_toolchain_updates --target nightly --target torbrowser-android-armv7 + @tools/toolchain-updates/list-toolchain-updates create_glean_deps_tarball: submodule-update $(rbm) build glean-parser --target alpha --target torbrowser-android-armv7 ===================================== doc/MAKEFILE.txt ===================================== @@ -179,10 +179,9 @@ and compare them with the checksum from sha256sums-unsigned-build.txt. * Toolchain updates * ********************* -list_toolchain_updates-{application-services,firefox-linux,firefox-macos, - firefox-windows,geckoview} -------------------------------------------------------------------------- -List toolchain updates required for building selected component/platform. +list_toolchain_updates +---------------------- +List toolchain updates required. firefox-{linux-x86_64,linux-aarch64,windows-x86_64,windows-i686, ===================================== doc/how-to-update-glean-parser.txt ===================================== @@ -8,8 +8,8 @@ There are three steps involved in the process: 1) -Running `make list_toolchain_updates-geckoview` should tell -you which version of `glean_parser` is needed. +Running `make list_toolchain_updates` should tell you which version of +`glean_parser` is needed. 2) ===================================== projects/android-ndk/config ===================================== @@ -3,7 +3,7 @@ filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.[% c("co version: 29.0.14206865 var: - release: 29 + release: r29 setup: | mkdir -p /var/tmp/dist tar -C /var/tmp/dist -xf $rootdir/[% c("compiler_tarfile") %] @@ -13,6 +13,6 @@ var: export PATH=$PATH:$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/bin input_files: - - URL: 'https://dl.google.com/android/repository/android-ndk-r[% c("var/release") %]-linux.zip' + - URL: 'https://dl.google.com/android/repository/android-ndk-[% c("var/release") %]-linux.zip' name: ndk sha256sum: 4abbbcdc842f3d4879206e9695d52709603e52dd68d3c1fff04b3b5e7a308ecf ===================================== projects/application-services/config ===================================== @@ -27,11 +27,16 @@ var: nss_sha256sum: fb5aa56fa35d963d4c65278328e2e9c99c2484c86f0e41537412477739dcf997 cargo_vendor_include_config: 1 + application-services_versions_infos: '[% exec(c("basedir") _ "/tools/toolchain-updates/extract-application-services-versions-infos") %]' + targets: nightly: git_hash: '[% c("version") %]-TORBROWSER' tag_gpg_id: 0 + list_toolchain_updates: + git_hash: 'v[% c("version") %]' + steps: build: filename: '[% project %]-[% c("version") %]-[% c("var/rebuild_date") %]-[% c("var/build_id") %].tar.[% c("compress_tar") %]' @@ -97,9 +102,3 @@ steps: filename: 'gradle-dependencies-list-[% c("version") %].txt' var: generate_gradle_dependencies_list: 1 - - list_toolchain_updates: - git_hash: 'v[% c("version") %]' - input_files: [] - container: - use_container: 0 ===================================== projects/application-services/list_toolchain_updates_checks deleted ===================================== @@ -1,64 +0,0 @@ -#!/bin/bash - -# compileSdkVersion -read -d '' p << 'EOF' || true -if (m/^\\s*compileSdkMajorVersion:\\s([^"]*),/) { - $major = $1; -} -if (m/^\\s*compileSdkMinorVersion:\\s([^"]*),/) { - $minor = $1; -} -if ($major && $minor) { - print "$major.$minor.0"; - exit 0; -} -EOF -needed=$(cat build.gradle | perl -ne "$p") -current=[% pc("android-sdk", "version") %] -check_update_needed compileSdkVersion "$needed" "$current" - - -# gradle -read -d '' p << 'EOF' || true -if (m|distributionUrl=https\\\\://services.gradle.org/distributions/gradle-(.*)-.*.zip|) { - print $1; - exit; -} -EOF -needed=$(cat gradle/wrapper/gradle-wrapper.properties | perl -ne "$p") -current='[% c("var/gradle_version") %]' -check_update_needed gradle "$needed" "$current" - - -# nss-nspr -read -d '' p << 'EOF' || true -if (m/NSS_ARCHIVE="nss-(.*-with-nspr-.*)\\.tar\\.gz"/) { - print $1; - exit; -} -EOF -needed=$(cat libs/build-all.sh | perl -ne "$p") -current='[% c("var/nss_version") %]-with-nspr-[% c("var/nspr_version") %]' -check_update_needed nss-nspr "$needed" "$current" - -# nss-sha256sum -read -d '' p << 'EOF' || true -if (m/NSS_SHA256="([^"]+)"/) { - print $1; - exit; -} -EOF -needed=$(cat libs/build-all.sh | perl -ne "$p") -current='[% c("var/nss_sha256sum") %]' -check_update_needed nss-sha256sum "$needed" "$current" - -# uniffi -p='while (<>) { - if (m/^\s*uniffi\s*=\s*\{[^}]*version\s*=\s*"([^"]*)"/) { - print $1; - exit; - } -}' -needed=$(cat components/autofill/Cargo.toml | perl -ne "$p") -current='[% pc("uniffi-rs", "version") %]' -check_update_needed uniffi "$needed" "$current" ===================================== projects/common/list_toolchain_updates deleted ===================================== @@ -1,38 +0,0 @@ -#!/bin/bash - -set -e - -rootdir="$(pwd)" -no_updates="$rootdir/no_updates.txt" -updates="$rootdir/updates.txt" - -tar -xf [% project %]-[% c('version') %].tar.[% c('compress_tar') %] -cd [% project %]-[% c('version') %] - -function check_update_needed() { - name="$1" - v1="$2" - v2="$3" - if test "$v1" = "$v2" - then - echo "* $name ($v1)" >> "$no_updates" - else - echo "* $name needs to be updated to $v1 (currently at $v2)" >> "$updates" - fi -} - -[% INCLUDE list_toolchain_updates_checks %] - -echo '### Component: [% project %] ([% c("git_hash") %])' - -if test -f "$updates" -then - echo "The following components need to be updated:" - cat "$updates" - echo -fi -if test -f "$no_updates" -then - echo "The following components don't need to be updated:" - cat "$no_updates" -fi ===================================== projects/common/list_toolchain_updates-common-firefox-geckoview deleted ===================================== @@ -1,178 +0,0 @@ -#!/bin/bash - -# rust -[% IF c("var/linux") || c("var/android"); - SET toolchain_alias='linux64-rust'; - ELSIF c("var/windows"); - SET toolchain_alias='mingw32-rust'; - ELSIF c("var/macos"); - SET toolchain_alias='linux64-rust-macos'; - END; %] -read -d '' p << 'EOF' || true -sub as_array { - ref $_[0] eq 'ARRAY' ? $_[0] : [ $_[0] ]; -} -my $d = YAML::XS::LoadFile('taskcluster/kinds/toolchain/rust.yml'); -foreach my $t (keys %$d) { - next unless ref $d->{$t}{run}{'toolchain-alias'} eq 'HASH'; - my $talias = as_array($d->{$t}{run}{'toolchain-alias'}{'by-project'}{default}); - if (grep { $_ eq '[% toolchain_alias %]' } @$talias) { - my $channel; - foreach my $arg (@{$d->{$t}{run}{arguments}}) { - if ($arg eq '--channel') { - $channel = 1; - next; - } - if ($channel) { - print $arg; - exit; - } - } - } -} -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("rust", "version") %]' -check_update_needed rust "$needed" "$current" - - -# clang -[% IF c("var/linux") || c("var/android"); - SET toolchain_alias='linux64-clang'; - ELSIF c("var/windows"); - SET toolchain_alias='linux64-clang-mingw-x64'; - ELSIF c("var/macos"); - SET toolchain_alias='macosx64-clang'; - END; %] -read -d '' p << 'EOF' || true -sub as_array { - ref $_[0] eq 'ARRAY' ? $_[0] : [ $_[0] ]; -} -my $d = YAML::XS::LoadFile('taskcluster/kinds/toolchain/clang.yml'); -my $clang_toolchain; -my $clang_fetch; -TOOLCHAIN: foreach my $t (keys %$d) { - my $aliases = as_array($d->{$t}{run}{'toolchain-alias'}{'by-project'}{default}); - foreach my $alias (@$aliases) { - if ($alias eq '[% toolchain_alias %]') { - foreach my $fetch (@{$d->{$t}{fetches}{toolchain}}) { - $clang_toolchain = $fetch if $fetch =~ m/^.*-clang-.*/; - } - foreach my $fetch (@{$d->{$t}{fetches}{fetch}}) { - $clang_fetch = $fetch if $fetch =~ m/^clang-.*/; - } - last TOOLCHAIN; - } - } -} - -if (!$clang_toolchain && !$clang_fetch) { - print STDERR "Error: could not find clang toolchain\\n"; - exit 1; -} - -if (!$clang_fetch) { - foreach my $fetch (@{$d->{$clang_toolchain}{fetches}{fetch}}) { - $clang_fetch = $fetch if $fetch =~ m/^clang-.*/; - } -} - -if (!$clang_fetch) { - print STDERR "Error: could not find clang fetch\\n"; - exit 1; -} - -my $fetch = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -print $fetch->{$clang_fetch}{fetch}{revision}; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("llvm-project", "git_hash") %]' -check_update_needed clang "$needed" "$current" - - -# cbindgen -read -d '' p << 'EOF' || true -my $fetch = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -my $revision; -T: foreach my $t (keys %$fetch) { - if ($fetch->{$t}{'fetch-alias'} eq 'cbindgen') { - $revision = $fetch->{$t}{fetch}{revision}; - last T; - } -} -exit 1 unless $revision; -print $revision; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("cbindgen", "git_hash") %]' -check_update_needed cbindgen "$needed" "$current" - - -# nasm -read -d '' p << 'EOF' || true -my $nasm = YAML::XS::LoadFile('taskcluster/kinds/toolchain/nasm.yml'); -my $linux64 = 'linux64-nasm'; -print substr $nasm->{$linux64}{'fetches'}{'fetch'}[0], 5; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("nasm", "version") %]' -check_update_needed nasm "$needed" "$current" - - -# node -read -d '' p << 'EOF' || true -sub l { - ref $_[0] eq 'ARRAY' ? $_[0] : [ $_[0] ]; -} - -my $d = YAML::XS::LoadFile('taskcluster/kinds/toolchain/node.yml'); -my $node_toolchain; -T: foreach my $t (keys %$d) { - foreach my $alias (@{l($d->{$t}{run}{'toolchain-alias'})}) { - if ($alias eq 'linux64-node') { - foreach my $fetch (@{$d->{$t}{fetches}{fetch}}) { - if ($fetch =~ m/^nodejs-.*/) { - $node_toolchain = $fetch; - last T; - } - } - } - } -} - -exit 1 unless $node_toolchain; - -my $fetch = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -my ($version) = $fetch->{$node_toolchain}{fetch}{url} =~ m|^https://nodejs.org/dist/v([^/]+)/|; -print "$version\\n"; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("node", "version") %]' -check_update_needed node "$needed" "$current" - - -# Python -read -d '' p << 'EOF' || true -if (m/^\\s*MINIMUM_MINOR_VERSION = ([0-9]+)/) { - print $1; - exit; -} -EOF -needed=3.$(cat python/mozboot/bin/bootstrap.py | perl -ne "$p") -current="3.9" # 3.11.x on Debian bookworm, 3.13.11 on our python project -check_update_needed python "$needed" "$current" - - -# binutils -read -d '' p << 'EOF' || true - -my $d = YAML::XS::LoadFile('taskcluster/kinds/toolchain/misc.yml'); -my $f = $d->{'linux64-binutils'}{fetches}{fetch}[0]; -my ($version) = $f =~ m|^binutils-([0-9\\.]+)|; -exit 1 unless $version; -print "$version\\n"; -EOF - -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("binutils", "version") %]' -check_update_needed binutils "$needed" "$current" ===================================== projects/firefox/config ===================================== @@ -63,6 +63,8 @@ var: windows_rs_version: '0.62.2' windows_rs_sha256sum: 527fadee13e0c05939a6a05d5bd6eec6cd2e3dbd648b9f8e447c6518133d8580 + firefox_versions_infos: '[% exec(c("basedir") _ "/tools/toolchain-updates/extract-firefox-versions-infos") %]' + steps: src-tarballs: filename: 'src-[% project %]-[% c("version") %].tar.xz' @@ -90,13 +92,6 @@ steps: nightly: version: '[% c("abbrev") %]' - list_toolchain_updates: - git_hash: '[% c("var/upstream_firefox_commit") %]' - tag_gpg_id: 0 - input_files: [] - container: - use_container: 0 - targets: firefoxbrowser: git_hash: '[% c("var/upstream_firefox_commit") %]' @@ -168,6 +163,10 @@ targets: - python3-zstandard - wine + list_toolchain_updates: + git_hash: '[% c("var/upstream_firefox_commit") %]' + tag_gpg_id: 0 + input_files: - project: container-image - filename: 'mozconfig' ===================================== projects/firefox/list_toolchain_updates_checks deleted ===================================== @@ -1,157 +0,0 @@ -#!/bin/bash - -[% INCLUDE 'list_toolchain_updates-common-firefox-geckoview' %] - -[% IF c("var/macos") %] -# macosx-sdk-version -read -d '' p << 'EOF' || true -sub l { - ref $_[0] eq 'ARRAY' ? $_[0] : [ $_[0] ]; -} - -my $d = YAML::XS::LoadFile('taskcluster/kinds/toolchain/macos-sdk.yml'); - -foreach my $t (keys %$d) { - foreach my $alias (@{l($d->{$t}{run}{'toolchain-alias'})}) { - if ($alias eq 'macosx64-sdk') { - if ($t =~ m/^macosx64-sdk-(.*)/) { - print "$1"; - exit 0; - } - } - } -} - -exit 1; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("macosx-toolchain", "version") %]' -check_update_needed macos-sdk-version "$needed" "$current" - -# macos-sdk-sha512sum -read -d '' p << 'EOF' || true -sub l { - ref $_[0] eq 'ARRAY' ? $_[0] : [ $_[0] ]; -} - -my $d = YAML::XS::LoadFile('taskcluster/kinds/toolchain/macos-sdk.yml'); - -foreach my $t (keys %$d) { - foreach my $alias (@{l($d->{$t}{run}{'toolchain-alias'})}) { - if ($alias eq 'macosx64-sdk') { - if ($t =~ m/^macosx64-sdk-(.*)/) { - print $d->{$t}{run}{'arguments'}[1]; - exit 0; - } - } - } -} - -exit 1; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("macosx-toolchain", "var/sdk_sha512sum") %]' -check_update_needed macos-sdk-sha512sum "$needed" "$current" - -# cctools -read -d '' p << 'EOF' || true -my $d = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -print $d->{'cctools-port'}{fetch}{revision}; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("cctools", "git_hash") %]' -check_update_needed cctools "$needed" "$current" - -# libdispatch -read -d '' p << 'EOF' || true -my $d = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -print $d->{'libdispatch'}{fetch}{revision}; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("libdispatch", "git_hash") %]' -check_update_needed libtapi "$needed" "$current" - -# libtapi -read -d '' p << 'EOF' || true -my $d = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -print $d->{'libtapi'}{fetch}{revision}; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("libtapi", "git_hash") %]' -check_update_needed libtapi "$needed" "$current" - -# End of macOS checks -[% END -%] - - -[% IF c("var/windows") %] -# mingw-w64-clang -read -d '' p << 'EOF' || true -my $d = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -print $d->{'mingw-w64'}{fetch}{revision}; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("mingw-w64-clang", "git_hash") %]' -check_update_needed mingw-w64-clang "$needed" "$current" - - -# fxc2 -read -d '' p << 'EOF' || true -my $d = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -print $d->{fxc2}{fetch}{revision}; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("fxc2", "git_hash") %]' -check_update_needed fxc2 "$needed" "$current" - -read -d '' p << 'EOF' || true -my $d = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -my $url = $d->{'windows-rs'}{fetch}{url}; -if ($url =~ m|crates/windows/(.+)/download$|) { - print $1; - exit 0; -} -print STDERR "windows-rs url not found ($url)\\n"; -exit 1 -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% c("var/windows_rs_version") %]' -check_update_needed windows-rs "$needed" "$current" - -read -d '' p << 'EOF' || true -my $d = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -my $sha256 = $d->{'windows-rs'}{fetch}{sha256}; -exit 1 unless $sha256; -print $sha256; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% c("var/windows_rs_sha256sum") %]' -check_update_needed windows-rs-sha256sum "$needed" "$current" - -# windows-app-sdk -read -d '' p << 'EOF' || true -my $d = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -my $sha256 = $d->{'windowsappsdk-redist'}{fetch}{sha256}; -exit 1 unless $sha256; -print $sha256; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("windows-app-sdk", "var/windowsappsdk_sha256sum") %]' -check_update_needed windowsappsdk_sha256sum "$needed" "$current" - -# End of Windows checks -[% END -%] - - -[% IF c("var/rlbox") %] -# wasi-sdk -read -d '' p << 'EOF' || true -my $d = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); -print $d->{'wasi-sdk'}{fetch}{revision}; -EOF -needed=$(perl -MYAML::XS -e "$p") -current='[% pc("wasi-sysroot", "git_hash") %]' -check_update_needed wasi-sdk "$needed" "$current" - -# End RLBox -[% END -%] ===================================== projects/geckoview/config ===================================== @@ -155,13 +155,6 @@ steps: name: bundletool sha256sum: '[% c("var/bundletool_hash") %]' - list_toolchain_updates: - git_hash: '[% pc("firefox", "var/upstream_firefox_commit") %]' - tag_gpg_id: 0 - input_files: [] - container: - use_container: 0 - input_files: - project: container-image - filename: 'mozconfig' ===================================== projects/geckoview/list_toolchain_updates_checks deleted ===================================== @@ -1,124 +0,0 @@ -#!/bin/bash - -[% INCLUDE 'list_toolchain_updates-common-firefox-geckoview' %] - -# ndk version -read -d '' p << 'EOF' || true -if (m/^\\s*NDK_VERSION\\s*=\\s*"(.+)"/) { - print $1; - exit; -} -EOF -needed=$(cat python/mozboot/mozboot/android.py | perl -ne "$p") -current='r[% pc("android-ndk", "var/release") %]' -check_update_needed ndk_release_name "$needed" "$current" - - -# build_tools -read -d '' p << 'EOF' || true -if (m/build-tools;(.+)$/) { - print $1; - exit; -} -EOF -needed=$(cat python/mozboot/mozboot/android-packages.txt | perl -ne "$p") -current='[% pc("android-sdk", "version") %]' -check_update_needed build_tools "$needed" "$current" - - -# cmdline-tools -read -d '' p << 'EOF' || true -my $v, $s; -while (<>) { - if (m/^\\s*CMDLINE_TOOLS_VERSION_STRING\\s*=\\s*"(.+)"/) { - $s = $1; - } - if (m/^\\s*CMDLINE_TOOLS_VERSION\\s*=\\s*"(.+)"/) { - $v = $1; - } - if ($v && $s) { - print "$s-$v"; - exit; - } -} -EOF -needed=$(cat python/mozboot/mozboot/android.py | perl -e "$p") -current='[% pc("android-sdk", "var/commandlinetools_version_string") %]-[% pc("android-sdk", "var/commandlinetools_version") %]' -check_update_needed cmdline-tools "$needed" "$current" - - -# min-android -read -d '' p << 'EOF' || true -use Path::Tiny; -my $f; -foreach (path('build/moz.configure/android-ndk.configure')->lines_utf8) { - if ($_ eq "def min_android_version():\\n") { - $f = $_; - next; - } else { - next unless $f; - } - m/return "([0-9]+)"/; - if ($1) { - print $1; - exit; - } -} -EOF -needed=$(perl -e "$p") -current='[% c("var/android_min_api") %]' -check_update_needed min-android "$needed" "$current" - - -# gradle -read -d '' p << 'EOF' || true -if (m{distributionUrl=https\\\\://services.gradle.org/distributions/gradle-(.*)-(bin|all).zip}) { - print $1; - exit; -} -EOF -needed=$(cat gradle/wrapper/gradle-wrapper.properties | perl -ne "$p") -current='[% c("var/gradle_version") %]' -check_update_needed gradle "$needed" "$current" - - -# bundletool -read -d '' p << 'EOF' || true -if (m{^BUNDLETOOL_VERSION\\s*=\\s*"(.+)"}) { - print $1; - exit; -} -EOF -needed=$(cat python/mozboot/mozboot/android.py | perl -ne "$p") -current='[% c("var/bundletool_version") %]' -check_update_needed bundletool "$needed" "$current" - - -# glean version -read -d '' p << 'EOF' || true -if (m{^glean\\s*=\\s*"(.+)"}) { - print $1; - exit; -} -EOF -needed=$(cat gradle/libs.versions.toml | perl -ne "$p") -current='[% pc("glean", "version") %]' -check_update_needed glean "$needed" "$current" - - -# glean-parser version -read -d '' p << 'EOF' || true -if (m{^glean_parser-(.+)\.dist-info}) { - print $1; - exit; -} -EOF -needed=$(ls third_party/python/glean_parser | perl -ne "$p") -current='[% pc("glean-parser", "version") %]' -check_update_needed glean-parser "$needed" "$current" - -# terser - -needed=$(jq -r '.dependencies.terser' tools/terser/package.json) -current='[% pc("terser", "version") %]' -check_update_needed terser "$needed" "$current" ===================================== projects/python/config ===================================== @@ -19,6 +19,17 @@ var: tar -C /var/tmp/dist -xf $rootdir/[% c("python_tarfile") %] export PATH=/var/tmp/dist/python/bin:"$PATH" + # This variable is used by tools/toolchain-updates/extract-firefox-versions-infos + # to notice when the minimum python version required by firefox is + # changing, so that we can review if we need to update python. + # + # Currently this custom python build is used on Linux only. + # + # Debian includes the following python versions: + # bookworm (used for Android): 3.11.2 + # trixie (windows, macos): 3.13.5 + firefox_minimum_python_version: 3.9 + input_files: - project: container-image - name: python ===================================== rbm.conf ===================================== @@ -14,10 +14,6 @@ container: build: 1 steps: - list_toolchain_updates: - build_log: '-' - list_toolchain_updates: '[% INCLUDE list_toolchain_updates %]' - cargo_vendor: filename: '[% project %]-vendor-[% c("version") %]-[% c("var/build_id") %].tar.[% c("compress_tar") %]' output_dir: "out/[% project %]/cargo_vendor" ===================================== tools/list_toolchain_updates deleted ===================================== @@ -1,35 +0,0 @@ -#!/bin/bash -# -# This script runs all `make list_toolchain_updates-*` commands, save -# their outputs to a file (tmp/toolchain_updates.txt) and print it -# (again) at the end. -# -set -e - -script_dir=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) -cd "$script_dir"/.. - -output_file="$script_dir/../tmp/toolchain_updates.txt" - -rm -f "$output_file" - -function list_toolchain_updates { - for cmd in \ - list_toolchain_updates-firefox-linux \ - list_toolchain_updates-firefox-windows \ - list_toolchain_updates-firefox-macos \ - list_toolchain_updates-application-services \ - list_toolchain_updates-geckoview - do - line_title="** $cmd **" - line_size=$(echo "$line_title" | wc -m) - line_line=$(perl -e 'print "*" x ('$line_size' - 1), "\n"') - echo "$line_line"; echo "$line_title"; echo "$line_line"; echo - make -s $cmd - echo; echo; echo "$line_line"; echo "$line_line"; echo; echo - done -} - -list_toolchain_updates | tee -a "$output_file" - -echo "See also tmp/toolchain_updates.txt" ===================================== tools/toolchain-updates/ExtractVersionsInfos.pm ===================================== @@ -0,0 +1,41 @@ +#!/usr/bin/perl -w +package ExtractVersionsInfos; + +use Exporter; +our @ISA = qw/Exporter/; +our @EXPORT = qw/*version_infos *rbm_os_target set_version_info + get_version_info set_rbm_info set_error as_array/; + +our %version_infos; + +our %rbm_os_target = ( + linux => 'torbrowser-linux-x86_64', + windows => 'torbrowser-windows-x86_64', + macos => 'torbrowser-macos-aarch64', + android => 'torbrowser-android-aarch64', +); + +sub set_version_info { + my ($name, $value) = @_; + $version_infos{$name}->{expected_value} = $value; +} + +sub get_version_info { + return $version_infos{$_[0]}->{expected_value}; +} + +sub set_rbm_info { + my ($name, $rbm_info) = @_; + $version_infos{$name}->{rbm_info} = $rbm_info; +} + +sub set_error { + $version_infos{$_[0]}->{error} = 1; +} + +sub as_array { + return [] unless defined $_[0]; + return ref $_[0] eq 'ARRAY' ? $_[0] : [ $_[0] ]; +} + +1; ===================================== tools/toolchain-updates/check-versions ===================================== @@ -0,0 +1,117 @@ +#!/usr/bin/perl -w +# +# This script takes one or more json file as command line arguments, as +# created by the `extract-*-versions-infos` files. For each component +# defined in the json this script checks the expected value and compares +# it with the current value in the rbm config. It then lists all +# components that are up-to-date, components that need an update, and +# components where there was an error (error when extracting the value +# from the sources tree, or getting the value from the rbm config). + +use strict; + +use English; +use FindBin; +use YAML::XS qw(LoadFile); +use JSON qw/decode_json/; +use Path::Tiny; +use Capture::Tiny qw(capture); + +my @base_targets = qw/nightly/; +my @default_targets = qw/torbrowser-linux-x86_64/; +my $rbm = "$FindBin::Bin/../../rbm/rbm"; + +sub exit_error { + print STDERR "Error: ", $_[0], "\n"; + exit (exists $_[1] ? $_[1] : 1); +} + +sub capture_exec { + my @cmd = @_; + my ($stdout, $stderr, $exit) = capture { + system(@cmd); + }; + return ($stdout, $stderr, $exit == 0, $exit) if wantarray(); + return $stdout; +} + +if (!@ARGV) { + exit_error "usage: $0 <json-file...>"; +} + +my %version_infos; +foreach my $input_file (@ARGV) { + my $infos = decode_json path($input_file)->slurp_utf8; + exit_error "Error reading $input_file" unless $infos; + %version_infos = ( %version_infos, %$infos ); +} + +my %component_need_update; +my %component_ok; +my %component_error_extract; +my %component_error_rbm; + +foreach my $component (keys %version_infos) { + print "Checking $component\n"; + my $rbm_info = $version_infos{$component}{rbm_info}; + my @targets = ( @base_targets, + $rbm_info->{targets} ? @{$rbm_info->{targets}} : @default_targets ); + @targets = map { ('--target', $_ ) } @targets; + my @cmd = ($rbm, 'showconf', $rbm_info->{project}, $rbm_info->{option}, + @targets); + if ($version_infos{$component}{error}) { + $component_error_extract{$component} = 1; + next; + } + my ($stdout, $stderr, $success) = capture_exec(@cmd); + if (!$success) { + $component_error_rbm{$component} = $stderr; + next; + } + chomp $stdout; + if ($stdout eq $version_infos{$component}{expected_value}) { + $component_ok{$component} = $stdout; + next; + } + $component_need_update{$component}{expected_value} = + $version_infos{$component}->{expected_value}; + $component_need_update{$component}{current_value} = $stdout; +} + +if (!%component_need_update && !%component_error_extract && !%component_error_rbm) { + print "Everything is up-to-date\n"; + exit 0; +} + +if (%component_ok) { + print "The following components are up-to-date:\n"; + foreach my $component (sort keys %component_ok) { + print "- $component ($component_ok{$component})\n"; + } + print "\n"; +} + +if (%component_error_extract) { + print "There was an error extracting the following components info from source tree:\n"; + foreach my $component (sort keys %component_error_extract) { + print "- $component\n"; + } + print "\n"; +} + +if (%component_error_rbm) { + print "There was an error extracting the following components from rbm config:\n"; + foreach my $component (sort keys %component_error_rbm) { + print "- $component\n"; + } + print "\n"; +} + +if (%component_need_update) { + print "The following components need an update:\n"; + foreach my $component (sort keys %component_need_update) { + print "- $component\n"; + print " current: $component_need_update{$component}{current_value}\n"; + print " expected: $component_need_update{$component}{expected_value}\n"; + } +} ===================================== tools/toolchain-updates/extract-application-services-versions-infos ===================================== @@ -0,0 +1,119 @@ +#!/usr/bin/perl -w +# +# This script extract version informations from an application-services +# tree. It will output on stdout a json file containing version +# information found in the firefox tree. The json file also include +# information about which rbm project, option and targets can be used +# to query (and later update) the current value in `tor-browser-build` +# (this part is done by the `check-versions` script). +# +# This script should be run from the root of an application-services +# tree, without argument. + +use strict; +use English; +use YAML::XS qw(LoadFile); +use Path::Tiny; +use JSON qw/decode_json/; +use FindBin; +use lib $FindBin::Bin; +use ExtractVersionsInfos; + +## compileSdkVersion +sub get_compilesdk_info { + my $component = 'appservices/android-sdk'; + set_rbm_info($component, { + project => 'android-sdk', + option => 'version', + }); + my @lines = path('build.gradle')->lines_utf8; + my ($major, $minor); + foreach my $line (@lines) { + if ($line =~ m/^\s*MINIMUM_MINOR_VERSION = ([0-9]+)/) { + set_version_info('python', "3.$1"); + return; + } + if ($line =~ m/^\s*compileSdkMajorVersion:\s([^"]*),/) { + $major = $1; + } elsif ($line =~ m/^\s*compileSdkMinorVersion:\s([^"]*),/) { + $minor = $1; + } + if ($major && $minor) { + set_version_info($component, "$major.$minor.0"); + return; + } + } + set_error($component); +} + +## gradle +sub get_gradle_info { + my $component = 'appservices/gradle'; + set_rbm_info($component, { + project => 'application-services', + option => 'var/gradle_version', + }); + my @lines = path('gradle/wrapper/gradle-wrapper.properties')->lines_utf8; + foreach my $line (@lines) { + if ($line =~ m|distributionUrl=https\\://services.gradle.org/distributions/gradle-(.*)-.*.zip|) { + set_version_info($component, $1); + return; + } + } + set_error($component); +} + +# nss-nspr +sub get_nss_info { + my %component_option = ( + nss => 'var/nss_version', + nspr => 'var/nspr_version', + 'nss-sha256sum' => 'var/nss_sha256sum', + ); + foreach my $component (keys %component_option) { + set_rbm_info($component, { + project => 'application-services', + option => $component_option{$component}, + }); + } + my ($nss, $sha256sum); + my @lines = path('libs/build-all.sh')->lines_utf8; + foreach my $line (@lines) { + if ($line =~ m/NSS_ARCHIVE="nss-(.*)-with-nspr-(.*)\.tar\.gz"/) { + set_version_info('nss', $1); + set_version_info('nspr', $2); + $nss = 1; + } + if ($line =~ m/NSS_SHA256="([^"]+)"/) { + set_version_info('nss-sha256sum', $1); + $sha256sum = 1; + } + return if ($nss && $sha256sum); + } + set_error('nss') unless $nss; + set_error('nspr') unless $nss; + set_error('nss-sha256sum') unless $sha256sum; +} + +## uniffi +sub get_uniffi_info { + my $component = 'uniffi'; + set_rbm_info($component, { + project => 'uniffi-rs', + option => 'version', + }); + my @lines = path('components/autofill/Cargo.toml')->lines_utf8; + foreach my $line (@lines) { + if ($line =~ m/^\s*uniffi\s*=\s*\{[^}]*version\s*=\s*"([^"]*)"/) { + set_version_info($component, $1); + return; + } + } + set_error($component); +} + +get_compilesdk_info; +get_gradle_info; +get_nss_info; +get_uniffi_info; +print JSON->new->utf8->canonical->pretty->encode(\%version_infos); ===================================== tools/toolchain-updates/extract-firefox-versions-infos ===================================== @@ -0,0 +1,481 @@ +#!/usr/bin/perl -w +# +# This script extracts version informations from a firefox tree. It will +# output on stdout a json file containing version information found in +# the firefox tree. The json file also include information about which +# rbm project, option and targets can be used to query (and later update) +# the current value in `tor-browser-build` (this part is done by the +# `check-versions` script). +# +# This script should be run from the root of a firefox tree, without +# argument. +# +# +# == How to add extraction of a new version information == +# +# - Copy the `get_glean_info` function (as an example of basic extraction +# function) and rename it with the name of the new component +# - Add the new function to the list of functions being called, at the +# end of the file +# - Update `$component` with the name of the component +# - Update the `set_rbm_info` call with the rbm project, option and +# target (optional) used query the current version in `tor-browser-build`. +# - Update the file path in the `my @lines = ...` line to the path in +# the source tree where the information can be found +# - Update the regexp to match the line containing the information we +# want to extract. The part between parenthesis will be stored as `$1`, +# which we can use in the `set_version_info` call to save the +# information we extracted. If the information was not found, +# `set_error` is used to register an error. +# +# Alternatively it should be possible to write an additional script to +# extract some informations from the sources, and output them in the +# same json format as this script. + +use strict; +use English; +use YAML::XS qw(LoadFile); +use Path::Tiny; +use JSON qw/decode_json/; +use FindBin; +use lib $FindBin::Bin; +use ExtractVersionsInfos; + +# taskcluster/kinds/toolchain/rust.yml +# +## Rust ## +sub get_rust_info { + my $infos = YAML::XS::LoadFile('taskcluster/kinds/toolchain/rust.yml'); + my %toolchain_aliases = ( + linux => 'linux64-rust', + windows => 'mingw32-rust', + macos => 'linux64-rust-macos', + ); + foreach my $os (keys %toolchain_aliases) { + TOOLCHAIN_ALIAS: foreach my $t (keys %$infos) { + next unless ref $infos->{$t}{run}{'toolchain-alias'} eq 'HASH'; + my $talias = as_array($infos->{$t}{run}{'toolchain-alias'}{'by-project'}{default}); + if (grep { $_ eq $toolchain_aliases{$os} } @$talias) { + my $channel; + foreach my $arg (@{$infos->{$t}{run}{arguments}}) { + if ($arg eq '--channel') { + $channel = 1; + next; + } + if ($channel) { + set_version_info("rust-$os", $arg); + last TOOLCHAIN_ALIAS; + } + } + } + } + + set_error("rust-$os") unless get_version_info("rust-$os"); + set_rbm_info("rust-$os", { + project => 'rust', + option => 'version', + targets => [ $rbm_os_target{$os} ], + }); + } +} + + +# taskcluster/kinds/toolchain/clang.yml +# +## clang ## +sub get_clang_info { + my %toolchain_aliases = ( + linux => 'linux64-clang', + windows => 'linux64-clang-mingw-x64', + macos => 'macosx64-clang', + ); + my $infos = YAML::XS::LoadFile('taskcluster/kinds/toolchain/clang.yml'); + OS: foreach my $os (keys %toolchain_aliases) { + my $clang_toolchain; + my $clang_fetch; + TOOLCHAIN: foreach my $t (keys %$infos) { + next unless ref $infos->{$t}{run}{'toolchain-alias'} eq 'HASH'; + next unless $infos->{$t}{run}{'toolchain-alias'}{'by-project'}{default}; + my $aliases = as_array($infos->{$t}{run}{'toolchain-alias'}{'by-project'}{default}); + foreach my $alias (@$aliases) { + if ($alias eq $toolchain_aliases{$os}) { + foreach my $fetch (@{$infos->{$t}{fetches}{toolchain}}) { + $clang_toolchain = $fetch if $fetch =~ m/^.*-clang-.*/; + } + foreach my $fetch (@{$infos->{$t}{fetches}{fetch}}) { + $clang_fetch = $fetch if $fetch =~ m/^clang-.*/; + } + last TOOLCHAIN; + } + } + } + + if (!$clang_toolchain && !$clang_fetch) { + print STDERR "Error: could not find clang toolchain for $toolchain_aliases{$os}\n"; + set_error("clang-$os"); + next OS; + } + + if (!$clang_fetch) { + foreach my $fetch (@{$infos->{$clang_toolchain}{fetches}{fetch}}) { + $clang_fetch = $fetch if $fetch =~ m/^clang-.*/; + } + } + + if (!$clang_fetch) { + print STDERR "Error: could not find clang fetch for $toolchain_aliases{$os}\n"; + set_error("clang-$os"); + next OS; + } + + my $fetch = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); + set_version_info("clang-$os", $fetch->{$clang_fetch}{fetch}{revision}); + + set_rbm_info("clang-$os", { + project => 'llvm-project', + option => 'git_hash', + targets => [ $rbm_os_target{$os} ], + }); + } +} + +## cbindgen +sub get_cbindgen_info { + my $fetch = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); + my $revision; + T: foreach my $t (keys %$fetch) { + next unless $fetch->{$t}{'fetch-alias'}; + if ($fetch->{$t}{'fetch-alias'} eq 'cbindgen') { + $revision = $fetch->{$t}{fetch}{revision}; + last T; + } + } + return 0 unless $revision; + set_version_info('cbindgen', $revision); + set_rbm_info('cbindgen', { + project => 'cbindgen', + option => 'git_hash', + }); +} + +## nasm +sub get_nasm_info { + my $nasm = YAML::XS::LoadFile('taskcluster/kinds/toolchain/nasm.yml'); + my $fetch = $nasm->{'linux64-nasm'}{'fetches'}{'fetch'}[0]; + return set_error('nasm') unless $fetch; + set_version_info('nasm', substr $fetch, 5); + set_rbm_info('nasm', { + project => 'nasm', + option => 'version', + }); +} + +## node +sub get_node_info { + my $d = YAML::XS::LoadFile('taskcluster/kinds/toolchain/node.yml'); + my $node_toolchain; + T: foreach my $t (keys %$d) { + foreach my $alias (@{as_array($d->{$t}{run}{'toolchain-alias'})}) { + if ($alias eq 'linux64-node') { + foreach my $fetch (@{$d->{$t}{fetches}{fetch}}) { + if ($fetch =~ m/^nodejs-.*/) { + $node_toolchain = $fetch; + last T; + } + } + } + } + } + + return set_error('node') unless $node_toolchain; + + my $fetch = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); + my ($version) = $fetch->{$node_toolchain}{fetch}{url} =~ m|^https://nodejs.org/dist/v([^/]+)/|; + return set_error('node') unless $version; + set_version_info('node', $version); + set_rbm_info('node', { + project => 'node', + option => 'version', + }); +} + +## Python +sub get_python_info { + my $component = 'minimum_python'; + set_rbm_info($component, { + project => 'python', + option => 'var/firefox_minimum_python_version', + }); + my @lines = path('python/mozboot/bin/bootstrap.py')->lines_utf8; + foreach my $line (@lines) { + if ($line =~ m/^\s*MINIMUM_MINOR_VERSION = ([0-9]+)/) { + set_version_info($component, "3.$1"); + return; + } + } + set_error($component); +} + +## binutils +sub get_binutils_info { + my $component = 'binutils'; + set_rbm_info($component, { + project => 'binutils', + option => 'version', + }); + my $d = YAML::XS::LoadFile('taskcluster/kinds/toolchain/misc.yml'); + my $f = $d->{'linux64-binutils'}{fetches}{fetch}[0]; + my ($version) = $f =~ m|^binutils-([0-9\\.]+)|; + return set_error($component) unless $version; + set_version_info($component, $version); +} + +## macosx-sdk +sub get_macosx_sdk_info { + set_rbm_info('macosx-sdk-version', { + project => 'macosx-toolchain', + option => 'version', + targets => [ $rbm_os_target{macos} ], + }); + set_rbm_info('macosx-sdk-sha512sum', { + project => 'macosx-toolchain', + option => 'var/sdk_sha512sum', + targets => [ $rbm_os_target{macos} ], + }); + my $d = YAML::XS::LoadFile('taskcluster/kinds/toolchain/macos-sdk.yml'); + + foreach my $t (keys %$d) { + foreach my $alias (@{as_array($d->{$t}{run}{'toolchain-alias'})}) { + if ($alias eq 'macosx64-sdk') { + if ($t =~ m/^macosx64-sdk-(.*)/) { + set_version_info('macosx-sdk-version', $1); + set_version_info('macosx-sdk-sha512sum', $d->{$t}{run}{'arguments'}[1]); + return; + } + } + } + } + + set_error('macosx-sdk-version'); + set_error('macosx-sdk-sha512sum'); +} + +## taskcluster/kinds/fetch/toolchains.yml +sub get_toolchains_fetch_info { + my @projects = qw/cctools libdispatch libtapi mingw-w64-clang fxc2 wasi-sysroot/; + # projects which have a different name in toolchains.yml + my %toolchain_names = ( + cctools => 'cctools-port', + 'wasi-sysroot' => 'wasi-sdk', + 'mingw-w64-clang' => 'mingw-w64', + ); + # projects which have an different rbm option than git_hash + my %rbm_option = ( + 'windows-app-sdk' => 'var/windowsappsdk_sha256sum', + ); + my $d = YAML::XS::LoadFile('taskcluster/kinds/fetch/toolchains.yml'); + foreach my $name (@projects) { + set_rbm_info($name, + { + project => $name, + option => ($rbm_option{$name} ? $rbm_option{$name} : 'git_hash'), + }); + my $toolchain_name = $toolchain_names{$name} ? $toolchain_names{$name} : $name; + my $revision = $d->{$toolchain_name}{fetch}{revision}; + if ($revision) { + set_version_info($name, $revision); + } else { + set_error($name); + } + } + + set_rbm_info('windows-rs', { + project => 'firefox', + option => 'var/windows_rs_version', + }); + my $winrs_url = $d->{'windows-rs'}{fetch}{url}; + if ($winrs_url =~ m|crates/windows/(.+)/download$|) { + set_version_info('windows-rs', $1); + } else { + set_error('windows-rs'); + } + + set_rbm_info('windows-rs-sha256sum', { + project => 'firefox', + option => 'var/windows_rs_sha256sum', + }); + my $winrs_sha256 = $d->{'windows-rs'}{fetch}{sha256}; + if ($winrs_sha256) { + set_version_info('windows-rs-sha256sum', $winrs_sha256); + } else { + set_error('windows-rs-sha256sum'); + } +} + +## Extract infos from python/mozboot/mozboot/android.py +sub get_mozboot_android_info { + my %component_project = ( + ndk => 'android-ndk', + commandlinetools_version_string => 'android-sdk', + commandlinetools_version => 'android-sdk', + bundletool => 'geckoview', + ); + my %component_option = ( + ndk => 'var/release', + commandlinetools_version_string => 'var/commandlinetools_version_string', + commandlinetools_version => 'var/commandlinetools_version', + bundletool => 'var/bundletool_version', + ); + my %component_NAME = ( + ndk => 'NDK_VERSION', + commandlinetools_version_string => 'CMDLINE_TOOLS_VERSION_STRING', + commandlinetools_version => 'CMDLINE_TOOLS_VERSION', + bundletool => 'BUNDLETOOL_VERSION', + ); + foreach my $component (keys %component_project) { + set_rbm_info($component, { + project => $component_project{$component}, + option => $component_option{$component}, + targets => [ $rbm_os_target{android} ], + }); + } + my %versions; + my @lines = path('python/mozboot/mozboot/android.py')->lines_utf8; + foreach my $line (@lines) { + foreach my $component (keys %component_NAME) { + if ($line =~ m/^\s*$component_NAME{$component}\s*=\s*"(.+)"/) { + $versions{$component} = $1; + } + } + } + foreach my $component (keys %component_NAME) { + if ($versions{$component}) { + set_version_info($component, $versions{$component}); + } else { + set_error($component); + } + } +} + +sub get_build_tools_info { + my $component = 'build_tools'; + set_rbm_info($component, { + project => 'android-sdk', + option => 'version', + targets => [ $rbm_os_target{android} ], + }); + my @lines = path('python/mozboot/mozboot/android-packages.txt')->lines_utf8; + foreach my $line (@lines) { + if ($line =~ m/build-tools;(.+)$/) { + set_version_info($component, $1); + return; + } + } + set_error($component); +} + +sub get_min_android_info { + my $component = 'min-android'; + set_rbm_info($component, { + project => 'geckoview', + option => 'var/android_min_api', + targets => [ $rbm_os_target{android} ], + }); + my @lines = path('build/moz.configure/android-ndk.configure')->lines_utf8; + my $f; + foreach my $line (@lines) { + if ($line eq "def min_android_version():\n") { + $f = $line; + next; + } + next unless $f; + if ($line =~ m/return "([0-9]+)"/) { + set_version_info($component, $1); + return; + } + } + set_error($component); +} + +sub get_gradle_info { + my $component = 'geckoview/gradle'; + set_rbm_info($component, { + project => 'geckoview', + option => 'var/gradle_version', + targets => [ $rbm_os_target{android} ], + }); + my @lines = path('gradle/wrapper/gradle-wrapper.properties')->lines_utf8; + foreach my $line (@lines) { + if ($line =~ m{distributionUrl=https\\://services.gradle.org/distributions/gradle-(.*)-(bin|all).zip}) { + set_version_info($component, $1); + return; + } + } + set_error($component); +} + +sub get_glean_info { + my $component = 'glean'; + set_rbm_info($component, { + project => 'glean', + option => 'version', + targets => [ $rbm_os_target{android} ], + }); + my @lines = path('gradle/libs.versions.toml')->lines_utf8; + foreach my $line (@lines) { + if ($line =~ m{^glean\s*=\s*"(.+)"}) { + set_version_info($component, $1); + return; + } + } + set_error($component); +} + +sub get_glean_parser_info { + my $component = 'glean-parser'; + set_rbm_info($component, { + project => 'glean-parser', + option => 'version', + targets => [ $rbm_os_target{android} ], + }); + my $files = path('third_party/python/glean_parser'); + foreach my $file ($files->children) { + if ($file->basename =~ m{^glean_parser-(.+)\.dist-info}) { + set_version_info($component, $1); + return; + } + } + set_error($component); +} + +sub get_terser_info { + my $component = 'terser'; + set_rbm_info($component, { + project => 'terser', + option => 'version', + targets => [ $rbm_os_target{android} ], + }); + my $version_infos = decode_json path('tools/terser/package.json')->slurp_utf8; + if ($version_infos->{dependencies}{terser}) { + set_version_info($component, $version_infos->{dependencies}{terser}); + } else { + set_error($component); + } +} + +get_rust_info; +get_clang_info; +get_cbindgen_info; +get_nasm_info; +get_node_info; +get_python_info; +get_macosx_sdk_info; +get_toolchains_fetch_info; +get_mozboot_android_info; +get_build_tools_info; +get_min_android_info; +get_gradle_info; +get_glean_info; +get_glean_parser_info; +get_terser_info; +print JSON->new->utf8->canonical->pretty->encode(\%version_infos); ===================================== tools/toolchain-updates/list-toolchain-updates ===================================== @@ -0,0 +1,26 @@ +#!/bin/bash +# +# Run this script to check which toolchain updates are needed. +# +# It can be run with `make list_toolchain_updates`. + +set -e + +script_dir=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) +tbb_dir="$script_dir/../.." +tmpdir=$(mktemp -d) +trap "rm -Rf $tmpdir" EXIT + +firefox_infos="$tmpdir/firefox-infos.json" +appservices_infos="$tmpdir/application-services-infos.json" + +echo "Extracting firefox versions infos" +"$tbb_dir"/rbm/rbm showconf --target list_toolchain_updates --target torbrowser-linux-x86_64 \ + firefox var/firefox_versions_infos > "$firefox_infos" + +echo "Extracting application-services versions infos" +"$tbb_dir"/rbm/rbm showconf --target list_toolchain_updates --target torbrowser-android-aarch64 \ + application-services var/application-services_versions_infos > "$appservices_infos" + +echo "Checking needed updates" +"$script_dir"/check-versions "$firefox_infos" "$appservices_infos" View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/commit/7f... -- View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/commit/7f... You're receiving this email because of your account on gitlab.torproject.org.