Pier Angelo Vendrame pushed to branch main at The Tor Project / Applications / tor-browser-build Commits: 4be76d68 by Beatriz Rizental at 2025-12-09T17:27:56+01:00 Bug 41485: Add a script to retrieve gradle dependencies. - - - - - 5bbb5bef by Pier Angelo Vendrame at 2025-12-09T17:29:02+01:00 Bug 41485: Automate the Gradle deps update (Glean). - - - - - 0e05b1e9 by Pier Angelo Vendrame at 2025-12-09T17:29:24+01:00 Bug 41485: Automate the Gradle deps update (Application Services). - - - - - 09f06adb by Pier Angelo Vendrame at 2025-12-09T17:29:42+01:00 Bug 41485: Automate the Gradle deps update (GeckoView). - - - - - 40f3459b by Pier Angelo Vendrame at 2025-12-09T17:35:18+01:00 Bug 41485: Documentation for the Gradle deps process. - - - - - 12 changed files: - − doc/how-to-create-gradle-dependencies-list.txt - + doc/how-to-update-gradle-dependencies-list.md - projects/application-services/build - projects/application-services/config - + projects/common/gen-gradle-deps-file.py - − projects/common/gen_gradle_deps_file.sh - projects/geckoview/build - projects/geckoview/build_ac_fenix - projects/geckoview/build_common - projects/geckoview/config - projects/glean/build - projects/glean/config Changes: ===================================== doc/how-to-create-gradle-dependencies-list.txt deleted ===================================== @@ -1,48 +0,0 @@ -### Updating Gradle Dependencies -If additional Android dependencies are required by the project's build, then -the Gradle build will fail due to missing dependencies. To find out what the -missing dependencies are take the following steps. - -1) Updating gradle dependencies for `application-services` - -The following makefile rules can be used: - - make get_gradle_dependencies_list-application-services - -which should create the gradle-dependencies-list.txt files in the -corresponding out/$project directory, using nightly's branch. - - -2) Updating gradle dependencies for `geckoview` - -The `geckoview` project doesn't have (yet) a makefile rule to generate the -gradle-dependencies-list.txt file, so a few more steps are needed. - -For the geckoview project, comment out the following line in the project's -build_common file: - - export GRADLE_MAVEN_REPOSITORIES="file://$gradle_repo","file://$gradle_repo/maven2" - -Also modify the gradle flags to include the debug option so the download logs will show up: - - export GRADLE_FLAGS="--no-daemon --debug" - -Otherwise, when calling gradle in the project's build script replace the -`--offline` flag with `--debug` and remove any `-Dmaven.repo.local` arguments. - -Finally, allow network access during the build by setting -`container/disable_network/build` to `0` in rbm.conf, and rerun the build. - -Dependent artifacts will show up as downloads in the logs. You can pull out -these dependencies into a list by passing the log file to the gradle dependency -list script in the projects/common directory: - -`projects/common/gen_gradle_deps_file.sh /path/to/log/file` - - -3) Copying the resulting `gradle-dependencies-list.txt` - -Copy the resulting `gradle-dependencies-list.txt` (from step 1. or 2.) -over the one in the respective project. Then, in the project's config -file, increment the `var/gradle_dependencies_version` and make sure to -restore the project's build file back to original if you modified it. ===================================== doc/how-to-update-gradle-dependencies-list.md ===================================== @@ -0,0 +1,99 @@ +# Gradle Dependencies + +## TL; DR + +1. Open `projects/<project name>/config`. +2. Look for `# generate_gradle_dependencies_list: 1` and uncomment it. +3. Run `rbm/rbm build <project name> --target nightly --target torbrowser-android-<arch>`. + For GeckoView, make sure to run a single-arch build to catch all dependencies + at once. + Testbuilds (i.e., `make torbrowser-testbuild-android-<arch>`) are single-arch + by default, but you can also set `var/android_single_arch`. +4. Find the corresponding output: `ls -lt out/<project name> | head`. +5. Extract it. +6. Move the `gradle-dependencies-list.txt` you just extracted to + `projects/<project name>/`. +7. Bump `var/gradle_dependencies_version`. + The preferred format is `<project version>-<list version>`. + +Theoretically, it should be also possible to set +`generate_gradle_dependencies_list: 1` in `rbm.local.conf`, run a full build and +update all dependencies at once. + +## Rationale + +Many of our Android projects use Gradle to build, which also handles +dependencies. + +For reproducibility purposes, we need to stabilize them, so we version a list of +dependencies that are known for working in a file called +`gradle-dependencies-list.txt`. + +This file has a simple format: + +``` +# Lines starting with '#' are comments +<sha265> | <URL> +``` + +In addition to guaranteeing reproducibility, this makes sure we do not get +poisoned dependencies in subsequent builds. + +## Creating and/or updating the list + +Gradle does not have official tooling to create such lists. + +Instead, we pass `--info` to make sure Gradle logs information about downloads. +Therefore, these logs need to be captured to a file. Most commonly, Gradle is +invoked directly and in that case appending something like +`2>&1 | tee -a gradle-output.log` to the command line is enough. + +After that, we parse the logs with the `projects/common/gen-gradle-deps-file.py` +script: + +```shell +gen-gradle-deps-file.py gradle-output.log +``` + +This generates a new `gradle-dependencies-list.txt`. + +This procedure can be run outside a tor-browser-build container, but for best compatibility, +it should run inside it. + +Most of our Android projects have been equipped with a flag +(`var/generate_gradle_dependencies_list`) to do this automatically. + +Therefore, it is possible to just enable this flag, and grab an updated file +from the output artifact. + +New Android projects should be setup to do this. + +> **Important**: dependencies are keyed and cached. So, after updating the list, +> you must also bump `var/gradle_dependencies_version`. + +This used to be an integer, but to avoid clashes between the main and +maintenance branches, we decided to switch to a +`<project version>-<list version>` format. +Usually, the list version will be `1`. + +## Consuming the list + +`projects/common/fetch-gradle-dependencies` is the consumer of the list. + +It ignores the hash of the `.pom` files, as they might change, but it checks the +hashes of all the other artifacts. + +You can wire it in your project with code that looks like this: + +```yaml +- filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]' + name: gradle-dependencies + exec: '[% INCLUDE "fetch-gradle-dependencies" %]' + enable: '[% !c("var/generate_gradle_dependencies_list") %]' +``` + +Then, you will have to patch your project to look for offline dependencies, too. +But this is project-specific. + +Generally speaking, you will have to add a `mavenLocal()` entry to the +`repositories` section in some `build.gradle` file. ===================================== projects/application-services/build ===================================== @@ -24,7 +24,7 @@ export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64 [% INCLUDE 'fake-git' %] -[% IF !c('var/fetch_gradle_dependencies') %] +[% IF !c('var/generate_gradle_dependencies_list') -%] tar -xf [% c('input_files_by_name/glean') %] gradle_repo=/var/tmp/dist/gradle-dependencies @@ -33,7 +33,7 @@ export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64 cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true cp -rl $gradle_repo/m2/* $gradle_repo || true cp -rl $gradle_repo/maven2/* $gradle_repo || true -[% END %] +[% END -%] tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.[% c('compress_tar') %] @@ -41,24 +41,26 @@ mv /var/tmp/build/[% project %]-[% c('version') %] $builddir cd $builddir -# Prepare the offline build. +[% IF !c('var/generate_gradle_dependencies_list') -%] + # Prepare the offline build. -tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %] -export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels -# Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will still -# try to install Glean with pip. -cat > $rootdir/pip.conf << 'EOF' + tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %] + export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels + # Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will + # still try to install Glean with pip. + cat > $rootdir/pip.conf << 'EOF' [global] find-links = /var/tmp/build/application-services/glean-wheels index-url = no-index = yes EOF -export PIP_CONFIG_FILE=$rootdir/pip.conf -# Move the directory for hardcoding the path in .cargo/config. -tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %] -# Make sure our vendored crates are used for offline builds. -cat vendor/cargo-config.toml >> .cargo/config.toml -rm vendor/cargo-config.toml + export PIP_CONFIG_FILE=$rootdir/pip.conf + # Move the directory for hardcoding the path in .cargo/config. + tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %] + # Make sure our vendored crates are used for offline builds. + cat vendor/cargo-config.toml >> .cargo/config.toml + rm vendor/cargo-config.toml +[% END -%] tar -xf $rootdir/[% c('input_files_by_name/uniffi-rs') %] @@ -68,30 +70,33 @@ ln -s $rootdir/[% c("input_files_by_name/nss") %] ./ ./build-all.sh android popd -[% IF c('var/fetch_gradle_dependencies') %] - # XXX: `assemble` is still not enough to see all fetched dependencies via - # Gradle's --debug. See: tor-browser-build#40056. - gradle --debug --no-daemon assemble > gradle.log 2>&1 || ( cat gradle.log; exit 1 ) - $rootdir/gen_gradle_deps_file.sh gradle.log - mv gradle-dependencies-list.txt '[% dest_dir _ "/" _ c("filename") %]' -[% ELSE %] - # We need to set `LC_ALL` and `LANG` to something that is not ASCII as encoding - # otherwise `click` barfs. See: https://click.palletsprojects.com/python3/ - export LC_ALL=C.UTF-8 - export LANG=C.UTF-8 - gradle_flags="--offline --no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies" - gradle $gradle_flags assembleRelease - gradle $gradle_flags publish - mv build/maven $distdir - - pushd components/support/nimbus-fml - cargo build --release - popd - cp target/release/nimbus-fml $distdir - - cd /var/tmp/dist - [% c('tar', { - tar_src => [ project ], - tar_args => '-caf ' _ dest_dir _ '/' _ c('filename'), +# We need to set `LC_ALL` and `LANG` to something that is not ASCII as encoding +# otherwise `click` barfs. See: https://click.palletsprojects.com/python3/ +export LC_ALL=C.UTF-8 +export LANG=C.UTF-8 +[% IF !c('var/generate_gradle_dependencies_list') -%] + deps_flags="--offline -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies" +[% ELSE -%] + deps_flags="--info" + gradle_logs=/var/tmp/build/gradle-logs.log +[% END -%] +gradle_flags="$deps_flags --no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen" +gradle $gradle_flags assembleRelease [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %] +gradle $gradle_flags publish +mv build/maven $distdir + +pushd components/support/nimbus-fml +cargo build --release +popd +cp target/release/nimbus-fml $distdir + +[% IF c('var/generate_gradle_dependencies_list') -%] + cd $distdir + $rootdir/gen-gradle-deps-file.py $gradle_logs +[% END -%] + +cd /var/tmp/dist +[% c('tar', { + tar_src => [ project ], + tar_args => '-caf ' _ dest_dir _ '/' _ c('filename'), }) %] -[% END %] ===================================== projects/application-services/config ===================================== @@ -14,6 +14,8 @@ gpg_keyring: git_submodule: 1 container: use_container: 1 + disable_network: + build: '[% !c("var/generate_gradle_dependencies_list") %]' var: build_number: 1 @@ -23,6 +25,9 @@ var: nss_version: '3.117' nspr_version: '4.37' cargo_vendor_include_config: 1 + # Uncomment this to run an online build to grab an updated + # gradle-dependencies-list.txt. + # generate_gradle_dependencies_list: 1 targets: nightly: @@ -61,7 +66,7 @@ steps: - project: glean name: glean pkg_type: build - enable: '[% !c("var/fetch_gradle_dependencies") %]' + enable: '[% !c("var/generate_gradle_dependencies_list") %]' # Only Application Services currently requires build tools 35. # So, download them only here, rather than adding them to the shared # toolchain. @@ -75,28 +80,22 @@ steps: - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]' name: gradle-dependencies exec: '[% INCLUDE "fetch-gradle-dependencies" %]' - enable: '[% !c("var/fetch_gradle_dependencies") %]' + enable: '[% !c("var/generate_gradle_dependencies_list") %]' - URL: '[% pc("glean-parser", "var/glean_wheels_url") %]' name: glean-wheels sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]' - enable: '[% !c("var/fetch_gradle_dependencies") %]' + enable: '[% !c("var/generate_gradle_dependencies_list") %]' - name: cargo_vendor project: application-services pkg_type: cargo_vendor norec: sha256sum: 8a93a189985ad28c334970dc5ab386c81d4a415f3777c2941248f836f4af44fd - enable: '[% !c("var/fetch_gradle_dependencies") %]' - - filename: gen_gradle_deps_file.sh - enable: '[% c("var/fetch_gradle_dependencies") %]' + enable: '[% !c("var/generate_gradle_dependencies_list") %]' + - filename: gen-gradle-deps-file.py + enable: '[% c("var/generate_gradle_dependencies_list") %]' list_toolchain_updates: git_hash: 'v[% c("version") %]' input_files: [] container: use_container: 0 - - get_gradle_dependencies_list: - filename: 'gradle-dependencies-list-[% c("version") %].txt' - get_gradle_dependencies_list: '[% INCLUDE build %]' - var: - fetch_gradle_dependencies: 1 ===================================== projects/common/gen-gradle-deps-file.py ===================================== @@ -0,0 +1,127 @@ +#!/usr/bin/env python3 + +import argparse +import hashlib +import re +import sys +import urllib.request +from concurrent.futures import ThreadPoolExecutor, as_completed + + +def download_and_hash(url): + try: + sha256_hash = hashlib.sha256() + with urllib.request.urlopen(url) as f: + sha256_hash.update(f.read()) + return url, sha256_hash.hexdigest(), None + except Exception as e: + return url, None, str(e) + + +def parse_arguments(): + parser = argparse.ArgumentParser( + description=""" +This script generates a `gradle-dependencies-list.txt` file as expected by +Tor Browser Build to download dependencies for offline builds. + +It parses a log file from a Gradle build, determines which dependencies were +downloaded and from where, re-downloads them, and calculates their SHA256 hash. +Downloads are parallelized — by default, 4 workers are used, but this can be +configured via the optional `--workers` argument. + +--- + +Usage: + +First, run the exact Gradle command that will be used inside the offline +tor-browser-build container. Make sure to set the log level to `info` and +capture the full output to a log file. + +`--refresh-dependencies` can be pass to Gradle to force all dependencies to be +re-downloaded if needed (e.g., if running this script outside containers or +outside tor-browser-build). + +Example: + + ./gradlew --info clean assembleRelease 2>&1 | tee -a out.log + +Then run this script with the generated log: + + ./gen-gradle-deps-file.py out.log + +This will create a `gradle-dependencies-list.txt` file in the current +directory. +""", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + + parser.add_argument( + "log_file", help="Path to the Gradle log file (e.g., out.log)" + ) + parser.add_argument( + "--workers", + type=int, + default=4, + help="Number of parallel download workers (default: 4).", + ) + parser.add_argument( + "--output", + default="gradle-dependencies-list.txt", + help="Output file path (default: gradle-dependencies-list.txt).", + ) + + return parser.parse_args() + + +def extract_urls(log_file): + pattern = re.compile(rf"Downloading (https://\S+) to ") + urls = [] + with open(log_file, "r") as file: + for line in file: + match = pattern.search(line.strip()) + if match: + urls.append(match.group(1)) + return urls + + +def main(): + args = parse_arguments() + + urls = extract_urls(args.log_file) + if not urls: + print("⚠️ No matching download lines found in the log file.") + return + + print( + f"🔍 Found {len(urls)} URLs. Starting downloads using {args.workers} workers...\n" + ) + + results = [] + with ThreadPoolExecutor(max_workers=args.workers) as executor: + futures = { + executor.submit(download_and_hash, url): url for url in urls + } + for future in as_completed(futures): + url, sha256, error = future.result() + if error: + print(f"[ERROR] {url}\n Reason: {error}") + else: + print(f"[OK] {url}\n SHA256: {sha256}") + results.append((sha256, url)) + + if results: + with open(args.output, "w") as out_file: + out_file.write( + "# Don't forget to update var/gradle_dependencies_version when modifying this file\n" + ) + out_file.write("sha256sum | url\n") + for sha256, url in sorted(results, key=lambda x: x[1]): + out_file.write(f"{sha256} | {url}\n") + print(f"\n✅ {len(urls)} dependencies written to: {args.output}") + else: + print("\n⚠️ No successful downloads to write.") + sys.exit(1) + + +if __name__ == "__main__": + main() ===================================== projects/common/gen_gradle_deps_file.sh deleted ===================================== @@ -1,93 +0,0 @@ -#!/bin/bash - -# Copyright (c) 2020, The Tor Project, Inc. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are -# met: -# -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# -# * Redistributions in binary form must reproduce the above -# copyright notice, this list of conditions and the following disclaimer -# in the documentation and/or other materials provided with the -# distribution. -# -# * Neither the names of the copyright owners nor the names of its -# contributors may be used to endorse or promote products derived from -# this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -# Usage: -# 1) Point to a log file with all the dependency download attempts (for its -# generation see: doc/how-to-create-gradle-dependencies-list.txt) -# 2) Double-check that you get the same SHA-256 sums when downloaded from a -# different network location. E.g. by using `torsocks` with this script after -# having made a copy of `gradle-dependencies-list.txt` from 1) and comparing -# the two .txt files. - -export LC_ALL=C -log="$1" - -# Step 1: Extract all the download attempts out of the log file, ignore the ones -# for maven-metadata.xml and module files. We don't need those. -cat $log | grep "Performing HTTP" | grep -o "https://.*" | \ - grep -vE "\.module|maven-metadata\.xml" | sort | uniq > dl-attempts - -# Step 2: Fetch all the dependencies and calculate the SHA-256 sum -declare -A URLs -function dl_url { - local url="$1" - test -n "${URLs[$url]}" && return 0 - URLs[$url]=1 - wget -U "" $url - test $? -eq 0 || return 1 - local fn=$(basename "$url") - local sha256=`sha256sum $fn | cut -d ' ' -f 1` - echo "$sha256 | $url" >> deps - rm $fn -} - -while read line -do - dl_url "$line" - # If we downloaded a .pom file, also try to get corresponding .jar and - # .aar files - bname=$(basename "$line" .pom) - dname=$(dirname "$line") - if test "$line" != "$dname/$bname" - then - dl_url "$dname/$bname.aar" - dl_url "$dname/$bname.jar" - fi -done < dl-attempts - -# Step 3: Add the header at the beginning of the final dependency file. -echo "# On how to update dependencies see doc/how-to-create-gradle\ --dependencies-list.txt" > gradle-dependencies-list.txt -echo "# Don't forget to update var/gradle_dependencies_version when modifying \ -this file" >> gradle-dependencies-list.txt -echo "sha256sum | url" >> gradle-dependencies-list.txt - -# Step 4: Keep only successfully downloaded artifacts, remove duplicates, and -# sort based on download URL. -grep ^[a-f0-9] deps | rev | sort -t/ -u -k1,4 | rev | \ - sort -k 3 >> gradle-dependencies-list.txt - -# Step 5: Clean up -rm dl-attempts -rm deps - -exit 0 ===================================== projects/geckoview/build ===================================== @@ -23,7 +23,7 @@ echo "Starting ./mach configure $(date)" [% IF !c("var/rlbox") -%]--without-wasm-sandboxed-libraries[% END %] echo "Starting ./mach build $(date)" -./mach build --verbose +./mach build --verbose [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %] [% IF c("var/dev_artifacts") -%] echo "Building development artifacts" @@ -50,6 +50,11 @@ echo "Build finished, copying the AAR to the to the destination directory $(date [% IF !c("var/android_single_arch") -%] mkdir -p "$outdir/[% project %]" cp obj-*/gradle/target.maven.zip $outdir/[% project %]/ + + [% IF c('var/generate_gradle_dependencies_list') -%] + cd $outdir/[% project %] + $rootdir/gen-gradle-deps-file.py $gradle_logs + [% END -%] [% ELSE -%] [% INCLUDE 'build_ac_fenix' %] [% END -%] ===================================== projects/geckoview/build_ac_fenix ===================================== @@ -8,7 +8,7 @@ objdir=$(cd obj-* && pwd) echo "Building Android Components $(date)" pushd mobile/android/android-components -gradle $GRADLE_FLAGS assembleGecko -x lint +gradle $GRADLE_FLAGS assembleGecko -x lint [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %] popd # The build might fail with "file exists" otherwise. @@ -36,7 +36,7 @@ variant='[% c("var/variant") %]' version_name="[% c('var/torbrowser_version') %] ([% c('var/geckoview_version') %])" echo "Building $variant Fenix APK" -gradle $GRADLE_FLAGS -PversionName="$version_name" "assemble$variant" +gradle $GRADLE_FLAGS -PversionName="$version_name" "assemble$variant" [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %] echo "Build finished, copying the APK(s) to the destination directory $(date)" mkdir -p $outdir/[% project %] @@ -46,7 +46,7 @@ echo "Building non optimized $variant Fenix APK for testing" gradle $GRADLE_FLAGS -PversionName="$version_name" -PdisableOptimization "assemble$variant" echo "Building Fenix instrumentation tests" -gradle $GRADLE_FLAGS -PversionName="$version_name" -PtestBuildType="$variant" -PdisableOptimization assembleAndroidTest +gradle $GRADLE_FLAGS -PversionName="$version_name" -PtestBuildType="$variant" -PdisableOptimization assembleAndroidTest [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %] echo "Test build finished, copying the APKs to the destination directory $(date)" @@ -55,3 +55,8 @@ mv $objdir/gradle/build/mobile/android/fenix/app/outputs/apk/fenix/$variant/*.ap mv $objdir/gradle/build/mobile/android/fenix/app/outputs/apk/androidTest/fenix/$variant/*.apk $outdir/[% project %]/tests popd + +[% IF c('var/generate_gradle_dependencies_list') -%] + cd $outdir/[% project %] + $rootdir/gen-gradle-deps-file.py $gradle_logs +[% END -%] ===================================== projects/geckoview/build_common ===================================== @@ -14,8 +14,6 @@ export PATH="/var/tmp/dist/node/bin:$PATH" export LC_ALL=C.UTF-8 export LANG=C.UTF-8 -tar -xf [% c('input_files_by_name/glean') %] - export TOR_EXPERT_BUNDLE_AAR=$rootdir/[% c('input_files_by_name/tor-expert-bundle-aar') %]/tor-expert-bundle.aar tar -C /var/tmp/dist -xf [% c('input_files_by_name/application-services') %] @@ -29,25 +27,34 @@ tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.[% c('compress_ta [% c("var/set_MOZ_BUILD_DATE") %] export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64 -gradle_repo=/var/tmp/dist/gradle-dependencies -export GRADLE_MAVEN_REPOSITORIES="file://$gradle_repo","file://$gradle_repo/maven2" -# Set the Maven local repository because Gradle ignores our overriding of $HOME. -# It is only used for the local pubblication of single-arch AARs. -export GRADLE_FLAGS="--no-daemon --offline -Dmaven.repo.local=$distdir/[% project %]" -# Move the Gradle repo to a hard-coded location. The location is embedded in -# the file chrome/toolkit/content/global/buildconfig.html so it needs to be -# standardized for reproducibility. -mv $rootdir/[% c('input_files_by_name/gradle-dependencies') %] $gradle_repo -cp -rl $rootdir/glean/maven/* $gradle_repo -cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true -cp -rl $gradle_repo/m2/* $gradle_repo || true -cp -rl $gradle_repo/maven2/* $gradle_repo || true - -tar -xf [% c('input_files_by_name/glean-wheels') %] -export GLEAN_PYTHON_WHEELS_DIR=$rootdir/glean-wheels - -tar -xf [% c('input_files_by_name/oss-licenses-plugin') %] -cp -rl oss-licenses-plugin/. $gradle_repo + +[% IF !c('var/generate_gradle_dependencies_list') -%] + gradle_repo=/var/tmp/dist/gradle-dependencies + export GRADLE_MAVEN_REPOSITORIES="file://$gradle_repo","file://$gradle_repo/maven2" + # Set the Maven local repository because Gradle ignores our overriding of $HOME. + # It is only used for the local pubblication of single-arch AARs. + export GRADLE_FLAGS="--no-daemon --offline -Dmaven.repo.local=$distdir/[% project %]" + # Move the Gradle repo to a hard-coded location. The location is embedded in + # the file chrome/toolkit/content/global/buildconfig.html so it needs to be + # standardized for reproducibility. + mv $rootdir/[% c('input_files_by_name/gradle-dependencies') %] $gradle_repo + + tar -xf [% c('input_files_by_name/glean') %] + cp -rl $rootdir/glean/maven/* $gradle_repo + + cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true + cp -rl $gradle_repo/m2/* $gradle_repo || true + cp -rl $gradle_repo/maven2/* $gradle_repo || true + + tar -xf [% c('input_files_by_name/glean-wheels') %] + export GLEAN_PYTHON_WHEELS_DIR=$rootdir/glean-wheels + + tar -xf [% c('input_files_by_name/oss-licenses-plugin') %] + cp -rl oss-licenses-plugin/. $gradle_repo +[% ELSE -%] + gradle_logs=/var/tmp/build/gradle-logs.log + export GRADLE_FLAGS="--no-daemon --info" +[% END -%] # We unbreak mach, see: https://bugzilla.mozilla.org/show_bug.cgi?id=1656993. export MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=system ===================================== projects/geckoview/config ===================================== @@ -48,6 +48,9 @@ var: has_l10n: '[% !c("var/testbuild") && !c("var/locales").empty %]' l10n-changesets: '[% exec("git --no-pager show " _ c("git_hash") _ ":browser/locales/l10n-changesets.json", { exec_noco => 1 }) %]' mozconfig_file: 'mozconfig-[% c("var/osname") %]' + # Uncomment this to run an online build to grab an updated + # gradle-dependencies-list.txt. + # generate_gradle_dependencies_list: 1 targets: release: @@ -87,9 +90,11 @@ steps: - project: glean name: glean pkg_type: build + enable: '[% !c("var/generate_gradle_dependencies_list") %]' - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]' name: gradle-dependencies exec: '[% INCLUDE "fetch-gradle-dependencies" %]' + enable: '[% !c("var/generate_gradle_dependencies_list") %]' - name: geckoview_armv7 project: geckoview pkg_type: build @@ -123,15 +128,19 @@ steps: - URL: '[% pc("glean-parser", "var/glean_wheels_url") %]' name: glean-wheels sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]' + enable: '[% !c("var/generate_gradle_dependencies_list") %]' - name: tor-expert-bundle-aar project: tor-expert-bundle-aar pkg_type: build - project: oss-licenses-plugin name: oss-licenses-plugin pkg_type: build + enable: '[% !c("var/generate_gradle_dependencies_list") %]' - URL: 'https://github.com/google/bundletool/releases/download/[% c("var/bundletool_version") %]/bundletool-all-[% c("var/bundletool_version") %].jar' name: bundletool sha256sum: '[% c("var/bundletool_hash") %]' + - filename: gen-gradle-deps-file.py + enable: '[% c("var/generate_gradle_dependencies_list") %]' list_toolchain_updates: git_hash: FIREFOX_NIGHTLY_146_END @@ -167,14 +176,15 @@ input_files: - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]' name: gradle-dependencies exec: '[% INCLUDE "fetch-gradle-dependencies" %]' + enable: '[% !c("var/generate_gradle_dependencies_list") %]' - URL: '[% pc("glean-parser", "var/glean_wheels_url") %]' name: glean-wheels sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]' - project: glean name: glean - pkg_type: build - project: oss-licenses-plugin name: oss-licenses-plugin + enable: '[% !c("var/generate_gradle_dependencies_list") %]' - name: tor-expert-bundle-aar project: tor-expert-bundle-aar - project: firefox-l10n @@ -192,3 +202,5 @@ input_files: - URL: 'https://github.com/google/bundletool/releases/download/[% c("var/bundletool_version") %]/bundletool-all-[% c("var/bundletool_version") %].jar' name: bundletool sha256sum: '[% c("var/bundletool_hash") %]' + - filename: gen-gradle-deps-file.py + enable: '[% c("var/generate_gradle_dependencies_list") %]' ===================================== projects/glean/build ===================================== @@ -28,7 +28,7 @@ export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64 [% INCLUDE 'fake-git' %] -[% IF !c('var/fetch_gradle_dependencies') %] +[% IF !c('var/generate_gradle_dependencies_list') -%] gradle_repo=/var/tmp/dist/gradle-dependencies mv $rootdir/[% c('input_files_by_name/gradle-dependencies') %] $gradle_repo cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true @@ -42,23 +42,24 @@ mv /var/tmp/build/[% project %]-[% c('version') %] $builddir cd $builddir -# Prepare the offline build. - -tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %] -export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels -# Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will still -# try to install Glean with pip. +[% IF !c('var/generate_gradle_dependencies_list') -%] + # Prepare the offline build. + tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %] + export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels + # Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will still + # try to install Glean with pip. cat > $rootdir/pip.conf << 'EOF' [global] find-links = /var/tmp/build/glean/glean-wheels index-url = no-index = yes EOF -export PIP_CONFIG_FILE=$rootdir/pip.conf -# Move the directory for hardcoding the path in .cargo/config. -tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %] -# Make sure our vendored crates are used for offline builds. -cat >> .cargo/config.toml << 'EOF' + + export PIP_CONFIG_FILE=$rootdir/pip.conf + # Move the directory for hardcoding the path in .cargo/config. + tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %] + # Make sure our vendored crates are used for offline builds. + cat >> .cargo/config.toml << 'EOF' [source.crates-io] replace-with = "vendored-sources" @@ -70,16 +71,27 @@ directory = "/var/tmp/build/glean/vendor" offline=true EOF + deps_flags="--offline -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies" +[% ELSE -%] + deps_flags="--info" + gradle_logs=/var/tmp/build/gradle-logs.log +[% END -%] + tar -xf $rootdir/[% c('input_files_by_name/uniffi-rs') %] patch -p1 < $rootdir/use-custom-gradle-repo.diff patch -p1 < $rootdir/use-uniffi-noop.diff -gradle_flags="--offline --no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies" -gradle $gradle_flags :glean:assembleRelease +gradle_flags="--no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen $deps_flags" +gradle $gradle_flags :glean:assembleRelease [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %] gradle $gradle_flags publish mv build/maven $distdir +[% IF c('var/generate_gradle_dependencies_list') -%] + cd $distdir + $rootdir/gen-gradle-deps-file.py $gradle_logs +[% END -%] + cd /var/tmp/dist [% c('tar', { tar_src => [ project ], ===================================== projects/glean/config ===================================== @@ -6,11 +6,16 @@ filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.[% c("co version: 66.1.1 container: use_container: 1 + disable_network: + build: '[% !c("var/generate_gradle_dependencies_list") %]' var: # This should be updated when the list of gradle dependencies is changed. gradle_dependencies_version: 66.1.1-1 gradle_version: 8.14.3 + # Uncomment this to run an online build to grab an updated + # gradle-dependencies-list.txt. + # generate_gradle_dependencies_list: 1 steps: build: @@ -36,6 +41,7 @@ steps: - URL: '[% pc("glean-parser", "var/glean_wheels_url") %]' name: glean-wheels sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]' + enable: '[% !c("var/generate_gradle_dependencies_list") %]' - name: cargo_vendor project: glean pkg_type: cargo_vendor @@ -44,5 +50,8 @@ steps: - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]' name: gradle-dependencies exec: '[% INCLUDE "fetch-gradle-dependencies" %]' + enable: '[% !c("var/generate_gradle_dependencies_list") %]' + - filename: gen-gradle-deps-file.py + enable: '[% c("var/generate_gradle_dependencies_list") %]' - filename: use-custom-gradle-repo.diff - filename: use-uniffi-noop.diff View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/compare/c... -- View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/compare/c... You're receiving this email because of your account on gitlab.torproject.org.