Pier Angelo Vendrame pushed to branch main at The Tor Project / Applications / tor-browser-build

Commits:

12 changed files:

Changes:

  • doc/how-to-create-gradle-dependencies-list.txt deleted
    1 
    -### Updating Gradle Dependencies
    2 
    -If additional Android dependencies are required by the project's build, then
    3 
    -the Gradle build will fail due to missing dependencies. To find out what the
    4 
    -missing dependencies are take the following steps.
    5 
    -
    6 
    -1) Updating gradle dependencies for `application-services`
    7 
    -
    8 
    -The following makefile rules can be used:
    9 
    -
    10 
    -  make get_gradle_dependencies_list-application-services
    11 
    -
    12 
    -which should create the gradle-dependencies-list.txt files in the
    13 
    -corresponding out/$project directory, using nightly's branch.
    14 
    -
    15 
    -
    16 
    -2) Updating gradle dependencies for `geckoview`
    17 
    -
    18 
    -The `geckoview` project doesn't have (yet) a makefile rule to generate the
    19 
    -gradle-dependencies-list.txt file, so a few more steps are needed.
    20 
    -
    21 
    -For the geckoview project, comment out the following line in the project's
    22 
    -build_common file:
    23 
    -
    24 
    -   export GRADLE_MAVEN_REPOSITORIES="file://$gradle_repo","file://$gradle_repo/maven2"
    25 
    -
    26 
    -Also modify the gradle flags to include the debug option so the download logs will show up:
    27 
    -
    28 
    -   export GRADLE_FLAGS="--no-daemon --debug"
    29 
    -
    30 
    -Otherwise, when calling gradle in the project's build script replace the
    31 
    -`--offline` flag with `--debug` and remove any `-Dmaven.repo.local` arguments.
    32 
    -
    33 
    -Finally, allow network access during the build by setting
    34 
    -`container/disable_network/build` to `0` in rbm.conf, and rerun the build.
    35 
    -
    36 
    -Dependent artifacts will show up as downloads in the logs. You can pull out
    37 
    -these dependencies into a list by passing the log file to the gradle dependency
    38 
    -list script in the projects/common directory:
    39 
    -
    40 
    -`projects/common/gen_gradle_deps_file.sh /path/to/log/file`
    41 
    -
    42 
    -
    43 
    -3) Copying the resulting `gradle-dependencies-list.txt`
    44 
    -
    45 
    -Copy the resulting `gradle-dependencies-list.txt` (from step 1. or 2.)
    46 
    -over the one in the respective project. Then, in the project's config
    47 
    -file, increment the `var/gradle_dependencies_version` and make sure to
    48 
    -restore the project's build file back to original if you modified it.

  • doc/how-to-update-gradle-dependencies-list.md
    1 
    +# Gradle Dependencies
    2 
    +
    3 
    +## TL; DR
    4 
    +
    5 
    +1. Open `projects/<project name>/config`.
    6 
    +2. Look for `# generate_gradle_dependencies_list: 1` and uncomment it.
    7 
    +3. Run `rbm/rbm build <project name> --target nightly --target torbrowser-android-<arch>`.
    8 
    +   For GeckoView, make sure to run a single-arch build to catch all dependencies
    9 
    +   at once.
    10 
    +   Testbuilds (i.e., `make torbrowser-testbuild-android-<arch>`) are single-arch
    11 
    +   by default, but you can also set `var/android_single_arch`.
    12 
    +4. Find the corresponding output: `ls -lt out/<project name> | head`.
    13 
    +5. Extract it.
    14 
    +6. Move the `gradle-dependencies-list.txt` you just extracted to
    15 
    +   `projects/<project name>/`.
    16 
    +7. Bump `var/gradle_dependencies_version`.
    17 
    +   The preferred format is `<project version>-<list version>`.
    18 
    +
    19 
    +Theoretically, it should be also possible to set
    20 
    +`generate_gradle_dependencies_list: 1` in `rbm.local.conf`, run a full build and
    21 
    +update all dependencies at once.
    22 
    +
    23 
    +## Rationale
    24 
    +
    25 
    +Many of our Android projects use Gradle to build, which also handles
    26 
    +dependencies.
    27 
    +
    28 
    +For reproducibility purposes, we need to stabilize them, so we version a list of
    29 
    +dependencies that are known for working in a file called
    30 
    +`gradle-dependencies-list.txt`.
    31 
    +
    32 
    +This file has a simple format:
    33 
    +
    34 
    +```
    35 
    +# Lines starting with '#' are comments
    36 
    +<sha265> | <URL>
    37 
    +```
    38 
    +
    39 
    +In addition to guaranteeing reproducibility, this makes sure we do not get
    40 
    +poisoned dependencies in subsequent builds.
    41 
    +
    42 
    +## Creating and/or updating the list
    43 
    +
    44 
    +Gradle does not have official tooling to create such lists.
    45 
    +
    46 
    +Instead, we pass `--info` to make sure Gradle logs information about downloads.
    47 
    +Therefore, these logs need to be captured to a file. Most commonly, Gradle is
    48 
    +invoked directly and in that case appending something like
    49 
    +`2>&1 | tee -a gradle-output.log` to the command line is enough.
    50 
    +
    51 
    +After that, we parse the logs with the `projects/common/gen-gradle-deps-file.py`
    52 
    +script:
    53 
    +
    54 
    +```shell
    55 
    +gen-gradle-deps-file.py gradle-output.log
    56 
    +```
    57 
    +
    58 
    +This generates a new `gradle-dependencies-list.txt`.
    59 
    +
    60 
    +This procedure can be run outside a tor-browser-build container, but for best compatibility,
    61 
    +it should run inside it.
    62 
    +
    63 
    +Most of our Android projects have been equipped with a flag
    64 
    +(`var/generate_gradle_dependencies_list`) to do this automatically.
    65 
    +
    66 
    +Therefore, it is possible to just enable this flag, and grab an updated file
    67 
    +from the output artifact.
    68 
    +
    69 
    +New Android projects should be setup to do this.
    70 
    +
    71 
    +> **Important**: dependencies are keyed and cached. So, after updating the list,
    72 
    +> you must also bump `var/gradle_dependencies_version`.
    73 
    +
    74 
    +This used to be an integer, but to avoid clashes between the main and
    75 
    +maintenance branches, we decided to switch to a
    76 
    +`<project version>-<list version>` format.
    77 
    +Usually, the list version will be `1`.
    78 
    +
    79 
    +## Consuming the list
    80 
    +
    81 
    +`projects/common/fetch-gradle-dependencies` is the consumer of the list.
    82 
    +
    83 
    +It ignores the hash of the `.pom` files, as they might change, but it checks the
    84 
    +hashes of all the other artifacts.
    85 
    +
    86 
    +You can wire it in your project with code that looks like this:
    87 
    +
    88 
    +```yaml
    89 
    +- filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'
    90 
    +  name: gradle-dependencies
    91 
    +  exec: '[% INCLUDE "fetch-gradle-dependencies" %]'
    92 
    +  enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    93 
    +```
    94 
    +
    95 
    +Then, you will have to patch your project to look for offline dependencies, too.
    96 
    +But this is project-specific.
    97 
    +
    98 
    +Generally speaking, you will have to add a `mavenLocal()` entry to the
    99 
    +`repositories` section in some `build.gradle` file.

  • projects/application-services/build
    ... ... @@ -24,7 +24,7 @@ export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64
    24 24
    25 25 
     [% INCLUDE 'fake-git' %]
    26 26
    27 
    -[% IF !c('var/fetch_gradle_dependencies') %]
    27 
    +[% IF !c('var/generate_gradle_dependencies_list') -%]
    28 28 
       tar -xf [% c('input_files_by_name/glean') %]
    29 29
    30 30 
       gradle_repo=/var/tmp/dist/gradle-dependencies
    ... ... @@ -33,7 +33,7 @@ export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64
    33 33 
       cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true
    34 34 
       cp -rl $gradle_repo/m2/* $gradle_repo || true
    35 35 
       cp -rl $gradle_repo/maven2/* $gradle_repo || true
    36 
    -[% END %]
    36 
    +[% END -%]
    37 37
    38 38 
     tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.[% c('compress_tar') %]
    39 39
    ... ... @@ -41,24 +41,26 @@ mv /var/tmp/build/[% project %]-[% c('version') %] $builddir
    41 41
    42 42 
     cd $builddir
    43 43
    44 
    -# Prepare the offline build.
    44 
    +[% IF !c('var/generate_gradle_dependencies_list') -%]
    45 
    +  # Prepare the offline build.
    45 46
    46 
    -tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %]
    47 
    -export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels
    48 
    -# Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will still
    49 
    -# try to install Glean with pip.
    50 
    -cat > $rootdir/pip.conf << 'EOF'
    47 
    +  tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %]
    48 
    +  export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels
    49 
    +  # Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will
    50 
    +  # still try to install Glean with pip.
    51 
    +  cat > $rootdir/pip.conf << 'EOF'
    51 52 
     [global]
    52 53 
     find-links = /var/tmp/build/application-services/glean-wheels
    53 54 
     index-url =
    54 55 
     no-index = yes
    55 56 
     EOF
    56 
    -export PIP_CONFIG_FILE=$rootdir/pip.conf
    57 
    -# Move the directory for hardcoding the path in .cargo/config.
    58 
    -tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %]
    59 
    -# Make sure our vendored crates are used for offline builds.
    60 
    -cat vendor/cargo-config.toml >> .cargo/config.toml
    61 
    -rm vendor/cargo-config.toml
    57 
    +  export PIP_CONFIG_FILE=$rootdir/pip.conf
    58 
    +  # Move the directory for hardcoding the path in .cargo/config.
    59 
    +  tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %]
    60 
    +  # Make sure our vendored crates are used for offline builds.
    61 
    +  cat vendor/cargo-config.toml >> .cargo/config.toml
    62 
    +  rm vendor/cargo-config.toml
    63 
    +[% END -%]
    62 64
    63 65 
     tar -xf $rootdir/[% c('input_files_by_name/uniffi-rs') %]
    64 66
    ... ... @@ -68,30 +70,33 @@ ln -s $rootdir/[% c("input_files_by_name/nss") %] ./
    68 70 
     ./build-all.sh android
    69 71 
     popd
    70 72
    71 
    -[% IF c('var/fetch_gradle_dependencies') %]
    72 
    -  # XXX: `assemble` is still not enough to see all fetched dependencies via
    73 
    -  # Gradle's --debug. See: tor-browser-build#40056.
    74 
    -  gradle --debug --no-daemon assemble > gradle.log 2>&1 || ( cat gradle.log; exit 1 )
    75 
    -  $rootdir/gen_gradle_deps_file.sh gradle.log
    76 
    -  mv gradle-dependencies-list.txt '[% dest_dir _ "/" _ c("filename") %]'
    77 
    -[% ELSE %]
    78 
    -  # We need to set `LC_ALL` and `LANG` to something that is not ASCII as encoding
    79 
    -  # otherwise `click` barfs. See: https://click.palletsprojects.com/python3/
    80 
    -  export LC_ALL=C.UTF-8
    81 
    -  export LANG=C.UTF-8
    82 
    -  gradle_flags="--offline --no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies"
    83 
    -  gradle $gradle_flags assembleRelease
    84 
    -  gradle $gradle_flags publish
    85 
    -  mv build/maven $distdir
    86 
    -
    87 
    -  pushd components/support/nimbus-fml
    88 
    -  cargo build --release
    89 
    -  popd
    90 
    -  cp target/release/nimbus-fml $distdir
    91 
    -
    92 
    -  cd /var/tmp/dist
    93 
    -  [% c('tar', {
    94 
    -      tar_src => [ project ],
    95 
    -      tar_args => '-caf ' _ dest_dir _ '/' _ c('filename'),
    73 
    +# We need to set `LC_ALL` and `LANG` to something that is not ASCII as encoding
    74 
    +# otherwise `click` barfs. See: https://click.palletsprojects.com/python3/
    75 
    +export LC_ALL=C.UTF-8
    76 
    +export LANG=C.UTF-8
    77 
    +[% IF !c('var/generate_gradle_dependencies_list') -%]
    78 
    +  deps_flags="--offline -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies"
    79 
    +[% ELSE -%]
    80 
    +  deps_flags="--info"
    81 
    +  gradle_logs=/var/tmp/build/gradle-logs.log
    82 
    +[% END -%]
    83 
    +gradle_flags="$deps_flags --no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen"
    84 
    +gradle $gradle_flags assembleRelease [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
    85 
    +gradle $gradle_flags publish
    86 
    +mv build/maven $distdir
    87 
    +
    88 
    +pushd components/support/nimbus-fml
    89 
    +cargo build --release
    90 
    +popd
    91 
    +cp target/release/nimbus-fml $distdir
    92 
    +
    93 
    +[% IF c('var/generate_gradle_dependencies_list') -%]
    94 
    +  cd $distdir
    95 
    +  $rootdir/gen-gradle-deps-file.py $gradle_logs
    96 
    +[% END -%]
    97 
    +
    98 
    +cd /var/tmp/dist
    99 
    +[% c('tar', {
    100 
    +    tar_src => [ project ],
    101 
    +    tar_args => '-caf ' _ dest_dir _ '/' _ c('filename'),
    96 102 
       }) %]
    97 
    -[% END %]

  • projects/application-services/config
    ... ... @@ -14,6 +14,8 @@ gpg_keyring:
    14 14 
     git_submodule: 1
    15 15 
     container:
    16 16 
       use_container: 1
    17 
    +  disable_network:
    18 
    +    build: '[% !c("var/generate_gradle_dependencies_list") %]'
    17 19
    18 20 
     var:
    19 21 
       build_number: 1
    ... ... @@ -23,6 +25,9 @@ var:
    23 25 
       nss_version: '3.117'
    24 26 
       nspr_version: '4.37'
    25 27 
       cargo_vendor_include_config: 1
    28 
    +  # Uncomment this to run an online build to grab an updated
    29 
    +  # gradle-dependencies-list.txt.
    30 
    +  # generate_gradle_dependencies_list: 1
    26 31
    27 32 
     targets:
    28 33 
       nightly:
    ... ... @@ -61,7 +66,7 @@ steps:
    61 66 
           - project: glean
    62 67 
             name: glean
    63 68 
             pkg_type: build
    64 
    -        enable: '[% !c("var/fetch_gradle_dependencies") %]'
    69 
    +        enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    65 70 
           # Only Application Services currently requires build tools 35.
    66 71 
           # So, download them only here, rather than adding them to the shared
    67 72 
           # toolchain.
    ... ... @@ -75,28 +80,22 @@ steps:
    75 80 
           - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'
    76 81 
             name: gradle-dependencies
    77 82 
             exec: '[% INCLUDE "fetch-gradle-dependencies" %]'
    78 
    -        enable: '[% !c("var/fetch_gradle_dependencies") %]'
    83 
    +        enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    79 84 
           - URL: '[% pc("glean-parser", "var/glean_wheels_url") %]'
    80 85 
             name: glean-wheels
    81 86 
             sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]'
    82 
    -        enable: '[% !c("var/fetch_gradle_dependencies") %]'
    87 
    +        enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    83 88 
           - name: cargo_vendor
    84 89 
             project: application-services
    85 90 
             pkg_type: cargo_vendor
    86 91 
             norec:
    87 92 
               sha256sum: 8a93a189985ad28c334970dc5ab386c81d4a415f3777c2941248f836f4af44fd
    88 
    -        enable: '[% !c("var/fetch_gradle_dependencies") %]'
    89 
    -      - filename: gen_gradle_deps_file.sh
    90 
    -        enable: '[% c("var/fetch_gradle_dependencies") %]'
    93 
    +        enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    94 
    +      - filename: gen-gradle-deps-file.py
    95 
    +        enable: '[% c("var/generate_gradle_dependencies_list") %]'
    91 96
    92 97 
       list_toolchain_updates:
    93 98 
         git_hash: 'v[% c("version") %]'
    94 99 
         input_files: []
    95 100 
         container:
    96 101 
           use_container: 0
    97 
    -
    98 
    -  get_gradle_dependencies_list:
    99 
    -    filename: 'gradle-dependencies-list-[% c("version") %].txt'
    100 
    -    get_gradle_dependencies_list: '[% INCLUDE build %]'
    101 
    -    var:
    102 
    -      fetch_gradle_dependencies: 1

  • projects/common/gen-gradle-deps-file.py
    1 
    +#!/usr/bin/env python3
    2 
    +
    3 
    +import argparse
    4 
    +import hashlib
    5 
    +import re
    6 
    +import sys
    7 
    +import urllib.request
    8 
    +from concurrent.futures import ThreadPoolExecutor, as_completed
    9 
    +
    10 
    +
    11 
    +def download_and_hash(url):
    12 
    +    try:
    13 
    +        sha256_hash = hashlib.sha256()
    14 
    +        with urllib.request.urlopen(url) as f:
    15 
    +            sha256_hash.update(f.read())
    16 
    +        return url, sha256_hash.hexdigest(), None
    17 
    +    except Exception as e:
    18 
    +        return url, None, str(e)
    19 
    +
    20 
    +
    21 
    +def parse_arguments():
    22 
    +    parser = argparse.ArgumentParser(
    23 
    +        description="""
    24 
    +This script generates a `gradle-dependencies-list.txt` file as expected by
    25 
    +Tor Browser Build to download dependencies for offline builds.
    26 
    +
    27 
    +It parses a log file from a Gradle build, determines which dependencies were
    28 
    +downloaded and from where, re-downloads them, and calculates their SHA256 hash.
    29 
    +Downloads are parallelized — by default, 4 workers are used, but this can be
    30 
    +configured via the optional `--workers` argument.
    31 
    +
    32 
    +---
    33 
    +
    34 
    +Usage:
    35 
    +
    36 
    +First, run the exact Gradle command that will be used inside the offline
    37 
    +tor-browser-build container. Make sure to set the log level to `info` and
    38 
    +capture the full output to a log file.
    39 
    +
    40 
    +`--refresh-dependencies` can be pass to Gradle to force all dependencies to be
    41 
    +re-downloaded if needed (e.g., if running this script outside containers or
    42 
    +outside tor-browser-build).
    43 
    +
    44 
    +Example:
    45 
    +
    46 
    +    ./gradlew --info clean assembleRelease 2>&1 | tee -a out.log
    47 
    +
    48 
    +Then run this script with the generated log:
    49 
    +
    50 
    +    ./gen-gradle-deps-file.py out.log
    51 
    +
    52 
    +This will create a `gradle-dependencies-list.txt` file in the current
    53 
    +directory.
    54 
    +""",
    55 
    +        formatter_class=argparse.RawDescriptionHelpFormatter,
    56 
    +    )
    57 
    +
    58 
    +    parser.add_argument(
    59 
    +        "log_file", help="Path to the Gradle log file (e.g., out.log)"
    60 
    +    )
    61 
    +    parser.add_argument(
    62 
    +        "--workers",
    63 
    +        type=int,
    64 
    +        default=4,
    65 
    +        help="Number of parallel download workers (default: 4).",
    66 
    +    )
    67 
    +    parser.add_argument(
    68 
    +        "--output",
    69 
    +        default="gradle-dependencies-list.txt",
    70 
    +        help="Output file path (default: gradle-dependencies-list.txt).",
    71 
    +    )
    72 
    +
    73 
    +    return parser.parse_args()
    74 
    +
    75 
    +
    76 
    +def extract_urls(log_file):
    77 
    +    pattern = re.compile(rf"Downloading (https://\S+) to ")
    78 
    +    urls = []
    79 
    +    with open(log_file, "r") as file:
    80 
    +        for line in file:
    81 
    +            match = pattern.search(line.strip())
    82 
    +            if match:
    83 
    +                urls.append(match.group(1))
    84 
    +    return urls
    85 
    +
    86 
    +
    87 
    +def main():
    88 
    +    args = parse_arguments()
    89 
    +
    90 
    +    urls = extract_urls(args.log_file)
    91 
    +    if not urls:
    92 
    +        print("⚠️ No matching download lines found in the log file.")
    93 
    +        return
    94 
    +
    95 
    +    print(
    96 
    +        f"🔍 Found {len(urls)} URLs. Starting downloads using {args.workers} workers...\n"
    97 
    +    )
    98 
    +
    99 
    +    results = []
    100 
    +    with ThreadPoolExecutor(max_workers=args.workers) as executor:
    101 
    +        futures = {
    102 
    +            executor.submit(download_and_hash, url): url for url in urls
    103 
    +        }
    104 
    +        for future in as_completed(futures):
    105 
    +            url, sha256, error = future.result()
    106 
    +            if error:
    107 
    +                print(f"[ERROR] {url}\n  Reason: {error}")
    108 
    +            else:
    109 
    +                print(f"[OK] {url}\n  SHA256: {sha256}")
    110 
    +                results.append((sha256, url))
    111 
    +
    112 
    +    if results:
    113 
    +        with open(args.output, "w") as out_file:
    114 
    +            out_file.write(
    115 
    +                "# Don't forget to update var/gradle_dependencies_version when modifying this file\n"
    116 
    +            )
    117 
    +            out_file.write("sha256sum | url\n")
    118 
    +            for sha256, url in sorted(results, key=lambda x: x[1]):
    119 
    +                out_file.write(f"{sha256} | {url}\n")
    120 
    +        print(f"\n✅ {len(urls)} dependencies written to: {args.output}")
    121 
    +    else:
    122 
    +        print("\n⚠️ No successful downloads to write.")
    123 
    +        sys.exit(1)
    124 
    +
    125 
    +
    126 
    +if __name__ == "__main__":
    127 
    +    main()

  • projects/common/gen_gradle_deps_file.sh deleted
    1 
    -#!/bin/bash
    2 
    -
    3 
    -# Copyright (c) 2020, The Tor Project, Inc.
    4 
    -#
    5 
    -# Redistribution and use in source and binary forms, with or without
    6 
    -# modification, are permitted provided that the following conditions are
    7 
    -# met:
    8 
    -#
    9 
    -#     * Redistributions of source code must retain the above copyright
    10 
    -# notice, this list of conditions and the following disclaimer.
    11 
    -#
    12 
    -#     * Redistributions in binary form must reproduce the above
    13 
    -# copyright notice, this list of conditions and the following disclaimer
    14 
    -# in the documentation and/or other materials provided with the
    15 
    -# distribution.
    16 
    -#
    17 
    -#     * Neither the names of the copyright owners nor the names of its
    18 
    -# contributors may be used to endorse or promote products derived from
    19 
    -# this software without specific prior written permission.
    20 
    -#
    21 
    -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    22 
    -# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    23 
    -# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
    24 
    -# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
    25 
    -# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
    26 
    -# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
    27 
    -# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
    28 
    -# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
    29 
    -# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
    30 
    -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
    31 
    -# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    32 
    -
    33 
    -# Usage:
    34 
    -# 1) Point to a log file with all the dependency download attempts (for its
    35 
    -#    generation see: doc/how-to-create-gradle-dependencies-list.txt)
    36 
    -# 2) Double-check that you get the same SHA-256 sums when downloaded from a
    37 
    -#    different network location. E.g. by using `torsocks` with this script after
    38 
    -#    having made a copy of `gradle-dependencies-list.txt` from 1) and comparing
    39 
    -#    the two .txt files.
    40 
    -
    41 
    -export LC_ALL=C
    42 
    -log="$1"
    43 
    -
    44 
    -# Step 1: Extract all the download attempts out of the log file, ignore the ones
    45 
    -# for maven-metadata.xml and module files. We don't need those.
    46 
    -cat $log | grep "Performing HTTP" | grep -o "https://.*" | \
    47 
    -  grep -vE "\.module|maven-metadata\.xml" | sort | uniq > dl-attempts
    48 
    -
    49 
    -# Step 2: Fetch all the dependencies and calculate the SHA-256 sum
    50 
    -declare -A URLs
    51 
    -function dl_url {
    52 
    -  local url="$1"
    53 
    -  test -n "${URLs[$url]}" && return 0
    54 
    -  URLs[$url]=1
    55 
    -  wget -U "" $url
    56 
    -  test $? -eq 0 || return 1
    57 
    -  local fn=$(basename "$url")
    58 
    -  local sha256=`sha256sum $fn | cut -d ' ' -f 1`
    59 
    -  echo "$sha256 | $url" >> deps
    60 
    -  rm $fn
    61 
    -}
    62 
    -
    63 
    -while read line
    64 
    -do
    65 
    -  dl_url "$line"
    66 
    -  # If we downloaded a .pom file, also try to get corresponding .jar and
    67 
    -  # .aar files
    68 
    -  bname=$(basename "$line" .pom)
    69 
    -  dname=$(dirname "$line")
    70 
    -  if test "$line" != "$dname/$bname"
    71 
    -  then
    72 
    -    dl_url "$dname/$bname.aar"
    73 
    -    dl_url "$dname/$bname.jar"
    74 
    -  fi
    75 
    -done < dl-attempts
    76 
    -
    77 
    -# Step 3: Add the header at the beginning of the final dependency file.
    78 
    -echo "# On how to update dependencies see doc/how-to-create-gradle\
    79 
    --dependencies-list.txt" > gradle-dependencies-list.txt
    80 
    -echo "# Don't forget to update var/gradle_dependencies_version when modifying \
    81 
    -this file" >> gradle-dependencies-list.txt
    82 
    -echo "sha256sum | url" >> gradle-dependencies-list.txt
    83 
    -
    84 
    -# Step 4: Keep only successfully downloaded artifacts, remove duplicates, and
    85 
    -# sort based on download URL.
    86 
    -grep ^[a-f0-9] deps | rev | sort -t/ -u -k1,4 | rev | \
    87 
    -  sort -k 3 >> gradle-dependencies-list.txt
    88 
    -
    89 
    -# Step 5: Clean up
    90 
    -rm dl-attempts
    91 
    -rm deps
    92 
    -
    93 
    -exit 0

  • projects/geckoview/build
    ... ... @@ -23,7 +23,7 @@ echo "Starting ./mach configure $(date)"
    23 23 
       [% IF !c("var/rlbox") -%]--without-wasm-sandboxed-libraries[% END %]
    24 24
    25 25 
     echo "Starting ./mach build $(date)"
    26 
    -./mach build --verbose
    26 
    +./mach build --verbose [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
    27 27
    28 28 
     [% IF c("var/dev_artifacts") -%]
    29 29 
       echo "Building development artifacts"
    ... ... @@ -50,6 +50,11 @@ echo "Build finished, copying the AAR to the to the destination directory $(date
    50 50 
     [% IF !c("var/android_single_arch") -%]
    51 51 
       mkdir -p "$outdir/[% project %]"
    52 52 
       cp obj-*/gradle/target.maven.zip $outdir/[% project %]/
    53 
    +
    54 
    +  [% IF c('var/generate_gradle_dependencies_list') -%]
    55 
    +    cd $outdir/[% project %]
    56 
    +    $rootdir/gen-gradle-deps-file.py $gradle_logs
    57 
    +  [% END -%]
    53 58 
     [% ELSE -%]
    54 59 
     [% INCLUDE 'build_ac_fenix' %]
    55 60 
     [% END -%]

  • projects/geckoview/build_ac_fenix
    ... ... @@ -8,7 +8,7 @@ objdir=$(cd obj-* && pwd)
    8 8
    9 9 
     echo "Building Android Components $(date)"
    10 10 
     pushd mobile/android/android-components
    11 
    -gradle $GRADLE_FLAGS assembleGecko -x lint
    11 
    +gradle $GRADLE_FLAGS assembleGecko -x lint [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
    12 12 
     popd
    13 13
    14 14 
     # The build might fail with "file exists" otherwise.
    ... ... @@ -36,7 +36,7 @@ variant='[% c("var/variant") %]'
    36 36 
     version_name="[% c('var/torbrowser_version') %] ([% c('var/geckoview_version') %])"
    37 37
    38 38 
     echo "Building $variant Fenix APK"
    39 
    -gradle $GRADLE_FLAGS -PversionName="$version_name" "assemble$variant"
    39 
    +gradle $GRADLE_FLAGS -PversionName="$version_name" "assemble$variant" [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
    40 40
    41 41 
     echo "Build finished, copying the APK(s) to the destination directory $(date)"
    42 42 
     mkdir -p $outdir/[% project %]
    ... ... @@ -46,7 +46,7 @@ echo "Building non optimized $variant Fenix APK for testing"
    46 46 
     gradle $GRADLE_FLAGS -PversionName="$version_name" -PdisableOptimization "assemble$variant"
    47 47
    48 48 
     echo "Building Fenix instrumentation tests"
    49 
    -gradle $GRADLE_FLAGS -PversionName="$version_name" -PtestBuildType="$variant" -PdisableOptimization assembleAndroidTest
    49 
    +gradle $GRADLE_FLAGS -PversionName="$version_name" -PtestBuildType="$variant" -PdisableOptimization assembleAndroidTest [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
    50 50
    51 51 
     echo "Test build finished, copying the APKs to the destination directory $(date)"
    52 52
    ... ... @@ -55,3 +55,8 @@ mv $objdir/gradle/build/mobile/android/fenix/app/outputs/apk/fenix/$variant/*.ap
    55 55 
     mv $objdir/gradle/build/mobile/android/fenix/app/outputs/apk/androidTest/fenix/$variant/*.apk $outdir/[% project %]/tests
    56 56
    57 57 
     popd
    58 
    +
    59 
    +[% IF c('var/generate_gradle_dependencies_list') -%]
    60 
    +  cd $outdir/[% project %]
    61 
    +  $rootdir/gen-gradle-deps-file.py $gradle_logs
    62 
    +[% END -%]

  • projects/geckoview/build_common
    ... ... @@ -14,8 +14,6 @@ export PATH="/var/tmp/dist/node/bin:$PATH"
    14 14 
     export LC_ALL=C.UTF-8
    15 15 
     export LANG=C.UTF-8
    16 16
    17 
    -tar -xf [% c('input_files_by_name/glean') %]
    18 
    -
    19 17 
     export TOR_EXPERT_BUNDLE_AAR=$rootdir/[% c('input_files_by_name/tor-expert-bundle-aar') %]/tor-expert-bundle.aar
    20 18
    21 19 
     tar -C /var/tmp/dist -xf [% c('input_files_by_name/application-services') %]
    ... ... @@ -29,25 +27,34 @@ tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.[% c('compress_ta
    29 27 
     [% c("var/set_MOZ_BUILD_DATE") %]
    30 28
    31 29 
     export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64
    32 
    -gradle_repo=/var/tmp/dist/gradle-dependencies
    33 
    -export GRADLE_MAVEN_REPOSITORIES="file://$gradle_repo","file://$gradle_repo/maven2"
    34 
    -# Set the Maven local repository because Gradle ignores our overriding of $HOME.
    35 
    -# It is only used for the local pubblication of single-arch AARs.
    36 
    -export GRADLE_FLAGS="--no-daemon --offline -Dmaven.repo.local=$distdir/[% project %]"
    37 
    -# Move the Gradle repo to a hard-coded location. The location is embedded in
    38 
    -# the file chrome/toolkit/content/global/buildconfig.html so it needs to be
    39 
    -# standardized for reproducibility.
    40 
    -mv $rootdir/[% c('input_files_by_name/gradle-dependencies') %] $gradle_repo
    41 
    -cp -rl $rootdir/glean/maven/* $gradle_repo
    42 
    -cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true
    43 
    -cp -rl $gradle_repo/m2/* $gradle_repo || true
    44 
    -cp -rl $gradle_repo/maven2/* $gradle_repo || true
    45 
    -
    46 
    -tar -xf [% c('input_files_by_name/glean-wheels') %]
    47 
    -export GLEAN_PYTHON_WHEELS_DIR=$rootdir/glean-wheels
    48 
    -
    49 
    -tar -xf [% c('input_files_by_name/oss-licenses-plugin') %]
    50 
    -cp -rl oss-licenses-plugin/. $gradle_repo
    30 
    +
    31 
    +[% IF !c('var/generate_gradle_dependencies_list') -%]
    32 
    +  gradle_repo=/var/tmp/dist/gradle-dependencies
    33 
    +  export GRADLE_MAVEN_REPOSITORIES="file://$gradle_repo","file://$gradle_repo/maven2"
    34 
    +  # Set the Maven local repository because Gradle ignores our overriding of $HOME.
    35 
    +  # It is only used for the local pubblication of single-arch AARs.
    36 
    +  export GRADLE_FLAGS="--no-daemon --offline -Dmaven.repo.local=$distdir/[% project %]"
    37 
    +  # Move the Gradle repo to a hard-coded location. The location is embedded in
    38 
    +  # the file chrome/toolkit/content/global/buildconfig.html so it needs to be
    39 
    +  # standardized for reproducibility.
    40 
    +  mv $rootdir/[% c('input_files_by_name/gradle-dependencies') %] $gradle_repo
    41 
    +
    42 
    +  tar -xf [% c('input_files_by_name/glean') %]
    43 
    +  cp -rl $rootdir/glean/maven/* $gradle_repo
    44 
    +
    45 
    +  cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true
    46 
    +  cp -rl $gradle_repo/m2/* $gradle_repo || true
    47 
    +  cp -rl $gradle_repo/maven2/* $gradle_repo || true
    48 
    +
    49 
    +  tar -xf [% c('input_files_by_name/glean-wheels') %]
    50 
    +  export GLEAN_PYTHON_WHEELS_DIR=$rootdir/glean-wheels
    51 
    +
    52 
    +  tar -xf [% c('input_files_by_name/oss-licenses-plugin') %]
    53 
    +  cp -rl oss-licenses-plugin/. $gradle_repo
    54 
    +[% ELSE -%]
    55 
    +  gradle_logs=/var/tmp/build/gradle-logs.log
    56 
    +  export GRADLE_FLAGS="--no-daemon --info"
    57 
    +[% END -%]
    51 58
    52 59 
     # We unbreak mach, see: https://bugzilla.mozilla.org/show_bug.cgi?id=1656993.
    53 60 
     export MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=system

  • projects/geckoview/config
    ... ... @@ -48,6 +48,9 @@ var:
    48 48 
       has_l10n: '[% !c("var/testbuild") && !c("var/locales").empty %]'
    49 49 
       l10n-changesets: '[% exec("git --no-pager show " _ c("git_hash") _ ":browser/locales/l10n-changesets.json", { exec_noco => 1 }) %]'
    50 50 
       mozconfig_file: 'mozconfig-[% c("var/osname") %]'
    51 
    +  # Uncomment this to run an online build to grab an updated
    52 
    +  # gradle-dependencies-list.txt.
    53 
    +  # generate_gradle_dependencies_list: 1
    51 54
    52 55 
     targets:
    53 56 
       release:
    ... ... @@ -87,9 +90,11 @@ steps:
    87 90 
           - project: glean
    88 91 
             name: glean
    89 92 
             pkg_type: build
    93 
    +        enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    90 94 
           - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'
    91 95 
             name: gradle-dependencies
    92 96 
             exec: '[% INCLUDE "fetch-gradle-dependencies" %]'
    97 
    +        enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    93 98 
           - name: geckoview_armv7
    94 99 
             project: geckoview
    95 100 
             pkg_type: build
    ... ... @@ -123,15 +128,19 @@ steps:
    123 128 
           - URL: '[% pc("glean-parser", "var/glean_wheels_url") %]'
    124 129 
             name: glean-wheels
    125 130 
             sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]'
    131 
    +        enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    126 132 
           - name: tor-expert-bundle-aar
    127 133 
             project: tor-expert-bundle-aar
    128 134 
             pkg_type: build
    129 135 
           - project: oss-licenses-plugin
    130 136 
             name: oss-licenses-plugin
    131 137 
             pkg_type: build
    138 
    +        enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    132 139 
           - URL: 'https://github.com/google/bundletool/releases/download/[% c("var/bundletool_version") %]/bundletool-all-[% c("var/bundletool_version") %].jar'
    133 140 
             name: bundletool
    134 141 
             sha256sum: '[% c("var/bundletool_hash") %]'
    142 
    +      - filename: gen-gradle-deps-file.py
    143 
    +        enable: '[% c("var/generate_gradle_dependencies_list") %]'
    135 144
    136 145 
       list_toolchain_updates:
    137 146 
         git_hash: FIREFOX_NIGHTLY_146_END
    ... ... @@ -167,14 +176,15 @@ input_files:
    167 176 
       - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'
    168 177 
         name: gradle-dependencies
    169 178 
         exec: '[% INCLUDE "fetch-gradle-dependencies" %]'
    179 
    +    enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    170 180 
       - URL: '[% pc("glean-parser", "var/glean_wheels_url") %]'
    171 181 
         name: glean-wheels
    172 182 
         sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]'
    173 183 
       - project: glean
    174 184 
         name: glean
    175 
    -    pkg_type: build
    176 185 
       - project: oss-licenses-plugin
    177 186 
         name: oss-licenses-plugin
    187 
    +    enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    178 188 
       - name: tor-expert-bundle-aar
    179 189 
         project: tor-expert-bundle-aar
    180 190 
       - project: firefox-l10n
    ... ... @@ -192,3 +202,5 @@ input_files:
    192 202 
       - URL: 'https://github.com/google/bundletool/releases/download/[% c("var/bundletool_version") %]/bundletool-all-[% c("var/bundletool_version") %].jar'
    193 203 
         name: bundletool
    194 204 
         sha256sum: '[% c("var/bundletool_hash") %]'
    205 
    +  - filename: gen-gradle-deps-file.py
    206 
    +    enable: '[% c("var/generate_gradle_dependencies_list") %]'

  • projects/glean/build
    ... ... @@ -28,7 +28,7 @@ export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64
    28 28
    29 29 
     [% INCLUDE 'fake-git' %]
    30 30
    31 
    -[% IF !c('var/fetch_gradle_dependencies') %]
    31 
    +[% IF !c('var/generate_gradle_dependencies_list') -%]
    32 32 
       gradle_repo=/var/tmp/dist/gradle-dependencies
    33 33 
       mv $rootdir/[% c('input_files_by_name/gradle-dependencies') %] $gradle_repo
    34 34 
       cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true
    ... ... @@ -42,23 +42,24 @@ mv /var/tmp/build/[% project %]-[% c('version') %] $builddir
    42 42
    43 43 
     cd $builddir
    44 44
    45 
    -# Prepare the offline build.
    46 
    -
    47 
    -tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %]
    48 
    -export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels
    49 
    -# Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will still
    50 
    -# try to install Glean with pip.
    45 
    +[% IF !c('var/generate_gradle_dependencies_list') -%]
    46 
    +  # Prepare the offline build.
    47 
    +  tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %]
    48 
    +  export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels
    49 
    +  # Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will still
    50 
    +  # try to install Glean with pip.
    51 51 
     cat > $rootdir/pip.conf << 'EOF'
    52 52 
     [global]
    53 53 
     find-links = /var/tmp/build/glean/glean-wheels
    54 54 
     index-url =
    55 55 
     no-index = yes
    56 56 
     EOF
    57 
    -export PIP_CONFIG_FILE=$rootdir/pip.conf
    58 
    -# Move the directory for hardcoding the path in .cargo/config.
    59 
    -tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %]
    60 
    -# Make sure our vendored crates are used for offline builds.
    61 
    -cat >> .cargo/config.toml << 'EOF'
    57 
    +
    58 
    +  export PIP_CONFIG_FILE=$rootdir/pip.conf
    59 
    +  # Move the directory for hardcoding the path in .cargo/config.
    60 
    +  tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %]
    61 
    +  # Make sure our vendored crates are used for offline builds.
    62 
    +  cat >> .cargo/config.toml << 'EOF'
    62 63
    63 64 
     [source.crates-io]
    64 65 
     replace-with = "vendored-sources"
    ... ... @@ -70,16 +71,27 @@ directory = "/var/tmp/build/glean/vendor"
    70 71 
     offline=true
    71 72 
     EOF
    72 73
    74 
    +  deps_flags="--offline -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies"
    75 
    +[% ELSE -%]
    76 
    +  deps_flags="--info"
    77 
    +  gradle_logs=/var/tmp/build/gradle-logs.log
    78 
    +[% END -%]
    79 
    +
    73 80 
     tar -xf $rootdir/[% c('input_files_by_name/uniffi-rs') %]
    74 81
    75 82 
     patch -p1 < $rootdir/use-custom-gradle-repo.diff
    76 83 
     patch -p1 < $rootdir/use-uniffi-noop.diff
    77 84
    78 
    -gradle_flags="--offline --no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies"
    79 
    -gradle $gradle_flags :glean:assembleRelease
    85 
    +gradle_flags="--no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen $deps_flags"
    86 
    +gradle $gradle_flags :glean:assembleRelease [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
    80 87 
     gradle $gradle_flags publish
    81 88 
     mv build/maven $distdir
    82 89
    90 
    +[% IF c('var/generate_gradle_dependencies_list') -%]
    91 
    +  cd $distdir
    92 
    +  $rootdir/gen-gradle-deps-file.py $gradle_logs
    93 
    +[% END -%]
    94 
    +
    83 95 
     cd /var/tmp/dist
    84 96 
     [% c('tar', {
    85 97 
         tar_src => [ project ],

  • projects/glean/config
    ... ... @@ -6,11 +6,16 @@ filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.[% c("co
    6 6 
     version: 66.1.1
    7 7 
     container:
    8 8 
       use_container: 1
    9 
    +  disable_network:
    10 
    +    build: '[% !c("var/generate_gradle_dependencies_list") %]'
    9 11
    10 12 
     var:
    11 13 
       # This should be updated when the list of gradle dependencies is changed.
    12 14 
       gradle_dependencies_version: 66.1.1-1
    13 15 
       gradle_version: 8.14.3
    16 
    +  # Uncomment this to run an online build to grab an updated
    17 
    +  # gradle-dependencies-list.txt.
    18 
    +  # generate_gradle_dependencies_list: 1
    14 19
    15 20 
     steps:
    16 21 
       build:
    ... ... @@ -36,6 +41,7 @@ steps:
    36 41 
           - URL: '[% pc("glean-parser", "var/glean_wheels_url") %]'
    37 42 
             name: glean-wheels
    38 43 
             sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]'
    44 
    +        enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    39 45 
           - name: cargo_vendor
    40 46 
             project: glean
    41 47 
             pkg_type: cargo_vendor
    ... ... @@ -44,5 +50,8 @@ steps:
    44 50 
           - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'
    45 51 
             name: gradle-dependencies
    46 52 
             exec: '[% INCLUDE "fetch-gradle-dependencies" %]'
    53 
    +        enable: '[% !c("var/generate_gradle_dependencies_list") %]'
    54 
    +      - filename: gen-gradle-deps-file.py
    55 
    +        enable: '[% c("var/generate_gradle_dependencies_list") %]'
    47 56 
           - filename: use-custom-gradle-repo.diff
    48 57 
           - filename: use-uniffi-noop.diff

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.torproject.org. Manage all notifications · Help Notification message regarding https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/compare/cf5a3f623f94cf5bc093db61fe64f9b38b03fce0...40f3459b2d37266dd417fca1b937329aa92ec32d at 1765298341