brizental pushed to branch main at The Tor Project / Applications / tor-browser-bundle-testsuite Commits: 2e6652ed by Beatriz Rizental at 2026-04-27T18:18:48+02:00 Bug 40091: Add CI for testing Nightly builds This commit creates the base jobs for running tests in multiple platforms as well as adds a first job that runs marionette tests. - - - - - 7 changed files: - + .gitlab-ci.yml - + .gitlab/README.md - + .gitlab/_base.yml - + .gitlab/_inputs.yml - + .gitlab/scripts/after_script.py - + .gitlab/scripts/before_script.py - + .gitlab/test_marionette.yml Changes: ===================================== .gitlab-ci.yml ===================================== @@ -0,0 +1,33 @@ +spec: + include: + - local: '.gitlab/_inputs.yml' +--- +stages: + - test-marionette + +default: + artifacts: + # Regardless of test outcome, all files inside the `logs` folder are saved. + when: always + paths: + - logs/ + expire_in: 1 week + +include: + - local: '.gitlab/_base.yml' + inputs: + mozharness_url: $[[ inputs.mozharness_url ]] + android_x86_64_installer_url: $[[ inputs.android_x86_64_installer_url ]] + android_x86_64_package_name: $[[ inputs.android_x86_64_package_name ]] + android_x86_64_artifacts_url: $[[ inputs.android_x86_64_artifacts_url ]] + android_x86_64_sha256sums_url: $[[ inputs.android_x86_64_sha256sums_url ]] + debian_x86_64_installer_url: $[[ inputs.debian_x86_64_installer_url ]] + debian_x86_64_artifacts_url: $[[ inputs.debian_x86_64_artifacts_url ]] + debian_x86_64_sha256sums_url: $[[ inputs.debian_x86_64_sha256sums_url ]] + macos_x86_64_installer_url: $[[ inputs.macos_x86_64_installer_url ]] + macos_x86_64_artifacts_url: $[[ inputs.macos_x86_64_artifacts_url ]] + macos_x86_64_sha256sums_url: $[[ inputs.macos_x86_64_sha256sums_url ]] + windows_x86_64_installer_url: $[[ inputs.windows_x86_64_installer_url ]] + windows_x86_64_artifacts_url: $[[ inputs.windows_x86_64_artifacts_url ]] + windows_x86_64_sha256sums_url: $[[ inputs.windows_x86_64_sha256sums_url ]] + - local: '.gitlab/test_marionette.yml' ===================================== .gitlab/README.md ===================================== @@ -0,0 +1,108 @@ +# Tor Browser Nightly CI + +This directory holds the the GitLab CI configuration for the Tor Browser nightlies test pipeline. + +The top-level entry point is [`.gitlab-ci.yml`](../.gitlab-ci.yml). That file defines the pipeline stages and includes the files in this directory. + +## Files + +- [`_inputs.yml`](./_inputs.yml): Shared `spec:inputs` definitions. Both the top-level pipeline and [`_base.yml`](./_base.yml) include this file so they agree on the input schema. +- [`_base.yml`](./_base.yml): Hidden base jobs for each platform. These jobs define runner tags, `rules`, setup steps, and platform-specific variables. +- [`test_*.yml`](./test_*.yml): The actual test jobs. These are small jobs that extend the platform base jobs from [`_base.yml`](./_base.yml) and add the test command to run. + +All jobs here use Mozharness. Mozharness is Mozilla's Python-based automation harness, and Mozilla keeps test runner scripts like `marionette.py` and `android_emulator_unittest.py` under `testing/mozharness/scripts`. + +> ### How Inputs Flow +> +> 1. [`../.gitlab-ci.yml`](../.gitlab-ci.yml) includes [`_inputs.yml`](./_inputs.yml) in its `spec`. +> 2. [`../.gitlab-ci.yml`](../.gitlab-ci.yml) includes [`_base.yml`](./_base.yml) and forwards all input values with `include:inputs`. +> 3. [`_base.yml`](./_base.yml) also includes [`_inputs.yml`](./_inputs.yml) in its own `spec`, so it can use `$[[ inputs.* ]]`. +> 4. Jobs in [`test_marionette.yml`](./test_marionette.yml) extend the hidden base jobs from [`_base.yml`](./_base.yml), inheriting the platform setup and variables like `INSTALLER_URL` and `ARTIFACTS_URL`. +> +> This explicit forwarding is required because inputs are scoped to the file that declares them. + +## Running A Job + +These jobs are meant to be started as triggered pipelines. The base jobs in [`_base.yml`](./_base.yml) only run when `CI_PIPELINE_SOURCE == "trigger"` and the platform-specific inputs are present. + +To create a trigger token in GitLab: + +1. Open the project in GitLab. +2. Go to `Settings` > `CI/CD`. +3. Expand `Pipeline trigger tokens`. +4. Create a new token and copy the value. + +GitLab's trigger API accepts an `inputs` map, which matches the names defined in [`_inputs.yml`](./_inputs.yml). A typical JSON request looks like this: + +```bash +curl --request POST \ + --header "Content-Type: application/json" \ + --data '{ + "inputs": { + "mozharness_url": "https://example.org/mozharness.zip", + "debian_x86_64_installer_url": "https://example.org/tor-browser-linux-x86_64.tar.xz", + "debian_x86_64_artifacts_url": "https://example.org/debian-artifacts", + "debian_x86_64_sha256sums_url": "https://example.org/sha256sums.txt" + } + }' \ + "https://gitlab.example.org/api/v4/projects/$PROJECT_ID/trigger/pipeline?toke..." +``` + +You can also trigger multiple platform jobs in a single pipeline by providing more than one platform's input set in the same request: + +```bash +curl --request POST \ + --header "Content-Type: application/json" \ + --data '{ + "inputs": { + "mozharness_url": "https://example.org/mozharness.zip", + "debian_x86_64_installer_url": "https://example.org/tor-browser-linux-x86_64.tar.xz", + "debian_x86_64_artifacts_url": "https://example.org/debian-artifacts", + "debian_x86_64_sha256sums_url": "https://example.org/debian-sha256sums.txt", + "macos_x86_64_installer_url": "https://example.org/TorBrowser.dmg", + "macos_x86_64_artifacts_url": "https://example.org/macos-artifacts", + "macos_x86_64_sha256sums_url": "https://example.org/macos-sha256sums.txt", + "windows_x86_64_installer_url": "https://example.org/tor-browser-install.exe", + "windows_x86_64_artifacts_url": "https://example.org/windows-artifacts", + "windows_x86_64_sha256sums_url": "https://example.org/windows-sha256sums.txt" + } + }' \ + "https://gitlab.example.org/api/v4/projects/$PROJECT_ID/trigger/pipeline?toke..." +``` + +### Accepted Inputs + +The pipeline accepts the following inputs: + +- `mozharness_url`: URL to the `mozharness.zip` archive used by every job. +- `debian_x86_64_installer_url`: URL to the Debian Tor Browser build to test. +- `debian_x86_64_artifacts_url`: URL to the Debian test artifacts, including `target.test_packages.json`. +- `debian_x86_64_sha256sums_url`: URL to the Debian `sha256sums.txt` file. Optional. When provided, the job checks this URL is reachable before running the full test suite. +- `macos_x86_64_installer_url`: URL to the macOS Tor Browser build to test. +- `macos_x86_64_artifacts_url`: URL to the macOS test artifacts, including `target.test_packages.json`. +- `macos_x86_64_sha256sums_url`: URL to the macOS `sha256sums.txt` file. Optional. When provided, the job checks this URL is reachable before running the full test suite. +- `windows_x86_64_installer_url`: URL to the Windows Tor Browser build to test. +- `windows_x86_64_artifacts_url`: URL to the Windows test artifacts, including `target.test_packages.json`. +- `windows_x86_64_sha256sums_url`: URL to the Windows `sha256sums.txt` file. Optional. When provided, the job checks this URL is reachable before running the full test suite. +- `android_x86_64_installer_url`: URL to the Android APK to test. +- `android_x86_64_artifacts_url`: URL to the Android test artifacts, including `target.test_packages.json`. +- `android_x86_64_sha256sums_url`: URL to the Android `sha256sums.txt` file. Optional. When provided, the job checks this URL is reachable before running the full test suite. +- `android_x86_64_package_name`: Android package name to test. + +All inputs default to the empty string. The platform-specific inputs are optional when triggering the pipeline as a whole, but the installer URL, artifacts URL, and (for Android) package name must be provided together for that platform's job to run. + +### How Job Selection Works + +Each hidden base job in [`_base.yml`](./_base.yml) has a `rules` section that checks two things: + +- the pipeline was started with `CI_PIPELINE_SOURCE == "trigger"` +- the inputs required for that platform are not empty + +That means the set of jobs in the pipeline depends on the inputs you provide: + +- If you provide `mozharness_url`, `debian_x86_64_installer_url`, and `debian_x86_64_artifacts_url`, the Debian Marionette job is included. +- If you provide the macOS installer and artifacts inputs, the macOS Marionette job is included. +- If you provide the Windows installer and artifacts inputs, the Windows Marionette job is included. +- If you provide the Android installer URL, artifacts URL, and package name, the Android Marionette job is included. + +If a platform-specific input is missing or left as an empty string, that platform's job is skipped entirely rather than starting and failing later. In other words, one triggered pipeline can run one platform, several platforms, or all platforms, depending on which input set you send. ===================================== .gitlab/_base.yml ===================================== @@ -0,0 +1,86 @@ +spec: + include: + - local: '.gitlab/_inputs.yml' +--- +variables: + MOZHARNESS_URL: "$[[ inputs.mozharness_url ]]" + +.common: + before_script: + - python3 .gitlab/scripts/before_script.py + + after_script: + - python3 .gitlab/scripts/after_script.py + +.debian-x86_64: + extends: .common + + tags: + - debian-trixie-x86_64 + + rules: + - if: $CI_PIPELINE_SOURCE == "trigger" && "$[[ inputs.mozharness_url ]]" != "" && "$[[ inputs.debian_x86_64_installer_url ]]" != "" && "$[[ inputs.debian_x86_64_artifacts_url ]]" != "" + when: on_success + - when: never + + variables: + MOZ_FETCHES_DIR: "/home/gitlab-runner/fetches" + MINIDUMP_STACKWALK_URL: "https://github.com/rust-minidump/rust-minidump/releases/download/v0.26.1/min..." + MINIDUMP_STACKWALK_SHA256: "f789997f086dab6e46c46ab4560fc28676124613770ddac3b89a6e2e0e4963d8" + INSTALLER_URL: "$[[ inputs.debian_x86_64_installer_url ]]" + ARTIFACTS_URL: "$[[ inputs.debian_x86_64_artifacts_url ]]" + SHA256SUMS_URL: "$[[ inputs.debian_x86_64_sha256sums_url ]]" + +.macos-x86_64: + extends: .common + + tags: + - macos-sonoma-x86_64 + + rules: + - if: $CI_PIPELINE_SOURCE == "trigger" && "$[[ inputs.mozharness_url ]]" != "" && "$[[ inputs.macos_x86_64_installer_url ]]" != "" && "$[[ inputs.macos_x86_64_artifacts_url ]]" != "" + when: on_success + - when: never + + variables: + MOZ_FETCHES_DIR: "/Users/gitlab-runner/fetches" + MINIDUMP_STACKWALK_URL: "https://github.com/rust-minidump/rust-minidump/releases/download/v0.26.1/min..." + MINIDUMP_STACKWALK_SHA256: "7a3e274a09bc35ccc35e9d3dcd60e738ab14e54ea3306be94a4a8dedaf7a468d" + INSTALLER_URL: "$[[ inputs.macos_x86_64_installer_url ]]" + ARTIFACTS_URL: "$[[ inputs.macos_x86_64_artifacts_url ]]" + SHA256SUMS_URL: "$[[ inputs.macos_x86_64_sha256sums_url ]]" + +.windows-x86_64: + extends: .common + + tags: + - windows-11-x86_64 + + rules: + - if: $CI_PIPELINE_SOURCE == "trigger" && "$[[ inputs.mozharness_url ]]" != "" && "$[[ inputs.windows_x86_64_installer_url ]]" != "" && "$[[ inputs.windows_x86_64_artifacts_url ]]" != "" + when: on_success + - when: never + + variables: + MOZ_FETCHES_DIR: 'C:\windoes\fetches' + MINIDUMP_STACKWALK_URL: 'https://github.com/rust-minidump/rust-minidump/releases/download/v0.26.1/min...' + MINIDUMP_STACKWALK_SHA256: "dc6da411047ef15b784bffe0f9bf7ecb9221186c4c8fd655e3d8a3fbba0a9c61" + INSTALLER_URL: "$[[ inputs.windows_x86_64_installer_url ]]" + ARTIFACTS_URL: "$[[ inputs.windows_x86_64_artifacts_url ]]" + SHA256SUMS_URL: "$[[ inputs.windows_x86_64_sha256sums_url ]]" + +.android-x86_64: + extends: .debian-x86_64 + + rules: + - if: '$CI_PIPELINE_SOURCE == "trigger" && "$[[ inputs.mozharness_url ]]" != "" && "$[[ inputs.android_x86_64_installer_url ]]" != "" && "$[[ inputs.android_x86_64_artifacts_url ]]" != "" && "$[[ inputs.android_x86_64_package_name ]]" != ""' + when: on_success + - when: never + + variables: + # TODO (tor-browser-bundle-testsuite#40101): Stop hardcoding the filename here. + ANDROID_SDK_REPACK_URL: "https://tb-build-06.torproject.org/~tb-builder/tor-browser-build/out/android..." + INSTALLER_URL: "$[[ inputs.android_x86_64_installer_url ]]" + ARTIFACTS_URL: "$[[ inputs.android_x86_64_artifacts_url ]]" + PACKAGE_NAME: "$[[ inputs.android_x86_64_package_name ]]" + SHA256SUMS_URL: "$[[ inputs.android_x86_64_sha256sums_url ]]" ===================================== .gitlab/_inputs.yml ===================================== @@ -0,0 +1,61 @@ +inputs: + mozharness_url: + type: string + default: "" + description: "Location of the mozharness.zip archive" + + android_x86_64_installer_url: + type: string + default: "" + description: "Location of the Android APK to test" + android_x86_64_package_name: + type: string + default: "" + description: "Name of the Android package to test" + android_x86_64_artifacts_url: + type: string + default: "" + description: "Location of Android test artifacts" + android_x86_64_sha256sums_url: + type: string + default: "" + description: "Location of the Android sha256sums.txt file" + + debian_x86_64_installer_url: + type: string + default: "" + description: "Location of the Debian build to test" + debian_x86_64_artifacts_url: + type: string + default: "" + description: "Location of Debian test artifacts" + debian_x86_64_sha256sums_url: + type: string + default: "" + description: "Location of the Debian sha256sums.txt file" + + macos_x86_64_installer_url: + type: string + default: "" + description: "Location of the macOS build to test" + macos_x86_64_artifacts_url: + type: string + default: "" + description: "Location of macOS test artifacts" + macos_x86_64_sha256sums_url: + type: string + default: "" + description: "Location of the macOS sha256sums.txt file" + + windows_x86_64_installer_url: + type: string + default: "" + description: "Location of the Windows build to test" + windows_x86_64_artifacts_url: + type: string + default: "" + description: "Location of Windows test artifacts" + windows_x86_64_sha256sums_url: + type: string + default: "" + description: "Location of the Windows sha256sums.txt file" ===================================== .gitlab/scripts/after_script.py ===================================== @@ -0,0 +1,15 @@ +import shutil +from pathlib import Path + +if __name__ == "__main__": + # This directory will be saved as an artifact after each job. + logs_dir = Path("logs") + logs_dir.mkdir(parents=True, exist_ok=True) + + # This is the directory created by mozharness with a lot of useful debug things. + blobber_dir = Path("build") / "blobber_upload_dir" + if blobber_dir.exists(): + shutil.copytree(blobber_dir, logs_dir, dirs_exist_ok=True) + print(f"Copied {blobber_dir} -> {logs_dir}") + else: + print(f"No blobber_upload_dir found at {blobber_dir}, skipping") ===================================== .gitlab/scripts/before_script.py ===================================== @@ -0,0 +1,166 @@ +import hashlib +import os +import shutil +import subprocess +import time +import urllib.request +import zipfile +from pathlib import Path + +SPECULATIVE_DOWNLOAD_RETRY_ATTEMPTS = 3 +SPECULATIVE_DOWNLOAD_RETRY_DELAY = 300 # 5min + +# Mozilla hardcodes this version in their testing code and +# expects the emulator to have a this exact android version. +EMULATOR_ANDROID_VERSION = 34 + + +def download_file(url: str, dest: Path, sha256: str = "") -> None: + print(f"Downloading {url} -> {dest}") + urllib.request.urlretrieve(url, dest) + if sha256: + actual = hashlib.sha256(dest.read_bytes()).hexdigest() + if actual != sha256: + raise ValueError( + f"SHA256 mismatch for {dest}: expected {sha256}, got {actual}" + ) + + +def extract_tar(tar_path: Path, dest_dir: Path) -> None: + subprocess.run( + # --strip-components=1: all the archives we fetch have contents nested inside a + # top-level directory rather than at the root, so we strip it on extraction. + ["tar", "-xf", str(tar_path), "-C", str(dest_dir), "--strip-components=1"], + check=True, + ) + + +def check_sha256sums_url(sha256sums_url: str) -> None: + if not sha256sums_url: + return + + for attempt in range(SPECULATIVE_DOWNLOAD_RETRY_ATTEMPTS): + try: + dest = Path("sha256sums") + download_file(sha256sums_url, dest) + print( + f"SHA256SUMS downloaded: {len(dest.read_bytes().splitlines())} entries" + ) + return + except Exception as e: + print(f"Attempt {attempt + 1}: {e}") + + if attempt < SPECULATIVE_DOWNLOAD_RETRY_ATTEMPTS - 1: + print(f"Retrying in {SPECULATIVE_DOWNLOAD_RETRY_DELAY}s...") + time.sleep(SPECULATIVE_DOWNLOAD_RETRY_DELAY) + + raise RuntimeError(f"SHA256SUMS not available after retries: {sha256sums_url}") + + +def download_mozharness(mozharness_url: str) -> None: + zip_path = Path("mozharness.zip") + download_file(mozharness_url, zip_path) + with zipfile.ZipFile(zip_path) as zf: + zf.extractall("mozharness") + print("Extracted mozharness/") + + +def download_minidump_stackwalk( + minidump_stackwalk_url: str, moz_fetches_dir: str, sha256: str = "" +) -> None: + dest_dir = Path(moz_fetches_dir) / "minidump-stackwalk" + dest_dir.mkdir(parents=True, exist_ok=True) + + tar_path = Path("minidump-stackwalk.tar.xz") + download_file(minidump_stackwalk_url, tar_path, sha256) + extract_tar(tar_path, dest_dir) + print(f"Extracted minidump-stackwalk -> {dest_dir}") + + +def setup_android_sdk( + moz_fetches_dir: str, + android_sdk_repack_url: str, +) -> None: + android_sdk_dir = Path(moz_fetches_dir) / "android-sdk-linux" + android_sdk_dir.mkdir(parents=True, exist_ok=True) + + tar_zst_path = Path("android-sdk.tar.zst") + download_file(android_sdk_repack_url, tar_zst_path) + extract_tar(tar_zst_path, android_sdk_dir) + + # Remove fake emulator binary added by tor-browser-build + emulator_dir = android_sdk_dir / "emulator" + if emulator_dir.exists(): + shutil.rmtree(emulator_dir) + print("Removed fake emulator binary") + + bin_dir = next((android_sdk_dir / "cmdline-tools").glob("*/bin")) + sdkmanager = bin_dir / "sdkmanager" + avdmanager = bin_dir / "avdmanager" + system_image = ( + f"system-images;android-{EMULATOR_ANDROID_VERSION};google_apis;x86_64" + ) + + print("Installing emulator and system images via sdkmanager...") + subprocess.run( + [str(sdkmanager), "emulator", system_image], + # 10 is an arbitrary number. We just want to repeat the "y" + # as many times as needed to accept all licenses. + input="y\n" * 10, + text=True, + stdout=subprocess.DEVNULL, + check=False, + ) + + android_device_dir = Path(moz_fetches_dir) / "android-device" + android_device_dir.mkdir(parents=True, exist_ok=True) + + # The AVD must be named exactly like this, otherwise upstream test code can't find it. + avd_name = f"mozemulator-android{EMULATOR_ANDROID_VERSION}-x86_64" + print(f"Creating AVD: {avd_name}") + subprocess.run( + [str(avdmanager), "create", "avd", "-n", avd_name, "-k", system_image], + # At the end of this command it will prompt the user for a custom + # configuration, inputting "no" will let it know we don't want it. + input="no\n", + text=True, + stdout=subprocess.DEVNULL, + check=False, + ) + + avd_source = Path.home() / ".android" / "avd" + shutil.move(str(avd_source), str(android_device_dir)) + print(f"Moved AVD -> {android_device_dir / 'avd'}") + + +def before_script( + mozharness_url: str, + minidump_stackwalk_url: str, + minidump_stackwalk_sha256: str, + moz_fetches_dir: str, + sha256sums_url: str = "", + android_sdk_repack_url: str = "", +) -> None: + check_sha256sums_url(sha256sums_url) + download_mozharness(mozharness_url) + download_minidump_stackwalk( + minidump_stackwalk_url, moz_fetches_dir, minidump_stackwalk_sha256 + ) + + if android_sdk_repack_url: + setup_android_sdk( + moz_fetches_dir, + android_sdk_repack_url, + ) + + +if __name__ == "__main__": + before_script( + mozharness_url=os.environ["MOZHARNESS_URL"], + minidump_stackwalk_url=os.environ["MINIDUMP_STACKWALK_URL"], + minidump_stackwalk_sha256=os.environ.get("MINIDUMP_STACKWALK_SHA256", ""), + moz_fetches_dir=os.environ["MOZ_FETCHES_DIR"], + sha256sums_url=os.environ.get("SHA256SUMS_URL", ""), + # Android-only stuff. + android_sdk_repack_url=os.environ.get("ANDROID_SDK_REPACK_URL", ""), + ) ===================================== .gitlab/test_marionette.yml ===================================== @@ -0,0 +1,39 @@ +.marionette-desktop: + stage: test-marionette + script: + - > + python3 "$(pwd)/mozharness/scripts/marionette.py" + --config-file "$(pwd)/mozharness/configs/marionette/prod_config.py" + --installer-url "$INSTALLER_URL" + --test-packages-url "$ARTIFACTS_URL/target.test_packages.json" + --tag tor + --headless + +debian-x86_64_marionette: + extends: + - .debian-x86_64 + - .marionette-desktop + +macos-x86_64_marionette: + extends: + - .macos-x86_64 + - .marionette-desktop + +windows-x86_64_marionette: + extends: + - .windows-x86_64 + - .marionette-desktop + +android-x86_64_marionette: + extends: .android-x86_64 + stage: test-marionette + script: + - > + python3 "$(pwd)/mozharness/scripts/android_emulator_unittest.py" + --config-file "$(pwd)/mozharness/configs/android/android_common.py" + --config-file "$(pwd)/mozharness/configs/android/android14-x86_64.py" + --test-suite=marionette + --installer-url "$INSTALLER_URL" + --test-packages-url "$ARTIFACTS_URL/target.test_packages.json" + --package-name "$PACKAGE_NAME" + --tag tor View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/... -- View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/... You're receiving this email because of your account on gitlab.torproject.org. Manage all notifications: https://gitlab.torproject.org/-/profile/notifications | Help: https://gitlab.torproject.org/help