richard pushed to branch main at The Tor Project / Applications / tor-browser-build Commits: 7789b280 by Pier Angelo Vendrame at 2024-05-07T10:41:54+00:00 Bug 41001: Change some config files to make automation easier. - - - - - cbbcdf08 by Pier Angelo Vendrame at 2024-05-07T10:41:54+00:00 Bug 41001: Refactor fetch_allowed_addons.py. In this commit we change this file to be able to use it as a Python module from other scripts. Also, we lint it with black. - - - - - 51c8ccd3 by Pier Angelo Vendrame at 2024-05-07T10:41:54+00:00 Bug 41001: Refactored fetch-changelog.py. Also, added Zstandard to the possible updates. - - - - - 70539485 by Pier Angelo Vendrame at 2024-05-07T10:41:54+00:00 Bug 41001: Renamed fetch-changelog.py. After the refactor, fetch-changelog.py can be used as a Python module, but to do so we need to replace the dash in its name with something else (e.g., an underscore). - - - - - 48d9469a by Pier Angelo Vendrame at 2024-05-07T10:41:54+00:00 Bug 41001: Refactored fetch-manual.py. Allow the script to possibly run as a Python module. - - - - - 8e155939 by Pier Angelo Vendrame at 2024-05-07T10:41:54+00:00 Bug 41001: Renamed fetch-manual.py to update_manual.py. This allows us to import it in other Python scripts. - - - - - 9f583c64 by Pier Angelo Vendrame at 2024-05-07T10:41:54+00:00 Bug 41001: Add a release preparation script. - - - - - a34dcb00 by Pier Angelo Vendrame at 2024-05-07T10:41:54+00:00 Bug 41001: Added a checklist to the relprep MR template. Since the release preparation is becoming an almost fully automated procedure, it is necessary to be more careful during the review. This new checklist in the release preparation MR should help to spot any errors. - - - - - 16 changed files: - + .gitattributes - .gitlab/issue_templates/Release Prep - Mullvad Browser Alpha.md - .gitlab/issue_templates/Release Prep - Tor Browser Alpha.md - .gitlab/merge_request_templates/relprep.md - projects/browser/config - projects/firefox/config - projects/go/config - projects/openssl/config - rbm.conf - tools/.gitignore - − tools/fetch-changelogs.py - − tools/fetch-manual.py - tools/fetch_allowed_addons.py - + tools/fetch_changelogs.py - + tools/relprep.py - + tools/update_manual.py Changes: ===================================== .gitattributes ===================================== @@ -0,0 +1 @@ +projects/browser/allowed_addons.json -diff ===================================== .gitlab/issue_templates/Release Prep - Mullvad Browser Alpha.md ===================================== @@ -67,14 +67,14 @@ Mullvad Browser Alpha (and Nightly) are on the `main` branch - [ ] Update `ChangeLog-MB.txt` - [ ] Ensure `ChangeLog-MB.txt` is sync'd between alpha and stable branches - [ ] Check the linked issues: ask people to check if any are missing, remove the not fixed ones - - [ ] Run `./tools/fetch-changelogs.py $(ISSUE_NUMBER) --date $date $updateArgs` + - [ ] Run `./tools/fetch_changelogs.py $(ISSUE_NUMBER) --date $date $updateArgs` - Make sure you have `requests` installed (e.g., `apt install python3-requests`) - The first time you run this script you will need to generate an access token; the script will guide you - `$updateArgs` should be these arguments, depending on what you actually updated: - [ ] `--firefox` (be sure to include esr at the end if needed, which is usually the case) - [ ] `--no-script` - [ ] `--ublock` - - E.g., `./tools/fetch-changelogs.py 41029 --date 'December 19 2023' --firefox 115.6.0esr --no-script 11.4.29 --ublock 1.54.0` + - E.g., `./tools/fetch_changelogs.py 41029 --date 'December 19 2023' --firefox 115.6.0esr --no-script 11.4.29 --ublock 1.54.0` - `--date $date` is optional, if omitted it will be the date on which you run the command - [ ] Copy the output of the script to the beginning of `ChangeLog-MB.txt` and adjust its output - [ ] Open MR with above changes, using the template for release preparations ===================================== .gitlab/issue_templates/Release Prep - Tor Browser Alpha.md ===================================== @@ -78,6 +78,10 @@ Tor Browser Alpha (and Nightly) are on the `main` branch - [ ] Check for zlib updates here: https://github.com/madler/zlib/releases - [ ] **(Optional)** If new tag available, update `projects/zlib/config` - [ ] `version` : update to next release tag + - [ ] Check for Zstandard updates here: https://github.com/facebook/zstd/releases + - [ ] **(Optional)** If new tag available, update `projects/zstd/config` + - [ ] `version` : update to next release tag + - [ ] `git_hash`: update to the commit corresponding to the tag (we don't check signatures for Zstandard) - [ ] Check for tor updates here : https://gitlab.torproject.org/tpo/core/tor/-/tags - [ ] ***(Optional)*** Update `projects/tor/config` - [ ] `version` : update to latest `-alpha` tag or release tag if newer (ping dgoulet or ahf if unsure) @@ -86,18 +90,17 @@ Tor Browser Alpha (and Nightly) are on the `main` branch - [ ] ***(Optional)*** Update `projects/go/config` - [ ] `version` : update go version - [ ] `input_files/sha256sum` for `go` : update sha256sum of archive (sha256 sums are displayed on the go download page) - - [ ] Check for manual updates by running (from `tor-browser-build` root): `./tools/fetch-manual.py` + - [ ] Check for manual updates by running (from `tor-browser-build` root): `./tools/update_manual.py` - [ ] ***(Optional)*** If new version is available: - [ ] Upload the downloaded `manual_$PIPELINEID.zip` file to `tb-build-02.torproject.org` + - The script will tell if it's necessary to - [ ] Deploy to `tb-builder`'s `public_html` directory: - `sudo -u tb-builder cp manual_$PIPELINEID.zip ~tb-builder/public_html/.` - - [ ] Update `projects/manual/config`: - - [ ] Change the `version` to `$PIPELINEID` - - [ ] Update `sha256sum` in the `input_files` section + - [ ] Add `projects/manual/config` to the stage area if the script updated it. - [ ] Update `ChangeLog-TBB.txt` - [ ] Ensure `ChangeLog-TBB.txt` is sync'd between alpha and stable branches - [ ] Check the linked issues: ask people to check if any are missing, remove the not fixed ones - - [ ] Run `./tools/fetch-changelogs.py $(ISSUE_NUMBER) --date $date $updateArgs` + - [ ] Run `./tools/fetch_changelogs.py $(ISSUE_NUMBER) --date $date $updateArgs` - Make sure you have `requests` installed (e.g., `apt install python3-requests`) - The first time you run this script you will need to generate an access token; the script will guide you - `$updateArgs` should be these arguments, depending on what you actually updated: @@ -106,8 +109,9 @@ Tor Browser Alpha (and Nightly) are on the `main` branch - [ ] `--no-script` - [ ] `--openssl` - [ ] `--zlib` + - [ ] `--zstd` - [ ] `--go` - - E.g., `./tools/fetch-changelogs.py 41028 --date 'December 19 2023' --firefox 115.6.0esr --tor 0.4.8.10 --no-script 11.4.29 --zlib 1.3 --go 1.21.5 --openssl 3.0.12` + - E.g., `./tools/fetch_changelogs.py 41028 --date 'December 19 2023' --firefox 115.6.0esr --tor 0.4.8.10 --no-script 11.4.29 --zlib 1.3 --go 1.21.5 --openssl 3.0.12` - `--date $date` is optional, if omitted it will be the date on which you run the command - [ ] Copy the output of the script to the beginning of `ChangeLog-TBB.txt` and adjust its output - [ ] Open MR with above changes, using the template for release preparations ===================================== .gitlab/merge_request_templates/relprep.md ===================================== @@ -1,10 +1,43 @@ -## Merge Info - -### Related Issues +## Related Issues - tor-browser-build#xxxxx - tor-browser-build#xxxxx +## Self-review + reviewer's template + +- [ ] `rbm.conf` updates: + - [ ] `var/torbrowser_version` + - [ ] `var/torbrowser_build`: should be `build1`, unless bumping a previous release preparation + - [ ] `var/browser_release_date`: must not be in the future when we start building + - [ ] `var/torbrowser_incremental_from` (not needed for Android-only releases) +- [ ] Tag updates: + - [ ] [Firefox](https://gitlab.torproject.org/tpo/applications/tor-browser/-/tags) + - [ ] Geckoview - should match Firefox + - [ ] [Firefox Android](https://gitlab.torproject.org/tpo/applications/firefox-android/-/tags) + - Tags might be speculative in the release preparation: i.e., they might not exist yet. +- [ ] Addon updates: + - [ ] [NoScript](https://addons.mozilla.org/en-US/firefox/addon/noscript/) + - [ ] [uBlock Origin](https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/) (Mullvad Browser only) + - [ ] [Mullvad Browser Extension](https://github.com/mullvad/browser-extension/releases) (Mullvad Browser only) + - For AMO extension (NoScript and uBlock), updating the version in the URL is not enough, check that also a numeric ID from the URL has changed +- [ ] Tor and dependencies updates (Tor Browser only) + - [ ] [Tor](https://gitlab.torproject.org/tpo/core/tor/-/tags) + - [ ] [OpenSSL](https://www.openssl.org/source/): we stay on the latest LTS channel (currently 3.0.x) + - [ ] [zlib](https://github.com/madler/zlib/releases) + - [ ] [Zstandard](https://github.com/facebook/zstd/releases) (Android only, at least for now) + - [ ] [Go](https://go.dev/dl): avoid major updates, unless planned +- [ ] Manual version update (Tor Browser only, optional) +- [ ] Changelogs + - [ ] Changelogs must be in sync between stable and alpha + - [ ] Check the browser name + - [ ] Check the version + - [ ] Check the release date + - [ ] Check we include only the platform we're releasing for (e.g., no Android in desktop-only releases) + - [ ] Check all the updates from above are reported in the changelogs + - [ ] Check for major errors + - If you find errors such as platform or category (build system) please adjust the issue label accordingly + - You can run `tools/relprep.py --only-changelogs --date $date $version` to update only the changelogs + ## Review ### Request Reviewer ===================================== projects/browser/config ===================================== @@ -111,7 +111,7 @@ input_files: name: ublock-origin sha256sum: 9928e79a52cecf7cfa231fdb0699c7d7a427660d94eb10d711ed5a2f10d2eb89 enable: '[% c("var/mullvad-browser") %]' - - URL: https://github.com/mullvad/browser-extension/releases/download/v0.9.0-firefo... + - URL: https://cdn.mullvad.net/browser-extension/0.9.0/mullvad-browser-extension-0.... name: mullvad-extension sha256sum: 65bf235aa1015054ae0a54a40c5a663e67fe1d0f0799e7b4726f98cccc7f3eab enable: '[% c("var/mullvad-browser") %]' ===================================== projects/firefox/config ===================================== @@ -17,7 +17,8 @@ var: firefox_platform_version: 115.10.0 firefox_version: '[% c("var/firefox_platform_version") %]esr' browser_series: '13.5' - browser_branch: '[% c("var/browser_series") %]-1' + browser_rebase: 1 + browser_branch: '[% c("var/browser_series") %]-[% c("var/browser_rebase") %]' browser_build: 2 branding_directory_prefix: 'tb' copyright_year: '[% exec("git show -s --format=%ci").remove("-.*") %]' ===================================== projects/go/config ===================================== @@ -1,11 +1,14 @@ # vim: filetype=yaml sw=2 -version: '[% IF c("var/use_go_1_20") %]1.20.14[% ELSE %]1.21.9[% END %]' +# When Windows 7 goes EOL, just update this field +version: '[% IF c("var/use_go_1_20") %][% c("var/go_1_20") %][% ELSE %][% c("var/go_1_21") %][% END %]' filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.[% c("compress_tar") %]' container: use_container: 1 var: use_go_1_20: 0 + go_1_21: 1.21.9 + go_1_20: 1.20.14 setup: | mkdir -p /var/tmp/dist tar -C /var/tmp/dist -xf $rootdir/[% c("go_tarfile") %] @@ -119,13 +122,11 @@ input_files: - name: '[% c("var/compiler") %]' project: '[% c("var/compiler") %]' enable: '[% ! c("var/linux") %]' - - URL: 'https://go.dev/dl/go[% c("version") %].src.tar.gz' - # 1.21 series + - URL: 'https://go.dev/dl/go[% c("var/go_1_21") %].src.tar.gz' name: go sha256sum: 58f0c5ced45a0012bce2ff7a9df03e128abcc8818ebabe5027bb92bafe20e421 enable: '[% !c("var/use_go_1_20") %]' - - URL: 'https://go.dev/dl/go[% c("version") %].src.tar.gz' - # 1.20 series + - URL: 'https://go.dev/dl/go[% c("var/go_1_20") %].src.tar.gz' name: go sha256sum: 1aef321a0e3e38b7e91d2d7eb64040666cabdcc77d383de3c9522d0d69b67f4e enable: '[% c("var/use_go_1_20") %]' ===================================== projects/openssl/config ===================================== @@ -34,3 +34,4 @@ input_files: project: '[% c("var/compiler") %]' - URL: 'https://www.openssl.org/source/openssl-[% c("version") %].tar.gz' sha256sum: 88525753f79d3bec27d2fa7c66aa0b92b3aa9498dafd93d7cfa4b3780cdae313 + name: openssl ===================================== rbm.conf ===================================== @@ -75,16 +75,16 @@ buildconf: var: torbrowser_version: '13.5a7' torbrowser_build: 'build2' - torbrowser_incremental_from: - - '13.5a6' - - '13.5a5' - - '13.5a4' # This should be the date of when the build is started. For the build # to be reproducible, browser_release_date should always be in the past. browser_release_date: '2024/04/25 12:00:00' browser_release_date_timestamp: '[% USE date; date.format(c("var/browser_release_date"), "%s") %]' updater_enabled: 1 build_mar: 1 + torbrowser_incremental_from: + - '13.5a6' + - '13.5a5' + - '13.5a4' mar_channel_id: '[% c("var/projectname") %]-torproject-[% c("var/channel") %]' # By default, we sort the list of installed packages. This allows sharing ===================================== tools/.gitignore ===================================== @@ -1,3 +1,4 @@ _repackaged +__pycache__ .changelogs_token local ===================================== tools/fetch-changelogs.py deleted ===================================== @@ -1,276 +0,0 @@ -#!/usr/bin/env python3 -import argparse -from datetime import datetime -import enum -from pathlib import Path -import re -import sys - -import requests - - -GITLAB = "https://gitlab.torproject.org" -API_URL = f"{GITLAB}/api/v4" -PROJECT_ID = 473 - -is_mb = False -project_order = { - "tor-browser-spec": 0, - # Leave 1 free, so we can redefine mullvad-browser when needed. - "tor-browser": 2, - "tor-browser-build": 3, - "mullvad-browser": 4, - "rbm": 5, -} - - -class EntryType(enum.IntFlag): - UPDATE = 0 - ISSUE = 1 - - -class Platform(enum.IntFlag): - WINDOWS = 8 - MACOS = 4 - LINUX = 2 - ANDROID = 1 - DESKTOP = 8 | 4 | 2 - ALL_PLATFORMS = 8 | 4 | 2 | 1 - - -class ChangelogEntry: - def __init__(self, type_, platform, num_platforms, is_build): - self.type = type_ - self.platform = platform - self.num_platforms = num_platforms - self.is_build = is_build - - def get_platforms(self): - if self.platform == Platform.ALL_PLATFORMS: - return "All Platforms" - platforms = [] - if self.platform & Platform.WINDOWS: - platforms.append("Windows") - if self.platform & Platform.MACOS: - platforms.append("macOS") - if self.platform & Platform.LINUX: - platforms.append("Linux") - if self.platform & Platform.ANDROID: - platforms.append("Android") - return " + ".join(platforms) - - def __lt__(self, other): - if self.type != other.type: - return self.type < other.type - if self.type == EntryType.UPDATE: - # Rely on sorting being stable on Python - return False - if self.project == other.project: - return self.number < other.number - return project_order[self.project] < project_order[other.project] - - -class UpdateEntry(ChangelogEntry): - def __init__(self, name, version): - if name == "Firefox" and not is_mb: - platform = Platform.DESKTOP - num_platforms = 3 - elif name == "GeckoView": - platform = Platform.ANDROID - num_platforms = 3 - else: - platform = Platform.ALL_PLATFORMS - num_platforms = 4 - super().__init__( - EntryType.UPDATE, platform, num_platforms, name == "Go" - ) - self.name = name - self.version = version - - def __str__(self): - return f"Updated {self.name} to {self.version}" - - -class Issue(ChangelogEntry): - def __init__(self, j): - self.title = j["title"] - self.project, self.number = ( - j["references"]["full"].rsplit("/", 2)[-1].split("#") - ) - self.number = int(self.number) - platform = 0 - num_platforms = 0 - if "Desktop" in j["labels"]: - platform = Platform.DESKTOP - num_platforms += 3 - else: - if "Windows" in j["labels"]: - platform |= Platform.WINDOWS - num_platforms += 1 - if "MacOS" in j["labels"]: - platform |= Platform.MACOS - num_platforms += 1 - if "Linux" in j["labels"]: - platform |= Platform.LINUX - num_platforms += 1 - if "Android" in j["labels"]: - if is_mb and num_platforms == 0: - raise Exception( - f"Android-only issue on Mullvad Browser: {j['references']['full']}!" - ) - elif not is_mb: - platform |= Platform.ANDROID - num_platforms += 1 - if not platform or (is_mb and platform == Platform.DESKTOP): - platform = Platform.ALL_PLATFORMS - num_platforms = 4 - is_build = "Build System" in j["labels"] - super().__init__(EntryType.ISSUE, platform, num_platforms, is_build) - - def __str__(self): - return f"Bug {self.number}: {self.title} [{self.project}]" - - -def sorted_issues(issues): - issues = [sorted(v) for v in issues.values()] - return sorted( - issues, - key=lambda group: (group[0].num_platforms << 8) | group[0].platform, - reverse=True, - ) - - -parser = argparse.ArgumentParser() -parser.add_argument("issue_version") -parser.add_argument("--date", help="The date of the release") -parser.add_argument("--firefox", help="New Firefox version (if we rebased)") -parser.add_argument("--tor", help="New Tor version (if updated)") -parser.add_argument("--no-script", help="New NoScript version (if updated)") -parser.add_argument("--openssl", help="New OpenSSL version (if updated)") -parser.add_argument("--ublock", help="New uBlock version (if updated)") -parser.add_argument("--zlib", help="New zlib version (if updated)") -parser.add_argument("--go", help="New Go version (if updated)") -args = parser.parse_args() - -if not args.issue_version: - parser.print_help() - sys.exit(1) - -token_file = Path(__file__).parent / ".changelogs_token" -if not token_file.exists(): - print( - f"Please add your personal GitLab token (with 'read_api' scope) to {token_file}" - ) - print( - f"Please go to {GITLAB}/-/profile/personal_access_tokens and generate it." - ) - token = input("Please enter the new token: ").strip() - if not token: - print("Invalid token!") - sys.exit(2) - with token_file.open("w") as f: - f.write(token) -with token_file.open() as f: - token = f.read().strip() -headers = {"PRIVATE-TOKEN": token} - -version = args.issue_version -r = requests.get( - f"{API_URL}/projects/{PROJECT_ID}/issues?labels=Release Prep", - headers=headers, -) -if r.status_code == 401: - print("Unauthorized! Has your token expired?") - sys.exit(3) -issue = None -issues = [] -for i in r.json(): - if i["title"].find(version) != -1: - issues.append(i) -if len(issues) == 1: - issue = issues[0] -elif len(issues) > 1: - print("More than one matching issue found:") - for idx, i in enumerate(issues): - print(f" {idx + 1}) #{i['iid']} - {i['title']}") - print("Please use the issue id.") - sys.exit(4) -else: - iid = version - version = "CHANGEME!" - if iid[0] == "#": - iid = iid[1:] - try: - int(iid) - r = requests.get( - f"{API_URL}/projects/{PROJECT_ID}/issues?iids={iid}", - headers=headers, - ) - if r.ok and r.json(): - issue = r.json()[0] - version_match = re.search(r"\b[0-9]+\.[.0-9a]+\b", issue["title"]) - if version_match: - version = version_match.group() - except ValueError: - pass -if not issue: - print( - "Release preparation issue not found. Please make sure it has ~Release Prep." - ) - sys.exit(5) -if "Sponsor 131" in issue["labels"]: - is_mb = True - project_order["mullvad-browser"] = 1 -iid = issue["iid"] - -linked = {} -linked_build = {} - - -def add_entry(entry): - target = linked_build if entry.is_build else linked - if entry.platform not in target: - target[entry.platform] = [] - target[entry.platform].append(entry) - - -if args.firefox: - add_entry(UpdateEntry("Firefox", args.firefox)) - if not is_mb: - add_entry(UpdateEntry("GeckoView", args.firefox)) -if args.tor and not is_mb: - add_entry(UpdateEntry("Tor", args.tor)) -if args.no_script: - add_entry(UpdateEntry("NoScript", args.no_script)) -if not is_mb: - if args.openssl: - add_entry(UpdateEntry("OpenSSL", args.openssl)) - if args.zlib: - add_entry(UpdateEntry("zlib", args.zlib)) - if args.go: - add_entry(UpdateEntry("Go", args.go)) -elif args.ublock: - add_entry(UpdateEntry("uBlock Origin", args.ublock)) - -r = requests.get( - f"{API_URL}/projects/{PROJECT_ID}/issues/{iid}/links", headers=headers -) -for i in r.json(): - add_entry(Issue(i)) - -linked = sorted_issues(linked) -linked_build = sorted_issues(linked_build) - -name = "Mullvad" if is_mb else "Tor" -date = args.date if args.date else datetime.now().strftime("%B %d %Y") -print(f"{name} Browser {version} - {date}") -for issues in linked: - print(f" * {issues[0].get_platforms()}") - for i in issues: - print(f" * {i}") -if linked_build: - print(" * Build System") - for issues in linked_build: - print(f" * {issues[0].get_platforms()}") - for i in issues: - print(f" * {i}") ===================================== tools/fetch-manual.py deleted ===================================== @@ -1,83 +0,0 @@ -#!/usr/bin/env python3 -import hashlib -from pathlib import Path -import sys - -import requests -import yaml - - -GITLAB = "https://gitlab.torproject.org" -API_URL = f"{GITLAB}/api/v4" -PROJECT_ID = 23 -REF_NAME = "main" - - -token_file = Path(__file__).parent / ".changelogs_token" -if not token_file.exists(): - print("This scripts uses the same access token as fetch-changelog.py.") - print("However, the file has not been found.") - print( - "Please run fetch-changelog.py to get the instructions on how to " - "generate it." - ) - sys.exit(1) -with token_file.open() as f: - headers = {"PRIVATE-TOKEN": f.read().strip()} - -r = requests.get(f"{API_URL}/projects/{PROJECT_ID}/jobs", headers=headers) -if r.status_code == 401: - print("Unauthorized! Maybe the token has expired.") - sys.exit(2) -found = False -for job in r.json(): - if job["ref"] != REF_NAME: - continue - for art in job["artifacts"]: - if art["filename"] == "artifacts.zip": - found = True - break - if found: - break -if not found: - print("Cannot find a usable job.") - sys.exit(3) - -pipeline_id = job["pipeline"]["id"] -conf_file = Path(__file__).parent.parent / "projects/manual/config" -with conf_file.open() as f: - config = yaml.load(f, yaml.SafeLoader) -if int(config["version"]) == int(pipeline_id): - print( - "projects/manual/config is already using the latest pipeline. Nothing to do." - ) - sys.exit(0) - -manual_dir = Path(__file__).parent.parent / "out/manual" -manual_dir.mkdir(0o755, parents=True, exist_ok=True) -manual_file = manual_dir / f"manual_{pipeline_id}.zip" -sha256 = hashlib.sha256() -if manual_file.exists(): - with manual_file.open("rb") as f: - while chunk := f.read(8192): - sha256.update(chunk) - print("You already have the latest manual version in your out directory.") - print("Please update projects/manual/config to:") -else: - print("Downloading the new version of the manual...") - url = f"{API_URL}/projects/{PROJECT_ID}/jobs/artifacts/{REF_NAME}/download?job={job['name']}" - r = requests.get(url, headers=headers, stream=True) - # https://stackoverflow.com/a/16696317 - r.raise_for_status() - with manual_file.open("wb") as f: - for chunk in r.iter_content(chunk_size=8192): - f.write(chunk) - sha256.update(chunk) - print(f"File downloaded as {manual_file}.") - print( - "Please upload it to tb-build-02.torproject.org:~tb-builder/public_html/. and then update projects/manual/config:" - ) -sha256 = sha256.hexdigest() - -print(f"\tversion: {pipeline_id}") -print(f"\tSHA256: {sha256}") ===================================== tools/fetch_allowed_addons.py ===================================== @@ -5,33 +5,49 @@ import json import base64 import sys +NOSCRIPT = "{73a6fe31-595d-460b-a920-fcc0f8843232}" + + def fetch(x): - with urllib.request.urlopen(x) as response: - return response.read() + with urllib.request.urlopen(x) as response: + return response.read() + def find_addon(addons, addon_id): - results = addons['results'] - for x in results: - addon = x['addon'] - if addon['guid'] == addon_id: - return addon - sys.exit("Error: cannot find addon " + addon_id) + results = addons["results"] + for x in results: + addon = x["addon"] + if addon["guid"] == addon_id: + return addon + def fetch_and_embed_icons(addons): - results = addons['results'] - for x in results: - addon = x['addon'] - icon_data = fetch(addon['icon_url']) - addon['icon_url'] = 'data:image/png;base64,' + str(base64.b64encode(icon_data), 'utf8') + results = addons["results"] + for x in results: + addon = x["addon"] + icon_data = fetch(addon["icon_url"]) + addon["icon_url"] = "data:image/png;base64," + str( + base64.b64encode(icon_data), "utf8" + ) + + +def fetch_allowed_addons(amo_collection=None): + if amo_collection is None: + amo_collection = "83a9cccfe6e24a34bd7b155ff9ee32" + url = f"https://services.addons.mozilla.org/api/v4/accounts/account/mozilla/collections/{amo_collection}/addons/" + data = json.loads(fetch(url)) + fetch_and_embed_icons(data) + data["results"].sort(key=lambda x: x["addon"]["guid"]) + return data + def main(argv): - amo_collection = argv[0] if argv else '83a9cccfe6e24a34bd7b155ff9ee32' - url = 'https://services.addons.mozilla.org/api/v4/accounts/account/mozilla/collecti...' + amo_collection + '/addons/' - data = json.loads(fetch(url)) - fetch_and_embed_icons(data) - data['results'].sort(key=lambda x: x['addon']['guid']) - find_addon(data, '{73a6fe31-595d-460b-a920-fcc0f8843232}') # Check that NoScript is present - print(json.dumps(data, indent=2, ensure_ascii=False)) + data = fetch_allowed_addons(argv[0] if len(argv) > 1 else None) + # Check that NoScript is present + if find_addon(data, NOSCRIPT) is None: + sys.exit("Error: cannot find NoScript.") + print(json.dumps(data, indent=2, ensure_ascii=False)) + if __name__ == "__main__": - main(sys.argv[1:]) + main(sys.argv[1:]) ===================================== tools/fetch_changelogs.py ===================================== @@ -0,0 +1,366 @@ +#!/usr/bin/env python3 +import argparse +from datetime import datetime +import enum +from pathlib import Path +import re +import sys + +import requests + + +GITLAB = "https://gitlab.torproject.org" +API_URL = f"{GITLAB}/api/v4" +PROJECT_ID = 473 +AUTH_HEADER = "PRIVATE-TOKEN" + + +class EntryType(enum.IntFlag): + UPDATE = 0 + ISSUE = 1 + + +class Platform(enum.IntFlag): + WINDOWS = 8 + MACOS = 4 + LINUX = 2 + ANDROID = 1 + DESKTOP = 8 | 4 | 2 + ALL_PLATFORMS = 8 | 4 | 2 | 1 + + +class ChangelogEntry: + def __init__(self, type_, platform, num_platforms, is_build, is_mb): + self.type = type_ + self.platform = platform + self.num_platforms = num_platforms + self.is_build = is_build + self.project_order = { + "tor-browser-spec": 0, + # Leave 1 free, so we can redefine mullvad-browser when needed. + "tor-browser": 2, + "tor-browser-build": 3, + "mullvad-browser": 1 if is_mb else 4, + "rbm": 5, + } + + def get_platforms(self): + if self.platform == Platform.ALL_PLATFORMS: + return "All Platforms" + platforms = [] + if self.platform & Platform.WINDOWS: + platforms.append("Windows") + if self.platform & Platform.MACOS: + platforms.append("macOS") + if self.platform & Platform.LINUX: + platforms.append("Linux") + if self.platform & Platform.ANDROID: + platforms.append("Android") + return " + ".join(platforms) + + def __lt__(self, other): + if self.num_platforms != other.num_platforms: + return self.num_platforms > other.num_platforms + if self.platform != other.platform: + return self.platform > other.platform + if self.type != other.type: + return self.type < other.type + if self.type == EntryType.UPDATE: + # Rely on sorting being stable on Python + return False + if self.project == other.project: + return self.number < other.number + return ( + self.project_order[self.project] + < self.project_order[other.project] + ) + + +class UpdateEntry(ChangelogEntry): + def __init__(self, name, version, is_mb): + if name == "Firefox" and not is_mb: + platform = Platform.DESKTOP + num_platforms = 3 + elif name == "GeckoView" or name == "Zstandard": + platform = Platform.ANDROID + num_platforms = 1 + else: + platform = Platform.ALL_PLATFORMS + num_platforms = 4 + super().__init__( + EntryType.UPDATE, platform, num_platforms, name == "Go", is_mb + ) + self.name = name + self.version = version + + def __str__(self): + return f"Updated {self.name} to {self.version}" + + +class Issue(ChangelogEntry): + def __init__(self, j, is_mb): + self.title = j["title"] + self.project, self.number = ( + j["references"]["full"].rsplit("/", 2)[-1].split("#") + ) + self.number = int(self.number) + platform = 0 + num_platforms = 0 + if "Desktop" in j["labels"]: + platform = Platform.DESKTOP + num_platforms += 3 + else: + if "Windows" in j["labels"]: + platform |= Platform.WINDOWS + num_platforms += 1 + if "MacOS" in j["labels"]: + platform |= Platform.MACOS + num_platforms += 1 + if "Linux" in j["labels"]: + platform |= Platform.LINUX + num_platforms += 1 + if "Android" in j["labels"]: + if is_mb and num_platforms == 0: + raise Exception( + f"Android-only issue on Mullvad Browser: {j['references']['full']}!" + ) + elif not is_mb: + platform |= Platform.ANDROID + num_platforms += 1 + if not platform or (is_mb and platform == Platform.DESKTOP): + platform = Platform.ALL_PLATFORMS + num_platforms = 4 + is_build = "Build System" in j["labels"] + super().__init__( + EntryType.ISSUE, platform, num_platforms, is_build, is_mb + ) + + def __str__(self): + return f"Bug {self.number}: {self.title} [{self.project}]" + + +class ChangelogBuilder: + + def __init__(self, auth_token, issue_or_version, is_mullvad=None): + self.headers = {AUTH_HEADER: auth_token} + self._find_issue(issue_or_version, is_mullvad) + + def _find_issue(self, issue_or_version, is_mullvad): + self.version = None + if issue_or_version[0] == "#": + self._fetch_issue(issue_or_version[1:], is_mullvad) + return + labels = "Release Prep" + if is_mullvad: + labels += ",Sponsor 131" + elif not is_mullvad and is_mullvad is not None: + labels += "¬[labels]=Sponsor 131" + r = requests.get( + f"{API_URL}/projects/{PROJECT_ID}/issues?labels={labels}&search={issue_or_version}&in=title", + headers=self.headers, + ) + r.raise_for_status() + issues = r.json() + if len(issues) == 1: + self.version = issue_or_version + self._set_issue(issues[0], is_mullvad) + elif len(issues) > 1: + raise ValueError( + "Multiple issues found, try to specify the browser." + ) + else: + self._fetch_issue(issue_or_version, is_mullvad) + + def _fetch_issue(self, number, is_mullvad): + try: + # Validate the string to be an integer + number = int(number) + except ValueError: + # This is called either as a last chance, or because we + # were given "#", so this error should be good. + raise ValueError("Issue not found") + r = requests.get( + f"{API_URL}/projects/{PROJECT_ID}/issues?iids[]={number}", + headers=self.headers, + ) + r.raise_for_status() + issues = r.json() + if len(issues) != 1: + # It should be only 0, since we used the number... + raise ValueError("Issue not found") + self._set_issue(issues[0], is_mullvad) + + def _set_issue(self, issue, is_mullvad): + has_s131 = "Sponsor 131" in issue["labels"] + if is_mullvad is not None and is_mullvad != has_s131: + raise ValueError( + "Inconsistency detected: a browser was explicitly specified, but the issue does not have the correct labels." + ) + self.issue_id = issue["iid"] + self.is_mullvad = has_s131 + + if self.version is None: + version_match = re.search(r"\b[0-9]+\.[.0-9a]+\b", issue["title"]) + if version_match: + self.version = version_match.group() + + def create(self, **kwargs): + self._find_linked() + self._add_updates(kwargs) + self._sort_issues() + name = "Mullvad" if self.is_mullvad else "Tor" + date = ( + kwargs["date"] + if kwargs.get("date") + else datetime.now().strftime("%B %d %Y") + ) + text = f"{name} Browser {self.version} - {date}\n" + prev_platform = "" + for issue in self.issues: + platform = issue.get_platforms() + if platform != prev_platform: + text += f" * {platform}\n" + prev_platform = platform + text += f" * {issue}\n" + if self.issues_build: + text += " * Build System\n" + prev_platform = "" + for issue in self.issues_build: + platform = issue.get_platforms() + if platform != prev_platform: + text += f" * {platform}\n" + prev_platform = platform + text += f" * {issue}\n" + return text + + def _find_linked(self): + self.issues = [] + self.issues_build = [] + + r = requests.get( + f"{API_URL}/projects/{PROJECT_ID}/issues/{self.issue_id}/links", + headers=self.headers, + ) + for i in r.json(): + self._add_issue(i) + + def _add_issue(self, gitlab_data): + self._add_entry(Issue(gitlab_data, self.is_mullvad)) + + def _add_entry(self, entry): + target = self.issues_build if entry.is_build else self.issues + target.append(entry) + + def _add_updates(self, updates): + names = { + "Firefox": "firefox", + } + if not self.is_mullvad: + names.update( + { + "GeckoView": "firefox", + "Tor": "tor", + "NoScript": "noscript", + "OpenSSL": "openssl", + "zlib": "zlib", + "Zstandard": "zstd", + "Go": "go", + } + ) + else: + names.update( + { + "Mullvad Browser Extension": "mb_extension", + "uBlock Origin": "ublock", + } + ) + for name, key in names.items(): + self._maybe_add_update(name, updates, key) + + def _maybe_add_update(self, name, updates, key): + if updates.get(key): + self._add_entry(UpdateEntry(name, updates[key], self.is_mullvad)) + + def _sort_issues(self): + self.issues.sort() + self.issues_build.sort() + + +def load_token(test=True, interactive=True): + token_path = Path(__file__).parent / ".changelogs_token" + + if token_path.exists(): + with token_path.open() as f: + token = f.read().strip() + elif interactive: + print( + f"Please add your personal GitLab token (with 'read_api' scope) to {token_path}" + ) + print( + f"Please go to {GITLAB}/-/profile/personal_access_tokens and generate it." + ) + token = input("Please enter the new token: ").strip() + if not token: + raise ValueError("Invalid token!") + with token_path.open("w") as f: + f.write(token) + if test: + r = requests.get(f"{API_URL}/version", headers={AUTH_HEADER: token}) + if r.status_code == 401: + raise ValueError("The loaded or provided token does not work.") + return token + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("issue_version") + parser.add_argument("-d", "--date", help="The date of the release") + parser.add_argument( + "-b", "--browser", choices=["tor-browser", "mullvad-browser"] + ) + parser.add_argument( + "--firefox", help="New Firefox version (if we rebased)" + ) + parser.add_argument("--tor", help="New Tor version (if updated)") + parser.add_argument( + "--noscript", "--no-script", help="New NoScript version (if updated)" + ) + parser.add_argument("--openssl", help="New OpenSSL version (if updated)") + parser.add_argument("--zlib", help="New zlib version (if updated)") + parser.add_argument("--zstd", help="New zstd version (if updated)") + parser.add_argument("--go", help="New Go version (if updated)") + parser.add_argument( + "--mb-extension", + help="New Mullvad Browser Extension version (if updated)", + ) + parser.add_argument("--ublock", help="New uBlock version (if updated)") + args = parser.parse_args() + + if not args.issue_version: + parser.print_help() + sys.exit(1) + + try: + token = load_token() + except ValueError: + print( + "Invalid authentication token. Maybe has it expired?", + file=sys.stderr, + ) + sys.exit(2) + is_mullvad = args.browser == "mullvad-browser" if args.browser else None + cb = ChangelogBuilder(token, args.issue_version, is_mullvad) + print( + cb.create( + date=args.date, + firefox=args.firefox, + tor=args.tor, + noscript=args.noscript, + openssl=args.openssl, + zlib=args.zlib, + zstd=args.zstd, + go=args.go, + mb_extension=args.mb_extension, + ublock=args.ublock, + ) + ) ===================================== tools/relprep.py ===================================== @@ -0,0 +1,761 @@ +#!/usr/bin/env python3 +import argparse +from collections import namedtuple +import configparser +from datetime import datetime, timezone +from hashlib import sha256 +import json +import locale +import logging +from pathlib import Path +import re +import sys +import xml.etree.ElementTree as ET + +from git import Repo +import requests +import ruamel.yaml + +from fetch_allowed_addons import NOSCRIPT, fetch_allowed_addons, find_addon +import fetch_changelogs +from update_manual import update_manual + + +logger = logging.getLogger(__name__) + + +ReleaseTag = namedtuple("ReleaseTag", ["tag", "version"]) + + +class Version: + def __init__(self, v): + self.v = v + m = re.match(r"(\d+\.\d+)([a\.])?(\d*)", v) + self.major = m.group(1) + self.minor = int(m.group(3)) if m.group(3) else 0 + self.is_alpha = m.group(2) == "a" + self.channel = "alpha" if self.is_alpha else "release" + + def __str__(self): + return self.v + + def __lt__(self, other): + if self.major != other.major: + # String comparison, but it should be fine until + # version 100 :) + return self.major < other.major + if self.is_alpha != other.is_alpha: + return self.is_alpha + # Same major, both alphas/releases + return self.minor < other.minor + + def __eq__(self, other): + return self.v == other.v + + def __hash__(self): + return hash(self.v) + + +def get_sorted_tags(repo): + return sorted( + [t.tag for t in repo.tags if t.tag], + key=lambda t: t.tagged_date, + reverse=True, + ) + + +def get_github_release(project, regex=""): + if regex: + regex = re.compile(regex) + url = f"https://github.com/{project}/releases.atom" + r = requests.get(url) + r.raise_for_status() + feed = ET.fromstring(r.text) + for entry in feed.findall("{http://www.w3.org/2005/Atom}entry"): + link = entry.find("{http://www.w3.org/2005/Atom}link").attrib["href"] + tag = link.split("/")[-1] + if regex: + m = regex.match(tag) + if m: + return m.group(1) + else: + return tag + + +class ReleasePreparation: + def __init__(self, repo_path, version, **kwargs): + logger.debug( + "Initializing. Version=%s, repo=%s, additional args=%s", + repo_path, + version, + kwargs, + ) + self.base_path = Path(repo_path) + self.repo = Repo(self.base_path) + + self.tor_browser = bool(kwargs.get("tor_browser", True)) + self.mullvad_browser = bool(kwargs.get("tor_browser", True)) + if not self.tor_browser and not self.mullvad_browser: + raise ValueError("Nothing to do") + self.android = kwargs.get("android", self.tor_browser) + if not self.tor_browser and self.android: + raise ValueError("Only Tor Browser supports Android") + + logger.debug( + "Tor Browser: %s; Mullvad Browser: %s; Android: %s", + self.tor_browser, + self.mullvad_browser, + self.android, + ) + + self.yaml = ruamel.yaml.YAML() + self.yaml.indent(mapping=2, sequence=4, offset=2) + self.yaml.width = 4096 + self.yaml.preserve_quotes = True + + self.version = Version(version) + + self.build_date = kwargs.get("build_date", datetime.now(timezone.utc)) + self.changelog_date = kwargs.get("changelog_date", self.build_date) + self.num_incrementals = kwargs.get("num_incrementals", 3) + + self.get_last_releases() + + logger.info("Checking you have a working GitLab token.") + self.gitlab_token = fetch_changelogs.load_token() + + def run(self): + self.branch_sanity_check() + + self.update_firefox() + if self.android: + self.update_firefox_android() + self.update_translations() + self.update_addons() + + if self.tor_browser: + self.update_tor() + self.update_openssl() + self.update_zlib() + if self.android: + self.update_zstd() + self.update_go() + self.update_manual() + + self.update_changelogs() + self.update_rbm_conf() + + logger.info("Release preparation complete!") + + def branch_sanity_check(self): + logger.info("Checking you are on an updated branch.") + + remote = None + for rem in self.repo.remotes: + if "tpo/applications/tor-browser-build" in rem.url: + remote = rem + break + if remote is None: + raise RuntimeError("Cannot find the tpo/applications remote.") + remote.fetch() + + branch_name = ( + "main" if self.version.is_alpha else f"maint-{self.version.major}" + ) + branch = remote.refs[branch_name] + base = self.repo.merge_base(self.repo.head, branch)[0] + if base != branch.commit: + raise RuntimeError( + "You are not working on a branch descending from " + f"f{branch_name}. " + "Please checkout the correct branch, or pull/rebase." + ) + logger.debug("Sanity check succeeded.") + + def update_firefox(self): + logger.info("Updating Firefox (and GeckoView if needed)") + config = self.load_config("firefox") + + tag_tb = None + tag_mb = None + if self.tor_browser: + tag_tb = self._get_firefox_tag(config, "tor-browser") + logger.debug( + "Tor Browser tag: ff=%s, rebase=%s, build=%s", + tag_tb[0], + tag_tb[1], + tag_tb[2], + ) + if self.mullvad_browser: + tag_mb = self._get_firefox_tag(config, "mullvad-browser") + logger.debug( + "Mullvad Browser tag: ff=%s, rebase=%s, build=%s", + tag_mb[0], + tag_mb[1], + tag_mb[2], + ) + if ( + tag_mb + and (not tag_tb or tag_tb[2] == tag_mb[2]) + and "browser_build" in config["targets"]["mullvadbrowser"]["var"] + ): + logger.debug( + "Tor Browser and Mullvad Browser are on the same build number, deleting unnecessary targets/mullvadbrowser/var/browser_build." + ) + del config["targets"]["mullvadbrowser"]["var"]["browser_build"] + elif tag_mb and tag_tb and tag_mb[2] != tag_tb[2]: + config["targets"]["mullvadbrowser"]["var"]["browser_build"] = ( + tag_mb[2] + ) + logger.debug( + "Mismatching builds for TBB and MB, will add targets/mullvadbrowser/var/browser_build." + ) + # We assume firefox version and rebase to be in sync + if tag_tb: + version = tag_tb[0] + rebase = tag_tb[1] + build = tag_tb[2] + elif tag_mb: + version = tag_mb[0] + rebase = tag_mb[1] + build = tag_mb[2] + platform = version[:-3] if version.endswith("esr") else version + config["var"]["firefox_platform_version"] = platform + config["var"]["browser_rebase"] = rebase + config["var"]["browser_build"] = build + self.save_config("firefox", config) + logger.debug("Firefox configuration saved") + + if self.android: + assert tag_tb + config = self.load_config("geckoview") + config["var"]["geckoview_version"] = tag_tb[0] + config["var"][ + "browser_branch" + ] = f"{self.version.major}-{tag_tb[1]}" + config["var"]["browser_build"] = tag_tb[2] + self.save_config("geckoview", config) + logger.debug("GeckoView configuration saved") + + def _get_firefox_tag(self, config, browser): + if browser == "mullvad-browser": + remote = config["targets"]["mullvadbrowser"]["git_url"] + else: + remote = config["git_url"] + repo = Repo(self.base_path / "git_clones/firefox") + repo.remotes["origin"].set_url(remote) + logger.debug("About to fetch Firefox from %s.", remote) + repo.remotes["origin"].fetch() + tags = get_sorted_tags(repo) + for t in tags: + m = re.match( + r"(\w+-browser)-([^-]+)-([\d\.]+)-(\d+)-build(\d+)", t.tag + ) + if ( + m + and m.group(1) == browser + and m.group(3) == self.version.major + ): + # firefox-version, rebase, build + return (m.group(2), int(m.group(4)), int(m.group(5))) + + def update_firefox_android(self): + logger.info("Updating firefox-android") + config = self.load_config("firefox-android") + repo = Repo(self.base_path / "git_clones/firefox-android") + repo.remotes["origin"].fetch() + tags = get_sorted_tags(repo) + for t in tags: + m = re.match( + r"firefox-android-([^-]+)-([\d\.]+)-(\d+)-build(\d+)", t.tag + ) + if not m or m.group(2) != self.version.major: + logger.debug("Discarding firefox-android tag: %s", t.tag) + continue + logger.debug("Using firefox-android tag: %s", t.tag) + config["var"]["fenix_version"] = m.group(1) + config["var"]["browser_branch"] = m.group(2) + "-" + m.group(3) + config["var"]["browser_build"] = int(m.group(4)) + break + self.save_config("firefox-android", config) + + def update_translations(self): + logger.info("Updating translations") + repo = Repo(self.base_path / "git_clones/translation") + repo.remotes["origin"].fetch() + config = self.load_config("translation") + targets = ["base-browser"] + if self.tor_browser: + targets.append("tor-browser") + targets.append("fenix") + if self.mullvad_browser: + targets.append("mullvad-browser") + for i in targets: + branch = config["steps"][i]["targets"]["nightly"]["git_hash"] + config["steps"][i]["git_hash"] = str( + repo.rev_parse(f"origin/{branch}") + ) + self.save_config("translation", config) + logger.debug("Translations updated") + + def update_addons(self): + logger.info("Updating addons") + config = self.load_config("browser") + + amo_data = fetch_allowed_addons() + logger.debug("Fetched AMO data") + if self.android: + with ( + self.base_path / "projects/browser/allowed_addons.json" + ).open("w") as f: + json.dump(amo_data, f, indent=2) + + noscript = find_addon(amo_data, NOSCRIPT) + logger.debug("Updating NoScript") + self.update_addon_amo(config, "noscript", noscript) + if self.mullvad_browser: + logger.debug("Updating uBlock Origin") + ublock = find_addon(amo_data, "uBlock0@raymondhill.net") + self.update_addon_amo(config, "ublock-origin", ublock) + logger.debug("Updating the Mullvad Browser extension") + self.update_mullvad_addon(config) + + self.save_config("browser", config) + + def update_addon_amo(self, config, name, addon): + addon = addon["current_version"]["files"][0] + assert addon["hash"].startswith("sha256:") + addon_input = self.find_input(config, name) + addon_input["URL"] = addon["url"] + addon_input["sha256sum"] = addon["hash"][7:] + + def update_mullvad_addon(self, config): + input_ = self.find_input(config, "mullvad-extension") + r = requests.get( + "https://cdn.mullvad.net/browser-extension/updates.json" + ) + r.raise_for_status() + + data = r.json() + updates = data["addons"]["{d19a89b9-76c1-4a61-bcd4-49e8de916403}"][ + "updates" + ] + url = updates[-1]["update_link"] + if input_["URL"] == url: + logger.debug("No need to update the Mullvad extension.") + return + input_["URL"] = url + + path = self.base_path / "out/browser" / url.split("/")[-1] + # The extension should be small enough to easily fit in memory :) + if not path.exists: + r = requests.get(url) + r.raise_for_status() + with path.open("wb") as f: + f.write(r.bytes) + with path.open("rb") as f: + input_["sha256sum"] = sha256(f.read()).hexdigest() + logger.debug("Mullvad extension downloaded and updated") + + def update_tor(self): + logger.info("Updating Tor") + databag = configparser.ConfigParser() + r = requests.get( + "https://gitlab.torproject.org/tpo/web/tpo/-/raw/main/databags/versions.ini" + ) + r.raise_for_status() + databag.read_string(r.text) + tor_stable = databag["tor-stable"]["version"] + tor_alpha = databag["tor-alpha"]["version"] + logger.debug( + "Found tor stable: %s, alpha: %s", + tor_stable, + tor_alpha if tor_alpha else "(empty)", + ) + if self.version.is_alpha and tor_alpha: + version = tor_alpha + else: + version = tor_stable + + config = self.load_config("tor") + if version != config["version"]: + config["version"] = version + self.save_config("tor", config) + logger.debug("Tor updated to %s and config saved", version) + else: + logger.debug( + "No need to update Tor (current version: %s).", version + ) + + def update_openssl(self): + logger.info("Updating OpenSSL") + config = self.load_config("openssl") + version = get_github_release("openssl/openssl", r"openssl-(3.0.\d+)") + if version == config["version"]: + logger.debug("No need to update OpenSSL, keeping %s.", version) + return + + config["version"] = version + + source = self.find_input(config, "openssl") + # No need to update URL, as it uses a variable. + hash_url = ( + f"https://www.openssl.org/source/openssl-{version}.tar.gz.sha256" + ) + r = requests.get(hash_url) + r.raise_for_status() + source["sha256sum"] = r.text.strip() + self.save_config("openssl", config) + logger.debug("Updated OpenSSL to %s and config saved.", version) + + def update_zlib(self): + logger.info("Updating zlib") + config = self.load_config("zlib") + version = get_github_release("madler/zlib", r"v([0-9\.]+)") + if version == config["version"]: + logger.debug("No need to update zlib, keeping %s.", version) + return + config["version"] = version + self.save_config("zlib", config) + logger.debug("Updated zlib to %s and config saved.", version) + + def update_zstd(self): + logger.info("Updating Zstandard") + config = self.load_config("zstd") + version = get_github_release("facebook/zstd", r"v([0-9\.]+)") + if version == config["version"]: + logger.debug("No need to update Zstandard, keeping %s.", version) + return + + repo = Repo(self.base_path / "git_clones/zstd") + repo.remotes["origin"].fetch() + tag = repo.rev_parse(f"v{version}") + + config["version"] = version + config["git_hash"] = tag.object.hexsha + self.save_config("zstd", config) + logger.debug( + "Updated Zstandard to %s (commit %s) and config saved.", + version, + config["git_hash"], + ) + + def update_go(self): + def get_major(v): + major = ".".join(v.split(".")[:2]) + if major.startswith("go"): + major = major[2:] + return major + + config = self.load_config("go") + # TODO: When Windows 7 goes EOL use config["version"] + major = get_major(config["var"]["go_1_21"]) + + r = requests.get("https://go.dev/dl/?mode=json") + r.raise_for_status() + go_versions = r.json() + data = None + for v in go_versions: + if get_major(v["version"]) == major: + data = v + break + if not data: + raise KeyError("Could not find information for our Go series.") + # Skip the "go" prefix in the version. + config["var"]["go_1_21"] = data["version"][2:] + + sha256sum = "" + for f in data["files"]: + if f["kind"] == "source": + sha256sum = f["sha256"] + break + if not sha256sum: + raise KeyError("Go source package not found.") + updated_hash = False + for input_ in config["input_files"]: + if "URL" in input_ and "var/go_1_21" in input_["URL"]: + input_["sha256sum"] = sha256sum + updated_hash = True + break + if not updated_hash: + raise KeyError("Could not find a matching entry in input_files.") + + self.save_config("go", config) + + def update_manual(self): + logger.info("Updating the manual") + update_manual(self.gitlab_token, self.base_path) + + def get_last_releases(self): + logger.info("Finding the previous releases.") + sorted_tags = get_sorted_tags(self.repo) + self.last_releases = {} + self.build_number = 1 + regex = re.compile(r"(\w+)-([\d\.a]+)-build(\d+)") + num_releases = 0 + for t in sorted_tags: + m = regex.match(t.tag) + project = m.group(1) + version = Version(m.group(2)) + build = int(m.group(3)) + if version == self.version: + # A previous tag, we can use it to bump our build. + if self.build_number == 1: + self.build_number = build + 1 + logger.debug( + "Found previous tag for the version we are preparing: %s. Bumping build number to %d.", + t.tag, + self.build_number, + ) + continue + key = (project, version.channel) + if key not in self.last_releases: + self.last_releases[key] = [] + skip = False + for rel in self.last_releases[key]: + # Tags are already sorted: higher builds should come + # first. + if rel.version == version: + skip = True + logger.debug( + "Additional build for a version we already found, skipping: %s", + t.tag, + ) + break + if skip: + continue + if len(self.last_releases[key]) != self.num_incrementals: + logger.debug( + "Found tag to potentially build incrementals from: %s.", + t.tag, + ) + self.last_releases[key].append(ReleaseTag(t, version)) + num_releases += 1 + if num_releases == self.num_incrementals * 4: + break + + def update_changelogs(self): + if self.tor_browser: + logger.info("Updating changelogs for Tor Browser") + self.make_changelogs("tbb") + if self.mullvad_browser: + logger.info("Updating changelogs for Mullvad Browser") + self.make_changelogs("mb") + + def make_changelogs(self, tag_prefix): + locale.setlocale(locale.LC_TIME, "C") + kwargs = {"date": self.changelog_date.strftime("%B %d %Y")} + prev_tag = self.last_releases[(tag_prefix, self.version.channel)][ + 0 + ].tag + self.check_update( + kwargs, prev_tag, "firefox", ["var", "firefox_platform_version"] + ) + if "firefox" in kwargs: + # Sometimes this might be incorrect for alphas, but let's + # keep it for now. + kwargs["firefox"] += "esr" + self.check_update_simple(kwargs, prev_tag, "tor") + self.check_update_simple(kwargs, prev_tag, "openssl") + self.check_update_simple(kwargs, prev_tag, "zlib") + self.check_update_simple(kwargs, prev_tag, "zstd") + try: + self.check_update(kwargs, prev_tag, "go", ["var", "go_1_21"]) + except KeyError as e: + logger.warning( + "Go: var/go_1_21 not found, marking Go as not updated.", + exc_info=e, + ) + pass + self.check_update_extensions(kwargs, prev_tag) + logger.debug("Changelog arguments for %s: %s", tag_prefix, kwargs) + cb = fetch_changelogs.ChangelogBuilder( + self.gitlab_token, str(self.version), is_mullvad=tag_prefix == "mb" + ) + changelogs = cb.create(**kwargs) + + path = f"projects/browser/Bundle-Data/Docs-{tag_prefix.upper()}/ChangeLog.txt" + stable_tag = self.last_releases[(tag_prefix, "release")][0].tag + alpha_tag = self.last_releases[(tag_prefix, "alpha")][0].tag + if stable_tag.tagged_date > alpha_tag.tagged_date: + last_tag = stable_tag + else: + last_tag = alpha_tag + logger.debug("Using %s to add the new changelogs to.", last_tag.tag) + last_changelogs = self.repo.git.show(f"{last_tag.tag}:{path}") + with (self.base_path / path).open("w") as f: + f.write(changelogs + "\n" + last_changelogs + "\n") + + def check_update(self, updates, prev_tag, project, key): + old_val = self.load_old_config(prev_tag.tag, project) + new_val = self.load_config(project) + for k in key: + old_val = old_val[k] + new_val = new_val[k] + if old_val != new_val: + updates[project] = new_val + + def check_update_simple(self, updates, prev_tag, project): + self.check_update(updates, prev_tag, project, ["version"]) + + def check_update_extensions(self, updates, prev_tag): + old_config = self.load_old_config(prev_tag, "browser") + new_config = self.load_config("browser") + keys = { + "noscript": "noscript", + "mb_extension": "mullvad-extension", + "ublock": "ublock-origin", + } + regex = re.compile(r"-([0-9\.]+).xpi$") + for update_key, input_name in keys.items(): + old_url = self.find_input(old_config, input_name)["URL"] + new_url = self.find_input(new_config, input_name)["URL"] + old_version = regex.findall(old_url)[0] + new_version = regex.findall(new_url)[0] + if old_version != new_version: + updates[update_key] = new_version + + def update_rbm_conf(self): + logger.info("Updating rbm.conf.") + releases = {} + browsers = { + "tbb": '[% IF c("var/tor-browser") %]{}[% END %]', + "mb": '[% IF c("var/mullvad-browser") %]{}[% END %]', + } + incremental_from = [] + for b in ["tbb", "mb"]: + for rel in self.last_releases[(b, self.version.channel)]: + if rel.version not in releases: + releases[rel.version] = {} + releases[rel.version][b] = str(rel.version) + for version in sorted(releases.keys(), reverse=True): + if len(releases[version]) == 2: + incremental_from.append(releases[version]["tbb"]) + logger.debug( + "Building incremental from %s for both browsers.", version + ) + else: + for b, template in browsers.items(): + maybe_rel = releases[version].get(b) + if maybe_rel: + logger.debug( + "Building incremental from %s only for %s.", + version, + b, + ) + incremental_from.append(template.format(maybe_rel)) + + separator = "\n--- |\n" + path = self.base_path / "rbm.conf" + with path.open() as f: + docs = f.read().split(separator, 2) + config = self.yaml.load(docs[0]) + config["var"]["torbrowser_version"] = str(self.version) + config["var"]["torbrowser_build"] = f"build{self.build_number}" + config["var"]["torbrowser_incremental_from"] = incremental_from + config["var"]["browser_release_date"] = self.build_date.strftime( + "%Y/%m/%d %H:%M:%S" + ) + with path.open("w") as f: + self.yaml.dump(config, f) + f.write(separator) + f.write(docs[1]) + + def load_config(self, project): + config_path = self.base_path / f"projects/{project}/config" + return self.yaml.load(config_path) + + def load_old_config(self, committish, project): + treeish = f"{committish}:projects/{project}/config" + return self.yaml.load(self.repo.git.show(treeish)) + + def save_config(self, project, config): + config_path = self.base_path / f"projects/{project}/config" + with config_path.open("w") as f: + self.yaml.dump(config, f) + + def find_input(self, config, name): + for entry in config["input_files"]: + if "name" in entry and entry["name"] == name: + return entry + raise KeyError(f"Input {name} not found.") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "-r", + "--repository", + type=Path, + default=Path(__file__).parent.parent, + help="Path to a tor-browser-build.git clone", + ) + parser.add_argument("--tor-browser", action="store_true") + parser.add_argument("--mullvad-browser", action="store_true") + parser.add_argument( + "--date", + help="Release date and optionally time for changelog purposes. " + "It must be understandable by datetime.fromisoformat.", + ) + parser.add_argument( + "--build-date", + help="Build date. It cannot not be in the future when running the build.", + ) + parser.add_argument( + "--incrementals", type=int, help="The number of incrementals to create" + ) + parser.add_argument( + "--only-changelogs", + action="store_true", + help="Only update the changelogs", + ) + parser.add_argument( + "--log-level", + choices=["debug", "info", "warning", "error"], + default="info", + help="Set the log level", + ) + parser.add_argument("version") + + args = parser.parse_args() + + # Logger adapted from https://stackoverflow.com/a/56944256. + log_level = getattr(logging, args.log_level.upper()) + logger.setLevel(log_level) + ch = logging.StreamHandler() + ch.setLevel(log_level) + ch.setFormatter( + logging.Formatter( + "%(asctime)s - %(name)s - %(levelname)s - %(message)s (%(filename)s:%(lineno)d)", + datefmt="%Y-%m-%d %H:%M:%S", + ) + ) + logger.addHandler(ch) + + tbb = bool(args.tor_browser) + mb = bool(args.mullvad_browser) + kwargs = {} + if tbb or mb: + kwargs["tor_browser"] = tbb + kwargs["mullvad_browser"] = mb + if args.date: + try: + kwargs["changelog_date"] = datetime.fromisoformat(args.date) + except ValueError: + print("Invalid date supplied.", file=sys.stderr) + sys.exit(1) + if args.build_date: + try: + kwargs["build_date"] = datetime.fromisoformat(args.date) + except ValueError: + print("Invalid date supplied.", file=sys.stderr) + sys.exit(1) + if args.incrementals: + kwargs["incrementals"] = args.incrementals + rp = ReleasePreparation(args.repository, args.version, **kwargs) + if args.only_changelogs: + logger.info("Updating only the changelogs") + rp.update_changelogs() + else: + logger.debug("Running a complete release preparation.") + rp.run() ===================================== tools/update_manual.py ===================================== @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +import hashlib +from pathlib import Path + +import requests +import ruamel.yaml + +from fetch_changelogs import load_token, AUTH_HEADER + + +GITLAB = "https://gitlab.torproject.org" +API_URL = f"{GITLAB}/api/v4" +PROJECT_ID = 23 +REF_NAME = "main" + + +def find_job(auth_token): + r = requests.get( + f"{API_URL}/projects/{PROJECT_ID}/jobs", + headers={AUTH_HEADER: auth_token}, + ) + r.raise_for_status() + for job in r.json(): + if job["ref"] != REF_NAME: + continue + for artifact in job["artifacts"]: + if artifact["filename"] == "artifacts.zip": + return job + + +def update_config(base_path, pipeline_id, sha256): + yaml = ruamel.yaml.YAML() + yaml.indent(mapping=2, sequence=4, offset=2) + yaml.width = 150 + yaml.preserve_quotes = True + + config_path = base_path / "projects/manual/config" + config = yaml.load(config_path) + if int(config["version"]) == pipeline_id: + return False + + config["version"] = pipeline_id + for input_file in config["input_files"]: + if input_file.get("name") == "manual": + input_file["sha256sum"] = sha256 + break + with config_path.open("w") as f: + yaml.dump(config, f) + return True + +def download_manual(url, dest): + r = requests.get(url, stream=True) + # https://stackoverflow.com/a/16696317 + r.raise_for_status() + sha256 = hashlib.sha256() + with dest.open("wb") as f: + for chunk in r.iter_content(chunk_size=8192): + f.write(chunk) + sha256.update(chunk) + return sha256.hexdigest() + + +def update_manual(auth_token, base_path): + job = find_job(auth_token) + if job is None: + raise RuntimeError("No usable job found") + pipeline_id = int(job["pipeline"]["id"]) + + manual_fname = f"manual_{pipeline_id}.zip" + url = f"https://build-sources.tbb.torproject.org/{manual_fname}" + r = requests.head(url) + needs_upload = r.status_code != 200 + + manual_dir = base_path / "out/manual" + manual_dir.mkdir(0o755, parents=True, exist_ok=True) + manual_file = manual_dir / manual_fname + if manual_file.exists(): + sha256 = hashlib.sha256() + with manual_file.open("rb") as f: + while chunk := f.read(8192): + sha256.update(chunk) + sha256 = sha256.hexdigest() + elif not needs_upload: + sha256 = download_manual(url, manual_file) + else: + url = f"{API_URL}/projects/{PROJECT_ID}/jobs/artifacts/{REF_NAME}/download?job={job['name']}" + sha256 = download_manual(url, manual_file) + + if needs_upload: + print(f"New manual version: {manual_file}.") + print( + "Please upload it to tb-build-02.torproject.org:~tb-builder/public_html/." + ) + + return update_config(base_path, pipeline_id, sha256) + + +if __name__ == "__main__": + if update_manual(load_token(), Path(__file__).parent.parent): + print("Manual config updated, remember to stage it!") View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/compare/3... -- This project does not include diff previews in email notifications. View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/compare/3... You're receiving this email because of your account on gitlab.torproject.org.