henry pushed to branch tor-browser-153.6.0esr-16.0-1 at The Tor Project / Applications / Tor Browser Commits: 0d103a1e by Henry Wilkes at 2026-11-03T15:03:48+00:00 fixup! Tor Browser localization migration scripts. TB 45433: Change the script to be more versatile for test runs. Instead of passing in a python module name for the recipe, you now pass in the file path to the recipe script. You can now change the reference directory with the --reference-directory argument, which is useful for testing scripts before the changes have reached the translation repository. - - - - - 7e33b574 by Henry Wilkes at 2026-11-04T14:51:14+00:00 fixup! Tor Browser localization migration scripts. TB 45433: Refactor the WeblateMetadata class. We are stricter about checking the return types from the weblate API, and making sure it doesn't make requests to unexpected URLs. We also limit the components to the browser components, and we slightly alter the `is_translated` public method. - - - - - 90fb8c89 by Henry Wilkes at 2026-11-04T14:51:16+00:00 fixup! Tor Browser localization migration scripts. TB 45433: Add type hints to the script. - - - - - fdcaef33 by Henry Wilkes at 2026-11-04T14:51:17+00:00 fixup! Tor Browser localization migration scripts. TB 45433: Update the migrate.py documentation. - - - - - 1 changed file: - tools/torbrowser/l10n/migrate.py Changes: ===================================== tools/torbrowser/l10n/migrate.py ===================================== @@ -1,58 +1,252 @@ """ -Migration script for running migration recipes. +Migration script for running Tor Browser migration recipes. -Requires the mozilla fluent.migrate package (available with pip). +Based on the Firefox migration script, adapted for Tor Browser. See +https://firefox-source-docs.mozilla.org/l10n/migrations/ for more context and +information on how to write the migration recipes. -NOTE: This will likely send queries to the weblate API as part of the run. -Weblate may have a query limit of 100 requests per day for unauthorised users. +## Concepts + ++ String. Here, when we say "string", it refers to a piece of text that can be + translated. In practice, you can think of it as a being defined by a triple: + a file path relative to a locale's directory, an ID, and some content. For + example `("./tor-browser.ftl", "my-id", "my-id =\\n .label=Content")`. + When a string is translated the file path and ID stay the same, whilst the + translatable part of the content can vary. ++ Reference files. These are the en-US source Fluent files that contain the + *manual* string changes made by a developer. For most migrations, there is + only one reference file. ++ Migration recipe. A collection of instructions that take some string inputs + and transforms them into new strings. These are meant to mirror the manual + changes that were made in the reference files, in a way that can be applied + to any locale. ++ Translation repository. This refers to a local copy of the `translation` + repository (https://gitlab.torproject.org/tpo/translation). This repository + contains a different branch for different collections of strings. For + Tor Browser we usually use `tor-browser`, `base-browser` and + `fenix-torbrowserstringsxml`. Each branch will contain an `en-US` directory, + which contains the "source" files for translators to translate. These source + files are taken from the `tor-browser` repository. The branches also contain + a directory for each locale that is to be translated, in a file structure that + reflects the source `en-US` directory. This is where the migration script will + write changes to the translated files. + + +## Usage + +You will likely want to run this with `./mach python`, so that it uses the +`fluent.migrate` and `fluent.syntax` packages provided with the `tor-browser` +repository. + +For example, you might run this as -Based on the firefox migration script, adapted for tor browser. See -https://firefox-source-docs.mozilla.org/l10n/migrations/ for more context and -information on how to write the migration recipes. +``` + ./mach python migrate.py --translation-git /home/user/translation --locales 'es-ES sr' ./migrations/bug-xxxx-my-migration-recipe.py +``` + +Here `/home/user/translation` is a local translation repository with one of its +branches checked out (see below). This will run the migration recipe given in +`./migrations/bug-xxxx-my-migration-recipe.py` for the `es-ES` and `sr` +sub-directories within the `translation/` directory. + +### Changing the reference location + +You can optionally run the script with `--reference_directory` to read the +reference en-US files from somewhere other than the `translation` directory. +This can be useful when your manual changes have yet to reach the translation +repository, so you want to point the script to read the reference from the +`tor-browser` working directory instead. E.g. + +``` + --reference-directory /home/user/tor-browser/toolkit/locales/en-US/toolkit/global/ +``` + + +## Recipes + +Migration recipes are meant to mirror some manual change that was made to the +en-US source strings, in a way that can be reproduced in another locale. +### Example 1 -Usage with an example: +For this example, a developer wants to remove the string content from +`old-strings.properties`: -We have some en-US file "updated.ftl" in the tor-browser repository that we have -added new strings to. Some of these strings are copies or concatenations of some -already existing strings in a `.dtd`, `.properties` or `.ftl` file (including -strings within "updated.ftl"). +``` +my.old.string=Some string content +my.old.title=This is the tooltip +``` + +And add them to `tor-browser.ftl` instead, as: -So we also write a migration recipe in -`./l10n_migrations/bug-xxxx-my-migration-recipe.py` -which includes a `migrate(ctx)` method. This method takes the old strings and -create new Messages or Terms for "updated.ftl" using `ctx.add_transforms`. These -should match what was already done manually in "updated.ftl". +``` +my-new-string = Some string content + .title = This is the tooltip +``` + +For the en-US source strings, the developer simply deletes the two lines in the +`old-strings.properties` file (or the whole file, if these are the last two). +And they manually write in the content of `my-new-string` to `tor-browser.ftl`. +These manual changes will then eventually reach the `translation` repository to +be translated. + +But to save translators making the same manual changes, or re-translating, the +developer can write a migration recipe that mirror's their manual changes. In +this case, this migration recipe would contain: + +``` +ctx.add_transforms( + "tor-browser.ftl", # The path to write to. + "tor-browser.ftl", # The path to the reference file. + transforms_from( + \"\"\" +my-new-string = { COPY(path, "my.old.string") } + .title = { COPY(path, "my.old.title") } +\"\"\", + path="old-strings.properties", # The path to read from. + ), +) +``` + +Note, the first argument of `add_transforms` is the target path, which is +`tor-browser.ftl` in this case. This means that the migration script will try to +write the resulting `my-new-string` to the locale's `./tor-browser.ftl` file. +For example, to `translation/es-ES/tor-browser.ftl`. + +The second argument refers to the *reference* path. In almost all cases the +reference file will have the same name and path as the target path, but it has a +different role. Here the path is relative to the reference directory, which +contains the en-US source files. In this case, by default this means that +`translation/en-US/tor-browser.ftl` will act as the reference, but the exact +path could be changed with the `--reference-directory` argument. The existing +`my-new-string` will be read from this file to confirm that the migration +recipe produces the correct result. + +The file `old-strings.properties` is used as an input. This means that the +existing strings in the locale's `old-strings.properties` file will be read to +construct the new string `my-new-string`. For example, it will be read from +`translation/es-ES/old-strings.properties`. + +The `COPY` acts like a Fluent function, which asks the transformation to copy +the content from the translated string directly. + +### Example 2 + +For this example, a developer wants to swap a string in `tor-browser.ftl` from: + +``` +my-old-string2 = Some string content + .title = This is the tooltip +``` + +to + +``` +my-new-string2 = + .label = Some string content + .title = This is the tooltip +``` -Then we copy the en-US file to our local translation repository, say at -`/home/user/translation/en-US/updated.ftl`. The en-US file is needed as a -reference during migration to know where in the new file the new Messages or -Terms should be placed. Moreover, the migration can be verified against the -en-US file to ensure it is working as intended. +The developer removes `my-old-string2` from the file and adds `my-new-string2` +instead. This manual change will then eventually reach the `translation` +repository for `my-new-string2` to be translated. -Then we run the migration for individual locales: +The developer also writes a migration recipe to mirror this change: ``` - python migrate_l10n.py --translation-git /home/user/translation --locales 'es-ES sr' l10n_migrations.bug-xxxx-my-migration-recipe +ctx.add_transforms( + "tor-browser.ftl", + "tor-browser.ftl", + transforms_from( + \"\"\" +my-new-string2 = + .label = { COPY_PATTERN(path, "my-old-string2") } + .title = { COPY_PATTERN(path, "my-old-string2.title") } +\"\"\", + path="tor-browser.ftl", + ), +) ``` -This will first run a mock migration using the en-US localization files. This is -to help ensure that the migration will work as intended. I.e. if we ran the -migration recipe for en-US, using it as both the source of old strings and as a -reference, then each transformation using these old strings should produce a -Message or Term that is identical to the one in the reference file. +Note that in this case the locale's `tor-browser.ftl` file is being used as both +the input and the output. + +Note that for a Fluent-to-Fluent migration we use `COPY_PATTERN` rather than +`COPY`. The second argument provides an ID that either point's to the +Fluent Messages' value (roughly, the text content) by giving the Message's ID, +or one of its attributes by adding a corresponding suffix to the ID. + +### Other examples + +Migration recipes can support a lot more, including more advanced +transformations like removing or replacing HTML tags from content. There are +some existing recipes in the `./migrations` directory that can be used as a +reference. + + +## translation directory + +In most cases, you will just need to check out a `translation` branch that +contains the translated strings used by the migration recipe, as well as the +Fluent file that will be written to. Usually, this will be a branch based on +the origin/upstream `tor-browser` or `base-browser` branches. + +However, if the migration recipe requires source files that are in a different +branch to the Fluent file that is being written to, then you will need to +manually copy those files into the working directory before running the script. + + +## Script procedure -Then this will run `migrate(ctx)` for the es-ES and sr locales. Some of the -transformations will not be completed for a locale if that locale does not -contain a translated version of the old strings the transformation requires -(because there is no string to copy). +### Test en-US migration -If one of the old strings is from a `.dtd` or `.properties` file, this will -check whether they are marked as translated on weblate using its API. This -prevents copying "placeholder" en-US values into the locale's fluent files. +The script will first run a test migration using the en-US strings. It will run +the migration using the files from `translation/en-US/` as the input. In the +previous example 1, this will read `my.old.string` and `my.old.title` from +`translation/en-US/old-strings.properties`. And in example 2, this will read +`my-old-string2` from `translation/en-US/tor-browser.ftl`. -Any warnings or errors will be logged to stderr in the console. +If any of the input strings are missing, this indicates that there is likely a +mistake in the recipe or the translation directory and the script will be +aborted without running the migration for the other locales. + +Otherwise, it then compares the result of this migration with the strings found +in the reference files. In the previous example 1, this will read the entry for +`my-new-string` from `<reference-dir>/tor-browser.ftl`, where `<reference-dir>` +is `translation/en-US/` unless `--reference-directory` is given to use a +different directory. If the reference entry for `my-new-string` differs from the +entry that the recipe automatically produced, then this indicates that there is +a mistake in the recipe and the script will be aborted. + +### Locale migrations + +If the test migration passed, then the recipe will be re-run for each of the +specified locales. If the locale is missing some strings or files that a single +transformation requires, then that transformation will be skipped and the +corresponding entry won't be written to the Fluent file. Otherwise the new +entries will be appended to the existing Fluent files. + +#### Non-Fluent strings and Weblate queries + +Some non-Fluent formats, like DTD and properties, will fill untranslated strings +with the source en-US content. And we don't want these untranslated strings to +be used for migrations. There is no metadata within the files to determine this, +and simply being the same as the en-US string is not a reliable indicator. + +So for these strings the script will additionally make a remote query to the +Weblate API to fetch the metadata that indicates whether the string is +considered translated or not. If a transformation requires a string that is +marked as untranslated, it will be skipped. + +NOTE: The Weblate metadata is not always reliable, so you should still manually +review the changes. + +NOTE: When testing before, the Weblate API was limited to 100 requests per day +per IP address for unauthorised users. If this becomes an issue you may want to +run the migrations in limited batches of locales with new IP addresses. Or, you +can create a Weblate account and change this script to pass an authorisation +token with the request. """ import argparse @@ -63,142 +257,299 @@ import logging import os import sys import urllib.request +from collections.abc import Iterator +from typing import Any, NamedTuple, Protocol, cast from fluent.migrate.context import MigrationContext from fluent.migrate.errors import EmptyLocalizationError, UnreadableReferenceError from fluent.syntax import ast, parse, serialize -def in_red(string): +def in_red(string: str) -> str: """ Wrap a string so that it is shown as red in the console. + :param string: The string to wrap. + :returns: The wrapped string. """ return f"\x1b[1;91m{string}\x1b[0m" +def assert_keys( + check_dict: dict[Any, Any], expected_keys: tuple[str, ...], url: str +) -> None: + """ + Verify that the dictionary contains *at least* the expected keys, + otherwise this throws. + :param check_dict: The dictionary to check. + :param url: The url where the dictionary was fetched from. + """ + for key in expected_keys: + if key not in check_dict: + raise ValueError(f"Missing '{key}' in response for {url}") + + +class WeblateUnit: + """ + Represents a Weblate "unit": roughly corresponds to a translated string. + + See https://docs.weblate.org/en/latest/api.html#units. + """ + + def __init__(self, data: dict[Any, Any], url: str) -> None: + """ + :param data: The Weblate data response. + :param url: The URL that the data came from. + """ + assert_keys(data, ("context", "translated"), url) + string_id = data["context"] + translated = data["translated"] + if not isinstance(string_id, str): + raise ValueError(f"Unexpected 'context' type in response for {url}") + if not isinstance(translated, bool): + raise ValueError(f"Unexpected 'translated' type in response for {url}") + + self.string_id = string_id + self.translated = translated + + +class WeblateTranslation: + """ + Represents a Weblate "translation": roughly corresponds to the translation + of a file for a given locale/language. + + See https://docs.weblate.org/en/latest/api.html#translations. + """ + + def __init__(self, data: dict[Any, Any], url: str) -> None: + """ + :param data: The Weblate data response. + :param url: The URL that the data came from. + """ + assert_keys(data, ("filename", "units_list_url"), url) + locale_path = data["filename"] + units_url = data["units_list_url"] + if not isinstance(locale_path, str): + raise ValueError(f"Unexpected 'filename' type in response for {url}") + if not isinstance(units_url, str): + raise ValueError(f"Unexpected 'units_list_url' type in response for {url}") + + self.locale_path = locale_path + self.units_url = units_url + # The units, indexed by their string ID. Lazy-loaded and `None` whilst + # unfilled. + self.units: None | dict[str, WeblateUnit] = None + + +class WeblateComponent: + """ + Represents a Weblate "component": roughly corresponds to a source (en-US) + file that can be translated. + + See https://docs.weblate.org/en/latest/api.html#components. + And https://docs.weblate.org/en/latest/user/basic.html. + """ + + def __init__(self, data: dict[Any, Any], url: str) -> None: + """ + :param data: The Weblate data response. + :param url: The URL that the data came from. + """ + assert_keys(data, ("template", "translations_url", "branch"), url) + template_path = data["template"] + translations_url = data["translations_url"] + branch = data["branch"] + if not isinstance(template_path, str): + raise ValueError(f"Unexpected 'template' type in response for {url}") + if not isinstance(translations_url, str): + raise ValueError( + f"Unexpected 'translations_url' type in response for {url}" + ) + if not isinstance(branch, str): + raise ValueError(f"Unexpected 'branch' type in response for {url}") + + self.branch = branch + # The path to the source (en-US) file. + self.template_path = template_path + # The URL to fetch the component's translations from. + self.translations_url = translations_url + # The translations data, indexed by the locale file paths. Lazy-loaded + # and `None` whilst unfilled. + self.translations: None | dict[str, WeblateTranslation] = None + + class WeblateMetadata: """ - Manages and fetches string metadata from weblate. + Manages and fetches string metadata from Weblate. """ - def __init__(self): + # The translation branches that we expect to be used for Weblate. + _BROWSER_BRANCHES = ["tor-browser", "base-browser", "fenix-torbrowserstringsxml"] + _WEBLATE_API_URL = "https://hosted.weblate.org/api/" + _WEBLATE_API_TOR_COMPONENTS_URL = _WEBLATE_API_URL + "projects/tor/components/" + + def __init__(self) -> None: + """ + Create a new instance for fetching Weblate metadata. + """ self.logger = logging.getLogger("weblate-metadata") self.logger.setLevel(logging.INFO) - # The _components property will be lazy-loaded with weblate data. - # - # On weblate, for monolingual formats, the component will have a - # "template" file, which matches the en-US file path in the - # "translation" git repo. So we can find a component through a matching - # template. - # - # Each translation in that component will have a "filename" that will - # match the locale's file path in the "translation" git repo. So we can - # find a translation through a matching filename. - # - # Each translation can be queried for its units, including metadata. - # Each unit's "context" will match the string's id in the corresponding - # file. - # - # Expect the final structure to be: - # { - # template: { - # "translations-url": str, # Used for API translations query. - # "translations": { - # filename: { - # "units-url": str, # Used for API units query. - # "units": { - # context: { - # "translated": bool, - # }, - # }, - # }, - # }, - # }, - # } - self._components = None - - def _get_weblate_response(self, url): + # The _components property will be lazy-loaded with Weblate data. + self._components: dict[str, WeblateComponent] | None = None + + def _get_weblate_response(self, url: str) -> Any: + """ + Make a single query to Weblate. + :param url: The URL to fetch. + :returns: The JSON response. + """ + if not url.startswith(self._WEBLATE_API_URL): + raise ValueError(f"Unexpected URL request: {url}") + weblate_request = urllib.request.Request( url, # NOTE: can set the request header: # "Authorization": "Token WEBLATE_TOKEN" - # to get around rate-limiting + # to get around rate-limiting. + # NOTE: We empty the User-Agent to override the default + # value of "Python-urllib/{major}.{minor}". headers={"User-Agent": ""}, ) - with urllib.request.urlopen(weblate_request, timeout=20) as response: + with urllib.request.urlopen(weblate_request) as response: return json.load(response) - def _get_from_weblate(self, url): - ret = [] + def _get_from_weblate(self, url: str) -> list[dict[Any, Any]]: + """ + Fetch a list of results from Weblate. + :param url: The initial URL to fetch a response from. + :returns: The returned list of results. + """ + ret: list[dict[Any, Any]] = [] while url: response = self._get_weblate_response(url) + if not isinstance(response, dict): + raise ValueError(f"Unexpected response type for {url}") + assert_keys(response, ("results", "next"), url) + + res_val = response["results"] + if not isinstance(res_val, list): + raise ValueError(f"Unexpected 'results' type for {url}") + for res_item in res_val: + if not isinstance(res_item, dict): + raise ValueError(f"Unexpected 'results' item type for {url}") + ret.append(res_item) + # Continue to fetch the next page, if it is present. - # Expect the "next" url to be empty + # Expect the "next" url to be null # or the same url with "?page=2", etc. - url = response["next"] - ret.extend(response["results"]) + next_val = response["next"] + if next_val is not None and not isinstance(next_val, str): + raise ValueError(f"Unexpected 'next' type in response for {url}") + url = "" if next_val is None else next_val return ret - def _get_components(self): + def _get_components(self) -> dict[str, WeblateComponent]: + """ + Get all the Tor Browser Weblate components. + + We identify each Weblate component by its template path: the en-US + file path in the translation repository. + + NOTE: We assume that the template path uniquely identifies the + component, even though this is not guaranteed in general by Weblate. + + :returns: A dictionary of components, indexed by their en-US template + path. + """ + url = self._WEBLATE_API_TOR_COMPONENTS_URL if self._components is None: self._components = { - comp["template"]: { - "translations": None, - "translations-url": comp["translations_url"], - } - for comp in self._get_from_weblate( - "https://hosted.weblate.org/api/projects/tor/components/" + comp.template_path: comp + for comp in ( + WeblateComponent(data, url) for data in self._get_from_weblate(url) ) - if comp["template"] + if comp.template_path and comp.branch in self._BROWSER_BRANCHES } return self._components - def _get_translations(self, template): - component = self._get_components().get(template, None) + def _get_translations( + self, template_path: str + ) -> dict[str, WeblateTranslation] | None: + """ + Get the Weblate translations for a given Weblate component. + + We identify each translation by its file path: the locale-specific path + that the translated file is written to. + + :param template_path: The component's en-US template path. + :returns: A dictionary of translations, indexed by their locale file + paths. Or `None` if no components were found. + """ + component = self._get_components().get(template_path, None) if not component: - self.logger.warning(f"No component in weblate for {template}.") + self.logger.warning(f"No component in Weblate for {template_path}.") return None - if component["translations"] is None: - component["translations"] = { - trans["filename"]: { - "units": None, - "units-url": trans["units_list_url"], - } - for trans in self._get_from_weblate(component["translations-url"]) + if component.translations is None: + url = component.translations_url + component.translations = { + trans.locale_path: trans + for trans in ( + WeblateTranslation(data, url) + for data in self._get_from_weblate(url) + ) } - return component["translations"] + return component.translations + + def _get_units( + self, template_path: str, locale_path: str + ) -> dict[str, WeblateUnit] | None: + """ + Get the Weblate units for a given Weblate translation. + + We identify each unit by its string ID. - def _get_units(self, template, file): - translation_dict = self._get_translations(template) + :param template_path: The containing component's en-US template path. + :param locale_path: The translation's locale file path. + :returns: A dictionary of units, indexed by their string ID. Or `None` + if no translations could be found. + """ + translation_dict = self._get_translations(template_path) if translation_dict is None: return None - translation = translation_dict.get(file, None) + translation = translation_dict.get(locale_path, None) if translation is None: - self.logger.warning(f"No translation in weblate for {file}.") + self.logger.warning(f"No translation in Weblate for {locale_path}.") return None - if translation["units"] is None: - translation["units"] = { - unit["context"]: { - "translated": unit["translated"], - } - for unit in self._get_from_weblate(translation["units-url"]) + if translation.units is None: + url = translation.units_url + translation.units = { + unit.string_id: unit + for unit in ( + WeblateUnit(data, url) for data in self._get_from_weblate(url) + ) } - return translation["units"] + return translation.units - def is_translated(self, template_path, locale_path, string_id): + def is_translated(self, file_path: str, locale: str, string_id: str) -> bool: """ - Whether the given string is marked as translated on weblate. + Test whether the given string is marked as translated on Weblate. + + :param file_path: The file to check the translation of, relative to the + locale's directory. + :param locale: The locale to check for. + :param string_id: The ID of the string to check. """ + template_path = os.path.join("en-US", file_path) + locale_path = os.path.join(locale, file_path) unit_dict = self._get_units(template_path, locale_path) if unit_dict is None: return False unit = unit_dict.get(string_id, None) if unit is None: - self.logger.warning(f"No unit in weblate for {locale_path}:{string_id}.") + self.logger.warning(f"No unit in Weblate for {locale_path}:{string_id}.") return False - return unit["translated"] + return unit.translated class TorBrowserMigrationContext(MigrationContext): @@ -206,12 +557,18 @@ class TorBrowserMigrationContext(MigrationContext): Extension which adds some extra methods to use for tor-browser. """ - def __init__(self, locale, reference_dir, localization_dir): + def __init__(self, locale: str, reference_dir: str, localization_dir: str) -> None: super().__init__(locale, reference_dir, localization_dir) - def _fluent_keys(self, resource): + def _fluent_keys(self, resource: ast.Resource) -> Iterator[str]: + """ + Generate all the keys within a Fluent resource, where a key is an + identifier for a Term, a Message, a Term attribute, or a Message + attribute. + :yield: The Fluent keys. + """ # ast.Resource, want to extract all Message and Term identifiers, as - # well as their + # well as their attributes. for entry in resource.body: if not isinstance(entry, (ast.Term, ast.Message)): continue @@ -223,12 +580,13 @@ class TorBrowserMigrationContext(MigrationContext): for attr in entry.attributes: yield f"{key}.{attr.id.name}" - def tb_get_available_strings(self): + def tb_get_available_strings(self) -> set[tuple[str, str]]: """ - Return all the (path, string_id) pairs for all loaded localization + Get all the (path, string_id) pairs for all loaded localization resources. + :returns: The string pairs. """ - all_strings = set() + all_strings: set[tuple[str, str]] = set() # ctx.localization_resources is a dict containing all the loaded # localization_resources that have been added during add_transforms. # { localization_path: resource } @@ -237,14 +595,13 @@ class TorBrowserMigrationContext(MigrationContext): all_strings.update((path, key) for key in self._fluent_keys(resource)) else: # dictionary of { identifies: value } in legacy resource. - # For tor-browser, - # Only include references that differ from the en-US strings. all_strings.update((path, key) for key in resource.keys()) return all_strings - def tb_get_missing_resources(self): + def tb_get_missing_resources(self) -> set[str]: """ - Return the missing localization resources. + Get the missing localization resources. + :returns: The set of resource paths. """ return set( path @@ -253,25 +610,51 @@ class TorBrowserMigrationContext(MigrationContext): if path not in self.localization_resources ) - def tb_get_transform(self, target_path, transform_id): + def tb_get_transform( + self, target_path: str, transform_id: str + ) -> ast.Message | ast.Term | None: """ - Find the transformation node with the given id for the given path. + Find a transformation node. + :param target_path: The target path of the transformation. + :param transform_id: The ID of the transformation, AKA the string ID. + :returns: The matching transformation node, or `None` if there is no + match. """ for node in self.transforms[target_path]: if node.id.name == transform_id: return node return None - def tb_get_reference_entry(self, target_path, entry_id): + def tb_get_reference_entry( + self, target_path: str, transform_id: str + ) -> ast.Message | ast.Term | None: """ - Find the reference node that would be used for the given id and path. + Find a reference Message or Term for a given transformation. + :param target_path: The target path of the transformation. + :param transform_id: The ID of the transformation, AKA the string ID. + :returns: The matching reference entry for the transformation, or `None` + if there is no match. """ for entry in self.reference_resources[target_path].body: - if isinstance(entry, (ast.Term, ast.Message)) and entry.id.name == entry_id: + if ( + isinstance(entry, (ast.Term, ast.Message)) + and entry.id.name == transform_id + ): return entry.clone() return None +class RecipeModule(Protocol): + def migrate(self, context: MigrationContext) -> None: ... + + +class FluentSyntaxError(NamedTuple): + path: str + message: str + line: int + sample: str + + class TorBrowserMigrator: """ Performs a tor-browser migration. @@ -279,26 +662,38 @@ class TorBrowserMigrator: def __init__( self, - en_US_dir, - locale_dirs, - migrate_module, - weblate_metadata, - ): + en_US_ref_dir: str, + en_US_target_dir: str, + locale_dirs: dict[str, str], + recipe_module: RecipeModule, + weblate_metadata: WeblateMetadata, + ) -> None: + """ + :param en_US_ref_dir: The path to the en-US directory where reference + Fluent entries for the recipe can be found. + :param en_US_target_dir: The path to the en-US directory where existing + strings used by the recipe can be found. + :param locale_dirs: The collection of locales to run the migrations for. + :param recipe_module: The python module that contains the recipe + instructions. + :param weblate_metadata: The object to fetch Weblate metadata from. + """ self.logger = logging.getLogger("tor-browser-migrator") self.logger.setLevel(logging.INFO) - self.en_US_dir = en_US_dir + self.en_US_ref_dir = en_US_ref_dir + self.en_US_target_dir = en_US_target_dir self.locale_dirs = locale_dirs - self.migrate_module = migrate_module + self.recipe_module = recipe_module self.weblate_metadata = weblate_metadata - def run(self): + def run(self) -> None: """ Run the migration. """ if not self._check_en_US_resources(): sys.exit(1) - fluent_errors = [] + fluent_errors: list[FluentSyntaxError] = [] for locale, locale_dir in self.locale_dirs.items(): if locale == "en-US": print("", file=sys.stderr) @@ -316,12 +711,18 @@ class TorBrowserMigrator: "Migration does not need to be run again, but the following " "syntax errors should be fixed manually.\n" + "\n".join( - f"{in_red(full_path)}: line {line}: {message}: [[{sample}]]" - for full_path, message, line, sample in fluent_errors + f"{in_red(err.path)}: line {err.line}: {err.message}: [[{err.sample}]]" + for err in fluent_errors ) ) - def _run_locale(self, locale, locale_dir): + def _run_locale(self, locale: str, locale_dir: str) -> list[FluentSyntaxError]: + """ + Run a migration for a given locale. + :param locale: The name of the locale. + :param locale_dir: The path to the locale's directory. + :returns: A list of Fluent errors in the written files. + """ print("\n\n", file=sys.stderr) self.logger.info(f"Migrating '{in_red(locale)}' locale\n") @@ -358,7 +759,7 @@ class TorBrowserMigrator: available_strings = ctx.tb_get_available_strings() wrote_file = False - errors = [] + errors: list[FluentSyntaxError] = [] for target_path, reference in ctx.reference_resources.items(): translated_ids = [ @@ -422,13 +823,13 @@ class TorBrowserMigrator: break # Strings in legacy formats might have an entry in the file # that is just a copy of the en-US strings. - # For these we want to check the weblate metadata to ensure + # For these we want to check the Weblate metadata to ensure # it is a translated string. if not path.endswith( ".ftl" ) and not self.weblate_metadata.is_translated( - os.path.join("en-US", path), - os.path.join(locale, path), + path, + locale, string_id, ): can_transform = False @@ -436,7 +837,7 @@ class TorBrowserMigrator: f"Skipping transform {target_path}:{transform_id} " f"for '{locale}' locale because the string " f"{path}:{string_id} has not been translated on " - "weblate." + "Weblate." ) break if not can_transform: @@ -463,7 +864,9 @@ class TorBrowserMigrator: wrote_file = True # Collect any fluent parsing errors from the newly written file. errors.extend( - (full_path, message, line, sample) + FluentSyntaxError( + path=full_path, message=message, line=line, sample=sample + ) for message, line, sample in self._fluent_errors(full_content) ) @@ -471,27 +874,42 @@ class TorBrowserMigrator: self.logger.info(f"No files written for '{locale}' locale.") return errors - def _fluent_errors(self, fluent): + def _fluent_errors(self, fluent_content: str) -> Iterator[tuple[str, int, str]]: """ - Verify that the given fluent string can be parsed correctly. + Parse the given content as a Fluent file and generate all the syntax + errors. + :param fluent_content: The file content to parse. + :yield: Each encountered Fluent syntax error as a triple of the message, + the line number, and a sample. """ - resource = parse(fluent) + resource = parse(fluent_content) for entry in resource.body: if not isinstance(entry, ast.Junk): continue for annotation in entry.annotations: - line = fluent[0 : annotation.span.start].count("\n") + 1 - sample_start = max(annotation.span.start - 15, 0) - sample = "…" + fluent[sample_start : sample_start + 30] + "…" - yield annotation.message, line, sample - - def _get_migration_context(self, locale, locale_dir): - prev_missing_resources = set() + start = 0 + if annotation.span: + start = annotation.span.start + line = fluent_content[0:start].count("\n") + 1 + sample_start = max(start - 15, 0) + sample = "…" + fluent_content[sample_start : sample_start + 30] + "…" + yield annotation.message or "", line, sample + + def _get_migration_context( + self, locale: str, locale_dir: str + ) -> TorBrowserMigrationContext: + """ + Create a new migration context. + :param locale: The name of the locale. + :param locale_dir: The directory path for the locale. + :returns: The new context. + """ + prev_missing_resources: set[str] = set() while True: - ctx = TorBrowserMigrationContext(locale, self.en_US_dir, locale_dir) + ctx = TorBrowserMigrationContext(locale, self.en_US_ref_dir, locale_dir) try: - self.migrate_module.migrate(ctx) + self.recipe_module.migrate(ctx) except EmptyLocalizationError: # This case will be handled by missing_resources below. # NOTE: At the time of writing, add_transforms only throws if @@ -543,20 +961,28 @@ class TorBrowserMigrator: # before localization_resources was fully populated. prev_missing_resources = missing_resources - def _remove_if_empty(self, path): + def _remove_if_empty(self, path: str) -> None: + """ + Remove a file if it is empty. + :param path: The path to remove. + """ if os.stat(path).st_size: self.logger.warning(f"{path} is no longer empty. Not deleting.") return os.remove(path) - def _check_en_US_resources(self): - # We pass in the en-US directory as the localization directory, as well - # as the reference directory. - ctx = TorBrowserMigrationContext("en-US", self.en_US_dir, self.en_US_dir) + def _check_en_US_resources(self) -> bool: + """ + Verify that the migration recipe works as expected using en-US strings. + :returns: Whether the recipe worked as expected. + """ + ctx = TorBrowserMigrationContext( + "en-US", self.en_US_ref_dir, self.en_US_target_dir + ) have_error = False try: - self.migrate_module.migrate(ctx) + self.recipe_module.migrate(ctx) except EmptyLocalizationError: # Handle with localization_resources check. # NOTE: This throwing may have prevented further add_transforms from @@ -566,7 +992,9 @@ class TorBrowserMigrator: have_error = True except UnreadableReferenceError: # Reference filename is printed before this. - self.logger.error("Missing an en-US reference file.") + self.logger.error( + "Missing an en-US reference file. Is --reference-directory correct?" + ) return False # Check each transform would create the same entry in the target as the @@ -602,7 +1030,8 @@ class TorBrowserMigrator: reference_entry = ctx.tb_get_reference_entry(target_path, transform_id) if reference_entry is None: self.logger.error( - f"Missing en-US reference entry for transform {transform_name}" + f"Missing en-US reference entry for transform {transform_name}. " + "Is --reference-directory correct?" ) have_error = True continue @@ -625,9 +1054,50 @@ class TorBrowserMigrator: return not have_error +def main(recipe_path: str, translation_dir: str, locales: str, ref_dir: str) -> None: + """ + The main method. + :param recipe_path: The path to the recipe script. + :param translation_dir: The path to the translation repository directory. + :param locales: The locales to run the recipe for, separated by spaces. + :param ref_dir: The path to the reference directory. + """ + if not os.path.isfile(recipe_path): + print(f"{in_red(recipe_path)} is not a file.", file=sys.stderr) + sys.exit(1) + + recipe_module_spec = importlib.util.spec_from_file_location("_recipe", recipe_path) + recipe_module = cast( + RecipeModule, importlib.util.module_from_spec(recipe_module_spec) + ) + recipe_module_spec.loader.exec_module(recipe_module) + + def check_dir(path: str) -> str: + if not os.path.isdir(path): + print(f"{in_red(path)} is not a directory.", file=sys.stderr) + sys.exit(1) + return path + + translation_dir = check_dir(translation_dir) + en_US_target_dir = check_dir(os.path.join(translation_dir, "en-US")) + en_US_ref_dir = en_US_target_dir if ref_dir is None else check_dir(ref_dir) + + TorBrowserMigrator( + en_US_ref_dir, + en_US_target_dir, + { + locale: check_dir(os.path.join(translation_dir, locale)) + for locale in (l.strip() for l in locales.split(" ")) + if locale + }, + recipe_module, + WeblateMetadata(), + ).run() + + parser = argparse.ArgumentParser( description="Run a tor-browser localization migration. " - "See documentation within migrate_l10n.py." + "See documentation within migrate.py." ) parser.add_argument( "--translation-git", @@ -638,35 +1108,30 @@ parser.add_argument( parser.add_argument( "--locales", required=True, - metavar="<locale1> <locale2> ...", - help="Set of locales to restrict the migration to, separated by space.", + metavar="'<locale1> <locale2> ...'", + help="Set of locales to run the migration for, separated by space.", ) parser.add_argument( - "migration", - help="Migration to run, given as a python module. " - 'E.g. "l10n_migrations.my-migration-script".', + "--reference-directory", + metavar="<dir>", + help="Location of the en-US directory that contains the reference changes " + "that the recipe corresponds to. E.g. if the recipe constructs the string " + "`my-new-string` in `./tor-browser.ftl`, it should point to a directory " + "that contains the en-US `tor-browser.ftl` with `my-new-string` already " + "within it. Defaults to the `en-US/` sub-directory within the " + "translation-git directory.", +) +parser.add_argument( + "recipe", + help="The path to the migration recipe to run.", ) parsed_args = parser.parse_args() - -def check_dir(path): - if not os.path.isdir(path): - print(f"{in_red(path)} is not a directory.", file=sys.stderr) - sys.exit(1) - return path - - -translation_dir = check_dir(os.path.abspath(parsed_args.translation_git)) - -TorBrowserMigrator( - check_dir(os.path.join(translation_dir, "en-US")), - { - locale: check_dir(os.path.join(translation_dir, locale)) - for locale in (l.strip() for l in parsed_args.locales.split(" ")) - if locale - }, - importlib.import_module(parsed_args.migration), - WeblateMetadata(), -).run() +main( + parsed_args.recipe, + parsed_args.translation_git, + parsed_args.locales, + parsed_args.reference_directory, +) View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser/-/compare/1f8c6d9... -- View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser/-/compare/1f8c6d9... You're receiving this email because of your account on gitlab.torproject.org. Manage all notifications: https://gitlab.torproject.org/-/profile/notifications | Help: https://gitlab.torproject.org/help