GitLab

at 2025-02-11T11:55:20+01:00

Bug 42669: [Part 1] Add uniffi-rs project

Apps team forks uniffi and applies a patch to make the Kotlin
bindings no-op.

Bug 42669: [Part 2] Use custom uniffi-rs in application-services

The apps team custom uniffi-rs implementation removes
the dependecy on application-services Rust libraries.

Therefore, this commit also include changes to skip
building and including these libs (aka libmegazord)
in the final Android application.

 cargo_vendor-lox: submodule-update

 	$(rbm) build lox-wasm --step cargo_vendor --target alpha --target torbrowser-linux-x86_64

+cargo_vendor-uniffi-rs: submodule-update

+	$(rbm) build uniffi-rs --step cargo_vendor --target alpha --target torbrowser-linux-x86_64

+

 go_vendor-snowflake: submodule-update

 	$(rbm) build snowflake --step go_vendor --target alpha --target torbrowser-linux-x86_64

 Application Services is a collection of Rust components to enable integration

 with Mozilla online services, such as the Mozilla account, sync, etc...

+Since all of the application-services features are disabled by the Tor Project browsers,

+we don't build or include these Rust libraries in them. Even though these libraries

+are written in Rust, they are consumed by Kotlin. Application Services uses uniffi

+to auto-generate the Kotlin code that communicates with Rust. We have developed a

+custom generator for uniffi (see `projects/uniffi-rs`) to generate no-op bindings,

+i.e., bindings that don't call the Rust code and are therefore a dead-end.

+

 We do not fork this project, because we apply a minimal set of patches mainly

 needed for offline builds.

 ## Vendored Rust dependencies

-Application Services is written mainly in Rust and it mnanages external

+Application Services is written mainly in Rust and it manages external

 dependencies through cargo.

+

+Although most Rust libraries aren't built by this project, the `nimbus-fml`

+project -- a command line tool used by other projects -- is still built.

+

 Reproduciblity is guaranteed by the provided `Cargo.lock`.

 We run offline builds, so we create the dependency archive in a separate step

 A procedure to create this file is documented in

 [tor-browser-build#40855](https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40855#note_2906041).

-## Other dependencies

-

-Finally, Application Services depends on two C libraries:

-[NSS](https://firefox-source-docs.mozilla.org/security/nss/index.html) and

-[SQLCipher](https://www.zetetic.net/sqlcipher/).

-We used to have separate tor-browser-build projects for them, but they were

-almost an exact copy of the scripts included in this repository.

-Keeping them updated wasn't trivial, so we decided to run Mozilla's scripts

-instead.

-

 # Caveats

 ## Git repository information

+diff --git a/components/fxa-client/android/src/main/java/mozilla/appservices/fxaclient/Config.kt b/components/fxa-client/android/src/main/java/mozilla/appservices/fxaclient/Config.kt

+index 78c16dd0..d2615fa7 100644

+--- a/components/fxa-client/android/src/main/java/mozilla/appservices/fxaclient/Config.kt

++++ b/components/fxa-client/android/src/main/java/mozilla/appservices/fxaclient/Config.kt

+@@ -13,4 +13,5 @@ fun FxaServer.contentUrl() = when (this) {

+     is FxaServer.China -> "https://accounts.firefox.com.cn"

+     is FxaServer.LocalDev -> "http://127.0.0.1:3030"

+     is FxaServer.Custom -> this.url

++    is FxaServer.__NOOP -> ""

+ }

+diff --git a/components/nimbus/android/src/main/java/org/mozilla/experiments/nimbus/Nimbus.kt b/components/nimbus/android/src/main/java/org/mozilla/experiments/nimbus/Nimbus.kt

+index 8c05be9c..6bf2ec15 100644

+--- a/components/nimbus/android/src/main/java/org/mozilla/experiments/nimbus/Nimbus.kt

++++ b/components/nimbus/android/src/main/java/org/mozilla/experiments/nimbus/Nimbus.kt

+@@ -532,6 +532,8 @@ open class Nimbus(

+                         ),

+                     )

+                 }

++

++                EnrollmentChangeEventType.__NOOP -> {}

+             }

+         }

+     }

+diff --git a/components/places/android/src/main/java/mozilla/appservices/places/PlacesConnection.kt b/components/places/android/src/main/java/mozilla/appservices/places/PlacesConnection.kt

+index 282f4e90..db8b00a3 100644

+--- a/components/places/android/src/main/java/mozilla/appservices/places/PlacesConnection.kt

++++ b/components/places/android/src/main/java/mozilla/appservices/places/PlacesConnection.kt

+@@ -289,6 +289,7 @@ internal fun VisitType.toInt(): Int {

+         VisitType.FRAMED_LINK -> 8

+         VisitType.RELOAD -> 9

+         VisitType.UPDATE_PLACE -> 10

++        VisitType.__NOOP -> -1

+     }

+ }

+

+diff --git a/megazords/full/android/build.gradle b/megazords/full/android/build.gradle

+index 822ce180..ccdc6c8a 100644

+--- a/megazords/full/android/build.gradle

++++ b/megazords/full/android/build.gradle

+@@ -54,6 +54,8 @@ configurations {

+ }

+

+ cargo {

++    cargoCommand = "${rootDir}/megazords/full/android/cargo-no-build.sh"

++

+     // The directory of the Cargo.toml to build.

+     module = '..'

+

+diff --git a/publish.gradle b/publish.gradle

+index 71f5d55b..027665ff 100644

+--- a/publish.gradle

++++ b/publish.gradle

+@@ -194,7 +194,7 @@ ext.configureUniFFIBindgen = { udlFilePath ->

+         def uniffiGeneratedPath = "generated/source/uniffi/${variant.name}/java"

+         def t = tasks.register("generate${variant.name.capitalize()}UniFFIBindings", Exec) {

+             workingDir project.rootDir

+-            commandLine '/usr/bin/env', 'cargo', 'uniffi-bindgen', 'generate', "${project.projectDir}/${udlFilePath}", '--language', 'kotlin', '--out-dir', "${buildDir}/${uniffiGeneratedPath}"

++            commandLine '/usr/bin/env', "${rootProject.projectDir}/uniffi-rs/uniffi-bindgen", 'generate', "${project.projectDir}/${udlFilePath}", '--language', 'kotlin', '--out-dir', "${buildDir}/${uniffiGeneratedPath}"

+             outputs.dir "${buildDir}/${uniffiGeneratedPath}"

+             // Re-generate if the interface definition changes.

+             inputs.file "${project.projectDir}/${udlFilePath}"
-diff --git a/libs/build-all.sh b/libs/build-all.sh

-index 650c1299..6c4e5404 100755

---- a/libs/build-all.sh

-+++ b/libs/build-all.sh

-@@ -128,6 +128,15 @@ echo $'\

-      fi

- ' | patch "${NSS_SRC_PATH}/nspr/configure"

- 

-+rm -f python

-+ln -s /usr/bin/python3 python

-+export PATH=$(pwd):$PATH

-+patch_13028=$(realpath bug_13028.patch)

-+pushd $NSS_SRC_PATH

-+# Apply our proxy bypass defense-in-depth here as well to be on the safe side.

-+patch -p2 < $patch_13028

-+popd

-+

- if [[ "${PLATFORM}" == "ios" ]]

- then

-   ./build-all-ios.sh "${SQLCIPHER_SRC_PATH}" "${NSS_SRC_PATH}"
 +++ b/components/support/nimbus-fml/src/intermediate_representation.rs

 @@ -237,10 +237,10 @@ pub struct FeatureManifest {

      pub(crate) about: AboutBlock,

- 

+

      #[serde(default)]

 -    pub(crate) imported_features: HashMap<ModuleId, BTreeSet<String>>,

 +    pub(crate) imported_features: BTreeMap<ModuleId, BTreeSet<String>>,

- 

+

      #[serde(default)]

 -    pub(crate) all_imports: HashMap<ModuleId, FeatureManifest>,

 +    pub(crate) all_imports: BTreeMap<ModuleId, FeatureManifest>,

  }

- 

+

  impl TypeFinder for FeatureManifest {

 diff --git a/components/support/nimbus-fml/src/parser.rs b/components/support/nimbus-fml/src/parser.rs

 index 49cace525..cdf692b86 100644

          // We associate only the feature ids with the manifest we're loading in this method.

 -        let mut imported_feature_id_map = HashMap::new();

 +        let mut imported_feature_id_map = BTreeMap::new();

- 

+

          for block in &frontend.imports {

              // 1. Load the imported manifests in to the hash map.

 @@ -328,7 +328,7 @@ impl Parser {

-From 2f0888c348561249d3083555db33c5619840dbfa Mon Sep 17 00:00:00 2001

-From: Mike Perry <mikeperry-git@torproject.org>

-Date: Mon, 29 Sep 2014 14:30:19 -0700

-Subject: [PATCH] Bug 13028: Prevent potential proxy bypass cases.

-

-It looks like these cases should only be invoked in the NSS command line

-tools, and not the browser, but I decided to patch them anyway because there

-literally is a maze of network function pointers being passed around, and it's

-very hard to tell if some random code might not pass in the proper proxied

-versions of the networking code here by accident.

-

-diff --git a/security/nss/lib/certhigh/ocsp.c b/security/nss/lib/certhigh/ocsp.c

-index cea8456606bf..86fa971cfbef 100644

---- a/security/nss/lib/certhigh/ocsp.c

-+++ b/security/nss/lib/certhigh/ocsp.c

-@@ -2932,6 +2932,14 @@ ocsp_ConnectToHost(const char *host, PRUint16 port)

-     PRNetAddr addr;

-     char *netdbbuf = NULL;

- 

-+    // XXX: Do we need a unittest ifdef here? We don't want to break the tests, but

-+    // we want to ensure nothing can ever hit this code in production.

-+#if 1

-+    printf("Tor Browser BUG: Attempted OSCP direct connect to %s, port %u\n", host,

-+            port);

-+    goto loser;

-+#endif

-+

-     sock = PR_NewTCPSocket();

-     if (sock == NULL)

-         goto loser;

-diff --git a/security/nss/lib/libpkix/pkix_pl_nss/module/pkix_pl_socket.c b/security/nss/lib/libpkix/pkix_pl_nss/module/pkix_pl_socket.c

-index e8698376b5be..85791d84a932 100644

---- a/security/nss/lib/libpkix/pkix_pl_nss/module/pkix_pl_socket.c

-+++ b/security/nss/lib/libpkix/pkix_pl_nss/module/pkix_pl_socket.c

-@@ -1334,6 +1334,13 @@ pkix_pl_Socket_Create(

-                     plContext),

-                     PKIX_COULDNOTCREATESOCKETOBJECT);

- 

-+        // XXX: Do we need a unittest ifdef here? We don't want to break the tests, but

-+        // we want to ensure nothing can ever hit this code in production.

-+#if 1

-+        printf("Tor Browser BUG: Attempted pkix direct socket connect\n");

-+        PKIX_ERROR(PKIX_PRNEWTCPSOCKETFAILED);

-+#endif

-+

-         socket->isServer = isServer;

-         socket->timeout = timeout;

-         socket->clientSock = NULL;

-@@ -1433,6 +1440,13 @@ pkix_pl_Socket_CreateByName(

- 

-         localCopyName = PL_strdup(serverName);

- 

-+        // XXX: Do we need a unittest ifdef here? We don't want to break the tests, but

-+        // we want to ensure nothing can ever hit this code in production.

-+#if 1

-+        printf("Tor Browser BUG: Attempted pkix direct connect to %s\n", serverName);

-+        PKIX_ERROR(PKIX_PRNEWTCPSOCKETFAILED);

-+#endif

-+

-         sepPtr = strchr(localCopyName, ':');

-         /* First strip off the portnum, if present, from the end of the name */

-         if (sepPtr) {

-@@ -1582,6 +1596,13 @@ pkix_pl_Socket_CreateByHostAndPort(

-         PKIX_ENTER(SOCKET, "pkix_pl_Socket_CreateByHostAndPort");

-         PKIX_NULLCHECK_THREE(hostname, pStatus, pSocket);

- 

-+        // XXX: Do we need a unittest ifdef here? We don't want to break the tests, but

-+        // we want to ensure nothing can ever hit this code in production.

-+#if 1

-+        printf("Tor Browser BUG: Attempted pkix direct connect to %s, port %u\n", hostname,

-+                portnum);

-+        PKIX_ERROR(PKIX_PRNEWTCPSOCKETFAILED);

-+#endif

- 

-         prstatus = PR_GetHostByName(hostname, buf, sizeof(buf), &hostent);

- 

--- 

-2.27.0

-
 offline=true

 EOF

-pushd libs

-ln -s $rootdir/[% c("input_files_by_name/nss") %] ./

-ln -s $rootdir/bug_13028.patch

-patch -p2 < $rootdir/apply-bug-13028.diff

-patch -p2 < $rootdir/no-ndk-lookup.diff

-./build-all.sh desktop

-./build-all.sh android

+tar -xf $rootdir/[% c('input_files_by_name/uniffi-rs') %]

+

+# We add a suffix to the version to make it super specific.

+# This is useful for developer builds, which seem to try to fetch

+# latest versions of dependencies sometimes.

+sed -i '$ s/$/-TORBROWSER/' version.txt

+

+cp $rootdir/cargo-no-build.sh megazords/full/android/

+chmod +x megazords/full/android/cargo-no-build.sh

+patch -p1 < $rootdir/a-s-noop.diff

+

+pushd components/external/glean

+cp $rootdir/cargo-no-build.sh glean-core/android-native/

+cp $builddir/uniffi-rs/uniffi-bindgen glean-core/android-native/

+patch -p1 < $rootdir/glean-noop.diff

 popd

 [% IF c('var/fetch_gradle_dependencies') %]

   export LANG=C.UTF-8

   patch -p1 < $rootdir/local-repository.diff

   patch -p1 < $rootdir/ohttp-no-git.diff

-  patch -p1 < $rootdir/bug40485.diff

+  patch -p1 < $rootdir/bug40485-nimbus-fml-reproducibility.diff

   patch -p1 < $rootdir/offline-nimbus-fml.diff

   gradle_flags="--offline --no-daemon"

   gradle $gradle_flags assembleRelease

+#!/bin/bash

+

+if [[ "$1" == "build" ]]; then

+    echo "cargo build is disabled."

+    exit 0

+else

+    command cargo "$@"

+fi
   gradle_dependencies_version: 11

   gradle_version: 8.8

   glean_parser: 14.0.1

-  nss_version: '3.101'

-  nspr_version: '4.35'

+  # This is the only project for which the rust lib will actually be built

+  cargo_vendor_subdir: components/support/nimbus-fml

 steps:

   build:

       # builds when build time differ a lot, we need to do periodic rebuilds:

       # https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/937#note_3009480

       rebuild_date: '2024-09-12'

-      arch_deps:

-        # Needed to build NSS

-        - gyp

     input_files:

       - project: container-image

         pkg_type: build

       - project: ninja

         name: ninja

         pkg_type: build

+      - project: uniffi-rs

+        name: uniffi-rs

+        pkg_type: build

       # Only Application Services currently requires build tools 30.0.3.

       # So, download them only here, rather than adding them to the shared

       # toolchain.

       - URL: 'https://dl.google.com/android/repository/build-tools_r30.0.3-linux.zip'

         name: build_tools_30

         sha256sum: 24593500aa95d2f99fb4f10658aae7e65cb519be6cd33fa164f15f27f3c4a2d6

-      # NSS version ans sha256 are in libs/build-all.sh.

-      - URL: 'https://ftp.mozilla.org/pub/security/nss/releases/NSS_[% c("var/nss_version") | replace("\\.", "_") %]_RTM/src/nss-[% c("var/nss_version") %]-with-nspr-[% c("var/nspr_version") %].tar.gz'

-        name: nss

-        sha256sum: 566faa9283ff3d9a7d6c44272df6e4330e3e06ca4e841a68840d31b27c9161c4

       - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'

         name: gradle-dependencies

         exec: '[% INCLUDE "fetch-gradle-dependencies" %]'

         enable: '[% !c("var/fetch_gradle_dependencies") %]'

       - filename: gen_gradle_deps_file.sh

         enable: '[% c("var/fetch_gradle_dependencies") %]'

-      - filename: bug_13028.patch

-      - filename: apply-bug-13028.diff

-      # Delete when this patch is included upstream

-      - filename: bug40485.diff

-      - filename: no-ndk-lookup.diff

+      # tor-browser-build#40485: Make sure the Kotlin output of nimbus-fml

+      # has deterministic order for reproducibility.

+      # Delete when this patch is included upstream.

+      - filename: bug40485-nimbus-fml-reproducibility.diff

       # as-ohttp-client lists both the version for ohttp and a git repo + rev,

       # but this breaks the vendoring for offline builds.

       - filename: ohttp-no-git.diff

       - filename: offline-nimbus-fml.diff

+      - filename: glean-noop.diff

+      - filename: a-s-noop.diff

+      - filename: cargo-no-build.sh

   list_toolchain_updates:

     git_hash: 'v[% c("version") %]'

+diff --git a/glean-core/android-native/build.gradle b/glean-core/android-native/build.gradle

+index 48769651..ade00ec3 100644

+--- a/glean-core/android-native/build.gradle

++++ b/glean-core/android-native/build.gradle

+@@ -57,6 +57,8 @@ android {

+ }

+

+ cargo {

++    cargoCommand = "${rootDir}/glean-core/android-native/cargo-no-build.sh"

++

+     // The directory of the Cargo.toml to build.

+     module = '../bundle-android'

+

+diff --git a/glean-core/android/build.gradle b/glean-core/android/build.gradle

+index 3568e40b..05f28296 100644

+--- a/glean-core/android/build.gradle

++++ b/glean-core/android/build.gradle

+@@ -221,7 +221,7 @@ android.libraryVariants.all { variant ->

+     def udlFilePath = "../src/glean.udl"

+     def t = tasks.register("generate${variant.name.capitalize()}UniFFIBindings", Exec) {

+         workingDir project.rootDir

+-        commandLine 'cargo', 'uniffi-bindgen', 'generate', '--no-format', "${project.projectDir}/${udlFilePath}", '--language', 'kotlin', '--out-dir', "${buildDir}/${uniffiGeneratedPath}"

++        commandLine '${rootDir}/glean-core/android-native/uniffi-bindgen', 'generate', '--no-format', "${project.projectDir}/${udlFilePath}", '--language', 'kotlin', '--out-dir', "${buildDir}/${uniffiGeneratedPath}"

+         outputs.dir "${buildDir}/${uniffiGeneratedPath}"

+         // Re-generate if the interface definition changes.

+         inputs.file "${project.projectDir}/../src/glean.udl"
 check_update_needed gradle "$needed" "$current"

-# nss-nspr

+# uniffi

 read -d '' p << 'EOF' || true

-if (m/NSS_ARCHIVE="nss-(.*-with-nspr-.*)\\.tar\\.gz"/) {

+if (m/^\\s*uniffi\\s*=\\s*"([^"]*)"/) {

   print $1;

   exit;

 }

 EOF

-needed=$(cat libs/build-all.sh | perl -ne "$p")

-current='[% c("var/nss_version") %]-with-nspr-[% c("var/nspr_version") %]'

-check_update_needed nss-nspr "$needed" "$current"
+needed=$(cat Cargo.toml | perl -ne "$p")

+current='[% pc("uniffi-rs", "version") %]'

+check_update_needed uniffi "$needed" "$current"
-diff --git a/libs/android_defaults.sh b/libs/android_defaults.sh

-index 2cfcc4206..9e3d3de97 100755

---- a/libs/android_defaults.sh

-+++ b/libs/android_defaults.sh

-@@ -1,11 +1,7 @@

- #!/usr/bin/env bash

- 

--# Find the NDK.

--pushd ..

--NDK_VERSION=$(./gradlew -q printNdkVersion | tail -1)

--export ANDROID_NDK_HOME="$ANDROID_HOME/ndk/$NDK_VERSION"

--export ANDROID_NDK_ROOT="$ANDROID_NDK_HOME"

--popd || exit

-+# Do not try to find the NDK, as it does not work offline.

-+# We already define the needed variables in our environment.

- 

- if [[ -z "${ANDROID_NDK_API_VERSION:-}" ]]; then

-     export ANDROID_NDK_API_VERSION=21
+#!/bin/bash

+[% c("var/set_default_env") -%]

+distdir=/var/tmp/dist

+builddir=/var/tmp/build/[% project %]

+mkdir -p $distdir/[% project %]

+tar -C /var/tmp/dist -xf $rootdir/[% c('input_files_by_name/rust') %]

+export PATH="/var/tmp/dist/rust/bin:$PATH"

+[% IF c("var/linux") -%]

+  export LD_LIBRARY_PATH="/var/tmp/dist/rust/lib:$LD_LIBRARY_PATH"

+[% END -%]

+mkdir -p /var/tmp/build

+tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.[% c('compress_tar') %]

+

+# Now prepare the offline build

+# Move the directory for hardcoding the path in .cargo/config

+mv /var/tmp/build/[% project %]-[% c('version') %] $builddir

+tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %]

+cd $builddir

+cat >> .cargo/config << 'EOF'

+[source.crates-io]

+replace-with = "vendored-sources"

+

+[source.vendored-sources]

+directory = "/var/tmp/build/uniffi-rs/vendor"

+EOF

+

+cargo build --release --frozen --target x86_64-unknown-linux-gnu

+mv target/x86_64-unknown-linux-gnu/release/uniffi-bindgen $distdir/[% project %]

+

+cd $distdir

+[% c('tar', {

+        tar_src => [ project ],

+        tar_args => '-caf ' _ dest_dir _ '/' _ c('filename'),

+    }) %]
+# vim: filetype=yaml sw=2

+version: 0.27.1

+git_hash: bfb52effb0292f16d4c030d622887781639cbd1f

+git_url: https://gitlab.torproject.org/tpo/applications/uniffi-rs.git

+container:

+  use_container: 1

+

+steps:

+  build:

+    filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.[% c("compress_tar") %]'

+    input_files:

+      - project: container-image

+        pkg_type: build

+      - project: rust

+        name: rust

+      - name: cargo_vendor

+        project: uniffi-rs

+        pkg_type: cargo_vendor

+        norec:

+          sha256sum: 12a61a3dbe65a3946ed58fff3d849d11c9ade5818e2ef5771d2a2a531e25f553
       export PATH="/var/tmp/dist/rust/bin:$PATH"

       tar -xf [% project %]-[% c('version') %].tar.[% c('compress_tar') %]

       cd [% project %]-[% c('version') %]

-      cargo vendor --locked vendor

+      cargo vendor --locked vendor --manifest-path [% IF c('var/cargo_vendor_subdir') %][% c('var/cargo_vendor_subdir') %]/[% END %]Cargo.toml

       [% c('tar', {

         tar_src => [ 'vendor' ],

         tar_args => '-caf ' _ dest_dir _ '/' _ c('filename'),

Pier Angelo Vendrame pushed to branch main at The Tor Project / Applications / tor-browser-build

Commits:

15 changed files:

Changes: