- tbb-commits - lists.torproject.org

[tor-browser-build/master] Bug 40055: Add documentation for Glean offline dependency update
by sysrqb＠torproject.org 04 Sep '20

04 Sep '20

commit 1de6d966381efa6b3e5ce5cf06295597171be6c1 Author: Georg Koppen <gk(a)torproject.org> Date: Wed Aug 26 10:21:37 2020 +0000 Bug 40055: Add documentation for Glean offline dependency update --- projects/common/how-to-update-glean-parser.txt | 44 ++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/projects/common/how-to-update-glean-parser.txt b/projects/common/how-to-update-glean-parser.txt new file mode 100644 index 0000000..a125f5f --- /dev/null +++ b/projects/common/how-to-update-glean-parser.txt @@ -0,0 +1,44 @@ +In order to be able to do offline builds for mobile we need to collect and +distribute Glean dependencies beforehand.[1] + +There are three steps involved in the process: + 1) Finding out the `glean_parser` version + 2) Downloading and verifying the dependencies + 3) Bundling the dependencies up and pushing them to the sources mirror + +1) + +The `glean_parser` version is available in glean-core/python/setup.py in the +Glean code repositoroy.[2] So, make sure which Glean version your +`android-components` tag is using (`mozilla_glean` in +buildSrc/src/main/java/Dependencies.kt has it) and then track `glean_parser` +down in the respective Glean tag. + +2) + +Create a directory `glean-wheels` and change into it. Then download packages for +`glean_parser` and its dependencies using + +`python3 -m pip download glean_parser==$glean_parser` + +($glean_parser being the version from step 1 above and Python 3 needs to be 3.6; +ideally, we take the artifact we built and will use during the build later on as +well.) + +There is no really good way (yet) to verify that `pip` gave us actually what we +wanted. We can download `glean_parser` and its dependencies using a different +network path, though, e.g. by using `torsocks`. Comparing the results of both +downloads should yield the exact same binaries. + +3) + +Update the `glean_parser` version in `rbm.conf, tar `glean-wheels` up and +compress it: + +`tar cfj glean-wheels-$glean_parser.tar.bz2 glean-wheels` + +Upload the compressed tarball to the sources mirror and update the SHA-256 sum of +it in the respective projects. + +[1] See: https://bugzilla.mozilla.org/show_bug.cgi?id=1651662 for details. +[2] https://github.com/mozilla/glean

1 0

[tor-browser-build/master] Bug 40055: Integrate building Glean in offline mode
by sysrqb＠torproject.org 04 Sep '20

04 Sep '20

commit e414bcd0a51859acc3fbdc48366b612b20425be3 Author: Georg Koppen <gk(a)torproject.org> Date: Thu Sep 3 09:46:55 2020 +0000 Bug 40055: Integrate building Glean in offline mode Using Glean 32.1.1 (via android-components 56.0.0 or later) did not land yet in the application-services tag which Fenix and android-components depend on. Thus, we patch application-services to include the earliest version of android-components shipping Glean 32.1.1. --- projects/android-components/build | 11 ++++++-- projects/android-components/config | 6 ++++ projects/application-services/1651662.patch | 22 +++++++++++++++ projects/application-services/build | 8 ++++++ projects/application-services/config | 9 +++++- .../gradle-dependencies-list.txt | 32 +++++++++++----------- projects/fenix/build | 7 +++++ projects/fenix/config | 6 ++++ rbm.conf | 2 ++ 9 files changed, 84 insertions(+), 19 deletions(-) diff --git a/projects/android-components/build b/projects/android-components/build index a024861..3cac44a 100644 --- a/projects/android-components/build +++ b/projects/android-components/build @@ -3,10 +3,11 @@ [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] distdir=/var/tmp/dist builddir=/var/tmp/build/[% project %] -mkdir -p $distdir/[% project %] -mkdir -p /var/tmp/build +mkdir $distdir/[% project %] +mkdir /var/tmp/build [% IF !c("var/fetch_gradle_dependencies") %] + [% pc('python', 'var/setup', { python_tarfile => c('input_files_by_name/python') }) %] # XXX: Make this more generic and not only for geckoview-beta. geckoview_version=$(find . -type f -name geckoview-beta-*.aar | cut -d \- -f 6 | cut -d \. -f 1-3) gradle_repo=$rootdir/[% c('input_files_by_name/gradle-dependencies') %] @@ -29,6 +30,12 @@ patch -p1 < $rootdir/git.patch # `assemble` instead for now. gradle --debug -no-daemon assemble [% ELSE %] + # Prepare Glean dependencies for offline build + tar -xjf $rootdir/glean-parser-[% c('var/glean_parser') %].tar.bz2 + # We need to set `LC_ALL` and `LANG` to something that is not ASCII as encoding + # otherwise `click` barfs. See: https://click.palletsprojects.com/python3/ + export LC_ALL=C.UTF-8 + export LANG=C.UTF-8 patch -p1 < $rootdir/mavenLocal.patch # Make sure our GeckoView dependency is used. XXX: Make this more generic and diff --git a/projects/android-components/config b/projects/android-components/config index 3bdca82..56380e6 100644 --- a/projects/android-components/config +++ b/projects/android-components/config @@ -23,6 +23,9 @@ input_files: - name: application-services project: application-services enable: '[% !c("var/fetch_gradle_dependencies") %]' + - name: python + project: python + enable: '[% !c("var/fetch_gradle_dependencies") %]' - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]' name: gradle-dependencies exec: '[% INCLUDE "fetch-gradle-dependencies" %]' @@ -30,3 +33,6 @@ input_files: - filename: git.patch - filename: mavenLocal.patch enable: '[% !c("var/fetch_gradle_dependencies") %]' + - URL: https://people.torproject.org/~gk/mirrors/sources/glean-parser-[% c('var/glean_parser') %].tar.bz2 + sha256sum: 69234aa1ae38fd5dda4de71bd5e536795073703c3ed5bb994b9b8485bd7a55dc + enable: '[% !c("var/fetch_gradle_dependencies") %]' diff --git a/projects/application-services/1651662.patch b/projects/application-services/1651662.patch new file mode 100644 index 0000000..0275da2 --- /dev/null +++ b/projects/application-services/1651662.patch @@ -0,0 +1,22 @@ +From 1818764e69145839ab33a2f885139ea320f2283f Mon Sep 17 00:00:00 2001 +From: Georg Koppen <gk(a)torproject.org> +Date: Wed, 26 Aug 2020 15:56:49 +0000 +Subject: [PATCH] 1651662 + + +diff --git a/build.gradle b/build.gradle +index 58a4ddae..c6e0be82 100644 +--- a/build.gradle ++++ b/build.gradle +@@ -4,7 +4,7 @@ buildscript { + ext.kotlin_version = '1.3.71' + ext.jna_version = '5.2.0' + ext.android_gradle_plugin_version = '3.6.0' +- ext.android_components_version = '47.0.0' ++ ext.android_components_version = '56.0.0' + + ext.build = [ + ndkVersion: "21.3.6528147", // Keep it in sync in TC Dockerfile. +-- +2.28.0 + diff --git a/projects/application-services/build b/projects/application-services/build index 051163f..5c6b5e0 100644 --- a/projects/application-services/build +++ b/projects/application-services/build @@ -29,6 +29,7 @@ export ANDROID_NDK_API_VERSION=[% pc("android-toolchain", "var/android_ndk_versi export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 [% IF !c('var/fetch_gradle_dependencies') %] + [% pc('python', 'var/setup', { python_tarfile => c('input_files_by_name/python') }) %] gradle_repo=$rootdir/[% c('input_files_by_name/gradle-dependencies') %] cp -r $gradle_repo/dl/android/maven2/* $gradle_repo cp -r $gradle_repo/m2/* $gradle_repo @@ -82,11 +83,18 @@ patch -p1 < $rootdir/no-git.patch # complexity. That's needed for running unit tests later on, though. patch -p1 < $rootdir/target.patch patch -p1 < $rootdir/1651660.patch +patch -p1 < $rootdir/1651662.patch [% IF c('var/fetch_gradle_dependencies') %] # XXX: `assemble` is still not enough to see all fetched dependencies via # Gradle's --debug. See: tor-browser-build#40056. $GRADLE_HOME/gradle-6.3/bin/gradle --debug --no-daemon assemble [% ELSE %] + # Prepare Glean dependencies for offline build + tar -xjf $rootdir/glean-parser-[% c('var/glean_parser') %].tar.bz2 + # We need to set `LC_ALL` and `LANG` to something that is not ASCII as encoding + # otherwise `click` barfs. See: https://click.palletsprojects.com/python3/ + export LC_ALL=C.UTF-8 + export LANG=C.UTF-8 patch -p1 < $rootdir/mavenLocal.patch gradle_flags="--offline --no-daemon -Dmaven.repo.local=$gradle_repo" $GRADLE_HOME/gradle-6.3/bin/gradle $gradle_flags assembleRelease diff --git a/projects/application-services/config b/projects/application-services/config index 50986bb..1d9fd79 100644 --- a/projects/application-services/config +++ b/projects/application-services/config @@ -8,7 +8,7 @@ var: container: use_container: 1 # This should be updated when the list of gradle dependencies is changed. - gradle_dependencies_version: 1 + gradle_dependencies_version: 2 # This should be updated when the list of rust dependencies is changed. rust_vendor_version: 1 # For armv7 we need a different target than the default one we use for @@ -71,10 +71,16 @@ input_files: name: sqlcipher-x86_64 target_prepend: - torbrowser-android-x86_64 + - name: python + project: python + enable: '[% !c("var/fetch_gradle_dependencies") %]' - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]' name: gradle-dependencies exec: '[% INCLUDE "fetch-gradle-dependencies" %]' enable: '[% !c("var/fetch_gradle_dependencies") %]' + - URL: https://people.torproject.org/~gk/mirrors/sources/glean-parser-[% c('var/glean_parser') %].tar.bz2 + sha256sum: 69234aa1ae38fd5dda4de71bd5e536795073703c3ed5bb994b9b8485bd7a55dc + enable: '[% !c("var/fetch_gradle_dependencies") %]' # `cargo vendor vendor` in the `application-services` directory has vendored # the sources. - URL: https://people.torproject.org/~gk/mirrors/sources/application-services-vend… c('var/rust_vendor_version') %].tar.bz2 @@ -84,3 +90,4 @@ input_files: enable: '[% !c("var/fetch_gradle_dependencies") %]' - filename: target.patch - filename: 1651660.patch + - filename: 1651662.patch diff --git a/projects/application-services/gradle-dependencies-list.txt b/projects/application-services/gradle-dependencies-list.txt index ffad4f0..3978277 100644 --- a/projects/application-services/gradle-dependencies-list.txt +++ b/projects/application-services/gradle-dependencies-list.txt @@ -403,22 +403,22 @@ b51f8867c92b6a722499557fc3a1fdea77bdf9ef574722fe90ce436a29559454 | https://jcent fb40265f982548212ff82e362e59732b2187ec6f0d80182885c14ef1f982827a | https://jcenter.bintray.com/org/sonatype/oss/oss-parent/9/oss-parent-9.pom 13009fb5ede3cf2be5a8d0f1602155aeaa0ce5ef5f9366892bd258d8d3d4d2b1 | https://jcenter.bintray.com/org/yaml/snakeyaml/1.23/snakeyaml-1.23.jar 1e1beb22ca906209700122b5cacf6f2719324538c5b1e3c27bf91564c8d31dbd | https://jcenter.bintray.com/org/yaml/snakeyaml/1.23/snakeyaml-1.23.pom -4bca00bc684d935cf3675d3437113d1284bb94412b6b9aea03194f946c719a9c | https://maven.mozilla.org/maven2/org/mozilla/components/concept-fetch/47.0.… -c2ec8bc73e7e1a86b53f1c8ac8e2f09eee5e31b6bdc291f31495e3080ab66261 | https://maven.mozilla.org/maven2/org/mozilla/components/concept-fetch/47.0.… -5df22d5e7864c593c0b78624304d3046514177cd819bada0ce5927ed22e38bdf | https://maven.mozilla.org/maven2/org/mozilla/components/service-glean/47.0.… -c062918be5f021628bf9705898a9e7598ed340f5ec971033ab0ea239ee6efaeb | https://maven.mozilla.org/maven2/org/mozilla/components/service-glean/47.0.… -c959adf0ae40a073b3db4e21c1f6a9a961bbb7d6cc33c569492e2268b8a473a1 | https://maven.mozilla.org/maven2/org/mozilla/components/support-base/47.0.0… -0f47f76b38975b5407b212f1aa4dd1daabde311989ff934360afa529c1c77d92 | https://maven.mozilla.org/maven2/org/mozilla/components/support-base/47.0.0… -2813838b7bb4d2973b3663f2aac08b36c3db9a372d266eb228fba98bfcb09017 | https://maven.mozilla.org/maven2/org/mozilla/components/support-ktx/47.0.0/… -0d72828ec6af2fe0b7021c2cada48204a47ccd6e47eeea35dd2b51c24df3b8ff | https://maven.mozilla.org/maven2/org/mozilla/components/support-ktx/47.0.0/… -bd295b27f098eed0aadf2ebbf3daf2356c6ebe73c2cde49597bdf33414936699 | https://maven.mozilla.org/maven2/org/mozilla/components/support-utils/47.0.… -10543160bfa1790c2d98ba8e0de1230e0feea1926fd4859751c931e3bd5e1c34 | https://maven.mozilla.org/maven2/org/mozilla/components/support-utils/47.0.… -e24f78d36c64695407ace7b499472306161b1abf47b65c99ec790f321a6663e1 | https://maven.mozilla.org/maven2/org/mozilla/components/tooling-glean-gradl… -6320746e6d914aa45d9f8e0de23bdbd81c8097999b3dd64796363ed878633600 | https://maven.mozilla.org/maven2/org/mozilla/components/tooling-glean-gradl… -68f97a68faf84fa03cf3e1898042ffefe17b37c4c02df9e754fd2a47e6746196 | https://maven.mozilla.org/maven2/org/mozilla/telemetry/glean/31.1.1/glean-3… -9d6e64de942f997293b5ed35337da9350798192b22d8928030f191174b9f77fa | https://maven.mozilla.org/maven2/org/mozilla/telemetry/glean/31.1.1/glean-3… -3c2868a395d588eef2a370e4cd889a3a71351cff8eb4dd557dee764bc9f6e90a | https://maven.mozilla.org/maven2/org/mozilla/telemetry/glean-gradle-plugin/… -7517604c7643984718d421c85df395ae9f69f74c78ebd825930b4bceccf8196c | https://maven.mozilla.org/maven2/org/mozilla/telemetry/glean-gradle-plugin/… +e14c614b65ae4fe6c6d26ce369d523cf1ff837ce697e5e3f47e8cdfa355357f9 | https://maven.mozilla.org/maven2/org/mozilla/components/concept-fetch/56.0.… +e7bd7b2e2013a080bcb3cb9db974185eb28dd87c0594394f951a51ab5d6fcf5d | https://maven.mozilla.org/maven2/org/mozilla/components/concept-fetch/56.0.… +e79f67b5180f06a4680ffea310ceb2c5b0af205a88b3fb1cc76ce96e5b4a7ea7 | https://maven.mozilla.org/maven2/org/mozilla/components/service-glean/56.0.… +32acc51221752051a03765ac0bbf26a1da663ba2f7b3fe69ae422d5aac228f81 | https://maven.mozilla.org/maven2/org/mozilla/components/service-glean/56.0.… +ae35db5670efe458d3b9646cab337f2f7667becacdb023b599cc62bfa95b59ec | https://maven.mozilla.org/maven2/org/mozilla/components/support-base/56.0.0… +d87cd2eea092430c4e5a95ff04c3213be89f54aa61c2c7f80fb6548e6dc38bab | https://maven.mozilla.org/maven2/org/mozilla/components/support-base/56.0.0… +96aeede35017fd79576c5faa1cb41efc8137616e20231b41934999ca711f4407 | https://maven.mozilla.org/maven2/org/mozilla/components/support-ktx/56.0.0/… +dbd706151fe820ed233792a6773b70efcceb42987c209e27575ef91d4552b8c4 | https://maven.mozilla.org/maven2/org/mozilla/components/support-ktx/56.0.0/… +22934a560b0bc34f01304efbdccc988d2e0fca039ae577894fcec52a7fdfd88e | https://maven.mozilla.org/maven2/org/mozilla/components/support-utils/56.0.… +a2c1e39e5e58f585c2908af2266953b6b43e51f2fabe45e7723bc177c9c49bc0 | https://maven.mozilla.org/maven2/org/mozilla/components/support-utils/56.0.… +0165b5554aa3427a2e4026e9ca78c1ea5021f1578d36839b37350dbf0f192d29 | https://maven.mozilla.org/maven2/org/mozilla/components/tooling-glean-gradl… +878570d1c4e5b06d2434b7133a14ce97fb516b7871c7637c44a4b087db291a49 | https://maven.mozilla.org/maven2/org/mozilla/components/tooling-glean-gradl… +5b078e136164082be6d49afa0d87f849c7b9f65e21125fa57056cd630756279b | https://maven.mozilla.org/maven2/org/mozilla/telemetry/glean/32.1.1/glean-3… +64371d71a4fb11d63d2a5ff60149c94c1d0e0b909de09ffe199f6872663976b2 | https://maven.mozilla.org/maven2/org/mozilla/telemetry/glean/32.1.1/glean-3… +ab514359d7686282b7a7c8463e81237ff98460a23df534a6c9663b463e65423f | https://maven.mozilla.org/maven2/org/mozilla/telemetry/glean-gradle-plugin/… +f724de3fc1c37d8ecb4bbb7957dd67a489d0321da71eebb6b8d605ec4a57755a | https://maven.mozilla.org/maven2/org/mozilla/telemetry/glean-gradle-plugin/… 470564ff0a452da1c96be4ca3a9cdeceaa2912537ee2a8def486f3b383d43f15 | https://plugins.gradle.org/m2/com/jetbrains/python/envs/com.jetbrains.pytho… 6438812f927b505a4c4ef2703b45433a1b99e45334526bfed25e71e1a79abb14 | https://plugins.gradle.org/m2/gradle/plugin/com/jetbrains/python/gradle-pyt… 317aa4c467a37d5fcab6657c326a4069ea8e8c6dda951ec2091ffb95c596a2a3 | https://plugins.gradle.org/m2/gradle/plugin/com/jetbrains/python/gradle-pyt… diff --git a/projects/fenix/build b/projects/fenix/build index 5e483c9..0cde15d 100644 --- a/projects/fenix/build +++ b/projects/fenix/build @@ -7,6 +7,7 @@ mkdir -p $distdir/[% project %] mkdir -p /var/tmp/build [% IF !c("var/fetch_gradle_dependencies") %] + [% pc('python', 'var/setup', { python_tarfile => c('input_files_by_name/python') }) %] # XXX: Make this more generic and not only for geckoview-beta. geckoview_version=$(find . -type f -name geckoview-beta-*.aar | cut -d \- -f 6 | cut -d \. -f 1-3) gradle_repo=$rootdir/[% c('input_files_by_name/gradle-dependencies') %] @@ -32,6 +33,12 @@ cp $rootdir/[% c('input_files_by_name/tor-android-service') %]/* app/ [% IF c("var/fetch_gradle_dependencies") %] gradle --debug --no-daemon app:assemble[% c('variant') %] -x lint [% ELSE %] + # Prepare Glean dependencies for offline build + tar -xjf $rootdir/glean-parser-[% c('var/glean_parser') %].tar.bz2 + # We need to set `LC_ALL` and `LANG` to something that is not ASCII as encoding + # otherwise `click` barfs. See: https://click.palletsprojects.com/python3/ + export LC_ALL=C.UTF-8 + export LANG=C.UTF-8 patch -p1 < $rootdir/mavenLocal.patch # Make sure our GeckoView dependency is used. XXX: Make this more generic and diff --git a/projects/fenix/config b/projects/fenix/config index 01d3a25..33d4e44 100644 --- a/projects/fenix/config +++ b/projects/fenix/config @@ -39,10 +39,16 @@ input_files: project: geckoview pkg_type: merge_aars enable: '[% !c("var/fetch_gradle_dependencies") %]' + - name: python + project: python + enable: '[% !c("var/fetch_gradle_dependencies") %]' - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]' name: gradle-dependencies enable: '[% !c("var/fetch_gradle_dependencies") %]' exec: '[% INCLUDE "fetch-gradle-dependencies" %]' + - URL: https://people.torproject.org/~gk/mirrors/sources/glean-parser-[% c('var/glean_parser') %].tar.bz2 + sha256sum: 69234aa1ae38fd5dda4de71bd5e536795073703c3ed5bb994b9b8485bd7a55dc + enable: '[% !c("var/fetch_gradle_dependencies") %]' - filename: mavenLocal.patch enable: '[% !c("var/fetch_gradle_dependencies") %]' - project: tor-android-service diff --git a/rbm.conf b/rbm.conf index 411e120..b5dff53 100644 --- a/rbm.conf +++ b/rbm.conf @@ -261,6 +261,8 @@ targets: # API 21 is the minimum we currently support for 64 bit on Android android_min_api_x86_64: 21 android_min_api_aarch64: 21 + # This is needed to get the offline build part for Glean right. + glean_parser: 1.28.1 # We only build snowflake on the alpha and nightly # channels for now. snowflake: '[% c("var/alpha") || c("var/nightly") %]'

1 0

[tor-browser-build/master] Bug 40062: Fix mach build bustage with 81.0bx branches
by gk＠torproject.org 04 Sep '20

04 Sep '20

commit 3437f2a22e2480d19bf9fdbe92537505169d86ed Author: Georg Koppen <gk(a)torproject.org> Date: Thu Aug 27 19:33:29 2020 +0000 Bug 40062: Fix mach build bustage with 81.0bx branches After https://bugzilla.mozilla.org/show_bug.cgi?id=1656993 landed we can either run `./mach create-mach-environment` or set `MACH_USE_SYSTEM_PYTHON` to fix build breakage caused by `mach`. We do the latter as the former fails in our no-network-scenario, too. --- projects/geckoview/build | 2 ++ projects/geckoview/config | 2 ++ 2 files changed, 4 insertions(+) diff --git a/projects/geckoview/build b/projects/geckoview/build index 9eee8c4..b07bd2b 100644 --- a/projects/geckoview/build +++ b/projects/geckoview/build @@ -47,6 +47,8 @@ cp -r $gradle_repo/m2/* $gradle_repo rm -f configure rm -f js/src/configure +# We unbreak mach, see: https://bugzilla.mozilla.org/show_bug.cgi?id=1656993. +export MACH_USE_SYSTEM_PYTHON=1 ./mach configure --with-tor-browser-version=[% c("var/torbrowser_version") %] ./mach build --verbose diff --git a/projects/geckoview/config b/projects/geckoview/config index 0a8be4e..5478574 100644 --- a/projects/geckoview/config +++ b/projects/geckoview/config @@ -79,6 +79,8 @@ steps: mv $rootdir/[% c('input_files_by_name/gradle-dependencies') %] $gradle_repo cp -r $gradle_repo/m2/* $gradle_repo + # We unbreak mach, see: https://bugzilla.mozilla.org/show_bug.cgi?id=1656993. + export MACH_USE_SYSTEM_PYTHON=1 # We still need to specify --tor-browser-version due to bug 34005. ./mach configure --with-tor-browser-version=[% c("var/torbrowser_version") %] ./mach build --verbose

1 0

[tor-browser-build/master] Bump GeckoView version to 81.0b2
by gk＠torproject.org 04 Sep '20

04 Sep '20

commit ca725ba9672606e53058d741d26c2947e8b22049 Author: Georg Koppen <gk(a)torproject.org> Date: Thu Aug 27 19:36:20 2020 +0000 Bump GeckoView version to 81.0b2 --- projects/geckoview/config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/geckoview/config b/projects/geckoview/config index 5478574..2e9455d 100644 --- a/projects/geckoview/config +++ b/projects/geckoview/config @@ -8,7 +8,7 @@ git_submodule: 1 gpg_keyring: torbutton.gpg var: - geckoview_version: '80.0b[% c("var/beta_version") %]' + geckoview_version: '81.0b[% c("var/beta_version") %]' beta_version: 2 torbrowser_branch: 10.0 copyright_year: '[% exec("git show -s --format=%ci").remove("-.*") %]'

1 0

[tor-browser/tor-browser-78.2.0esr-10.0-1] Bug 1650281 - P1: Widen `gCombinedSizes` once the buffers grow r=gerald
by gk＠torproject.org 03 Sep '20

03 Sep '20

commit 5479c1f3ed0aa57b8f389a909d42343c9eaae09f Author: Chun-Min Chang <chun.m.chang(a)gmail.com> Date: Tue Jul 21 23:38:57 2020 +0000 Bug 1650281 - P1: Widen `gCombinedSizes` once the buffers grow r=gerald The `gCombinedSizes` need to be enlarged once the inner buffer within `MemoryBlockCache` grows. Otherwise, when the `MemoryBlockCache` is released, subtracting the buffer-size of the `MemoryBlockCache` from `gCombinedSizes` lead to a underflow. Differential Revision: https://phabricator.services.mozilla.com/D84273 --- dom/media/MemoryBlockCache.cpp | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/dom/media/MemoryBlockCache.cpp b/dom/media/MemoryBlockCache.cpp index 2d31119dca0a..bf073e6769d0 100644 --- a/dom/media/MemoryBlockCache.cpp +++ b/dom/media/MemoryBlockCache.cpp @@ -114,6 +114,10 @@ bool MemoryBlockCache::EnsureBufferCanContain(size_t aContentLength) { // possibly bypass some future growths that would fit in this new capacity. mBuffer.SetLength(capacity); } + const size_t newSizes = gCombinedSizes += (extra + extraCapacity); + LOG("EnsureBufferCanContain(%zu) - buffer size %zu + requested %zu + bonus " + "%zu = %zu; combined sizes %zu", + aContentLength, initialLength, extra, extraCapacity, capacity, newSizes); mHasGrown = true; return true; }

1 0

[tor-browser/tor-browser-78.2.0esr-10.0-1] Bug 1650281 - P2: Make sure `gCombinedSizes` won't be underflowed r=gerald
by gk＠torproject.org 03 Sep '20

03 Sep '20

commit 965a049f6bb123f384e72029feded2550f20027d Author: Chun-Min Chang <chun.m.chang(a)gmail.com> Date: Tue Jul 21 23:39:14 2020 +0000 Bug 1650281 - P2: Make sure `gCombinedSizes` won't be underflowed r=gerald In any case, `gCombinedSizes` should be larger than or equal to the buffer within `MemoryClockCache`. Differential Revision: https://phabricator.services.mozilla.com/D84274 --- dom/media/MemoryBlockCache.cpp | 1 + 1 file changed, 1 insertion(+) diff --git a/dom/media/MemoryBlockCache.cpp b/dom/media/MemoryBlockCache.cpp index bf073e6769d0..2848a3f3812c 100644 --- a/dom/media/MemoryBlockCache.cpp +++ b/dom/media/MemoryBlockCache.cpp @@ -52,6 +52,7 @@ MemoryBlockCache::MemoryBlockCache(int64_t aContentLength) } MemoryBlockCache::~MemoryBlockCache() { + MOZ_ASSERT(gCombinedSizes >= mBuffer.Length()); size_t sizes = static_cast<size_t>(gCombinedSizes -= mBuffer.Length()); LOG("~MemoryBlockCache() - destroying buffer of size %zu; combined sizes now " "%zu",

1 0

[tor-browser/tor-browser-78.2.0esr-10.0-1] Merge branch 'bug_40109' into tor-browser-78.2.0esr-10.0-1
by gk＠torproject.org 03 Sep '20

03 Sep '20

commit 885968e7a8374d3215e48cd27621fa7928200a38 Merge: b1f49b0a36f1 965a049f6bb1 Author: Georg Koppen <gk(a)torproject.org> Date: Thu Sep 3 09:36:50 2020 +0000 Merge branch 'bug_40109' into tor-browser-78.2.0esr-10.0-1 dom/media/MemoryBlockCache.cpp | 5 +++++ 1 file changed, 5 insertions(+)

1 0

[tor-browser-spec/master] Merge branch !6 into master
by gk＠torproject.org 03 Sep '20

03 Sep '20

commit 62045ea62ab28d8b1e9c25b7dc2b5ebdb509e145 Merge: f857ef5 1b265d8 Author: Georg Koppen <gk(a)torproject.org> Date: Thu Sep 3 08:04:26 2020 +0000 Merge branch !6 into master .../xxx-canvas-tainting-for-fp-resistance.txt | 134 +++++++++++++++++++++ 1 file changed, 134 insertions(+)

1 0

[tor-browser-spec/master] Bug 40003: Document the Canvas Breakage Ideas conversation from [tbb-dev]
by gk＠torproject.org 03 Sep '20

03 Sep '20

commit 1b265d8711626b6511e8ce4a3300360d873d8da7 Author: Sanketh Menda <sanketh(a)c1own.com> Date: Mon Aug 17 16:15:16 2020 -0400 Bug 40003: Document the Canvas Breakage Ideas conversation from [tbb-dev] --- .../xxx-canvas-tainting-for-fp-resistance.txt | 134 +++++++++++++++++++++ 1 file changed, 134 insertions(+) diff --git a/proposals/ideas/xxx-canvas-tainting-for-fp-resistance.txt b/proposals/ideas/xxx-canvas-tainting-for-fp-resistance.txt new file mode 100644 index 0000000..0c4fd17 --- /dev/null +++ b/proposals/ideas/xxx-canvas-tainting-for-fp-resistance.txt @@ -0,0 +1,134 @@ +Filename: xxx-canvas-tainting-for-fp-resistance.txt +Title: Canvas Tainting for Fingerprinting Resistance +Authors: David Fifield, Matthew Finkel, Sanketh Menda, and Tom Ritter +Created: 21-Aug-2020 +Status: Draft + +1. Motivation + + Canvas permissions currently break the following workflow: + 1. user uploads image + 2. website tries to display the image back to the user + 3. this image is white or noise (if canvas fuzzing is enabled.) + + Here are some instances of this breakage: + 1. Uploading images to WhatsApp Web: + https://bugzilla.mozilla.org/show_bug.cgi?id=1631673 + 2. Image cropping in Expensify: + https://bugzilla.mozilla.org/show_bug.cgi?id=1456378 + 3. Uploading images to Craigslist: + https://bugzilla.mozilla.org/show_bug.cgi?id=1573834 + + This workflow does not seem to be invading the user's privacy so it would be + nice to support it. Currently, to unbreak this functionality, the user has to + give full canvas permissions which may be abused by websites. + + This proposal specifies an approach towards allowing some use of the canvas + without sacrificing privacy. + +2. Proposal + +2.1 The Key Idea + + Keep track of whether the canvas is safe (in terms of fingerprintability) to + extract or not, and allow extractions if it is safe. + + This idea was first proposed by Gijs Kruitbosch [0] in analogy to canvas + tainting used to prevent cross-origin image loading [1]. + +2.2 Canvas Tainting for Fingerprinting Resistance + + We attach a state variable `isFPSafe` to the canvas, define three classes + of operations: safe, unsafe, and extraction, and when the canvas is + manipulated, we perform the following depending on whether the operation is + safe or not. + + ``` + if (safe operation) pass; + elif (unsafe operation) isFPSafe = false; + elif (extraction operation) { + if (isFPSafe) extract; + else return placeholder; + } + ``` + + See the canvas element section of the HTML spec [2] for details on canvas + operations. + +2.2.1 Safe Operations + + - `canvas.getContext`: returns the canvas object + - `canvas.drawImage` for user uploaded images: draws the given image onto the + canvas + + We hypothesize that allowing these operations unbreaks the aforementioned + workflow (see Research Question 1) and is safe (see Research Question 2). + + These operations are also probably safe: + - `canvas.getContextAttributes` + +2.2.2 Unsafe Operations + + All operations that have not been classified as safe (i.e., not listed in + Section 2.2.1) are considered unsafe. + +2.2.3 Extraction Operations + + - .toDataURL`: returns a dataURL of the canvas + - `canvas.toBlob`: returns a blob of the canvas + +3. Concerns + +3.1 Scaling inside drawImage + + The specification [3] does not specify how to scale images, so the scaling + algorithm may be implementation dependent and is a possible fingerprinting + vector. + +3.2 Image format encoding + + The specification [4] does not specify how to do image format encoding so it + may be implementation dependent and is a possible fingerprinting vector. + + Gómez-Boix, Laperdrix, and Baudry [3, p.314] noticed that compressing a canvas + to an image format adds entropy. However, in the case of user uploaded images, + this might be less of an issue (see Research Question 3.) + +3.2.1 `quality` levels + + `quality` levels add another layer of uncertainty to image format encoding as + they are not standardized [4]. + +3.2.1 Wide-gamut colors + + This does not seem to be in the HTML standard yet but it seems like WebKit and + Chrome already support it so it is worth looking into (see Research Question + 3). + + WebKit: https://github.com/whatwg/html/issues/299 + Chrome: https://github.com/whatwg/html/issues/4167 + +4. Research + + We would like to perform experiments to answer the following questions. + 1. Does this approach unbreak the motivating workflow? + 2. Are "safe operations" (the ones listed in 2.2.1) safe against + fingerprinting attacks? + 3. How bad are issues mentioned in the concerns? + +5. Acknowledgements + + We thank Alex Catarineu and Gijs Kruitbosch for insightful conversations, + and Georg Koppen for valuable comments on drafts of this document. + +6. References + +[0]: https://bugzilla.mozilla.org/show_bug.cgi?id=1631673#c8 +[1]: https://developer.mozilla.org/en-US/docs/Web/HTML/CORS_enabled_image +[2]: https://html.spec.whatwg.org/multipage/canvas.html +[3]: https://html.spec.whatwg.org/multipage/canvas.html#drawing-images +[4]: +https://html.spec.whatwg.org/multipage/canvas.html#serialising-bitmaps-to-a-file +[5]: Hiding in the Crowd: an Analysis of the Effectiveness of Browser +Fingerprinting at Large Scale. Gómez-Boix, Laperdrix, and Baudry. WWW '18. +https://doi.org/10.1145/3178876.3186097

1 0

[tor-browser-build/master] Bug 40058: Build Fenix with tor-android-service and topl
by gk＠torproject.org 02 Sep '20

02 Sep '20

commit 6bb6ec3650e28c07d9076ffa1f72099393f9e44a Author: Alex Catarineu <acat(a)torproject.org> Date: Fri Aug 28 19:32:52 2020 +0200 Bug 40058: Build Fenix with tor-android-service and topl With this tor-onion-proxy-library and tor-android-service will build fat aars with all the archs we support. We copy those over the fenix/app/ folder so that the Fenix build system picks them up for the final apks. --- projects/fenix/build | 5 ++++ projects/fenix/config | 6 ++++- projects/fenix/gradle-dependencies-list.txt | 10 +++++--- projects/tor-android-service/build | 4 ++- projects/tor-android-service/config | 16 +++++++++++- projects/tor-onion-proxy-library/build | 38 ++++++++++++++++++----------- projects/tor-onion-proxy-library/config | 32 ++++++++++++++++++++++-- 7 files changed, 89 insertions(+), 22 deletions(-) diff --git a/projects/fenix/build b/projects/fenix/build index 9726bc4..fdc6499 100644 --- a/projects/fenix/build +++ b/projects/fenix/build @@ -24,6 +24,11 @@ mkdir -p /var/tmp/build tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.gz cd $builddir-[% c("version") %] + +# Move Android library dependencies so they will be included in the apk during the build +cp $rootdir/[% c('input_files_by_name/topl') %]/* app/ +cp $rootdir/[% c('input_files_by_name/tor-android-service') %]/* app/ + [% IF c("var/fetch_gradle_dependencies") %] gradle --debug --no-daemon app:assemble[% c('build_flavor') %] -x lint [% ELSE %] diff --git a/projects/fenix/config b/projects/fenix/config index 2f8ae66..7a8b2f9 100644 --- a/projects/fenix/config +++ b/projects/fenix/config @@ -16,7 +16,7 @@ var: container: use_container: 1 # This should be updated when the list of gradle dependencies is changed. - gradle_dependencies_version: 1 + gradle_dependencies_version: 2 # Switch to make it easier to grab all dependencies during a dry-run. fetch_gradle_dependencies: 0 @@ -47,3 +47,7 @@ input_files: exec: '[% INCLUDE "fetch-gradle-dependencies" %]' - filename: mavenLocal.patch enable: '[% !c("var/fetch_gradle_dependencies") %]' + - project: tor-android-service + name: tor-android-service + - project: tor-onion-proxy-library + name: topl diff --git a/projects/fenix/gradle-dependencies-list.txt b/projects/fenix/gradle-dependencies-list.txt index 86c4be0..f8a97a9 100644 --- a/projects/fenix/gradle-dependencies-list.txt +++ b/projects/fenix/gradle-dependencies-list.txt @@ -462,6 +462,8 @@ f014af749024c57989ca2aa51b0a986f9d86d2669d074c1dee068a8f8da19562 | https://jcent 943e12b100627804638fa285805a0ab788a680266531e650921ebfe4621a8bfa | https://jcenter.bintray.com/javax/inject/javax.inject/1/javax.inject-1.pom b5e60cd8b7b5ff01ce4a74c5dd008f4fbd14ced3495d0b47b85cfedc182211f2 | https://jcenter.bintray.com/javax/xml/bind/jaxb-api/2.2.12-b140109.1041/jax… 2092a7d8d9bc5698c59b094bdea46622915c48c83ae66d5b5ad549c7bf16155b | https://jcenter.bintray.com/javax/xml/bind/jaxb-api/2.2.12-b140109.1041/jax… +ec3a75bebddbf19ff56a281cf5d1ad146169dcaa0e69d7b14f4aaba2e7775f34 | https://jcenter.bintray.com/net/freehaven/tor/control/jtorctl/0.2/jtorctl-0… +3369726ca2b0e3736c741ff3c22e06f707a1007ff20ccc5b5ba5d0d9a01ead30 | https://jcenter.bintray.com/net/freehaven/tor/control/jtorctl/0.2/jtorctl-0… c83a9c71358d781539c34b38d76a88e3ec8e2d587c7f04b611acc9041919b310 | https://jcenter.bintray.com/net/java/dev/jna/jna/5.2.0/jna-5.2.0.aar aafe6935d6bedb89a6aa32563d441fbe305da1de193d6f71a45a0212acf9b756 | https://jcenter.bintray.com/net/java/dev/jna/jna/5.2.0/jna-5.2.0.pom 281440811268e65d9e266b3cc898297e214e04f09740d0386ceeb4a8923d63bf | https://jcenter.bintray.com/net/java/jvnet-parent/1/jvnet-parent-1.pom @@ -645,9 +647,11 @@ c0e82b220b0a52c71c7ca2a58c99a2530696c7b58b173052b9d48fe3efb10073 | https://jcent 0f8a1b116e760b8fe6389c51b84e4b07a70fc11082d4f936e453b583dd50b43b | https://jcenter.bintray.com/org/ow2/ow2/1.5/ow2-1.5.pom 6e58dad0b8565b95c6fb14b4bfbf570523d1c5290244cfb33822789fa53b1d25 | https://jcenter.bintray.com/org/python/jython-installer/2.7.1/jython-instal… 2a42db37f9a565f1baa833b7cb7e9f901bd9fd750d10b9bd7ca76b2385b22387 | https://jcenter.bintray.com/org/python/jython-installer/2.7.1/jython-instal… -baf3c7fe15fefeaf9e5b000d94547379dc48370f22a8797e239c127e7d7756ec | https://jcenter.bintray.com/org/slf4j/slf4j-api/1.7.24/slf4j-api-1.7.24.jar -d7a15266d22fadc1c81bbc3b62637801d116e484eb6c831217dc36c537be48d2 | https://jcenter.bintray.com/org/slf4j/slf4j-api/1.7.24/slf4j-api-1.7.24.pom -36367d127a586d502b60bf54db2d7397ec94573620eff523873ec12a12e54839 | https://jcenter.bintray.com/org/slf4j/slf4j-parent/1.7.24/slf4j-parent-1.7.… +fbd7b254e02d8aef60c418a5f0e14a783b38a16162caffb2d2a16ccd5d2c09b4 | https://jcenter.bintray.com/org/slf4j/slf4j-android/1.7.25/slf4j-android-1.… +bd9b9cb1a3987b1427f7a18babe7f92078e32bbe2e1dca6dced00cc0e3a077a9 | https://jcenter.bintray.com/org/slf4j/slf4j-android/1.7.25/slf4j-android-1.… +18c4a0095d5c1da6b817592e767bb23d29dd2f560ad74df75ff3961dbde25b79 | https://jcenter.bintray.com/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar +7cd9d7a0b5d93dfd461a148891b43509cf403a9c7f9fb49060d3554df1c81e1e | https://jcenter.bintray.com/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.pom +18f5c52120db036e88d6136f8839c832d074bdda95c756c6f429249d2db54ac6 | https://jcenter.bintray.com/org/slf4j/slf4j-parent/1.7.25/slf4j-parent-1.7.… b51f8867c92b6a722499557fc3a1fdea77bdf9ef574722fe90ce436a29559454 | https://jcenter.bintray.com/org/sonatype/oss/oss-parent/7/oss-parent-7.pom fb40265f982548212ff82e362e59732b2187ec6f0d80182885c14ef1f982827a | https://jcenter.bintray.com/org/sonatype/oss/oss-parent/9/oss-parent-9.pom d3f7f09989d5b0ce5c4791818ef937ee7663f1e359c2ef2d312f938aad0763da | https://jcenter.bintray.com/org/yaml/snakeyaml/1.24/snakeyaml-1.24.jar diff --git a/projects/tor-android-service/build b/projects/tor-android-service/build index 4fb41bc..df83673 100644 --- a/projects/tor-android-service/build +++ b/projects/tor-android-service/build @@ -17,7 +17,9 @@ cd /var/tmp/build/[% project %]-[% c('version') %] # Replace pre-packaged tor library with the latest build rm -fR service/src/main/jniLibs/* -tar -C service/src/main -xf $rootdir/[% c('input_files_by_name/tor') %]/tor.tar.gz +[% FOREACH arch = ['armv7', 'aarch64', 'x86', 'x86_64'] -%] + tar -C service/src/main -xf $rootdir/[% c('input_files_by_name/tor-' _ arch) %]/tor.tar.gz +[% END -%] # Replace libraries with the config dependencies rm service/libs/* diff --git a/projects/tor-android-service/config b/projects/tor-android-service/config index 1f833ba..e2bb2db 100644 --- a/projects/tor-android-service/config +++ b/projects/tor-android-service/config @@ -23,4 +23,18 @@ input_files: - project: tor-onion-proxy-library name: topl - project: tor - name: tor + name: tor-armv7 + target_prepend: + - torbrowser-android-armv7 + - project: tor + name: tor-aarch64 + target_prepend: + - torbrowser-android-aarch64 + - project: tor + name: tor-x86 + target_prepend: + - torbrowser-android-x86 + - project: tor + name: tor-x86_64 + target_prepend: + - torbrowser-android-x86_64 diff --git a/projects/tor-onion-proxy-library/build b/projects/tor-onion-proxy-library/build index 3e02fb4..edeb8ae 100644 --- a/projects/tor-onion-proxy-library/build +++ b/projects/tor-onion-proxy-library/build @@ -23,20 +23,30 @@ patch -p1 < $rootdir/gradle.patch patch -p1 < $rootdir/0001-Bug-33931-Filter-bridges-in-stream-by-type.patch patch -p1 < $rootdir/0001-Bug-30318-Add-snowflake-support.patch -# Extract obfs4proxy from TorBrowser/Tor/PluggableTransports/obfs4proxy -tar --strip-components=4 -xf $rootdir/[% c('input_files_by_name/obfs4') %] - -# Extract snowflake from TorBrowser/Tor/PluggableTransports/snowflake -tar --strip-components=4 -xf $rootdir/[% c('input_files_by_name/snowflake') %] - -# Overwrite the obfs4proxy binary provided by Pluto and add Snowflake -for d in external/pluto/bin/*; do - cp obfs4proxy $d/ - cp snowflake-client $d/ -done - -rm obfs4proxy -rm snowflake-client +[% FOREACH arch = ['armv7', 'aarch64', 'x86', 'x86_64'] -%] + # Extract obfs4proxy from TorBrowser/Tor/PluggableTransports/obfs4proxy + tar --strip-components=4 -xf $rootdir/[% c('input_files_by_name/obfs4-' _ arch) %] + + # Extract snowflake from TorBrowser/Tor/PluggableTransports/snowflake + tar --strip-components=4 -xf $rootdir/[% c('input_files_by_name/snowflake-' _ arch) %] + + # Overwrite the obfs4proxy binary provided by Pluto and add Snowflake + [% IF arch == "armv7" -%] + cp obfs4proxy external/pluto/bin/armeabi-v7a/ + cp obfs4proxy external/pluto/bin/armeabi/ + cp snowflake-client external/pluto/bin/armeabi-v7a/ + cp snowflake-client external/pluto/bin/armeabi/ + [% ELSIF arch == "aarch64" -%] + cp obfs4proxy external/pluto/bin/arm64-v8a/ + cp snowflake-client external/pluto/bin/arm64-v8a/ + [% ELSE -%] + cp obfs4proxy external/pluto/bin/[% arch %]/ + cp snowflake-client external/pluto/bin/[% arch %]/ + [% END -%] + + rm obfs4proxy + rm snowflake-client +[% END -%] # Build Android Libraries and Apps gradle --offline --no-daemon -P androidplugin=3.6.0 -Dmaven.repo.local=$gradle_repo assembleRelease -x lint diff --git a/projects/tor-onion-proxy-library/config b/projects/tor-onion-proxy-library/config index 58aa4f6..671e988 100644 --- a/projects/tor-onion-proxy-library/config +++ b/projects/tor-onion-proxy-library/config @@ -20,9 +20,37 @@ input_files: - name: '[% c("var/compiler") %]' project: '[% c("var/compiler") %]' - project: obfs4 - name: obfs4 + name: obfs4-armv7 + target_prepend: + - torbrowser-android-armv7 + - project: obfs4 + name: obfs4-aarch64 + target_prepend: + - torbrowser-android-aarch64 + - project: obfs4 + name: obfs4-x86 + target_prepend: + - torbrowser-android-x86 + - project: obfs4 + name: obfs4-x86_64 + target_prepend: + - torbrowser-android-x86_64 + - project: snowflake + name: snowflake-armv7 + target_prepend: + - torbrowser-android-armv7 + - project: snowflake + name: snowflake-aarch64 + target_prepend: + - torbrowser-android-aarch64 + - project: snowflake + name: snowflake-x86 + target_prepend: + - torbrowser-android-x86 - project: snowflake - name: snowflake + name: snowflake-x86_64 + target_prepend: + - torbrowser-android-x86_64 - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]' name: gradle-dependencies exec: '[% INCLUDE "fetch-gradle-dependencies" %]'

1 0