commit 061b9e7bf60d4c442346a404f23a80c01f2b6b9f Author: Georg Koppen <gk@torproject.org> Date: Fri May 15 13:54:58 2020 +0000 Bug 33932: Improve steps for collecting gradle dependencies --- .../how-to-create-gradle-dependencies-list.txt | 43 ++++--------- tools/gen_gradle_deps_file.sh | 74 ++++++++++++++++++++++ 2 files changed, 85 insertions(+), 32 deletions(-) diff --git a/projects/common/how-to-create-gradle-dependencies-list.txt b/projects/common/how-to-create-gradle-dependencies-list.txt index e85c56a..3ecdbf6 100644 --- a/projects/common/how-to-create-gradle-dependencies-list.txt +++ b/projects/common/how-to-create-gradle-dependencies-list.txt @@ -3,48 +3,27 @@ If additional Android dependencies are required by the project's build, then the Gradle build will fail due to missing dependencies. To find out what the missing dependencies are take the following steps. -For tor-onion-proxy-library and tor-android-service, replace the following line -in the respective project build file: - $GRADLE_HOME/gradle-4.10.2/bin/gradle --offline --no-daemon -P androidplugin=3.1.0 -Dmaven.repo.local=$gradle_repo assembleRelease -x lint -with - $GRADLE_HOME/gradle-4.10.2/bin/gradle --debug --no-daemon -P androidplugin=3.1.0 assembleRelease -x lint +When calling gradle in the project's build script replace the `--offline` flag +with `--debug` and remove any `-Dmaven.repo.local` arguments. For the firefox project, comment out the following line in the project's build file: - export GRADLE_MAVEN_REPOSITORIES="file://$rootdir/[% c('input_files_by_name/gradle-dependencies') %]" + export GRADLE_MAVEN_REPOSITORIES="file://$gradle_repo" Also modify the gradle flags to include the debug option so the download logs will show up: export GRADLE_FLAGS="--no-daemon --debug" -then allow network access during the build by setting -var/container/disable_network/build to 0 in rbm.conf, and rerun the build. +Then allow network access during the build by setting +`var/container/disable_network/build` to `0` in rbm.conf, and rerun the build. Dependent artifacts will show up as downloads in the logs. You can pull out -these dependencies into a list with the following command (replacing -"firefox-android-armv7.log" with the build log file name of the actual project): +these dependencies into a list by passing the log file to the gradle dependency +list script in the tools directory: -`cat logs/firefox-android-armv7.log | grep "Performing HTTP" | grep -o "https://.*" | sort | uniq > download-attempts.txt` +`./gen_gradle_deps_file.sh /path/to/log/file` -The download-attempts.txt file contains all the attempted downloads, so we need to find out which ones failed - -`cat logs/firefox-android-armv7.log | grep "Resource missing" | grep -o "https:.*[^]]" | sort | uniq > download-fails.txt` - -Now take the intersection. This removes failures from attempts, leaving just successful downloads - -`sort download-attempts.txt download-fails.txt | uniq -u | rev | sort -t/ -u -k1,4 | rev | sort > download-urls.txt` - -You will then need to add the new dependency URLs and SHA-256 values into the -projects/$project/gradle-dependencies-list.txt file. The format of this file is -pipe delimited - sha256sum | url - -Finally, in the project's config file increment the -var/gradle_dependencies_version and make sure to restore the project's build +Copy the resulting `gradle-dependencies-list.txt` over the one in the respective +project. Then, in the project's config file, increment the +`var/gradle_dependencies_version` and make sure to restore the project's build file back to original. - -It may also be the case that you wish to clean up old versions of the artifacts. -For this you will need to run the build with a -gradle-dependencies-list.txt file containing only the headers. Make sure to also -comment the GRADLE_MAVEN_REPOSITORIES line from the project's build file. You -can now proceed to reconstruct the list as given above. diff --git a/tools/gen_gradle_deps_file.sh b/tools/gen_gradle_deps_file.sh new file mode 100755 index 0000000..377676f --- /dev/null +++ b/tools/gen_gradle_deps_file.sh @@ -0,0 +1,74 @@ +#!/bin/sh + +# Copyright (c) 2020, The Tor Project, Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# +# * Neither the names of the copyright owners nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# Usage: +# 1) Point to a log file with all the dependency download attempts (for its +# generation see: projects/common/how-to-create-gradle-dependencies-list.txt) +# 2) Double-check that you get the same SHA-256 sums when downloaded from a +# different network location. E.g. by using `torsocks` with this script after +# having made a copy of `gradle-dependencies-list.txt` from 1) and comparing +# the two .txt files. + +log="$1" + +# Step 1: Extract all the download attempts out of the log file, ignore the ones +# for maven-metadata.xml files. We don't need those. +cat $log | grep "Performing HTTP" | grep -o "https://.*" | \ + grep -v "maven-metadata.xml" | sort | uniq > dl-attempts + +# Step 2: Fetch all the dependencies and calculate the SHA-256 sum +while read line +do + wget -U "" $line + fn=$(basename "$line") + sha256=`sha256sum $fn | cut -d ' ' -f 1` + echo "$sha256 | $line" >> deps + rm $fn +done < dl-attempts + +# Step 3: Add the header at the beginning of the final dependency file. +echo "# On how to update dependencies see projects/common/how-to-create-gradle\ +-dependencies-list.txt" > gradle-dependencies-list.txt +echo "# Don't forget to update var/gradle_dependencies_version when modifying \ +this file" >> gradle-dependencies-list.txt +echo "sha256sum | url" >> gradle-dependencies-list.txt + +# Step 4: Keep only successfully downloaded artifacts, remove duplicates, and +# sort based on download URL. +grep ^[a-f0-9] deps | rev | sort -t/ -u -k1,4 | rev | \ + sort -k 3 >> gradle-dependencies-list.txt + +# Step 5: Clean up +rm dl-attempts +rm deps + +exit 0