Pier Angelo Vendrame pushed to branch main at The Tor Project / Applications / tor-browser-build

Commits: b37e2ebd by Pier Angelo Vendrame at 2023-05-25T18:28:35+02:00 Bug 40777: Update Go to 1.20.4 and create a Go bootstrap project

The Go compiler is written in Go, but it is possible to bootstrap it from C wiht Go 1.4. Before this commit, we used to build Go 1.4 for all our platforms, so we built a compiler to throw it away several times.

Starting from Go 1.20, the team decided to require Go V-3 to bootstrap (i.e., Go 1.17 to bootrastrap Go 1.20). Bootstrapping is still feasible, but throwing away two compilers is not a good idea. So, we decided to create a Go bootstrap project, and build it only once (for Linux amd64) until it will be able to build more recent Go compilers.

- - - - -

5 changed files:

- + projects/go-bootstrap/build - + projects/go-bootstrap/config - projects/go/0001-Use-fixed-go-build-tmp-directory.patch - projects/go/build - projects/go/config

Changes:

===================================== projects/go-bootstrap/build ===================================== @@ -0,0 +1,38 @@ +#!/bin/bash +[% c("var/set_default_env") -%] + +# This project is expected to be built only on Linux. +# Then the actual Go compiler will be built for each platform. + +distdir=/var/tmp/dist/[% project %] +mkdir -p /var/tmp/build /var/tmp/dist + +# Building go 1.4.x +# This is needed to bootstrap the go that we actually use +# https://golang.org/doc/install/source#go14 +tar -C /var/tmp/build --transform='s,^go>,go1.4,' -xf $rootdir/[% c('input_files_by_name/go14') %] +cd /var/tmp/build/go1.4/src +# Disable cgo to avoid conflicts with newer GCC. cgo is not needed for the bootstrap go. +# https://github.com/golang/go/issues/13114#issuecomment-186922245 +# Disable CC etc. that are set up for cross builds. +CGO_ENABLED=0 CC= CFLAGS= LDFLAGS= ./make.bash +export GOROOT_BOOTSTRAP="/var/tmp/build/go1.4" + +cd $rootdir + +tar -C /var/tmp/dist -xf $rootdir/[% c('input_files_by_name/go') %] +export GOROOT="$distdir" +mv /var/tmp/dist/go $distdir +cd $distdir/src + +./make.bash + +# This directory is non-reproducible and doesn't exist in official Go releases, +# so remove it to preserve reproducibility of the output. +rm -rf $distdir/pkg/obj/go-build/ + +cd /var/tmp/dist +[% c('tar', { + tar_src => [ project ], + tar_args => '-czf ' _ dest_dir _ '/' _ c('filename'), + }) %]

===================================== projects/go-bootstrap/config ===================================== @@ -0,0 +1,14 @@ +# vim: filetype=yaml sw=2 +version: 1.19.9 +filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.gz' +container: + use_container: 1 + +input_files: + - project: container-image + - URL: 'https://golang.org/dl/go%5B% c("version") %].src.tar.gz' + name: go + sha256sum: 131190a4697a70c5b1d232df5d3f55a3f9ec0e78e40516196ffb3f09ae6a5744 + - URL: https://dl.google.com/go/go1.4-bootstrap-20171003.tar.gz + name: go14 + sha256sum: f4ff5b5eb3a3cae1c993723f3eab519c5bae18866b5e5f96fe1102f0cb5c3e52

===================================== projects/go/0001-Use-fixed-go-build-tmp-directory.patch ===================================== @@ -1,4 +1,4 @@ -From 958f724b1b434702c3d44ceaca646791674d07b1 Mon Sep 17 00:00:00 2001 +From 2c1c98d6404284df744a194dd23bd3660f365387 Mon Sep 17 00:00:00 2001 From: Georg Koppen gk@torproject.org Date: Fri, 17 Apr 2020 10:24:23 +0000 Subject: [PATCH] Use fixed go-build tmp directory @@ -6,15 +6,18 @@ Subject: [PATCH] Use fixed go-build tmp directory Use fixed go-build tmp directory, when the directory does not exist, in order to avoid some reproducibility issues where build directory gets embedded in generated binaries. +--- + src/cmd/go/internal/work/action.go | 10 +++++++--- + 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/src/cmd/go/internal/work/action.go b/src/cmd/go/internal/work/action.go -index 33b7818fb2..5e369d0f53 100644 +index 8beb1345d0..dccf39c1c1 100644 --- a/src/cmd/go/internal/work/action.go +++ b/src/cmd/go/internal/work/action.go -@@ -252,9 +252,13 @@ func (b *Builder) Init() { - if cfg.BuildN { - b.WorkDir = "$WORK" - } else { +@@ -279,9 +279,13 @@ func NewBuilder(workDir string) *Builder { + if !buildInitStarted { + panic("internal error: NewBuilder called before BuildInit") + } - tmp, err := os.MkdirTemp(cfg.Getenv("GOTMPDIR"), "go-build") - if err != nil { - base.Fatalf("go: creating work dir: %v", err) @@ -29,5 +32,5 @@ index 33b7818fb2..5e369d0f53 100644 if !filepath.IsAbs(tmp) { abs, err := filepath.Abs(tmp) -- -2.26.1 +2.39.2

===================================== projects/go/build ===================================== @@ -3,27 +3,8 @@ distdir=/var/tmp/dist/[% project %] mkdir -p /var/tmp/dist

-[% IF c("var/linux") %] - # Config options for hardening-wrapper - export DEB_BUILD_HARDENING=1 - export DEB_BUILD_HARDENING_STACKPROTECTOR=1 - export DEB_BUILD_HARDENING_FORTIFY=1 - export DEB_BUILD_HARDENING_FORMAT=1 - export DEB_BUILD_HARDENING_PIE=1 -[% END %] - -mkdir -p /var/tmp/build - -# Building go 1.4.x -# This is needed to bootstrap the go that we actually use -# https://golang.org/doc/install/source#go14 -tar -C /var/tmp/build --transform='s,^go>,go1.4,' -xf $rootdir/[% c('input_files_by_name/go14') %] -cd /var/tmp/build/go1.4/src -# Disable cgo to avoid conflicts with newer GCC. cgo is not needed for the bootstrap go. -# https://github.com/golang/go/issues/13114#issuecomment-186922245 -# Disable CC etc. that are set up for cross builds. -CGO_ENABLED=0 CC= CFLAGS= LDFLAGS= ./make.bash -export GOROOT_BOOTSTRAP="/var/tmp/build/go1.4" +tar -C /var/tmp/dist -xf $rootdir/[% c('input_files_by_name/go-bootstrap') %] +export GOROOT_BOOTSTRAP="/var/tmp/dist/go-bootstrap"

cd $rootdir [% IF ! c("var/linux") %]

===================================== projects/go/config ===================================== @@ -1,12 +1,10 @@ # vim: filetype=yaml sw=2 -version: 1.19.9 -filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.gz' +version: 1.20.4 +filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.gz' container: use_container: 1

var: - go14_version: 1.4.3 - setup: | mkdir -p /var/tmp/dist tar -C /var/tmp/dist -xf $rootdir/[% c("go_tarfile") %] @@ -121,9 +119,10 @@ input_files: enable: '[% ! c("var/linux") %]' - URL: 'https://golang.org/dl/go%5B% c("version") %].src.tar.gz' name: go - sha256sum: 131190a4697a70c5b1d232df5d3f55a3f9ec0e78e40516196ffb3f09ae6a5744 - - URL: 'https://golang.org/dl/go%5B% c("var/go14_version") %].src.tar.gz' - name: go14 - sha256sum: 9947fc705b0b841b5938c48b22dc33e9647ec0752bae66e50278df4f23f64959 + sha256sum: 9f34ace128764b7a3a4b238b805856cc1b2184304df9e5690825b0710f4202d6 + - project: go-bootstrap + name: go-bootstrap + target_replace: + '^.*browser-.*': 'basebrowser-linux-x86_64' - filename: 0001-Use-fixed-go-build-tmp-directory.patch enable: '[% c("var/android") %]'

View it on GitLab: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/commit/b3...