December 2016 - tbb-commits - lists.torproject.org

[tor-browser-bundle/hardened-builds] Bug 20147: (re-)dzip.sh: various improvements
by gk＠torproject.org 07 Dec '16

07 Dec '16

commit c74da40d46b10c7a757f8835f8177602ed77f8e4 Author: Rusty Bird <rustybird(a)openmailbox.org> Date: Wed Sep 21 12:05:39 2016 +0000 Bug 20147: (re-)dzip.sh: various improvements - Don't ignore errors - Quote $@ and $1 - Work with absolute filenames and filenames starting with a dash - Pass many files per chmod invocation (much faster) - Pass $UNZIPOPTS, like $ZIPOPTS - Reuse dzip.sh in re-dzip.sh The (re)generated zip files are identical. --- gitian/build-helpers/dzip.sh | 17 ++++++++--------- gitian/build-helpers/re-dzip.sh | 23 ++++++++++------------- 2 files changed, 18 insertions(+), 22 deletions(-) diff --git a/gitian/build-helpers/dzip.sh b/gitian/build-helpers/dzip.sh index 5772c8b..64fcdca 100755 --- a/gitian/build-helpers/dzip.sh +++ b/gitian/build-helpers/dzip.sh @@ -1,14 +1,13 @@ -#!/bin/sh +#!/bin/sh -e # Crappy deterministic zip wrapper export LC_ALL=C -ZIPFILE=$1 +ZIPFILE=${1:?} shift -[ -n "$REFERENCE_DATETIME" ] && \ - find $@ -exec touch --date="$REFERENCE_DATETIME" {} \; - -find $@ -executable -exec chmod 700 {} \; -find $@ ! -executable -exec chmod 600 {} \; - -find $@ | sort | zip $ZIPOPTS -X -@ "$ZIPFILE" +if [ -n "$REFERENCE_DATETIME" ]; then + find "$@" -exec touch --date="$REFERENCE_DATETIME" -- {} + +fi +find "$@" -executable -exec chmod 700 {} + +find "$@" ! -executable -exec chmod 600 {} + +find "$@" | sort | zip $ZIPOPTS -X -@ "$ZIPFILE" diff --git a/gitian/build-helpers/re-dzip.sh b/gitian/build-helpers/re-dzip.sh index 27828e9..58942c4 100755 --- a/gitian/build-helpers/re-dzip.sh +++ b/gitian/build-helpers/re-dzip.sh @@ -1,17 +1,14 @@ -#!/bin/sh +#!/bin/sh -e # Crappy deterministic zip repackager export LC_ALL=C -ZIPFILE=`basename $1` +ZIPFILE_BASENAME=$(basename -- "${1:?}") +TEMPDIR=tmp-re-dzip-$$ +RE_DZIP=$(readlink -f -- "$(which -- "$0")") +PATH=$PATH:$(dirname "$RE_DZIP") -mkdir tmp_dzip -cd tmp_dzip -unzip ../$1 -[ -n "$REFERENCE_DATETIME" ] && \ - find . -exec touch --date="$REFERENCE_DATETIME" {} \; -find . -executable -exec chmod 700 {} \; -find . ! -executable -exec chmod 600 {} \; -find . | sort | zip $ZIPOPTS -X -@ $ZIPFILE -mv $ZIPFILE ../$1 -cd .. -rm -rf tmp_dzip +mkdir "$TEMPDIR" +unzip $UNZIPOPTS -d "$TEMPDIR" -- "$1" +(cd "$TEMPDIR"; dzip.sh ./"$ZIPFILE_BASENAME" .) +mv -- "$TEMPDIR"/"$ZIPFILE_BASENAME" "$1" +rm -rf "$TEMPDIR"

1 0

[tor-browser-bundle/master] Bug 20147: (re-)dzip.sh: various improvements
by gk＠torproject.org 07 Dec '16

07 Dec '16

commit a38d827c12ce75d13e18f19fc8f5bac8aca28c55 Author: Rusty Bird <rustybird(a)openmailbox.org> Date: Wed Sep 21 12:05:39 2016 +0000 Bug 20147: (re-)dzip.sh: various improvements - Don't ignore errors - Quote $@ and $1 - Work with absolute filenames and filenames starting with a dash - Pass many files per chmod invocation (much faster) - Pass $UNZIPOPTS, like $ZIPOPTS - Reuse dzip.sh in re-dzip.sh The (re)generated zip files are identical. --- gitian/build-helpers/dzip.sh | 17 ++++++++--------- gitian/build-helpers/re-dzip.sh | 23 ++++++++++------------- 2 files changed, 18 insertions(+), 22 deletions(-) diff --git a/gitian/build-helpers/dzip.sh b/gitian/build-helpers/dzip.sh index 5772c8b..64fcdca 100755 --- a/gitian/build-helpers/dzip.sh +++ b/gitian/build-helpers/dzip.sh @@ -1,14 +1,13 @@ -#!/bin/sh +#!/bin/sh -e # Crappy deterministic zip wrapper export LC_ALL=C -ZIPFILE=$1 +ZIPFILE=${1:?} shift -[ -n "$REFERENCE_DATETIME" ] && \ - find $@ -exec touch --date="$REFERENCE_DATETIME" {} \; - -find $@ -executable -exec chmod 700 {} \; -find $@ ! -executable -exec chmod 600 {} \; - -find $@ | sort | zip $ZIPOPTS -X -@ "$ZIPFILE" +if [ -n "$REFERENCE_DATETIME" ]; then + find "$@" -exec touch --date="$REFERENCE_DATETIME" -- {} + +fi +find "$@" -executable -exec chmod 700 {} + +find "$@" ! -executable -exec chmod 600 {} + +find "$@" | sort | zip $ZIPOPTS -X -@ "$ZIPFILE" diff --git a/gitian/build-helpers/re-dzip.sh b/gitian/build-helpers/re-dzip.sh index 27828e9..58942c4 100755 --- a/gitian/build-helpers/re-dzip.sh +++ b/gitian/build-helpers/re-dzip.sh @@ -1,17 +1,14 @@ -#!/bin/sh +#!/bin/sh -e # Crappy deterministic zip repackager export LC_ALL=C -ZIPFILE=`basename $1` +ZIPFILE_BASENAME=$(basename -- "${1:?}") +TEMPDIR=tmp-re-dzip-$$ +RE_DZIP=$(readlink -f -- "$(which -- "$0")") +PATH=$PATH:$(dirname "$RE_DZIP") -mkdir tmp_dzip -cd tmp_dzip -unzip ../$1 -[ -n "$REFERENCE_DATETIME" ] && \ - find . -exec touch --date="$REFERENCE_DATETIME" {} \; -find . -executable -exec chmod 700 {} \; -find . ! -executable -exec chmod 600 {} \; -find . | sort | zip $ZIPOPTS -X -@ $ZIPFILE -mv $ZIPFILE ../$1 -cd .. -rm -rf tmp_dzip +mkdir "$TEMPDIR" +unzip $UNZIPOPTS -d "$TEMPDIR" -- "$1" +(cd "$TEMPDIR"; dzip.sh ./"$ZIPFILE_BASENAME" .) +mv -- "$TEMPDIR"/"$ZIPFILE_BASENAME" "$1" +rm -rf "$TEMPDIR"

1 0

[tor-browser/tor-browser-45.5.1esr-6.0-1] squash! Omnibox: Add DDG, Startpage, Disconnect, Youtube, Twitter; remove Amazon, eBay, bing
by gk＠torproject.org 07 Dec '16

07 Dec '16

commit 2c396200c8c659c6e2c3ab8e40851aa984da49eb Author: Georg Koppen <gk(a)torproject.org> Date: Tue Nov 29 09:43:16 2016 +0000 squash! Omnibox: Add DDG, Startpage, Disconnect, Youtube, Twitter; remove Amazon, eBay, bing This fixes bug 20809: the DuckDuckGo team has changed its server-side code in a way that lets users with JavaScript enabled use the default landing page while those without JavaScript available get redirected directly to the non-JS page. We adapt the search engine URLs accordingly. --- browser/locales/en-US/searchplugins/ddg-onion.xml | 4 ++-- browser/locales/en-US/searchplugins/ddg.xml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/browser/locales/en-US/searchplugins/ddg-onion.xml b/browser/locales/en-US/searchplugins/ddg-onion.xml index 009da7e..a1f6a28 100644 --- a/browser/locales/en-US/searchplugins/ddg-onion.xml +++ b/browser/locales/en-US/searchplugins/ddg-onion.xml @@ -3,9 +3,9 @@ <Description>Duck Duck Go Onion</Description> <InputEncoding>UTF-8</InputEncoding> <Image height="16" width="16">data:image/x-icon;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAMAAAAoLQ9TAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAB8lBMVEUAAADkRQzjPwPjQQXkRQ3iPwTiQQXgPQPeQgrcOwPVNgDVNQDWOgbTMwDRMgDQMwDSMwDRNwTQLgDRJgDSJwDSLgDSNwTjOgDiOADjOQDkPADhQAXzs5v+/fv////0vKbiRQvgPQHpdUr85NzuknPdKgDcIwDnZzj2w7HqeU/gPQLsimb/+PftjWn97Obpb0LdJQDeLQDtjmvsi2jgSBDnbULgOQD/39HgLQDeMgDpeFLgSBH0v670uqbaJQD2qImWvP/G1Ob5+/3u//+fvvXyp47dMwDaLwD0u6v0v6/aNQDiXi/aKQD3qozU7/8gSY2vvtg0ZK/OqLDaKQHYKgLgWTfaNADZMgDZMADZLADzqpD7//+xwdz//9H/5Bn/7Bn//ADofADYMADYMQDZOgPXLgDiZDj//97/0AD3tQDvlgHZOgbXLATXMADWMgDfXjLVLQD///z+0AD/3Rn/yRnwnQDcVjbVMQDyv67wuKTSJwDRHQD+8O/tg3/iQQDwhAHnawHWMADvtKfyva7XQxHga0bQGQD2vbH/u8LXIQCmPQzja07XQxLliGn99fPkcVHvhnGZ5VguvUU5wktBwCcAgxzydVv/8/XmiGngdlL+ysi3+I8LtCE80V6P3YmX4sDleljSNQLzr6D7sKPXNQTSIwAEAbMrAAAAF3RSTlMARqSkRvPz80PTpKRG3fPe3hio9/eoGP50jNsAAAABYktHRB5yCiArAAAAyElEQVQYGQXBvUqCYRiA4fu2V9Tn+UQddI3aCpxaOoU6iU4gcqqpoYbALXBuCuoYmttamq JDiEoh4YP+MOi6BNCh+uYKEGiOVNCXXxA2XDVV/UyfKbRCXTLQWAxbP2vt8Ue/uYDvfim91615sb2um6rqtrr/NFb1cUf1Ybd06areU6lSlYpK79jzK1SyJOkfhOl8JGEcqV5zoKrTRqO6yUzIzNu46ijdM1VV9bhuUJ/nZURExLRzUiPQm3kKXHi4BAEGOmOi78A/L1QoU/VHoTsAAAAldEVYdGRhdGU6Y3JlYXRlADIwMTQtMDEtMTlUMjA6MDE6MTEtMDU6MDAuET6cAAAAJXRFWHRkYXRlOm1vZGlmeQAyMDE0LTAxLTE5VDIwOjAxOjExLTA1OjAwX0yGIAAAAABJRU5ErkJggg==</Image> -<Url type="text/html" method="POST" template="http://3g2upl4pq6kufc4m.onion/html/"> +<Url type="text/html" method="POST" template="http://3g2upl4pq6kufc4m.onion"> <Param name="q" value="{searchTerms}"/> </Url> -<SearchForm>http://3g2upl4pq6kufc4m.onion/html/</SearchForm> +<SearchForm>http://3g2upl4pq6kufc4m.onion</SearchForm> </SearchPlugin> diff --git a/browser/locales/en-US/searchplugins/ddg.xml b/browser/locales/en-US/searchplugins/ddg.xml index 8318746..acf4c41 100644 --- a/browser/locales/en-US/searchplugins/ddg.xml +++ b/browser/locales/en-US/searchplugins/ddg.xml @@ -22,9 +22,9 @@ AScg7HAnIOz2JyDs/ycg7P8nIOz/JyDs/ycg7P8nIOz/JyDs9icg7HAnIOwBAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAJyDsJicg7IAnIOyzJyDs5icg7OYnIOyzJyDsgCcg7CYAAAAAAAAAAAAAAAAA AAAA+B8AAPAPAADAAwAAwAMAAIABAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAABAACAAQAAwAMAAMAD AADwDwAA+B8AAA==</Image> -<Url type="text/html" method="POST" template="https://duckduckgo.com/html"> +<Url type="text/html" method="POST" template="https://duckduckgo.com"> <Param name="q" value="{searchTerms}"/> </Url> -<SearchForm>https://duckduckgo.com/</SearchForm> +<SearchForm>https://duckduckgo.com</SearchForm> </SearchPlugin>

1 0

[tor-browser/tor-browser-45.5.1esr-6.5-1] squash! Omnibox: Add DDG, Startpage, Disconnect, Youtube, Twitter; remove Amazon, eBay, bing
by gk＠torproject.org 07 Dec '16

07 Dec '16

commit 7aa3d81033b34d92c3b01308ad9b700bccd33458 Author: Georg Koppen <gk(a)torproject.org> Date: Tue Nov 29 09:43:16 2016 +0000 squash! Omnibox: Add DDG, Startpage, Disconnect, Youtube, Twitter; remove Amazon, eBay, bing This fixes bug 20809: the DuckDuckGo team has changed its server-side code in a way that lets users with JavaScript enabled use the default landing page while those without JavaScript available get redirected directly to the non-JS page. We adapt the search engine URLs accordingly. --- browser/locales/en-US/searchplugins/ddg-onion.xml | 4 ++-- browser/locales/en-US/searchplugins/ddg.xml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/browser/locales/en-US/searchplugins/ddg-onion.xml b/browser/locales/en-US/searchplugins/ddg-onion.xml index 009da7e..a1f6a28 100644 --- a/browser/locales/en-US/searchplugins/ddg-onion.xml +++ b/browser/locales/en-US/searchplugins/ddg-onion.xml @@ -3,9 +3,9 @@ <Description>Duck Duck Go Onion</Description> <InputEncoding>UTF-8</InputEncoding> <Image height="16" width="16">data:image/x-icon;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAMAAAAoLQ9TAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAB8lBMVEUAAADkRQzjPwPjQQXkRQ3iPwTiQQXgPQPeQgrcOwPVNgDVNQDWOgbTMwDRMgDQMwDSMwDRNwTQLgDRJgDSJwDSLgDSNwTjOgDiOADjOQDkPADhQAXzs5v+/fv////0vKbiRQvgPQHpdUr85NzuknPdKgDcIwDnZzj2w7HqeU/gPQLsimb/+PftjWn97Obpb0LdJQDeLQDtjmvsi2jgSBDnbULgOQD/39HgLQDeMgDpeFLgSBH0v670uqbaJQD2qImWvP/G1Ob5+/3u//+fvvXyp47dMwDaLwD0u6v0v6/aNQDiXi/aKQD3qozU7/8gSY2vvtg0ZK/OqLDaKQHYKgLgWTfaNADZMgDZMADZLADzqpD7//+xwdz//9H/5Bn/7Bn//ADofADYMADYMQDZOgPXLgDiZDj//97/0AD3tQDvlgHZOgbXLATXMADWMgDfXjLVLQD///z+0AD/3Rn/yRnwnQDcVjbVMQDyv67wuKTSJwDRHQD+8O/tg3/iQQDwhAHnawHWMADvtKfyva7XQxHga0bQGQD2vbH/u8LXIQCmPQzja07XQxLliGn99fPkcVHvhnGZ5VguvUU5wktBwCcAgxzydVv/8/XmiGngdlL+ysi3+I8LtCE80V6P3YmX4sDleljSNQLzr6D7sKPXNQTSIwAEAbMrAAAAF3RSTlMARqSkRvPz80PTpKRG3fPe3hio9/eoGP50jNsAAAABYktHRB5yCiArAAAAyElEQVQYGQXBvUqCYRiA4fu2V9Tn+UQddI3aCpxaOoU6iU4gcqqpoYbALXBuCuoYmttamq JDiEoh4YP+MOi6BNCh+uYKEGiOVNCXXxA2XDVV/UyfKbRCXTLQWAxbP2vt8Ue/uYDvfim91615sb2um6rqtrr/NFb1cUf1Ybd06areU6lSlYpK79jzK1SyJOkfhOl8JGEcqV5zoKrTRqO6yUzIzNu46ijdM1VV9bhuUJ/nZURExLRzUiPQm3kKXHi4BAEGOmOi78A/L1QoU/VHoTsAAAAldEVYdGRhdGU6Y3JlYXRlADIwMTQtMDEtMTlUMjA6MDE6MTEtMDU6MDAuET6cAAAAJXRFWHRkYXRlOm1vZGlmeQAyMDE0LTAxLTE5VDIwOjAxOjExLTA1OjAwX0yGIAAAAABJRU5ErkJggg==</Image> -<Url type="text/html" method="POST" template="http://3g2upl4pq6kufc4m.onion/html/"> +<Url type="text/html" method="POST" template="http://3g2upl4pq6kufc4m.onion"> <Param name="q" value="{searchTerms}"/> </Url> -<SearchForm>http://3g2upl4pq6kufc4m.onion/html/</SearchForm> +<SearchForm>http://3g2upl4pq6kufc4m.onion</SearchForm> </SearchPlugin> diff --git a/browser/locales/en-US/searchplugins/ddg.xml b/browser/locales/en-US/searchplugins/ddg.xml index 8318746..acf4c41 100644 --- a/browser/locales/en-US/searchplugins/ddg.xml +++ b/browser/locales/en-US/searchplugins/ddg.xml @@ -22,9 +22,9 @@ AScg7HAnIOz2JyDs/ycg7P8nIOz/JyDs/ycg7P8nIOz/JyDs9icg7HAnIOwBAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAJyDsJicg7IAnIOyzJyDs5icg7OYnIOyzJyDsgCcg7CYAAAAAAAAAAAAAAAAA AAAA+B8AAPAPAADAAwAAwAMAAIABAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAABAACAAQAAwAMAAMAD AADwDwAA+B8AAA==</Image> -<Url type="text/html" method="POST" template="https://duckduckgo.com/html"> +<Url type="text/html" method="POST" template="https://duckduckgo.com"> <Param name="q" value="{searchTerms}"/> </Url> -<SearchForm>https://duckduckgo.com/</SearchForm> +<SearchForm>https://duckduckgo.com</SearchForm> </SearchPlugin>

1 0

[tor-browser-bundle/master] Moving Go building into utils
by gk＠torproject.org 07 Dec '16

07 Dec '16

commit af596cb55dbcd300c989d02be7fffd1a16a7fb8a Author: Georg Koppen <gk(a)torproject.org> Date: Wed Nov 23 13:26:45 2016 +0000 Moving Go building into utils --- .../linux/gitian-pluggable-transports.yml | 28 +++++----------------- gitian/descriptors/linux/gitian-utils.yml | 24 +++++++++++++++++++ gitian/mkbundle-linux.sh | 8 ++++++- 3 files changed, 37 insertions(+), 23 deletions(-) diff --git a/gitian/descriptors/linux/gitian-pluggable-transports.yml b/gitian/descriptors/linux/gitian-pluggable-transports.yml index 38804dd..c790a9f 100644 --- a/gitian/descriptors/linux/gitian-pluggable-transports.yml +++ b/gitian/descriptors/linux/gitian-pluggable-transports.yml @@ -45,13 +45,13 @@ files: - "zope.interface.zip" - "twisted.tar.bz2" - "parsley.tar.gz" -- "go14.tar.gz" -- "go.tar.gz" - "dzip.sh" - "gmp-linux32-utils.zip" - "gmp-linux64-utils.zip" - "openssl-linux32-utils.zip" - "openssl-linux64-utils.zip" +- "go-linux32-utils.zip" +- "go-linux64-utils.zip" script: | INSTDIR="$HOME/install" PTDIR="$INSTDIR/Tor/PluggableTransports" @@ -71,29 +71,13 @@ script: | unzip -d $INSTDIR gmp-linux$GBUILD_BITS-utils.zip unzip -d $INSTDIR openssl-linux$GBUILD_BITS-utils.zip + unzip -d $INSTDIR go-linux$GBUILD_BITS-utils.zip + export GOROOT="$INSTDIR/go" + export GOPATH="$HOME/go" + export PATH="$PATH:$GOROOT/bin" # FTE only needs libgmp.so.10 and no libgmpxx anymore. cp $INSTDIR/gmp/lib/libgmp.so.10 $INSTDIR/Tor - # Building go 1.4.x - # This is needed to bootstrap the go that we actually use - # https://golang.org/doc/install/source#go14 - tar xvf go14.tar.gz --transform='s,^go\>,go1.4,' - cd go1.4/src - # Disable cgo to avoid conflicts with newer GCC. cgo is not needed for the bootstrap go. - # https://github.com/golang/go/issues/13114#issuecomment-186922245 - CGO_ENABLED=0 ./make.bash - cd ../.. - export GOROOT_BOOTSTRAP="$PWD/go1.4" - - # Building go - # http://golang.org/doc/install/source#environment - export GOPATH="$HOME/go" - tar xvf go.tar.gz - cd go/src - ./make.bash - cd ../.. - export PATH="$PATH:$PWD/go/bin" - # Building pyptlib cd pyptlib find -type f -print0 | xargs -0 touch --date="$REFERENCE_DATETIME" diff --git a/gitian/descriptors/linux/gitian-utils.yml b/gitian/descriptors/linux/gitian-utils.yml index 7093a0d..b5246f4 100644 --- a/gitian/descriptors/linux/gitian-utils.yml +++ b/gitian/descriptors/linux/gitian-utils.yml @@ -33,6 +33,8 @@ files: - "gcc.tar.bz2" - "openssl.tar.gz" - "gmp.tar.bz2" +- "go14.tar.gz" +- "go.tar.gz" - "versions" - "dzip.sh" script: | @@ -122,6 +124,27 @@ script: | make install cd .. + # Building go 1.4.x + # This is needed to bootstrap the go that we actually use + # https://golang.org/doc/install/source#go14 + tar xvf go14.tar.gz --transform='s,^go\>,go1.4,' + cd go1.4/src + # Disable cgo to avoid conflicts with newer GCC. cgo is not needed for the bootstrap go. + # https://github.com/golang/go/issues/13114#issuecomment-186922245 + CGO_ENABLED=0 ./make.bash + cd ../.. + export GOROOT_BOOTSTRAP="$PWD/go1.4" + + # Building go + # http://golang.org/doc/install/source#environment + export GOROOT="$INSTDIR/go" + tar xvf go.tar.gz + cd go/src + ./make.bash + cd ../.. + # Let's get it zipped up. + mv go $INSTDIR + # Grabbing the remaining results and making sure timestamps don't spoil them # Since we stopped using libfaketime, the binutils, gcc, openssl, # libevent archives are no longer reproducible. The main reason @@ -133,4 +156,5 @@ script: | ~/build/dzip.sh openssl-$OPENSSL_VER-linux$GBUILD_BITS-utils.zip openssl ~/build/dzip.sh libevent-${LIBEVENT_TAG#release-}-linux$GBUILD_BITS-utils.zip libevent ~/build/dzip.sh gmp-$GMP_VER-linux$GBUILD_BITS-utils.zip gmp + ~/build/dzip.sh go-$GO_VER-linux$GBUILD_BITS-utils.zip go cp *utils.zip $OUTDIR/ diff --git a/gitian/mkbundle-linux.sh b/gitian/mkbundle-linux.sh index 430b666..156d2ba 100755 --- a/gitian/mkbundle-linux.sh +++ b/gitian/mkbundle-linux.sh @@ -110,7 +110,9 @@ if [ ! -f inputs/binutils-$BINUTILS_VER-linux32-utils.zip -o \ ! -f inputs/libevent-${LIBEVENT_TAG_ORIG#release-}-linux32-utils.zip -o \ ! -f inputs/libevent-${LIBEVENT_TAG_ORIG#release-}-linux64-utils.zip -o \ ! -f inputs/gmp-$GMP_VER-linux32-utils.zip -o \ - ! -f inputs/gmp-$GMP_VER-linux64-utils.zip ]; + ! -f inputs/gmp-$GMP_VER-linux64-utils.zip -o \ + ! -f inputs/go-$GO_VER-linux32-utils.zip -o \ + ! -f inputs/go-$GO_VER-linux64-utils.zip ]; then echo echo "****** Starting Utilities Component of Linux Bundle (1/6 for Linux) ******" @@ -135,6 +137,8 @@ then ln -sf libevent-${LIBEVENT_TAG_ORIG#release-}-linux64-utils.zip libevent-linux64-utils.zip ln -sf gmp-$GMP_VER-linux32-utils.zip gmp-linux32-utils.zip ln -sf gmp-$GMP_VER-linux64-utils.zip gmp-linux64-utils.zip + ln -sf go-$GO_VER-linux32-utils.zip go-linux32-utils.zip + ln -sf go-$GO_VER-linux64-utils.zip go-linux64-utils.zip cd .. #cp -a result/utils-linux-res.yml inputs/ else @@ -154,6 +158,8 @@ else ln -sf libevent-${LIBEVENT_TAG_ORIG#release-}-linux64-utils.zip libevent-linux64-utils.zip ln -sf gmp-$GMP_VER-linux32-utils.zip gmp-linux32-utils.zip ln -sf gmp-$GMP_VER-linux64-utils.zip gmp-linux64-utils.zip + ln -sf go-$GO_VER-linux32-utils.zip go-linux32-utils.zip + ln -sf go-$GO_VER-linux64-utils.zip go-linux64-utils.zip cd .. fi

1 0

[tor-browser-bundle/master] Bug 20758: Make the gb build directory deterministic
by gk＠torproject.org 06 Dec '16

06 Dec '16

commit 27cf6fc8837ab98cb702d8c4db093e7b8f9d1c07 Author: Georg Koppen <gk(a)torproject.org> Date: Tue Dec 6 21:38:57 2016 +0000 Bug 20758: Make the gb build directory deterministic The build directory `gb` uses is created with `ioutil.TempDir()` which results in a different one for every started build with high probability. We make it deterministic with this commit. --- gitian/descriptors/linux/gitian-sandbox.yml | 2 ++ gitian/patches/gb-build-dir.patch | 31 +++++++++++++++++++++++++++++ gitian/versions.nightly | 4 ++-- 3 files changed, 35 insertions(+), 2 deletions(-) diff --git a/gitian/descriptors/linux/gitian-sandbox.yml b/gitian/descriptors/linux/gitian-sandbox.yml index 0f4f78c..0c67126 100644 --- a/gitian/descriptors/linux/gitian-sandbox.yml +++ b/gitian/descriptors/linux/gitian-sandbox.yml @@ -29,6 +29,7 @@ files: - "go-linux32-utils.zip" - "go-linux64-utils.zip" - "dzip.sh" +- "gb-build-dir.patch" script: | INSTDIR="$HOME/install" mkdir $INSTDIR/sandbox @@ -59,6 +60,7 @@ script: | # Building gb cd gb + patch -p1 < ../gb-build-dir.patch find -type f -print0 | xargs -0 touch --date="$REFERENCE_DATETIME" mkdir -p "$GOPATH/src/github.com/constabulary/" ln -sf "$PWD" "$GOPATH/src/github.com/constabulary/gb" diff --git a/gitian/patches/gb-build-dir.patch b/gitian/patches/gb-build-dir.patch new file mode 100644 index 0000000..a52ca69 --- /dev/null +++ b/gitian/patches/gb-build-dir.patch @@ -0,0 +1,31 @@ +From a7b198e3a32509197150d7c2767262a7319ff339 Mon Sep 17 00:00:00 2001 +From: Georg Koppen <gk(a)torproject.org> +Date: Tue, 6 Dec 2016 21:34:16 +0000 +Subject: [PATCH] Make the gb build directory deterministic + + +diff --git a/context.go b/context.go +index 53c00da..d040082 100644 +--- a/context.go ++++ b/context.go +@@ -3,7 +3,6 @@ package gb + import ( + "fmt" + "io" +- "io/ioutil" + "os" + "os/exec" + "path/filepath" +@@ -136,7 +135,8 @@ func NewContext(p Project, opts ...func(*Context) error) (*Context, error) { + }, + GcToolchain(), + } +- workdir, err := ioutil.TempDir("", "gb") ++ workdir := filepath.Join(os.TempDir(), "gb") ++ err := os.Mkdir(workdir, 0777) + if err != nil { + return nil, err + } +-- +2.10.2 + diff --git a/gitian/versions.nightly b/gitian/versions.nightly index 304284d..2df094b 100755 --- a/gitian/versions.nightly +++ b/gitian/versions.nightly @@ -48,8 +48,8 @@ GO_X_CRYPTO_TAG=master GO_X_NET_TAG=master OBFS4_TAG=master NOTOFONTS_TAG=720e34851382ee3c1ef024d8dffb68ffbfb234c2 -ERRORS_TAG=master -GB_TAG=master +ERRORS_TAG=248dadf4e9068a0b3e79f02ed0a610d935de5302 +GB_TAG=06cc925cce6592e922dcc4839a8b44feb384e71e SANDBOX_TAG=master GITIAN_TAG=tor-browser-builder-4

1 0

[tor-browser-bundle/master] Bug 20352: Adding sandbox to our Gitian build
by gk＠torproject.org 06 Dec '16

06 Dec '16

commit 522cee6cb9c83be66cc2a6b5bbfefdb3c2bc3217 Author: Georg Koppen <gk(a)torproject.org> Date: Tue Dec 6 21:20:53 2016 +0000 Bug 20352: Adding sandbox to our Gitian build On Linux we start building Yawning's sandbox code while producing the Linux Tor Browser bundles. For that to work properly we need to use Debian Jessie images and install some packages from backports as not all dependencies are available in Wheezy. --- gitian/Makefile | 6 ++- gitian/descriptors/linux/gitian-sandbox.yml | 77 +++++++++++++++++++++++++++++ gitian/fetch-inputs.sh | 3 ++ gitian/make-vms.sh | 22 ++++++++- gitian/mkbundle-linux.sh | 44 +++++++++++++---- gitian/versions.nightly | 3 ++ 6 files changed, 141 insertions(+), 14 deletions(-) diff --git a/gitian/Makefile b/gitian/Makefile index a4a757f..5694cbf 100644 --- a/gitian/Makefile +++ b/gitian/Makefile @@ -101,8 +101,7 @@ prep-alpha: ./check-prerequisites.sh $(TORSOCKS) ./fetch-inputs.sh ../../gitian-builder/inputs/ versions.alpha - -clean: clean-utils clean-tor clean-browser clean-pt clean-bundle +clean: clean-utils clean-tor clean-browser clean-pt clean-sandbox clean-bundle rm -f ../../gitian-builder/inputs/*.yml rm -f ../../gitian-builder/inputs/bundle.inputs rm -f ../../gitian-builder/inputs/versions* @@ -124,6 +123,9 @@ clean-browser: clean-bundle clean-pt: clean-bundle rm -f ../../gitian-builder/inputs/pluggable-transports* +clean-sandbox: clean-bundle + rm -f ../../gitian-builder/inputs/sandbox-linux* + clean-bundle: rm -f ../../gitian-builder/inputs/bundle-* diff --git a/gitian/descriptors/linux/gitian-sandbox.yml b/gitian/descriptors/linux/gitian-sandbox.yml new file mode 100644 index 0000000..0f4f78c --- /dev/null +++ b/gitian/descriptors/linux/gitian-sandbox.yml @@ -0,0 +1,77 @@ +--- +name: "sandbox-linux" +distro: "debian" +suites: +- "jessie" +architectures: +- "i386" +- "amd64" +packages: +- "unzip" +- "zip" +- "hardening-wrapper" +# Needed for the sandboxing code +- "libx11-dev" +- "pkg-config" +- "libgtk-3-dev" +backports_packages: +- "libseccomp-dev" +- "libseccomp2" +reference_datetime: "2000-01-01 00:00:00" +remotes: +- "url": "https://github.com/pkg/error" + "dir": "errors" +- "url": "https://github.com/constabulary/gb" + "dir": "gb" +- "url": "https://git.schwanenlied.me/yawning/sandboxed-tor-browser" + "dir": "sandbox" +files: +- "go-linux32-utils.zip" +- "go-linux64-utils.zip" +- "dzip.sh" +script: | + INSTDIR="$HOME/install" + mkdir $INSTDIR/sandbox + export REFERENCE_DATETIME + export TZ=UTC + export LC_ALL=C + umask 0022 + + # Config options for hardening-wrapper for the stub + export DEB_BUILD_HARDENING=1 + export DEB_BUILD_HARDENING_STACKPROTECTOR=1 + export DEB_BUILD_HARDENING_FORTIFY=1 + export DEB_BUILD_HARDENING_FORMAT=1 + export DEB_BUILD_HARDENING_PIE=1 + + unzip -d $INSTDIR go-linux$GBUILD_BITS-utils.zip + export GOROOT="$INSTDIR/go" + export GOPATH="$HOME/go" + export PATH="$PATH:$GOROOT/bin:$GOPATH/bin" + + # Building errors + cd errors + find -type f -print0 | xargs -0 touch --date="$REFERENCE_DATETIME" + mkdir -p "$GOPATH/src/github.com/pkg/" + ln -sf "$PWD" "$GOPATH/src/github.com/pkg/errors" + go install github.com/pkg/errors + cd .. + + # Building gb + cd gb + find -type f -print0 | xargs -0 touch --date="$REFERENCE_DATETIME" + mkdir -p "$GOPATH/src/github.com/constabulary/" + ln -sf "$PWD" "$GOPATH/src/github.com/constabulary/gb" + go install github.com/constabulary/gb/cmd/gb + cd .. + + # XXX: Use own C compiler + # Building the sandbox + cd sandbox + make + cp bin/sandboxed-tor-browser $INSTDIR/sandbox + cd .. + + cd $INSTDIR + ~/build/dzip.sh sandbox-linux$GBUILD_BITS.zip sandbox/ + cp sandbox-linux$GBUILD_BITS.zip $OUTDIR/ diff --git a/gitian/fetch-inputs.sh b/gitian/fetch-inputs.sh index bf60ca3..b3d5c26 100755 --- a/gitian/fetch-inputs.sh +++ b/gitian/fetch-inputs.sh @@ -293,6 +293,9 @@ goxcrypto https://go.googlesource.com/crypto $GO_X_CRYPTO_TAG goxnet https://go.googlesource.com/net $GO_X_NET_TAG obfs4 https://git.torproject.org/pluggable-transports/obfs4.git $OBFS4_TAG noto-fonts https://github.com/googlei18n/noto-fonts $NOTOFONTS_TAG +errors https://github.com/pkg/errors $ERRORS_TAG +gb https://github.com/constabulary/gb $GB_TAG +sandbox https://git.schwanenlied.me/yawning/sandboxed-tor-browser $SANDBOX_TAG EOF # HTTPS-Everywhere is special, too. We need to initialize the git submodules and diff --git a/gitian/make-vms.sh b/gitian/make-vms.sh index 089865b..7edd2b4 100755 --- a/gitian/make-vms.sh +++ b/gitian/make-vms.sh @@ -32,7 +32,7 @@ build_and_test_vm() { then export LXC_SUITE=$dist export LXC_ARCH=$arch - if [ "$dist" = "wheezy" ]; + if [ "$dist" = "wheezy" -o "$dist" = "jessie" ]; then export DISTRO=debian ./bin/make-base-vm --distro debian --suite $dist --lxc --arch $arch @@ -41,7 +41,7 @@ build_and_test_vm() { ./bin/make-base-vm --suite $dist --lxc --arch $arch fi else - if [ "$dist" = "wheezy" ]; + if [ "$dist" = "wheezy" -o "$dist" = "jessie" ]; then export DISTRO=debian ./bin/make-base-vm --distro debian --suite $dist --arch $arch @@ -89,6 +89,24 @@ do echo done +while ! build_and_test_vm jessie i386 32 +do + stop-target 32 jessie + rm ./base-jessie-amd64* + echo + echo "Jessie i386 VM build failed... Trying again" + echo +done + +while ! build_and_test_vm jessie amd64 64 +do + stop-target 64 jessie + rm ./base-jessie-amd64* + echo + echo "Jessie amd64 VM build failed... Trying again" + echo +done + while ! build_and_test_vm precise i386 32 do stop-target 32 precise diff --git a/gitian/mkbundle-linux.sh b/gitian/mkbundle-linux.sh index f8aaec1..430b666 100755 --- a/gitian/mkbundle-linux.sh +++ b/gitian/mkbundle-linux.sh @@ -78,6 +78,7 @@ die_msg() { # the utils both if we verify tags and if we don't. LIBEVENT_TAG_ORIG=$LIBEVENT_TAG +SANDBOX_ORIG_TAG=$SANDBOX_TAG if [ "z$VERIFY_TAGS" = "z1" ]; then @@ -112,7 +113,7 @@ if [ ! -f inputs/binutils-$BINUTILS_VER-linux32-utils.zip -o \ ! -f inputs/gmp-$GMP_VER-linux64-utils.zip ]; then echo - echo "****** Starting Utilities Component of Linux Bundle (1/5 for Linux) ******" + echo "****** Starting Utilities Component of Linux Bundle (1/6 for Linux) ******" echo ./bin/gbuild -j $NUM_PROCS -m $VM_MEMORY --commit libevent=$LIBEVENT_TAG $DESCRIPTOR_DIR/linux/gitian-utils.yml @@ -138,7 +139,7 @@ then #cp -a result/utils-linux-res.yml inputs/ else echo - echo "****** SKIPPING already built Utilities Component of Linux Bundle (1/5 for Linux) ******" + echo "****** SKIPPING already built Utilities Component of Linux Bundle (1/6 for Linux) ******" echo # We might have built the utilities in the past but maybe the links are # pointing to the wrong version. Refresh them. @@ -160,7 +161,7 @@ if [ ! -f inputs/tor-linux32-gbuilt.zip -o \ ! -f inputs/tor-linux64-gbuilt.zip ]; then echo - echo "****** Starting Tor Component of Linux Bundle (2/5 for Linux) ******" + echo "****** Starting Tor Component of Linux Bundle (2/6 for Linux) ******" echo ./bin/gbuild -j $NUM_PROCS -m $VM_MEMORY --commit tor=$TOR_TAG $DESCRIPTOR_DIR/linux/gitian-tor.yml @@ -175,7 +176,7 @@ then #cp -a result/tor-linux-res.yml inputs/ else echo - echo "****** SKIPPING already built Tor Component of Linux Bundle (2/5 for Linux) ******" + echo "****** SKIPPING already built Tor Component of Linux Bundle (2/6 for Linux) ******" echo fi @@ -184,7 +185,7 @@ if [ ! -f inputs/tor-browser-linux32-gbuilt.zip -o \ ! -f inputs/tor-browser-linux64-gbuilt.zip ]; then echo - echo "****** Starting TorBrowser Component of Linux Bundle (3/5 for Linux) ******" + echo "****** Starting TorBrowser Component of Linux Bundle (3/6 for Linux) ******" echo ./bin/gbuild -j $NUM_PROCS -m $VM_MEMORY --commit tor-browser=$TORBROWSER_TAG,faketime=$FAKETIME_TAG $DESCRIPTOR_DIR/linux/gitian-firefox.yml @@ -200,7 +201,7 @@ then #cp -a result/torbrowser-linux-res.yml inputs/ else echo - echo "****** SKIPPING already built TorBrowser Component of Linux Bundle (3/5 for Linux) ******" + echo "****** SKIPPING already built TorBrowser Component of Linux Bundle (3/6 for Linux) ******" echo fi @@ -208,7 +209,7 @@ if [ ! -f inputs/pluggable-transports-linux32-gbuilt.zip -o \ ! -f inputs/pluggable-transports-linux64-gbuilt.zip ]; then echo - echo "****** Starting Pluggable Transports Component of Linux Bundle (4/5 for Linux) ******" + echo "****** Starting Pluggable Transports Component of Linux Bundle (4/6 for Linux) ******" echo ./bin/gbuild -j $NUM_PROCS -m $VM_MEMORY --commit pyptlib=$PYPTLIB_TAG,obfsproxy=$OBFSPROXY_TAG,libfte=$LIBFTE_TAG,fteproxy=$FTEPROXY_TAG,txsocksx=$TXSOCKSX_TAG,goptlib=$GOPTLIB_TAG,meek=$MEEK_TAG,ed25519=$GOED25519_TAG,siphash=$GOSIPHASH_TAG,goxcrypto=$GO_X_CRYPTO_TAG,goxnet=$GO_X_NET_TAG,obfs4=$OBFS4_TAG $DESCRIPTOR_DIR/linux/gitian-pluggable-transports.yml @@ -222,14 +223,35 @@ then #cp -a result/pluggable-transports-linux-res.yml inputs/ else echo - echo "****** SKIPPING already built Pluggable Transports Component of Linux Bundle (4/5 for Linux) ******" + echo "****** SKIPPING already built Pluggable Transports Component of Linux Bundle (4/6 for Linux) ******" + echo +fi + +if [ ! -f inputs/sandbox-linux32.zip -o \ + ! -f inputs/sandbox-linux64.zip ]; +then + echo + echo "****** Starting Sandbox Component of Linux Bundle (5/6 for Linux) ******" + echo + + ./bin/gbuild -j $NUM_PROCS -m $VM_MEMORY --commit errors=$ERRORS_TAG,gb=$GB_TAG,sandbox=$SANDBOX_TAG $DESCRIPTOR_DIR/linux/gitian-sandbox.yml + if [ $? -ne 0 ]; + then + exit 1 + fi + + cp -a build/out/sandbox-linux*.zip inputs/ + #cp -a result/sandbox-linux-res.yml inputs/ +else + echo + echo "****** SKIPPING already built Sandbox Component of Linux Bundle (5/6 for Linux) ******" echo fi if [ ! -f inputs/bundle-linux.gbuilt ]; then echo - echo "****** Starting Bundling+Localization of Linux Bundle (5/5 for Linux) ******" + echo "****** Starting Bundling+Localization of Linux Bundle (6/6 for Linux) ******" echo cd $WRAPPER_DIR && ./record-inputs.sh $VERSIONS_FILE && cd $GITIAN_DIR @@ -246,10 +268,12 @@ then cp -a build/out/*.mar $WRAPPER_DIR/$TORBROWSER_BUILDDIR/ || exit 1 cp -a inputs/mar-tools-linux*.zip $WRAPPER_DIR/$TORBROWSER_BUILDDIR/ || exit 1 cp -a inputs/*debug.zip $WRAPPER_DIR/$TORBROWSER_BUILDDIR/ || exit 1 + cp -a inputs/sandbox-linux32.zip $WRAPPER_DIR/$TORBROWSER_BUILDDIR/sandbox-linux32-${SANDBOX_ORIG_TAG}.zip || exit 1 + cp -a inputs/sandbox-linux64.zip $WRAPPER_DIR/$TORBROWSER_BUILDDIR/sandbox-linux64-${SANDBOX_ORIG_TAG}.zip || exit 1 touch inputs/bundle-linux.gbuilt else echo - echo "****** SKIPPING already built Bundling+Localization of Linux Bundle (5/5 for Linux) ******" + echo "****** SKIPPING already built Bundling+Localization of Linux Bundle (6/6 for Linux) ******" echo fi diff --git a/gitian/versions.nightly b/gitian/versions.nightly index f487cb5..304284d 100755 --- a/gitian/versions.nightly +++ b/gitian/versions.nightly @@ -48,6 +48,9 @@ GO_X_CRYPTO_TAG=master GO_X_NET_TAG=master OBFS4_TAG=master NOTOFONTS_TAG=720e34851382ee3c1ef024d8dffb68ffbfb234c2 +ERRORS_TAG=master +GB_TAG=master +SANDBOX_TAG=master GITIAN_TAG=tor-browser-builder-4

1 0

[gitian-builder/tor-browser-builder-4] Bug 20352: Enabling Jessie backports for libseccomp
by gk＠torproject.org 06 Dec '16

06 Dec '16

commit dd26189595159bb8f50185a754dd4d0bc61b5787 Author: Georg Koppen <gk(a)torproject.org> Date: Mon Dec 5 07:30:33 2016 +0000 Bug 20352: Enabling Jessie backports for libseccomp --- target-bin/bootstrap-fixup.in | 1 + 1 file changed, 1 insertion(+) diff --git a/target-bin/bootstrap-fixup.in b/target-bin/bootstrap-fixup.in index 738ed82..d8aaed3 100755 --- a/target-bin/bootstrap-fixup.in +++ b/target-bin/bootstrap-fixup.in @@ -13,6 +13,7 @@ elif [ $DISTRIB_NAME = "Debian" ]; then echo "deb http://HOSTIP:3142/ftp.debian.org/debian $DISTRIB_CODENAME main" > $1/etc/apt/sources.list echo "deb http://HOSTIP:3142/security.debian.org/ $DISTRIB_CODENAME/updates main" >> $1/etc/apt/sources.list echo "deb http://HOSTIP:3142/ftp.debian.org/debian $DISTRIB_CODENAME-updates main" >> $1/etc/apt/sources.list + echo "deb http://HOSTIP:3142/ftp.debian.org/debian $DISTRIB_CODENAME-backports main" >> $1/etc/apt/sources.list # grub-legacy conflicts grub-pc dependencies # No grub-legacy on Ubuntu, just on Debian # Work around bcron-run conflict due to cron being removed

1 0

[gitian-builder/tor-browser-builder-4] Bug 20352: add option to install backports packages
by gk＠torproject.org 06 Dec '16

06 Dec '16

commit afe8247f9fdf561b3e0d7701d2f49ef3bd5e6cb4 Author: Nicolas Vigier <boklm(a)torproject.org> Date: Mon Dec 5 13:42:04 2016 +0100 Bug 20352: add option to install backports packages The packages listed in the descriptors in 'backports_packages' are installed from the backports. --- bin/gbuild | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/bin/gbuild b/bin/gbuild index 512733a..e61fcb6 100755 --- a/bin/gbuild +++ b/bin/gbuild @@ -102,6 +102,11 @@ def build_one_configuration(distro, suite, arch, build_desc) info "Installing additional packages (log in var/install.log)" system! "on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install #{build_desc["packages"].join(" ")} > var/install.log 2>&1" + if build_desc.has_key?("backports_packages") + info "Installing additional backport packages (log in var/backports-install.log)" + system! "on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y -t #{suite}-backports install #{build_desc["backports_packages"].join(" ")} > var/backports-install.log 2>&1" + end + if @options[:upgrade] || system("on-target -u root '[ ! -e /var/cache/gitian/initial-upgrade ]'") info "Upgrading system, may take a while" system! "on-target -u root bash < target-bin/upgrade-system.sh > var/install.log 2>&1"

1 0

[tor-browser-bundle/hardened-builds] Bug 20903: Update gitian branch for nightlies if needed
by gk＠torproject.org 06 Dec '16

06 Dec '16

commit a0c4b74b735e2e2c71bfe9054cde5085ca9033c7 Author: Georg Koppen <gk(a)torproject.org> Date: Tue Dec 6 19:55:17 2016 +0000 Bug 20903: Update gitian branch for nightlies if needed --- gitian/fetch-inputs.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/gitian/fetch-inputs.sh b/gitian/fetch-inputs.sh index bbd2c88..65ebc11 100755 --- a/gitian/fetch-inputs.sh +++ b/gitian/fetch-inputs.sh @@ -253,7 +253,13 @@ cd .. git remote set-url origin https://git.torproject.org/builders/gitian-builder.git git fetch origin git fetch --tags origin # XXX - why do we fetch tags specifically? -git checkout "$GITIAN_TAG" +# If we're not verifying tags, then the branch requires an update. +if [ $VERIFY_TAGS -eq 0 ]; +then + git pull +else + git checkout "$GITIAN_TAG" +fi cd inputs while read dir url tag; do

1 0