- tbb-commits - lists.torproject.org

[tor-browser-build/master] README: add missing dependencies
by boklm＠torproject.org 31 Mar '17

31 Mar '17

commit 041df309831d73f69b7f2fc47b15211dda7e5e31 Author: Nicolas Vigier <boklm(a)torproject.org> Date: Fri Mar 31 12:57:04 2017 +0200 README: add missing dependencies --- README | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README b/README index 07725cf..08904fb 100644 --- a/README +++ b/README @@ -15,7 +15,7 @@ also be running. The sources of most components are downloaded using git, which needs to be installed. The sources of webrtc are downloaded using gclient, which -requires GTK+ 2.0 development files to be installed. +requires GTK+ 2.0 development files and curl to be installed. You also need a few perl modules installed: - YAML::XS @@ -27,6 +27,7 @@ You also need a few perl modules installed: - File::Temp - File::Slurp - File::Path +- File::Copy::Recursive - String::ShellQuote - Sort::Versions - Digest::SHA @@ -40,7 +41,7 @@ If you are running Debian or Ubuntu, you can install them with: libio-captureoutput-perl libfile-slurp-perl \ libstring-shellquote-perl libsort-versions-perl \ libdigest-sha-perl libdata-uuid-perl libdata-dump-perl \ - git libgtk2.0-dev + libfile-copy-recursive-perl git libgtk2.0-dev curl Starting a build

1 0

[tor-launcher/master] Bug 21546: Tor Launcher is multiprocessCompatible
by gk＠torproject.org 29 Mar '17

29 Mar '17

commit 336e7c8731296866dd67a090c010e41fcc26ef69 Author: Georg Koppen <gk(a)torproject.org> Date: Sun Mar 12 20:39:27 2017 +0000 Bug 21546: Tor Launcher is multiprocessCompatible --- src/install.rdf | 1 + 1 file changed, 1 insertion(+) diff --git a/src/install.rdf b/src/install.rdf index 493d976..881aa8a 100644 --- a/src/install.rdf +++ b/src/install.rdf @@ -8,6 +8,7 @@ <em:contributor>Pearl Crescent, LLC</em:contributor> <em:id>tor-launcher(a)torproject.org</em:id> <em:version>0.2.11.1</em:version> + <em:multiprocessCompatible>true</em:multiprocessCompatible> <em:homepageURL>https://www.torproject.org/projects/torbrowser.html</em:homepageURL> <em:updateURL>data:text/plain,</em:updateURL> <em:updateKey>-</em:updateKey>

1 0

[tor-browser-build/master] HTTPS Everywhere: Version bump - 5.2.13
by boklm＠torproject.org 27 Mar '17

27 Mar '17

commit bfc288d54a28f00e088ba9ce08bd718156d8ff2f Author: William Budington <bill(a)eff.org> Date: Sun Mar 26 07:12:49 2017 -0700 HTTPS Everywhere: Version bump - 5.2.13 --- projects/https-everywhere/config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/https-everywhere/config b/projects/https-everywhere/config index 17f9f82..7ef8568 100644 --- a/projects/https-everywhere/config +++ b/projects/https-everywhere/config @@ -1,5 +1,5 @@ # vim: filetype=yaml sw=2 -version: 5.2.11 +version: 5.2.13 git_url: https://git.torproject.org/https-everywhere.git git_hash: '[% c("version") %]' git_submodule: 1

1 0

[tor-browser-build/master] README: Add libgtk2.0-dev to the list of dependencies
by boklm＠torproject.org 26 Mar '17

26 Mar '17

commit 29f7cffa1199dd89da17a6d7792fe7f265d05a42 Author: Nicolas Vigier <boklm(a)torproject.org> Date: Sun Mar 26 14:45:01 2017 +0200 README: Add libgtk2.0-dev to the list of dependencies --- README | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README b/README index f4a74e0..07725cf 100644 --- a/README +++ b/README @@ -13,7 +13,9 @@ Your user account should have access to the docker command without using sudo, so it should be in the docker group. The docker daemon should also be running. -The sources are downloaded using git which needs to be installed. +The sources of most components are downloaded using git, which needs to +be installed. The sources of webrtc are downloaded using gclient, which +requires GTK+ 2.0 development files to be installed. You also need a few perl modules installed: - YAML::XS @@ -38,7 +40,7 @@ If you are running Debian or Ubuntu, you can install them with: libio-captureoutput-perl libfile-slurp-perl \ libstring-shellquote-perl libsort-versions-perl \ libdigest-sha-perl libdata-uuid-perl libdata-dump-perl \ - git + git libgtk2.0-dev Starting a build

1 0

[tor-browser-build/master] Bug 21655: add a script to clean old built components
by boklm＠torproject.org 21 Mar '17

21 Mar '17

commit 5d046ea3c4af539e36a886ee3a7b03c07f5d7b48 Author: Nicolas Vigier <boklm(a)torproject.org> Date: Mon Mar 20 18:29:03 2017 +0100 Bug 21655: add a script to clean old built components --- Makefile | 5 ++- README | 10 ++++- rbm | 2 +- rbm.local.conf.example | 28 ++++++++++++ tools/clean-old | 114 +++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 155 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index 9481537..b82309c 100644 --- a/Makefile +++ b/Makefile @@ -86,6 +86,9 @@ submodule-update: fetch: submodule-update $(rbm) fetch -clean-old: submodule-update +clean: submodule-update ./tools/clean-old +clean-dry-run: submodule-update + ./tools/clean-old --dry-run + diff --git a/README b/README index a14fb42..f4a74e0 100644 --- a/README +++ b/README @@ -171,8 +171,14 @@ file. Cleaning obsolete files and containers images --------------------------------------------- -There will be a script to clean old build files and containers that are -no longer used, but it has not been added yet. +You can run 'make clean' to clean old build files and containers that +are no longer used in current builds. Before doing that, you need to +configure the branches and build targets you are using in the +rbm.local.conf file. The cleaning script will checkout all the configured +branches to create a list of used build files, and delete the files +from the 'out' directory that are not used. If you want to see the list +of files and containers that would be removed without doing it, you can +use 'make clean-dry-run'. Multiple build directories on the same host diff --git a/rbm b/rbm index 845bb90..3f3886e 160000 --- a/rbm +++ b/rbm @@ -1 +1 @@ -Subproject commit 845bb904fb87b6c56ab789f35cd02bce32453f52 +Subproject commit 3f3886e1f210ad2853209c5aecd0951350a6f758 diff --git a/rbm.local.conf.example b/rbm.local.conf.example index 2a86647..78de08f 100644 --- a/rbm.local.conf.example +++ b/rbm.local.conf.example @@ -39,6 +39,34 @@ var: ### file. #sign_build_gpg_opts: '--local-user XXXXXXXX' + ### The clean configuration is used by the cleaning script to find the + ### branches and build targets you are using, to compute the list of + ### files that should be kept. + ### + ### If you only do alpha builds for all platforms, you can use the + ### following configuration: + #clean: + # HEAD: + # - project: release + # target: + # - alpha + # - torbrowser-all + # + ### If you are doing 'release' builds in the maint-7.0 branch and + ### 'alpha' builds in the master branch, you can use the following + ### configuration: + #clean: + # master: + # - project: release + # target: + # - alpha + # - torbrowser-all + # maint-7.0: + # - project: release + # target: + # - release + # - torbrowser-all + targets: ### testbuild is based on alpha by default. Uncomment this if you want it diff --git a/tools/clean-old b/tools/clean-old new file mode 100755 index 0000000..4d603fc --- /dev/null +++ b/tools/clean-old @@ -0,0 +1,114 @@ +#!/usr/bin/perl -w +use strict; +use IO::CaptureOutput qw(capture_exec); +use FindBin; +use lib "$FindBin::Bin/../rbm/lib"; +use RBM; +use File::Slurp qw(read_dir write_file read_file); +use Getopt::Long; + +my %options; +my @options_list = qw(dry-run! list-used-files=s project=s target=s@); +Getopt::Long::GetOptionsFromArray(\@ARGV, \%options, @options_list) || exit 1; + +sub clean_file { + my ($file, $used_files) = @_; + return if $used_files->{$file}; + if (-d $file) { + my @l = read_dir($file); + foreach my $subfile (@l) { + clean_file("$file/$subfile", $used_files); + } + @l = read_dir($file); + rmdir $file unless @l; + } else { + print "Removing file $file\n"; + unlink $file unless $options{'dry-run'}; + } +} + +sub clean_docker_images { + my ($dockerdir, $used_files) = @_; + my $imgprefix = RBM::project_config('docker-image', 'docker_image_prefix'); + my @imgs = read_dir($dockerdir); + foreach my $dockerimage (@imgs) { + next if $used_files->{"$dockerdir/$dockerimage"}; + my $img = "$imgprefix:$dockerimage"; + print "Cleaning docker image $img\n"; + next if $options{'dry-run'}; + my ($out, $err, $success) = capture_exec('docker', 'rmi', '-f', $img); + if (!$success) { + print STDERR "Error removing docker image $img:\n$err\n"; + exit 1; + } + unlink "$dockerdir/$dockerimage"; + } +} + +sub get_project_input_files { + my ($project, @targets) = @_; + print "Getting input files for $project ", join(' ', @targets), "\n"; + $RBM::config->{run}{target} = \@targets; + my $res = RBM::project_config($project, 'input_files_paths', + {error_if_undef => 1}); + return ref $res eq 'ARRAY' ? @$res : (); +} + +sub get_input_files { + my ($c) = @_; + my @files; + foreach my $p (@$c) { + my $tmp = File::Temp->new(); + my @args = ("$FindBin::Bin/$FindBin::Script", '--list-used-files', $tmp, + '--project', $p->{project}); + foreach my $target (@{$p->{target}}) { + push @args, ('--target', $target); + } + RBM::exit_error 'Error getting used files' unless system(@args) == 0; + push @files, map { chomp; $_ } read_file($tmp); + } + return @files; +} + +RBM::load_config; +RBM::load_system_config; +RBM::load_local_config; +RBM::set_default_env; + +if ($options{'list-used-files'}) { + if (!$options{project} || !$options{target}) { + RBM::exit_error 'Missing option project or target'; + } + my @files = get_project_input_files($options{project}, @{$options{target}}); + write_file($options{'list-used-files'}, join("\n", @files)); + exit 0; +} + +my $clean = RBM::project_config('clean', 'var/clean'); +if (!$clean) { + print STDERR "Clean configuration is missing. ", + "You should add it to rbm.local.conf.\n"; + exit 1; +} + +my @files = get_input_files($clean->{HEAD}) if $clean->{HEAD}; + +foreach my $branch (keys %$clean) { + next if $branch eq 'HEAD'; + print "Checking out $branch branch\n"; + RBM::exit_error("Error checking out $branch branch") + unless system('git', 'checkout', $branch) == 0; + RBM::exit_error("Error running git submodule update --init") + unless system('git', 'submodule', 'update', '--init') == 0; + push @files, get_input_files($clean->{$branch}); + RBM::exit_error('Error checking out @{-1}') + unless system('git', 'checkout', '@{-1}') == 0; + RBM::exit_error("Error running git submodule update --init") + unless system('git', 'submodule', 'update', '--init') == 0; +} +my %used_files = map { $_ => 1 } @files; +my $outdir = $RBM::config->{basedir} . '/out'; +# Don't clean docker-image files yet +$used_files{"$outdir/docker-image"} = 1; +clean_file($outdir, \%used_files); +clean_docker_images("$outdir/docker-image", \%used_files);

1 0

[rbm/master] Add the input_files_paths option
by boklm＠torproject.org 15 Mar '17

15 Mar '17

commit 3f3886e1f210ad2853209c5aecd0951350a6f758 Author: Nicolas Vigier <boklm(a)torproject.org> Date: Wed Mar 15 12:25:39 2017 +0100 Add the input_files_paths option --- NEWS | 2 ++ doc/rbm_config.asc | 6 ++++++ lib/RBM.pm | 23 ++++++++++++++++++++++- lib/RBM/DefaultConfig.pm | 1 + 4 files changed, 31 insertions(+), 1 deletion(-) diff --git a/NEWS b/NEWS index 5de36e0..4eac374 100644 --- a/NEWS +++ b/NEWS @@ -1,3 +1,5 @@ +- Add the input_files_paths option + - Add the ENV option - Add the build_log option diff --git a/doc/rbm_config.asc b/doc/rbm_config.asc index 48e505c..2e9b8ab 100644 --- a/doc/rbm_config.asc +++ b/doc/rbm_config.asc @@ -220,6 +220,12 @@ input_files_id:: file of type +project+, and the +filename+ and the sha256sum of the file for any other type of input file. +input_files_paths:: + The value of this option is an array of all the paths of input + files that currently exist and are used in the build of the + current project and its dependencies. This is useful when + cleaning old build files, to find which ones are still used. + timestamp:: This is the UNIX timestamp, set as modification time on files created such as the sources tarball and rpm spec file. The diff --git a/lib/RBM.pm b/lib/RBM.pm index 632a8ba..79e0670 100644 --- a/lib/RBM.pm +++ b/lib/RBM.pm @@ -731,8 +731,9 @@ sub input_files { my ($action, $project, $options, $dest_dir) = @_; my @res_copy; my %res_getfnames; + my @res_getfpaths; my $getfnames_noname = 0; - my $need_dl = 1; + my $need_dl = $action ne 'getfpaths'; my $input_files_id = ''; $options = {$options ? %$options : ()}; my $input_files = project_config($project, 'input_files', $options,); @@ -840,6 +841,25 @@ sub input_files { $input_files_id .= "\n"; next; } + if ($action eq 'getfpaths') { + push @res_getfpaths, $fname if $fname; + if ($file_gpg_id && $fname) { + my $sig_ext = $t->('sig_ext'); + $sig_ext = ref $sig_ext eq 'ARRAY' ? $sig_ext : [ $sig_ext ]; + foreach my $s (@$sig_ext) { + if (-f "$fname.$s") { + push @res_getfpaths, "$fname.$s"; + last; + } + } + } + if ($input_file->{project} && $t->('project')) { + my $r = RBM::project_step_config($t->('project'), 'input_files_paths', + {%$options, step => $t->('pkg_type'), %$input_file}); + push @res_getfpaths, @$r if @$r; + } + next; + } exit_error "Missing file $name" unless $fname; if ($t->('sha256sum') && $t->('sha256sum') ne sha256_hex(read_file($fname))) { @@ -886,6 +906,7 @@ sub input_files { return sha256_hex($input_files_id) if $action eq 'input_files_id'; return @res_copy if $action eq 'copy'; return \%res_getfnames if $action eq 'getfnames'; + return \@res_getfpaths if $action eq 'getfpaths'; } sub system_log { diff --git a/lib/RBM/DefaultConfig.pm b/lib/RBM/DefaultConfig.pm index c0764bd..cf0ed74 100644 --- a/lib/RBM/DefaultConfig.pm +++ b/lib/RBM/DefaultConfig.pm @@ -583,6 +583,7 @@ ZIP_END arch => \&get_arch, input_files_by_name => sub { RBM::input_files('getfnames', @_); }, input_files_id => sub { RBM::input_files('input_files_id', @_); }, + input_files_paths => sub { RBM::input_files('getfpaths', @_); }, steps => { srpm => 'rpm', 'deb-src' => 'deb',

1 0

[tor-browser-build/master] Add snowflake
by boklm＠torproject.org 14 Mar '17

14 Mar '17

commit 832c2ce17672d5cd1f554354902df670e6ee928a Author: Nicolas Vigier <boklm(a)torproject.org> Date: Mon Mar 13 19:39:02 2017 +0100 Add snowflake Only for linux-x86_64 for now, as it fails to build on linux-i686. --- .gitignore | 1 + projects/depot_tools/build | 8 +++ projects/depot_tools/config | 5 ++ projects/gcc/config | 2 +- projects/go-webrtc/config | 53 ++++++++++++++++++++ projects/go/config | 1 + projects/snowflake/build | 38 +++++++++++++++ projects/snowflake/config | 35 ++++++++++++++ projects/tor-browser/build | 3 ++ projects/tor-browser/config | 3 ++ projects/webrtc/build | 80 ++++++++++++++++++++++++++++++ projects/webrtc/config | 115 ++++++++++++++++++++++++++++++++++++++++++++ rbm.conf | 4 ++ 13 files changed, 347 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 9b6754f..0fe0aec 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ git_clones hg_clones +gclient out release alpha diff --git a/projects/depot_tools/build b/projects/depot_tools/build new file mode 100644 index 0000000..ac0f21b --- /dev/null +++ b/projects/depot_tools/build @@ -0,0 +1,8 @@ +#!/bin/bash +[% c("var/set_default_env") -%] +tar xf [% project %]-[% c("version") %].tar.gz +mv [% project %]-[% c("version") %] [% project %] +[% c('tar', { + tar_src => [ project ], + tar_args => '-czf ' _ dest_dir _ '/' _ c('filename'), + }) %] diff --git a/projects/depot_tools/config b/projects/depot_tools/config new file mode 100644 index 0000000..40a6d85 --- /dev/null +++ b/projects/depot_tools/config @@ -0,0 +1,5 @@ +# vim: filetype=yaml sw=2 +version: '[% c("abbrev") %]' +git_hash: 28216cd14b44716db5c83634afbdc6e90492652b +git_url: https://chromium.googlesource.com/chromium/tools/depot_tools.git +filename: '[% project %]-[% c("version") %].tar.gz' diff --git a/projects/gcc/config b/projects/gcc/config index bf73b7d..391e453 100644 --- a/projects/gcc/config +++ b/projects/gcc/config @@ -10,7 +10,7 @@ var: - libc6-dev-i386 setup: | mkdir -p /var/tmp/dist - tar -C /var/tmp/dist -xf [% c("compiler_tarfile") %] + tar -C /var/tmp/dist -xf $rootdir/[% c("compiler_tarfile") %] export PATH="/var/tmp/dist/gcc/bin:$PATH" [% IF c("var/osname") == "linux-i686" -%] export LD_LIBRARY_PATH=/var/tmp/dist/gcc/lib32 diff --git a/projects/go-webrtc/config b/projects/go-webrtc/config new file mode 100644 index 0000000..778d3db --- /dev/null +++ b/projects/go-webrtc/config @@ -0,0 +1,53 @@ +# vim: filetype=yaml sw=2 +version: '[% c("abbrev") %]' +git_url: https://github.com/keroserene/go-webrtc.git +git_hash: 69c5ca70533142494d145b43ef2f46ca94715557 +filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.gz' +remote_docker: 1 + +build: '[% c("projects/go/var/build_go_lib") %]' + +var: + go_lib: github.com/keroserene/go-webrtc + build_go_lib_pre: | + [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] + [% IF c("var/linux") %] + tar -C /var/tmp/dist -xf $rootdir/[% c('input_files_by_name/binutils') %] + export PATH="/var/tmp/dist/binutils/bin:$PATH" + [% END -%] + tar -C /var/tmp/dist -xf $rootdir/[% c('input_files_by_name/webrtc') %] + # Replace the prebuilt webrtc library with our own one. + rm -rf include/ lib/ + cp -a /var/tmp/dist/webrtc/{include,lib} . + # The go-webrtc package sets _GLIBCXX_USE_CXX11_ABI=0 for compatibility with + # the prebuilt libwebrtc-magic.a libraries. Since we build our own + # libwebrtc-magic.a with the C++11 ABI, we have to undo that setting here, + # using CGO_CXXFLAGS. + export CGO_CXXFLAGS=-D_GLIBCXX_USE_CXX11_ABI=1 + +targets: + master: + git_hash: master + linux-x86_64: + var: + arch_deps: + - pkg-config + - libx11-dev + linux-i686: + var: + arch_deps: + - pkg-config:i386 + - libx11-dev:i386 + - lib32stdc++6 + +input_files: + - project: docker-image + - name: go + project: go + - name: webrtc + project: webrtc + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + - name: binutils + project: binutils + enable: '[% c("var/linux") %]' diff --git a/projects/go/config b/projects/go/config index 5e55a09..870dc50 100644 --- a/projects/go/config +++ b/projects/go/config @@ -35,6 +35,7 @@ var: do patch -p1 < $p done + [% IF c("var/build_go_lib_pre"); GET c("var/build_go_lib_pre"); END; -%] [% IF c("var/go_lib_install") -%] [% FOREACH inst IN c("var/go_lib_install") %] go install [% inst %] diff --git a/projects/snowflake/build b/projects/snowflake/build new file mode 100644 index 0000000..6e08b4e --- /dev/null +++ b/projects/snowflake/build @@ -0,0 +1,38 @@ +#!/bin/bash +[% c("var/set_default_env") -%] +[% pc('go', 'var/setup', { go_tarfile => c('input_files_by_name/go') }) %] +[% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] +[% IF c("var/linux") %] + tar -C /var/tmp/dist -xf $rootdir/[% c('input_files_by_name/binutils') %] + export PATH="/var/tmp/dist/binutils/bin:$PATH" +[% END -%] +distdir=/var/tmp/dist/[% project %] +[% IF c("var/osx") %] + PTDIR="$distdir/Contents/MacOS/Tor/PluggableTransports" + DOCSDIR="$distdir/Contents/Resources/TorBrowser/Docs/snowflake" +[% ELSE %] + PTDIR="$distdir/TorBrowser/Tor/PluggableTransports" + DOCSDIR="$distdir/TorBrowser/Docs/snowflake" +[% END %] +mkdir -p $PTDIR $DOCSDIR + +tar -C /var/tmp/dist -xf [% c('input_files_by_name/go-webrtc') %] +tar -C /var/tmp/dist -xf [% c('input_files_by_name/uniuri') %] +tar -C /var/tmp/dist -xf [% c('input_files_by_name/goptlib') %] + +mkdir -p /var/tmp/build +tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.gz +cd /var/tmp/build/[% project %]-[% c('version') %] + +cd client +go build -ldflags '-s' +cp -a client[% IF c("var/windows") %].exe[% END %] $PTDIR/snowflake-client[% IF c("var/windows") %].exe[% END %] + +cd .. +cp -a README.md LICENSE $DOCSDIR + +cd $distdir +[% c('tar', { + tar_src => [ '.' ], + tar_args => '-czf ' _ dest_dir _ '/' _ c('filename'), + }) %] diff --git a/projects/snowflake/config b/projects/snowflake/config new file mode 100644 index 0000000..5ba006b --- /dev/null +++ b/projects/snowflake/config @@ -0,0 +1,35 @@ +# vim: filetype=yaml sw=2 +version: '[% c("abbrev") %]' +git_url: https://git.torproject.org/pluggable-transports/snowflake.git +git_hash: 6cecd31fd896eb26e64ad8bab8a9ea510ec3b21d +filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.gz' +remote_docker: 1 + +targets: + linux-i686: + var: + arch_deps: + - pkg-config + - libx11-dev:i386 + - lib32stdc++6 + linux-x86_64: + var: + arch_deps: + - pkg-config + - libx11-dev + +input_files: + - project: docker-image + - name: go + project: go + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + - name: binutils + project: binutils + enable: '[% c("var/linux") %]' + - name: go-webrtc + project: go-webrtc + - name: uniuri + project: uniuri + - name: goptlib + project: goptlib diff --git a/projects/tor-browser/build b/projects/tor-browser/build index 904f1b6..814f7af 100644 --- a/projects/tor-browser/build +++ b/projects/tor-browser/build @@ -52,6 +52,9 @@ mv [% c('input_files_by_name/noscript') %] $TBDIR/$EXTSPATH/{73a6fe31-595d-460b- tar -C $TBDIR -xf [% c('input_files_by_name/obfs4') %] tar -C $TBDIR -xf [% c('input_files_by_name/meek') %] mv $TBDIR/meek-http-helper(a)bamsoftware.com.xpi $TBDIR/$MEEKPROFILEPATH/extensions/ +[% IF c("var/snowflake") %] + tar -C $TBDIR -xf [% c('input_files_by_name/snowflake') -%] +[% END -%] tar -C $TBDIR[% IF c("var/osx") %]/Contents/Resources[% END %] -xf [% c('input_files_by_name/fonts') %] diff --git a/projects/tor-browser/config b/projects/tor-browser/config index 95f444d..f2c1d72 100644 --- a/projects/tor-browser/config +++ b/projects/tor-browser/config @@ -48,6 +48,9 @@ input_files: name: meek - project: obfs4 name: obfs4 + - project: snowflake + name: snowflake + enable: '[% c("var/snowflake") %]' - filename: Bundle-Data - URL: https://addons.cdn.mozilla.net/user-media/addons/722/noscript_security_suit… name: noscript diff --git a/projects/webrtc/build b/projects/webrtc/build new file mode 100644 index 0000000..5a70fbe --- /dev/null +++ b/projects/webrtc/build @@ -0,0 +1,80 @@ +#!/bin/bash +[% c("var/set_default_env") -%] +distdir=/var/tmp/dist/[% project %] +[% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] + +mkdir -p /var/tmp/dist + +[% IF c("var/linux") %] + tar -C /var/tmp/dist -xf $rootdir/[% c('input_files_by_name/binutils') %] + export PATH="/var/tmp/dist/binutils/bin:$PATH" +[% END -%] + +# Setting up depot_tools +# https://dev.chromium.org/developers/how-tos/install-depot-tools +tar -C /var/tmp/dist -xf $rootdir/[% c('input_files_by_name/depot_tools') %] +export PATH="$PATH:/var/tmp/dist/depot_tools" +# Disable automatic updating. +export DEPOT_TOOLS_UPDATE=0 + +mkdir -p /var/tmp/build +tar -C /var/tmp/build -xf webrtc-sources-[% c('version') %].tar.gz + +builddir=/var/tmp/build/[% project %]/src +cd $builddir +export GYP_CROSSCOMPILE=1 +export GYP_DEFINES="OS=[% c("var/webrtc/os") %] target_arch=[% c("var/webrtc/arch") %]" +# Do not use bundled utilities. +GYP_DEFINES+=" clang=0 host_clang=0 use_sysroot=0" +GYP_DEFINES+=" linux_use_bundled_gold=0 linux_use_gold_flags=0 linux_use_bundled_binutils=0" +# Avoid some dependencies. Some of these don't seem to work fully; for +# example even with use_alsa=0 we need libasound2-dev. +GYP_DEFINES+=" use_x11=0 use_gtk=0 use_gconf=0 use_alsa=0 use_pulseaudio=0 use_libpci=0 include_internal_audio_device=0 enabled_libjingle_device_manager=0" +# examples and tests also bring in dependencies. +GYP_DEFINES+=" include_examples=0 include_tests=0" +# embedded=1 is supposed to turn of various features; see +# https://bugs.chromium.org/p/chromium/issues/detail?id=318413. +GYP_DEFINES+=" embedded=1" +GYP_DEFINES+=" werror=" +[% IF c("var/linux") -%] + export CC=gcc + export CXX=g++ +[% END -%] +[% IF c("var/linux-i686") -%] + GYP_DEFINES+=" target_arch=ia32" + export CC='gcc -m32' + export CXX='g++ -m32' + export CFLAGS=-m32 + export CXXFLAGS=-m32 + export LDFLAGS=-m32 +[% END -%] +JAVA_HOME=/usr/lib/jvm/default-java webrtc/build/gyp_webrtc.py +ninja -C out/Release +# Run ninja once more, without include_tests=0, in order to build just the +# FakeAudioCaptureModule that go-webrtc uses. +GYP_DEFINES="$(echo "$GYP_DEFINES" | sed -e 's/include_tests=0//g')" +JAVA_HOME=/usr/lib/jvm/default-java webrtc/build/gyp_webrtc.py +ninja -C out/Release obj/webrtc/api/test/peerconnection_unittests.fakeaudiocapturemodule.o +# https://github.com/keroserene/go-webrtc/issues/23#issuecomment-175312648 +# dump_syms_regtest.o is actually an executable, not an object file. If not +# excluded, it results in the error: +# libwebrtc-linux-386-magic.a(dump_syms_regtest.o): unsupported ELF file type 2 +ar crs libwebrtc-magic.a $(find . -name '*.o' -not -name '*.main.o' -not -name 'dump_syms_regtest.o' | sort) + +mkdir -p $distdir +cd $distdir +mkdir -p include lib +cp -f $builddir/libwebrtc-magic.a [% c("var/webrtc/lib_path") %] +INCLUDE_DIR="$PWD/include" +cd $builddir +for h in $(find talk/ webrtc/ -type f -name '*.h'); +do + mkdir -p "$INCLUDE_DIR/$(dirname $h)" + cp -f "$h" "$INCLUDE_DIR/$h" +done + +cd /var/tmp/dist +[% c('tar', { + tar_src => [ project ], + tar_args => '-czf ' _ dest_dir _ '/' _ c('filename'), + }) %] diff --git a/projects/webrtc/config b/projects/webrtc/config new file mode 100644 index 0000000..52618c2 --- /dev/null +++ b/projects/webrtc/config @@ -0,0 +1,115 @@ +# vim: filetype=yaml sw=2 +version: '[% c("var/webrtc_tag") %]' +remote_docker: 1 +filename: 'webrtc-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.gz' + +var: + webrtc_tag: f33698296719f956497d2dbff81b5080864a8804 + +input_files: + - project: docker-image + - project: webrtc + pkg_type: fetch_sources + - project: depot_tools + name: depot_tools + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + - project: binutils + name: binutils + enable: '[% c("var/linux") %]' + +targets: + linux: + var: + webrtc: + os: linux + linux-i686: + var: + dockerbuild: "[% pc('docker-image', 'pre') %]" + sort_deps: 0 + arch_deps: + - default-jre-headless + - default-jdk + - lib32asound2-dev + - lib64expat1 + - libexpat1-dev:i386 + - libudev-dev:i386 + - libx11-dev:i386 + - libxext-dev:i386 + - libxrandr-dev:i386 + - lib32stdc++6 + - hardening-wrapper + webrtc: + arch: ia32 + lib_arch: 386 + lib_path: lib/libwebrtc-linux-386-magic.a + linux-x86_64: + var: + arch_deps: + - default-jre-headless + - default-jdk + - libasound2-dev + - libexpat1-dev + - libudev-dev + - libx11-dev + - libxext-dev + - libxrandr-dev + webrtc: + arch: x64 + lib_arch: amd64 + lib_path: lib/libwebrtc-linux-amd64-magic.a + +steps: + fetch_sources: + remote_docker: 0 + filename: 'webrtc-sources-[% c("var/webrtc_tag") %].tar.gz' + fetch_sources: | + #!/bin/bash + [% c("var/set_default_env") -%] + # WebRTC is special, having its own build system that brings in lots of Chromium dependencies. + # https://webrtc.org/native-code/development/ + tar xf [% c('input_files_by_name/depot_tools') %] + export PATH="$rootdir/depot_tools:$PATH" + # GYP_CROSSCOMPILE=1 and GYP_DEFINES="use_x11=0" prevent probing for certain dependencies. + # Use --no-history because the whole checkout with history is about 12 GB. + export GYP_CROSSCOMPILE=1 + export GYP_DEFINES="use_x11=0" + # JAVA_HOME is needed in a hook for libjingle. The readlink line tries to find the current JRE. + # default-java comes from the package default-jdk-headless. + export JAVA_HOME=/usr/lib/jvm/default-java + clone_dir='[% c("basedir") %]/gclient/webrtc' + mkdir -p "$clone_dir" + cd "$clone_dir" + if [ ! -d "src" ]; + then + # "fetch" is part of depot_tools. + #fetch --nohooks --no-history webrtc + # FIXME: To avoid the unconditional `gclient sync` in the call to fetch, + # we inline the result of a `fetch --dry-run` + gclient root + gclient config --spec 'solutions = [ + { + "managed": False, + "name": "src", + "url": "https://chromium.googlesource.com/external/webrtc.git", + "custom_deps": {}, + "deps_file": "DEPS", + "safesync_url": "", + }, + ] + ' + gclient sync --nohooks --no-history --with_branch_heads -r [% c("var/webrtc_tag") %] + cd src + git submodule foreach 'git config -f $toplevel/.git/config submodule.$name.ignore all' + git config --add remote.origin.fetch '+refs/tags/*:refs/tags/*' + git config diff.ignoreSubmodules all + cd .. + fi + # "gclient" is part of depot_tools. This download takes a long time the first time. + gclient sync --no-history --with_branch_heads -r [% c("var/webrtc_tag") %] + cd .. + tar --exclude .git -czf [% dest_dir _ '/' _ c('filename') %] webrtc + input_files: + - project: depot_tools + name: depot_tools + pkg_type: build diff --git a/rbm.conf b/rbm.conf index e5c69b0..e206193 100644 --- a/rbm.conf +++ b/rbm.conf @@ -105,6 +105,7 @@ targets: linux-x86_64: arch: x86_64 var: + linux-x86_64: 1 osname: linux-x86_64 deps: - build-essential @@ -115,9 +116,12 @@ targets: - libtool - zip - unzip + # We only build snowflake on linux-x86_64 for now + snowflake: 1 linux-i686: arch: i686 var: + linux-i686: 1 osname: linux-i686 configure_opt: '--host=i686-linux-gnu CFLAGS=-m32 CXXFLAGS=-m32 LDFLAGS=-m32' pre_pkginst: dpkg --add-architecture i386

1 0

[tor-browser-build/master] Update firefox, http-e, openssl, tor, torbutton versions
by boklm＠torproject.org 10 Mar '17

10 Mar '17

commit e040f4434ea48ad7be41d2a22f19dc58a3ac8f5b Author: Nicolas Vigier <boklm(a)torproject.org> Date: Fri Mar 10 11:44:33 2017 +0100 Update firefox, http-e, openssl, tor, torbutton versions --- projects/firefox-langpacks/config | 2 +- projects/firefox/config | 2 +- projects/https-everywhere/config | 2 +- projects/openssl/config | 10 ++-------- projects/tor/config | 2 +- projects/torbutton/config | 2 +- 6 files changed, 7 insertions(+), 13 deletions(-) diff --git a/projects/firefox-langpacks/config b/projects/firefox-langpacks/config index a645b92..dd96dcd 100644 --- a/projects/firefox-langpacks/config +++ b/projects/firefox-langpacks/config @@ -4,7 +4,7 @@ filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/buil var: ff_version: '[% pc("firefox", "var/firefox_version") %]' - ff_build: build1 + ff_build: build2 ff_arch: linux-i686 input_filename: 'dl-langpack-[% c("var/ff_arch") %]-[% c("version") %]' diff --git a/projects/firefox/config b/projects/firefox/config index 96be619..f7e89e5 100644 --- a/projects/firefox/config +++ b/projects/firefox/config @@ -8,7 +8,7 @@ gpg_keyring: torbutton.gpg remote_docker: 1 var: - firefox_version: 45.7.0esr + firefox_version: 45.8.0esr torbrowser_branch: 7.0 torbrowser_update_channel: alpha copyright_year: '[% exec("git show -s --format=%ci").remove("-.*") %]' diff --git a/projects/https-everywhere/config b/projects/https-everywhere/config index 557473d..17f9f82 100644 --- a/projects/https-everywhere/config +++ b/projects/https-everywhere/config @@ -1,5 +1,5 @@ # vim: filetype=yaml sw=2 -version: 5.2.9 +version: 5.2.11 git_url: https://git.torproject.org/https-everywhere.git git_hash: '[% c("version") %]' git_submodule: 1 diff --git a/projects/openssl/config b/projects/openssl/config index 637f3d1..8490b0b 100644 --- a/projects/openssl/config +++ b/projects/openssl/config @@ -1,5 +1,5 @@ # vim: filetype=yaml sw=2 -version: 1.0.2j +version: 1.0.2k filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.gz' remote_docker: 1 @@ -24,15 +24,9 @@ targets: pre_pkginst: dpkg --add-architecture i386 compiler: macosx-toolchain-gcc - nightly: - version: 1.0.2k - input_files: - project: docker-image - name: '[% c("var/compiler") %]' project: '[% c("var/compiler") %]' - URL: 'https://www.openssl.org/source/openssl-[% c("version") %].tar.gz' - sha256sum: e7aff292be21c259c6af26469c7a9b3ba26e9abaaffd325e3dccc9785256c431 - targets: - nightly: - sha256sum: 6b3977c61f2aedf0f96367dcfb5c6e578cf37e7b8d913b4ecb6643c3cb88d8c0 + sha256sum: 6b3977c61f2aedf0f96367dcfb5c6e578cf37e7b8d913b4ecb6643c3cb88d8c0 diff --git a/projects/tor/config b/projects/tor/config index 2fbbf24..55637dc 100644 --- a/projects/tor/config +++ b/projects/tor/config @@ -1,6 +1,6 @@ # vim: filetype=yaml sw=2 filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %]' -version: 0.3.0.2-alpha +version: 0.3.0.4-rc git_hash: 'tor-[% c("version") %]' git_url: https://git.torproject.org/tor.git gpg_keyring: tor.gpg diff --git a/projects/torbutton/config b/projects/torbutton/config index 5ff51f8..fdc9208 100644 --- a/projects/torbutton/config +++ b/projects/torbutton/config @@ -1,5 +1,5 @@ # vim: filetype=yaml sw=2 -version: 1.9.7 +version: 1.9.7.1 git_url: https://git.torproject.org/torbutton.git git_hash: '[% c("version") %]' gpg_keyring: torbutton.gpg

1 0

[tor-browser/tor-browser-45.8.0esr-6.5-2] fixup! Bug 13900: Remove 3rd party HTTP auth tokens.
by gk＠torproject.org 08 Mar '17

08 Mar '17

commit eb8f7fd0910e831d96dfdef4782e23c6c389e844 Author: Arthur Edelstein <arthuredelstein(a)gmail.com> Date: Fri Mar 3 21:47:20 2017 -0800 fixup! Bug 13900: Remove 3rd party HTTP auth tokens. The HostPartIsTheSame() is too strict about deciding what is first party. Instead we use IsThirdPartyURI() to check if the request is from the first party. Specifically, it considers an XHR to api.twitter.com from a twitter.com top-level page to be first-party. This fixes bug 21555 and 16450. --- netwerk/protocol/http/nsHttpChannel.cpp | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/netwerk/protocol/http/nsHttpChannel.cpp b/netwerk/protocol/http/nsHttpChannel.cpp index 3a9a126..e571394 100644 --- a/netwerk/protocol/http/nsHttpChannel.cpp +++ b/netwerk/protocol/http/nsHttpChannel.cpp @@ -6558,8 +6558,12 @@ nsHttpChannel::RemoveAuthorizationHeaderIfAppropriate() nsresult rv = thirdPartySvc->GetFirstPartyURIFromChannel(this, false, getter_AddRefs(firstPartyURI)); if (NS_SUCCEEDED(rv) && firstPartyURI) { - isAuthAllowed = (mURI == firstPartyURI) - || HostPartIsTheSame(firstPartyURI); + bool isThirdParty; + rv = thirdPartySvc->IsThirdPartyURI(firstPartyURI, mURI, &isThirdParty); + if (NS_SUCCEEDED(rv)) { + // Prevent third-party requests from sending auth tokens. + isAuthAllowed = !isThirdParty; + } } else { // We failed to get the first party URI. Check the document URI so // that we can allow authentication if the request originates from the

1 0

[tor-browser/tor-browser-45.8.0esr-7.0-1] fixup! Bug 13900: Remove 3rd party HTTP auth tokens.
by gk＠torproject.org 08 Mar '17

08 Mar '17

commit 340377b882408b5de24ff69bcd76e774d9844e4c Author: Arthur Edelstein <arthuredelstein(a)gmail.com> Date: Fri Mar 3 21:47:20 2017 -0800 fixup! Bug 13900: Remove 3rd party HTTP auth tokens. The HostPartIsTheSame() is too strict about deciding what is first party. Instead we use IsThirdPartyURI() to check if the request is from the first party. Specifically, it considers an XHR to api.twitter.com from a twitter.com top-level page to be first-party. This fixes bug 21555 and 16450. --- netwerk/protocol/http/nsHttpChannel.cpp | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/netwerk/protocol/http/nsHttpChannel.cpp b/netwerk/protocol/http/nsHttpChannel.cpp index 3a9a126..e571394 100644 --- a/netwerk/protocol/http/nsHttpChannel.cpp +++ b/netwerk/protocol/http/nsHttpChannel.cpp @@ -6558,8 +6558,12 @@ nsHttpChannel::RemoveAuthorizationHeaderIfAppropriate() nsresult rv = thirdPartySvc->GetFirstPartyURIFromChannel(this, false, getter_AddRefs(firstPartyURI)); if (NS_SUCCEEDED(rv) && firstPartyURI) { - isAuthAllowed = (mURI == firstPartyURI) - || HostPartIsTheSame(firstPartyURI); + bool isThirdParty; + rv = thirdPartySvc->IsThirdPartyURI(firstPartyURI, mURI, &isThirdParty); + if (NS_SUCCEEDED(rv)) { + // Prevent third-party requests from sending auth tokens. + isAuthAllowed = !isThirdParty; + } } else { // We failed to get the first party URI. Check the document URI so // that we can allow authentication if the request originates from the

1 0