[tbb-dev] Very very slow tor-browser-build build; catastrophic recursion in rbm?

David Fifield david at bamsoftware.com
Wed Feb 12 18:30:13 UTC 2020 

On Wed, Feb 12, 2020 at 02:40:38PM +0100, Nicolas Vigier wrote:
> > The above NYTProf output is not from a full build. It's from modifying
> > projects/snowflake/config only and then building snowflake for only one
> > target; still it took almost an hour.
> > 	perl -d:NYTProf rbm/rbm build snowflake --target testbuild --target torbrowser-linux-x86_64
> > 	nytprofhtml
> 
> Was it a new build of snowflake where some of the dependencies had to be
> built, or a build where only snowflake had to be built?
> 
> In the first case, I think part of the reason why it takes time is that
> snowflake has many small dependencies, and each one is built in a
> separate container, so some time is spent extracting/removing chroots.
> 
> If only snowflake and no other dependencies was built, then I don't know
> why it took almost one hour. In my case, adding a blank line to
> projects/snowflake/build and rebuilding it takes around 8 minutes.

It was a build of snowflake only, none of the dependencies. In other
words, I had already done a complete "make testbuild" (this is the one
that took all weekend), then I modified projects/snowflake/config, then
I did "rbm build snowflake" as shown above.

Overnight I did a "make testbuild" to build snowflake for the rest of
the targets, and it took 9 hours:
real    544m23.414s
user    482m3.219s
sys     73m29.899s
I'll put the top-level stdout at the end of this message; I'm pretty
sure it shows that only snowflake and tor-browser neede to be rebuilt,
and the android targets didn't need to be rebuilt at all because they
don't have snowflake as a dependency.

> > Notice how there are three loops of RBM::process_template before doing
> > an RBM::config_p that calls back into further RBM::process_template.
> 
> Yes, there is a lot of recursive calls.
> 
> For example the filename in projects/tor-browser/config is defined like
> this:
> 
>   filename: 'tor-browser-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %]'
> 
> computing this filename will require computing var/build_id, which is
> defined like this:
> 
> so this require recursively computing filenames of all other projects,
> which requires computing sha256sum of all input files.

Thank you for the explanation. It makes a lot of sense. I like this
feature of rbm a lot, that I can feel confident the right things will be
rebuilt when I change something.

I suppose Go projects are especially troublesome. In my build over the
weekend I was testing a change that added about 10 new transitive
dependencies to snowflake:
https://lists.torproject.org/pipermail/anti-censorship-team/2020-February/000069.html
https://gitweb.torproject.org/user/dcf/tor-browser-build.git/commit/?h=snowflake-turbotunnel&id=a0c7ffa70f09ca4d86e18a93483c1a378c3067b4
In my build last night (the one whose stdout is below), I removed all
those dependencies again and added only 1 dependency:
https://gitweb.torproject.org/user/dcf/tor-browser-build.git/commit/?h=snowflake-turbotunnel&id=b46fd762e0866fe01707e469eb997040f37c16a2

> I opened a ticket for this:
> https://trac.torproject.org/projects/tor/ticket/33283
> 
> I think this patch should avoid doing multiple `git checkout` and `git
> log` on the same projects:
> https://gitweb.torproject.org/user/boklm/rbm.git/commit/?h=bug_33283&id=ce6b9136fc9e006536b34844946d67aa92cc0c21

Thanks, I'll try the patch after I finish packaging the build from last
night.

----

Output of last night's "time make testbuild" with only snowflake
needing to be rebuilt. I had already manually built snowflake for
linux-x86_64. It was commit 5eff0f3ba406eb7d8cd90711edab7de7d7bda4fa in
https://git.torproject.org/user/dcf/snowflake.git.

```
# time make testbuild
git submodule update --init         
./rbm/rbm build release --target testbuild --target torbrowser-all
Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-android-armv7-143b27
Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-android-x86-b005b1
Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-android-x86_64-5512f4
Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-android-aarch64-0dec52
Building project tor-browser - tor-browser-9.5a4-linux-x86_64-7ef7ab
Using file /root/tor-browser-build/out/container-image/container-image_wheezy-amd64-a32817165f00.tar.gz
Using file /root/tor-browser-build/projects/tor-browser/run_scripts
Using directory /root/tor-browser-build/out/firefox/firefox-78e8d897cd2e-linux-x86_64-46c98c
Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-linux-x86_64-5408dc
Using file /root/tor-browser-build/out/https-everywhere/https-everywhere-2019.11.7-b894fe.xpi
Using file /root/tor-browser-build/out/fonts/fonts-720e34851382-074f0d.tar.gz
Using file /root/tor-browser-build/out/obfs4/obfs4-0.0.11-linux-x86_64-62b5ad.tar.gz
Using file /root/tor-browser-build/out/snowflake/snowflake-5eff0f3ba406-linux-x86_64-fa4d5c.tar.gz
Using directory /root/tor-browser-build/projects/tor-browser/Bundle-Data
Using file /root/tor-browser-build/out/tor-browser/noscript_security_suite-11.0.11-an+fx.xpi
Using file /root/tor-browser-build/projects/tor-browser/RelativeLink/start-tor-browser.desktop
Using file /root/tor-browser-build/projects/tor-browser/RelativeLink/execdesktop
Using file /root/tor-browser-build/projects/tor-browser/gtk3-settings.ini
Build log: /root/tor-browser-build/logs/tor-browser-linux-x86_64.log
Finished build of project tor-browser - tor-browser-9.5a4-linux-x86_64-7ef7ab
Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-linux-x86_64-7ef7ab
Building project tor-browser - tor-browser-9.5a4-linux-i686-56bb5f
Using file /root/tor-browser-build/out/container-image/container-image_wheezy-amd64-ae74865f5ad7.tar.gz
Using file /root/tor-browser-build/projects/tor-browser/run_scripts
Using directory /root/tor-browser-build/out/firefox/firefox-78e8d897cd2e-linux-i686-78f104
Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-linux-i686-8cabbd
Using file /root/tor-browser-build/out/https-everywhere/https-everywhere-2019.11.7-b894fe.xpi
Using file /root/tor-browser-build/out/fonts/fonts-720e34851382-074f0d.tar.gz
Using file /root/tor-browser-build/out/obfs4/obfs4-0.0.11-linux-i686-0d2a0b.tar.gz
Building project snowflake - snowflake-5eff0f3ba406-linux-i686-172b69.tar.gz
Created /root/tor-browser-build/tmp/rbm-7RbRC/snowflake-5eff0f3ba406.tar.gz
Using file /root/tor-browser-build/out/container-image/container-image_wheezy-amd64-f2bcf2cde274.tar.gz
Using file /root/tor-browser-build/out/go/go-1.12.13-6f6dc9.tar.gz
Using file /root/tor-browser-build/out/uniuri/uniuri-8902c56451e9-linux-i686-095a9f.tar.gz
Using file /root/tor-browser-build/out/goptlib/goptlib-v1.1.0-linux-i686-43ea9b.tar.gz
Using file /root/tor-browser-build/out/pion-webrtc/pion-webrtc-77c6e3b827e4-linux-i686-02e5a3.tar.gz
Using file /root/tor-browser-build/out/quic-go/quic-go-907071221cf9-linux-i686-4b046f.tar.gz
Build log: /root/tor-browser-build/logs/snowflake-linux-i686.log
Finished build of project snowflake - snowflake-5eff0f3ba406-linux-i686-172b69.tar.gz
Using file /root/tor-browser-build/out/snowflake/snowflake-5eff0f3ba406-linux-i686-172b69.tar.gz
Using directory /root/tor-browser-build/projects/tor-browser/Bundle-Data
Using file /root/tor-browser-build/out/tor-browser/noscript_security_suite-11.0.11-an+fx.xpi
Using file /root/tor-browser-build/projects/tor-browser/RelativeLink/start-tor-browser.desktop
Using file /root/tor-browser-build/projects/tor-browser/RelativeLink/execdesktop
Using file /root/tor-browser-build/projects/tor-browser/gtk3-settings.ini
Build log: /root/tor-browser-build/logs/tor-browser-linux-i686.log
Finished build of project tor-browser - tor-browser-9.5a4-linux-i686-56bb5f
Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-linux-i686-56bb5f
Building project tor-browser - tor-browser-9.5a4-windows-i686-465936
Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-860c78a36523.tar.gz
Using file /root/tor-browser-build/projects/tor-browser/run_scripts
Using directory /root/tor-browser-build/out/firefox/firefox-78e8d897cd2e-windows-i686-89cecd
Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-windows-i686-b41491
Using file /root/tor-browser-build/out/https-everywhere/https-everywhere-2019.11.7-46dd96.xpi
Using file /root/tor-browser-build/out/fonts/fonts-720e34851382-f954ac.tar.gz
Using file /root/tor-browser-build/out/obfs4/obfs4-0.0.11-windows-i686-72bc65.tar.gz
Building project snowflake - snowflake-5eff0f3ba406-windows-i686-dcf129.tar.gz
Created /root/tor-browser-build/tmp/rbm-oAH9V/snowflake-5eff0f3ba406.tar.gz
Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-18fd98cf5daf.tar.gz
Using file /root/tor-browser-build/out/go/go-1.12.13-4c7c84.tar.gz
Using file /root/tor-browser-build/out/uniuri/uniuri-8902c56451e9-windows-i686-1a6f4a.tar.gz
Using file /root/tor-browser-build/out/goptlib/goptlib-v1.1.0-windows-i686-852b18.tar.gz
Using file /root/tor-browser-build/out/pion-webrtc/pion-webrtc-77c6e3b827e4-windows-i686-dacf47.tar.gz
Using file /root/tor-browser-build/out/quic-go/quic-go-907071221cf9-windows-i686-01524c.tar.gz
Build log: /root/tor-browser-build/logs/snowflake-windows-i686.log
Finished build of project snowflake - snowflake-5eff0f3ba406-windows-i686-dcf129.tar.gz
Using file /root/tor-browser-build/out/snowflake/snowflake-5eff0f3ba406-windows-i686-dcf129.tar.gz
Using directory /root/tor-browser-build/projects/tor-browser/Bundle-Data
Using file /root/tor-browser-build/out/tor-browser/noscript_security_suite-11.0.11-an+fx.xpi
Using file /root/tor-browser-build/out/nsis/nsis-3.04-windows-i686-6d8d58.tar.gz
Using file /root/tor-browser-build/out/tbb-windows-installer/tbb-windows-installer-src-0.5-dbf177.tar.gz
Using file /root/tor-browser-build/projects/tor-browser/pe_checksum_fix.py
Using file /root/tor-browser-build/out/tor-browser/pefile-2017.11.5.tar.gz
Build log: /root/tor-browser-build/logs/tor-browser-windows-i686.log
Finished build of project tor-browser - tor-browser-9.5a4-windows-i686-465936
Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-windows-i686-465936
Building project tor-browser - tor-browser-9.5a4-windows-x86_64-9f74bd
Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-860c78a36523.tar.gz
Using file /root/tor-browser-build/projects/tor-browser/run_scripts
Using directory /root/tor-browser-build/out/firefox/firefox-78e8d897cd2e-windows-x86_64-ffc30f
Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-windows-x86_64-79026c
Using file /root/tor-browser-build/out/https-everywhere/https-everywhere-2019.11.7-46dd96.xpi
Using file /root/tor-browser-build/out/fonts/fonts-720e34851382-f954ac.tar.gz
Using file /root/tor-browser-build/out/obfs4/obfs4-0.0.11-windows-x86_64-bb2f23.tar.gz
Building project snowflake - snowflake-5eff0f3ba406-windows-x86_64-57b6ff.tar.gz
Created /root/tor-browser-build/tmp/rbm-eOk0z/snowflake-5eff0f3ba406.tar.gz
Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-18fd98cf5daf.tar.gz
Using file /root/tor-browser-build/out/go/go-1.12.13-846325.tar.gz
Using file /root/tor-browser-build/out/uniuri/uniuri-8902c56451e9-windows-x86_64-739ef5.tar.gz
Using file /root/tor-browser-build/out/goptlib/goptlib-v1.1.0-windows-x86_64-8eaaca.tar.gz
Using file /root/tor-browser-build/out/pion-webrtc/pion-webrtc-77c6e3b827e4-windows-x86_64-629396.tar.gz
Using file /root/tor-browser-build/out/quic-go/quic-go-907071221cf9-windows-x86_64-ae697f.tar.gz
Build log: /root/tor-browser-build/logs/snowflake-windows-x86_64.log
Finished build of project snowflake - snowflake-5eff0f3ba406-windows-x86_64-57b6ff.tar.gz
Using file /root/tor-browser-build/out/snowflake/snowflake-5eff0f3ba406-windows-x86_64-57b6ff.tar.gz
Using directory /root/tor-browser-build/projects/tor-browser/Bundle-Data
Using file /root/tor-browser-build/out/tor-browser/noscript_security_suite-11.0.11-an+fx.xpi
Using file /root/tor-browser-build/out/nsis/nsis-3.04-windows-x86_64-91a7bd.tar.gz
Using file /root/tor-browser-build/out/tbb-windows-installer/tbb-windows-installer-src-0.5-dbf177.tar.gz
Using file /root/tor-browser-build/projects/tor-browser/pe_checksum_fix.py
Using file /root/tor-browser-build/out/tor-browser/pefile-2017.11.5.tar.gz
Build log: /root/tor-browser-build/logs/tor-browser-windows-x86_64.log
Finished build of project tor-browser - tor-browser-9.5a4-windows-x86_64-9f74bd
Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-windows-x86_64-9f74bd
Building project tor-browser - tor-browser-9.5a4-osx-x86_64-db1da8
Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-1d614657ea1b.tar.gz
Using file /root/tor-browser-build/projects/tor-browser/run_scripts
Using directory /root/tor-browser-build/out/firefox/firefox-78e8d897cd2e-osx-x86_64-56070b
Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-osx-x86_64-cd810e
Using file /root/tor-browser-build/out/https-everywhere/https-everywhere-2019.11.7-46dd96.xpi
Using file /root/tor-browser-build/out/fonts/fonts-720e34851382-5830a6.tar.gz
Using file /root/tor-browser-build/out/obfs4/obfs4-0.0.11-osx-x86_64-9c01d4.tar.gz
Building project snowflake - snowflake-5eff0f3ba406-osx-x86_64-f9fc60.tar.gz
Created /root/tor-browser-build/tmp/rbm-hxgYM/snowflake-5eff0f3ba406.tar.gz
Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-c1192d34d787.tar.gz
Using file /root/tor-browser-build/out/go/go-1.12.13-bbe0e3.tar.gz
Using file /root/tor-browser-build/out/uniuri/uniuri-8902c56451e9-osx-x86_64-6d72b5.tar.gz
Using file /root/tor-browser-build/out/goptlib/goptlib-v1.1.0-osx-x86_64-ab0a8a.tar.gz
Using file /root/tor-browser-build/out/pion-webrtc/pion-webrtc-77c6e3b827e4-osx-x86_64-e18e7a.tar.gz
Using file /root/tor-browser-build/out/quic-go/quic-go-907071221cf9-osx-x86_64-a6f2f4.tar.gz
Build log: /root/tor-browser-build/logs/snowflake-osx-x86_64.log
Finished build of project snowflake - snowflake-5eff0f3ba406-osx-x86_64-f9fc60.tar.gz
Using file /root/tor-browser-build/out/snowflake/snowflake-5eff0f3ba406-osx-x86_64-f9fc60.tar.gz
Using directory /root/tor-browser-build/projects/tor-browser/Bundle-Data
Using file /root/tor-browser-build/out/tor-browser/noscript_security_suite-11.0.11-an+fx.xpi
Using file /root/tor-browser-build/out/libdmg-hfsplus/libdmg-hfsplus-dfd5e5cc3dc1-62fa3e.tar.gz
Build log: /root/tor-browser-build/logs/tor-browser-osx-x86_64.log
Finished build of project tor-browser - tor-browser-9.5a4-osx-x86_64-db1da8
Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-osx-x86_64-db1da8
Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-windows-i686-d97207
Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-windows-x86_64-18debd
Build log: /root/tor-browser-build/logs/release-linux-x86_64.log
                                    
real    544m23.414s                 
user    482m3.219s                  
sys     73m29.899s                  
```

More information about the tbb-dev mailing list