Re: [tbb-dev] [tor-dev] RFC: porting torbrowser (was: Re: GNU Guix and Tor Browser Packaging)
Hi,
It seems as if tbb-dev@lists.torproject.org is the list which would be more appropriate. If the 7 days without a reaction are simply due to the holidays in some countries, it's my mistake. If you need internal discussion about this to respond appropriately, let me know that you are reviewing this message at all. I have no expectation for "neoliberal optimized" reply times.
Thanks.
ng0 transcribed 7.9K bytes:
Hi!
ng0:
It's been taking a while but here come some comments to your mail:
Sounds good.
FWIW: I don't think "Sync" is working if you use Tor Browser as-is.
Because `about:license` is a page within the Firefox part of Tor Browser (originally) meant to cover exactly that part. There are other parts, like tor or NoScript, that come with their own licenses.
That said there is a bug on our side to update that page (https://trac.torproject.org/projects/tor/ticket/14936), and we could think about making it the central point where we show all the licenses the users get with the end product.
It seems Luke looked at the wrong file. If you are looking for preferences we set 000-tor-browser.js is a good starting point:
https://gitweb.torproject.org/tor-browser.git/tree/browser/app/profile/000-t...
Otherwise EME related things are disabled in our .mozconfig files in our tor-browser repo.
The changes you want to make and that are outlined above should not be an issue for calling the thing you are working on "Tor Browser". It would be nice if you could have a canonical URL where one can easily see all the changes you plan to apply (and are already applied). If there is such a place already could you point this list to it?
[snip]
Georg
Hi,
thanks for your reply.
Georg Koppen transcribed 7.4K bytes:
So it's basically a broken feature (in tor browser) and should be removed anyway? I'm busy on other fronts currently and had no time to look into this again. I hope to pick up work on this again in June or July depending on how the things which keep me occupied work out.
Sounds good.
Thanks, I'll compare with this page.
That's good to know, thank you.
The place where the package ends up finally would be the Guix master repository, the shared source of Guix and all system services and packages: https://git.savannah.gnu.org/cgit/guix.git/log/
I will pass you the link to my work repository as soon as there as something in a working state in the tor-browser branch in there.
Currently this non-working feature branch exists: http://qs3zumwfci4tntnd.onion/ng0/guix/commits/package/browser/torbrowser https://notabug.org/ng0/guix/commits/package/browser/torbrowser
which is due to be converted to an mirror-only repository of:
https://git.pragmatique.xyz/ng0-guix/log.html
I'll keep you in the loop.
[snip]
Georg
While I layed out the theory for working on a tor browser port last year, I was asked why it wouldn't make sense to use icecat as a base. I think with all due respects it takes to keep up with firefox/mozilla and their speed and teamsize, they have been occasionally slow in the past. It would be interesting to see how an alternative package of torbrowser based on icecat works out, but so far I leave that up to my developers repository as a would-be-could-be case to play around with (our icecat receives fast and regular patches from upstream firefox esr, that's why it could be an option). Your permission just was for the attempted replica of the build process of tor browser and I only focus on that officially for Guix inclusion.
ng0
Hi,
(top-posting and keeping the old email as reference in complete quote) I have now started working on a package for Guix. While I've had some exposure to the packaging of the ESR or Firefox and also some old code base fork of Firefox (Palemoon), I'm not sure if the problem I experience is something I can report or ask for at your side or elsewhere.
Building fails in the part of the build phase where interaction with quitter happens (for a lack of more insight into the actual codebase).
My package definition is not yet published (and early stages), I'll publish a public preview tomorrow.
The build error is curious for me, because Firefox ESR (not in upstream Guix) succeeds with builds. Icecat with patches from ESR applied in Guix master builds fine. I assume what I experience is Tor Browser specific.
I can send in a full build log in about 12 hours, for now there's this:
processing /tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/testing/marionette/jar.mn make[4]: Leaving directory '/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/ff/testing/marionette' make[4]: Entering directory '/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/ff/tools/quitter' mkdir -p '../../dist/xpi-stage/quitter/chrome/' /tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/ff/_virtualenv/bin/python -m mozbuild.action.jar_maker -d ../../dist/xpi-stage/quitter -t /tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout -f symlink -e --relativesrcdir=tools/quitter -c /tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/tools/quitter/en-US -DNDEBUG=1 -DTRIMMED=1 -DA11Y_LOG=1 -DACCESSIBILITY=1 -DATK_MAJOR_VERSION=2 -DATK_MINOR_VERSION=26 -DATK_REV_VERSION=1 -DBUILD_CTYPES=1 -DCROSS_COMPILE='' -DD_INO=d_ino -DENABLE_INTL_API=1 -DENABLE_MARIONETTE=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DEXPOSE_INTL_API=1 -DFIREFOX_VERSION=52.6.0 -DFORCE_PR_LOG=1 -DFUNCPROTO=15 -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_26 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_26 -DGL_PROVIDER_GLX=1 -DHAVE_64BIT_BUILD=1 -DHAVE_ALLOCA_H=1 -DHAVE_BYTESWAP_H=1 -DHAVE_CLOCK_MONOTONIC=1 -DHAVE_CPUID_H=1 -DHAVE_DIRENT_H=1 -DHAVE_DLADDR=1 -DHAVE_DLOPEN=1 -DHAVE_FONTCONFIG_FCFREETYPE_H=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_GLYPHSLOT_EMBOLDEN=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -DHAVE_GETOPT_H=1 -DHAVE_GMTIME_R=1 -DHAVE_I18N_LC_MESSAGES=1 -DHAVE_INTTYPES_H=1 -DHAVE_LANGINFO_CODESET=1 -DHAVE_LCHOWN=1 -DHAVE_LIBVPX=1 -DHAVE_LIBXSS=1 -DHAVE_LINUX_IF_ADDR_H=1 -DHAVE_LINUX_PERF_EVENT_H=1 -DHAVE_LINUX_QUOTA_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -DHAVE_LOCALECONV=1 -DHAVE_LOCALTIME_R=1 -DHAVE_LSTAT64=1 -DHAVE_MALLINFO=1 -DHAVE_MALLOC_H=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MEMALIGN=1 -DHAVE_MEMMEM=1 -DHAVE_NETINET_IN_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_POSIX_FADVISE=1 -DHAVE_POSIX_FALLOCATE=1 -DHAVE_POSIX_MEMALIGN=1 -DHAVE_PTHREAD_H=1 -DHAVE_RES_NINIT=1 -DHAVE_SETPRIORITY=1 -DHAVE_STAT64=1 -DHAVE_STDINT_H=1 -DHAVE_STRERROR=1 -DHAVE_STRNDUP=1 -DHAVE_SYSCALL=1 -DHAVE_SYS_QUEUE_H=1 -DHAVE_SYS_QUOTA_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_THREAD_TLS_KEYWORD=1 -DHAVE_TRUNCATE64=1 -DHAVE_UNISTD_H=1 -DHAVE_VALLOC=1 -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE__UNWIND_BACKTRACE=1 -DHAVE___CXA_DEMANGLE=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DMALLOC_H='<malloc.h>' -DMALLOC_USABLE_SIZE_CONST_PTR='' -DMOZILLA_UAVERSION='"52.0"' -DMOZILLA_VERSION='"52.6.0"' -DMOZILLA_VERSION_U=52.6.0 -DMOZ_ACCESSIBILITY_ATK=1 -DMOZ_ADDON_SIGNING=1 -DMOZ_APP_UA_NAME='""' -DMOZ_APP_UA_VERSION='"52.6.0"' -DMOZ_B2G_OS_NAME='""' -DMOZ_B2G_VERSION='"1.0.0"' -DMOZ_BUILD_APP=browser -DMOZ_BUNDLED_FONTS=1 -DMOZ_CONTENT_SANDBOX=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DMOZ_DATA_REPORTING=1 -DMOZ_DISTRIBUTION_ID='"org.mozilla"' -DMOZ_DLL_SUFFIX='".so"' -DMOZ_ENABLE_DBUS=1 -DMOZ_ENABLE_GIO=1 -DMOZ_ENABLE_PROFILER_SPS=1 -DMOZ_ENABLE_SIGNMAR=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_ENABLE_STARTUP_NOTIFICATION=1 -DMOZ_ENABLE_XREMOTE=1 -DMOZ_FEEDS=1 -DMOZ_FFMPEG=1 -DMOZ_FFVPX=1 -DMOZ_FMP4=1 -DMOZ_GAMEPAD=1 -DMOZ_GLUE_IN_PROGRAM=1 -DMOZ_GMP_SANDBOX=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_JSDOWNLOADS=1 -DMOZ_LIBAV_FFT=1 -DMOZ_LOGGING=1 -DMOZ_MACBUNDLE_ID=org.mozilla.torbrowser -DMOZ_MEMORY=1 -DMOZ_MEMORY_LINUX=1 -DMOZ_OFFICIAL_BRANDING=1 -DMOZ_PERMISSIONS=1 -DMOZ_PHOENIX=1 -DMOZ_PLACES=1 -DMOZ_PULSEAUDIO=1 -DMOZ_SAMPLE_TYPE_FLOAT32=1 -DMOZ_SANDBOX=1 -DMOZ_SECUREELEMENT=1 -DMOZ_SERVICES_HEALTHREPORT=1 -DMOZ_STATIC_JS=1 -DMOZ_SYSTEM_SQLITE=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_USER_DIR='".mozilla"' -DMOZ_VORBIS=1 -DMOZ_VPX_NO_MEM_REPORTING=1 -DMOZ_WEBGL_CONFORMANT=1 -DMOZ_WEBM_ENCODER=1 -DMOZ_WEBSPEECH=1 -DMOZ_WEBSPEECH_TEST_BACKEND=1 -DMOZ_WIDGET_GTK=2 -DMOZ_X11=1 -DMOZ_XUL=1 -DNO_NSPR_10_SUPPORT=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DRELEASE_OR_BETA=1 -DSPIDERMONKEY_PROMISE=1 -DSTATIC_JS_API=1 -DSTDC_HEADERS=1 -DTARGET_XPCOM_ABI='"x86_64-gcc3"' -DTOR_BROWSER_VERSION='"7.5"' -DUSE_SKIA=1 -DUSE_SKIA_GPU=1 -DVA_COPY=va_copy -DXP_LINUX=1 -DXP_UNIX=1 -D_REENTRANT=1 -DXPI_NAME=quitter -DAB_CD=en-US /tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/tools/quitter/jar.mn processing /tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/tools/quitter/jar.mn Traceback (most recent call last): File "/gnu/store/j4vj7h3wyb532g2j0axzjj43z2a0dg81-python-2.7.14/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/gnu/store/j4vj7h3wyb532g2j0axzjj43z2a0dg81-python-2.7.14/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/python/mozbuild/mozbuild/action/jar_maker.py", line 17, in <module> sys.exit(main(sys.argv[1:])) File "/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/python/mozbuild/mozbuild/action/jar_maker.py", line 13, in main return mozbuild.jar.main(args) File "/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/python/mozbuild/mozbuild/jar.py", line 597, in main jm.makeJar(infile, options.d) File "/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/python/mozbuild/mozbuild/jar.py", line 333, in makeJar self.processJarSection(info, jardir) File "/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/python/mozbuild/mozbuild/jar.py", line 391, in processJarSection jarinfo.chrome_manifests) File "/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/python/mozbuild/mozbuild/jar.py", line 281, in finalizeJar chromebasepath.format(chromebase), register) File "/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/python/mozbuild/mozbuild/jar.py", line 307, in updateManifest addEntriesToListFile(manifestPath, myregister.iterkeys()) File "/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/python/mozbuild/mozbuild/action/buildlist.py", line 36, in addEntriesToListFile with open(listFile, 'wb') as f: IOError: [Errno 13] Permission denied: '../../dist/xpi-stage/quitter/chrome.manifest' make[4]: *** [/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/config/rules.mk:1187: libs] Error 1 make[4]: Leaving directory '/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/ff/tools/quitter' make[3]: *** [/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/config/recurse.mk:79: tools/quitter/libs] Error 2 make[3]: Leaving directory '/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/ff' make[2]: *** [/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/config/recurse.mk:33: libs] Error 2 make[2]: Leaving directory '/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/ff' make[1]: *** [/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/config/rules.mk:523: default] Error 2 make[1]: Leaving directory '/tmp/guix-build-tor-browser-7.5.drv-0/tor-browser-7.5-checkout/ff' make: *** [client.mk:415: realbuild] Error 2 phase `build' failed after 2129.5 seconds builder for `/gnu/store/r66p868620dpvzmfymclz50qfdsvphyv-tor-browser-7.5.drv' failed with exit code 1 @ build-failed /gnu/store/r66p868620dpvzmfymclz50qfdsvphyv-tor-browser-7.5.drv - 1 builder for `/gnu/store/r66p868620dpvzmfymclz50qfdsvphyv-tor-browser-7.5.drv' failed with exit code 1 guix build: error: build failed: build of `/gnu/store/r66p868620dpvzmfymclz50qfdsvphyv-tor-browser-7.5.drv' failed
ng0 transcribed 9.7K bytes:
Georg Koppen transcribed 11K bytes:
Sorry, I just learned that we don't keep build logs on disk for failed builds.
The build is run with Guix, the rather hacky package definition so far is this commit: https://c.n0.is/ng0/ports/commit/?id=320fa06ca3af881ba7c1b26514f04eb69d56f0a... For reader convenience I'll attach the guile module to this email. It is in early stages and I was mainly focused on getting the configure and build stages running, the rest is inherited from an firefox-esr package and needs to be adjusted once I get there.
As you can read in the file, I use the following checkout url and commit/tag: + (uri (git-reference + (url "https://git.torproject.org/tor-browser.git") + (commit "tor-browser-52.6.0esr-7.5-2-build1")))
What .mozconfig file are you using (if any)?
The .mozconfig is programmatically assembled in the package definition. It boils down to these lines, ";;" is a commented line:
+ ". $topsrcdir/browser/config/mozconfig\n" + ;;"mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-@CONFIG_GUESS@\n" + "mk_add_options MOZ_OBJDIR=" (getenv "BUILD_OBJ_DIR") "\n" + "mk_add_options MOZ_APP_DISPLAYNAME="Tor Browser"\n" + "mk_add_options XARGS=" (which "xargs") "\n" + "mk_add_options MOZILLA_OFFICIAL=1\n" + "mk_add_options BUILD_OFFICIAL=1\n" + "ac_add_options --enable-extensions=default\n" + ;;"ac_add_options --enable-optimize\n" + ;;#ac_add_options --disable-optimize + "ac_add_options --enable-official-branding\n" + ;; # Let's support GTK2 for ESR52 + "ac_add_options --enable-default-toolkit=cairo-gtk2\n" + ;; ---- poncho didn't have those 2: + ;;"ac_add_options --enable-signmar\n" + ;;"ac_add_options --enable-verify-mar\n" + ;; ---- + "ac_add_options --disable-strip\n" + "ac_add_options --disable-install-strip\n" + "ac_add_options --disable-tests\n" + ;; "ac_add_options --disable-debug\n" + ;; ac_add_options --disable-maintenance-service + ;; ac_add_options --disable-crashreporter + "ac_add_options --disable-webrtc\n" + "ac_add_options --disable-eme\n" + + + "ac_add_options --with-app-name=torbrowser\n" + "ac_add_options --with-app-basename=torbrowser\n" + "ac_add_options --with-tor-browser-version=7.5\n" + "ac_add_options --disable-tor-browser-update\n" + ;;"ac_add_options --enable-tor-browser-data-outside-app-dir\n" + + ;; "ac_add_options --enable-default-toolkit=cairo-gtk3\n" + "ac_add_options --enable-gio\n" + "ac_add_options --enable-startup-notification\n" + "ac_add_options --enable-pulseaudio\n" + + "ac_add_options --disable-gconf\n" + "ac_add_options --disable-gnomeui\n" + + "ac_add_options --disable-maintenance-service\n" + "ac_add_options --disable-updater\n" + "ac_add_options --disable-necko-wifi\n" + ;;"ac_add_options --enable-safe-browsing\n" + "ac_add_options --disable-crashreporter\n" + + ;; Until 55 ESR we can disable this: + "ac_add_options --disable-rust\n" + + ;; Building with debugging symbols takes ~5GiB, so + ;; disable it. + "ac_add_options --disable-debug\n" + "ac_add_options --disable-debug-symbols\n" + ;;"ac_add_options --enable-pie\n" + + ;; Avoid bundled libraries. + "ac_add_options --with-system-zlib\n" + "ac_add_options --with-system-bz2\n" + "ac_add_options --with-system-jpeg\n" ; must be libjpeg-turbo + "ac_add_options --with-system-libevent\n" + "ac_add_options --with-system-libvpx\n" + "ac_add_options --with-system-icu\n" + "ac_add_options --with-system-nspr\n" + "ac_add_options --with-system-nss\n" + ;;"--with-system-harfbuzz" + ;;"--with-system-graphite2" + "ac_add_options --enable-system-pixman\n" + "ac_add_options --enable-system-cairo\n" + "ac_add_options --enable-system-ffi\n" + "ac_add_options --enable-system-hunspell\n" + "ac_add_options --enable-system-sqlite\n"))))
What are your commands to start the build?
The configure phase is: make --file=client.mk configure which is followed by the build phase: make -f client.mk realbuild
Georg
A complete build log up to the point where the failure happens can be found at https://d.n0.is/pub/tor-browser-buildlog-2018-03-09.txt (19MB).
I'm happy for any pointers in the right direction.
Thanks for your help.
ng0 transcribed 35K bytes:
On Fri, Mar 09, 2018 at 02:19:44PM +0000, ng0 wrote:
[...]
[...]
[...]
It's surprising creating '../../dist/xpi-stage/quitter/chrome/' is successful, but creating '../../dist/xpi-stage/quitter/chrome.manifest' fails. Do you have an apparmor or selinux policy enforced? I don't see anything immediately obvious that would specifically cause EPERM at this place during the build.
[...]
Is this included for a reason? Do you include additional extensions using this?
[...]
These are duplicated, and included/enabled, below.
[...]
We don't change these in the official build.
[...]
I noticed you don't enable bundled fonts. Is that on purpose?
Overall, this is exciting. I hope this is successful.
Hi,
Matthew Finkel transcribed 5.2K bytes:
There's no apparmor or SELinux present in our buildenvironment. I already had to fix up permissions for .mozconfig, but I figured it isn't connected to the later write error.
No, I just copied what poncho does: https://github.com/MeisterP/torbrowser-overlay/blob/master/www-client/torbro... poncho inherits this from the Gentoo Firefox ebuilds, but as we have no additional extentions here and do not expect them to appear in the near future, I can leave it out. Thanks for pointing it out.
Thanks, I'll check again. Actually ";;" is a comment in Guile and nothing that ends up in the file later on.
I know I'm missing one or two more seds Gentoo applies to change the profile folder, but: The exectuable ends up as "firefox" in that case (not changing the appname and the app-basename, right? And the folder for the profile in the $home will be firefox, not torbrowser?
Or could you explain this and why you don't change them?
[...]
I noticed you don't enable bundled fonts. Is that on purpose?
It's not on purpose. What do I search for to enable them? My idea was to get the fonts and other data from the release tarball, but that's just my first path. Fixes applied could lead to discover that the data from tarball approach poncho does for Gentoo is unnecessary.
Overall, this is exciting. I hope this is successful.
Thanks!
-
Georg Koppen -
Matthew Finkel -
ng0 -
ng0