August 2017 - tor-commits - lists.torproject.org

[tor-browser-build/master] Add a README.HACKING file
by boklm＠torproject.org 03 Aug '17

03 Aug '17

commit 41dc4091f04a16754b208af8edbb2e501a0a3831 Author: Nicolas Vigier <boklm(a)torproject.org> Date: Fri Aug 4 00:19:47 2017 +0200 Add a README.HACKING file Add a file to list the main things to know when making changes to the Tor Browser build. --- README | 7 ++ README.HACKING | 223 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 230 insertions(+) diff --git a/README b/README index e5fc45d..7a2a671 100644 --- a/README +++ b/README @@ -205,3 +205,10 @@ Common Build Errors You can look at the README.BUILD_ERRORS file for a list of common build errors and their solutions. + +Hacking on the Tor Browser build +-------------------------------- + +The file README.HACKING tries to list the main things to know when +making changes to the Tor Browser build. + diff --git a/README.HACKING b/README.HACKING new file mode 100644 index 0000000..2687ed1 --- /dev/null +++ b/README.HACKING @@ -0,0 +1,223 @@ +Hacking on Tor Browser Build +============================ + +This file tries to list the main things to know when making changes to +the Tor Browser build. + + +rbm documentation +----------------- + +If you go to directory rbm/doc, you can find the rbm documentation. You +can build it with 'make html' or 'make man'. + + +Using and defining options +-------------------------- + +Options can be used in templates with the following syntax: + + [% c("option_name") %] + +More details about the templates syntax can be found in +rbm/doc/rbm_templates.7 and [http://template-toolkit.org/docs/manual/index.html] + +Some options have a specific meaning to rbm. You can see their descriptions +in rbm/doc/rbm_config.7 and rbm/doc/rbm_input_files.7. + +When the option does not have a specific meaning to rbm, it is a good +idea to define it under var/ to avoid potential conflicts in option names +with a future version of rbm. + +The options can be defined in different places: + +- rbm.conf for options available to all components +- project/$project/config for options available to one component +- rbm.local.conf for user defined options + +In each of those places, an option can be defined: + +- at the root of the document +- under targets/$target/, in which case the definition only applies when + a specific target is defined + +The targets are usually used to select: + +- the platform: torbrowser-linux-x86_64, torbrowser-linux-i686, + torbrowser-windows-i686, torbrowser-osx-x86_64 +- the channel: release, nightly, alpha + +The targets torbrowser-linux-x86_64, torbrowser-linux-i686, +torbrowser-windows-i686, torbrowser-osx-x86_64 are special cases. They +do not contain options directly, instead they contain a list of other +targets. For instance, the torbrowser-linux-x86_64 target is pointing +to the linux-x86_64 and linux targets. You should define an option under +the linux target if it applies to Linux on both architectures, or under +the linux-x86_64 if it only applies to the x86_64 architecture. + +An option that is defined at the root of rbm.conf can be overrided by +an other definition under a target, or inside projects/$project/config. +You can find the complete priority order in rbm/doc/rbm_config.7. + + +Defining a project's filename +----------------------------- + +The filename option defines the output file or directory from the build +of a component. If the file or directory exists in the out/$component +directory, the component will not be rebuilt when it is used as a +dependency from an other component. + +The filename is usually something like this: + + filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.gz' + +The var/build_id value is a hash that is based on the build inputs of +the component: + - the build script (after template evaluation) + - the container definition + - the input files (the filename when it is a dependency on an other + project, the filename and hash of its content otherwise) +This means that each time the build script, the container or one of the +dependencies is modified, the output filename will change and a rebuild +of the component will be required. + +The version and var/osname values could be removed from the filename, +but they are useful to get an idea of what a file contains. + + +Adding some Linux/Windows/OSX specific changes to a build script +---------------------------------------------------------------- + +You can use the following template syntax in the build scripts: + + [% IF c("var/linux") -%] + # do something for linux + [% ELSIF c("var/windows") -%] + # do something for windows + [% ELSIF c("var/osx") -%] + # do something for osx + [% END -%] + +You can also use var/linux-x86_64 and var/linux-i686 for things that +only apply to x86_64 and i686 linux builds. You can use the var/release, +var/alpha and var/nightly options to do things depending on the channel. + +As an alternative you can define an option with a different value +depending on the target: + + targets: + linux: + var: + do_something: 'do something for linux' + windows-i686: + var: + do_something: 'do something for windows' + osx-x86_64: + var: + do_something: 'do something for osx' + +And in the build script, use: + + [% c("var/do_something") %] + + +Evaluating a component's build script +------------------------------------- + +If you want to look at the build script used to build a component for a +specific platform/channel, you can use the following command: + + $ ./rbm/rbm showconf $component build --target $channel --target $platform + +$component should be replaced by the name of the component. + +$channel should be one of the following: + + - release + + - alpha + + - nightly + +$platform should be one of the following: + + - torbrowser-linux-x86_64 + + - torbrowser-linux-i686 + + - torbrowser-windows-i686 + + - torbrowser-osx-x86_64 + +For example, to see the tor's build script for linux x86_64 on the alpha +channel, you can use: + + $ ./rbm/rbm showconf tor build --target alpha --target \ + torbrowser-linux-x86_64 + +If the component you are looking at has many dependencies, the display +can take some time as various build_id values need to be computed. If +you don't care about the accuracy of input and output file names, you +can add '--target no_build_id' to the command line. For instance if you +want to look at the build script for the tor-browser component (which +has a lot of dependencies), you can use: + + $ ./rbm/rbm showconf tor build --target alpha --target \ + torbrowser-linux-x86_64 --target no_build_id + +The same type of commands can be used to look at any option values, +replacing build with the name of the option you want to look at. For +instance if you want to know the output filename of tor on linux-x86_64 +on the alpha channel, you can use: + + $ ./rbm/rbm showconf tor filename --target alpha --target \ + torbrowser-linux-x86_64 + + +Building a single component +--------------------------- + +When you are working on some changes to a component, before creating a +full bundle with those changes, you might want to try a build of the +component you modified only. + +This can be done with the following command: + + $ ./rbm/rbm build $component --target $channel --target $platform + +See the previous section "Evaluating a component's build script" for a +list of possible values for $channel and $platform. + +For instance if you want to build tor for linux-x86_64 on the alpha +channel, you can run: + + $ ./rbm/rbm build tor --target alpha --target torbrowser-linux-x86_64 + +To find the resulting file from the build, you can use 'ls -ltr out/tor/' +to find the file with the last modification time. + + +Debugging a build error +----------------------- + +If the build fails, a debugging shell will automatically be opened in +the container used for the build (unless you set debug to 0 in +rbm.local.conf). + +Before using the debugging shell, you can first look at the build logs +in an other terminal (the filename of the log file is indicated in the +rbm output). + +In the debug shell, you can type 'bash' to get a shell with completion. +You can then look at the 'build' file, and copy paste some of its commands +to debug the problem. If you need to edit files, vim is not installed +in the build containers, but vi can be used instead. + +When you press ctrl+d (twice if you opened a bash shell) the debug shell +is closed and the container containing the failed build is removed. + +If you want to make a backup of the container's rootfs with the failed +build to look at it later, you should be able to find it in +/tmp/$UID/rbm-containers. +

1 0

[tor-browser-build/master] Add comment to explain why we build https-e once for all platforms
by boklm＠torproject.org 03 Aug '17

03 Aug '17

commit 84fd513bee8b345d636a3f9d6529fe683c40f0a1 Author: Nicolas Vigier <boklm(a)torproject.org> Date: Thu Aug 3 21:43:42 2017 +0200 Add comment to explain why we build https-e once for all platforms --- projects/https-everywhere/config | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/projects/https-everywhere/config b/projects/https-everywhere/config index 3652aaa..6ee1ee9 100644 --- a/projects/https-everywhere/config +++ b/projects/https-everywhere/config @@ -7,6 +7,14 @@ gpg_keyring: https-everywhere.gpg tag_gpg_id: 1 filename: "[% project %]-[% c('version') %]-[% c('var/build_id') %].xpi" var: + # HTTPS Everywhere is expected to be the same on all platforms. To avoid + # building the same thing 4 times, using 4 different container images + # (each one with a different suite or architecture), we set the container + # to wheezy/amd64 for all platforms. This allows us to create only one + # container image, and also build the extension only one time as the + # filename does not contain the platform, and var/build_id should be + # the same since there is now nothing platform specific in the build + # inputs. This allows us to save a little time and disk space. container: use_container: 1 suite: wheezy

1 0

[torspec/master] prop271: Note Paul's concerns on guard sampling biases from Wilmington.
by isis＠torproject.org 03 Aug '17

03 Aug '17

commit 28a8208232cac218b9bbbd95521ad7bd3c266b37 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 3 18:19:12 2017 +0000 prop271: Note Paul's concerns on guard sampling biases from Wilmington. --- proposals/271-another-guard-selection.txt | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/proposals/271-another-guard-selection.txt b/proposals/271-another-guard-selection.txt index f03a7c3..a0fba19 100644 --- a/proposals/271-another-guard-selection.txt +++ b/proposals/271-another-guard-selection.txt @@ -178,6 +178,23 @@ Implemented-In: 0.3.0.1-alpha if they have the "Guard" flag. Sampling is random but weighted by bandwidth. +[Paul Syverson in a conversation at the Wilmington Meeting 2017 says that +we should look into how we're doing this sampling. Essentially, his +concern is that, since we are sampling by bandwidth at first (when we +choose the `sampled` set), then later there is another bias—when trying to +build circuits (and hence marking guards as confirmed) we select those +which completed a usable circuit first (and hence have the lowest +latency)—that this sort of "doubly skewed" selection may "snub" some +low-consensus-weight guards and leave them unused completely. Thus the +issue is primarily that we're not allocating network resources +efficiently. Mine and Nick's guard algorithm simulation code never +checked what percentage of possible guards the algorithm reasonably +allowed clients to use; this would be an interesting thing to check in +simulation at some point. If it does turn out to be a problem, Paul's +intuition for a fix is to select uniformly at random to obtain the +`sampled` set, then weight by bandwidth when trying to build circuits and +marking guards as confirmed. —isis] + Once a path is built and a circuit established using this guard, it is marked as confirmed. Until this point, guards are first sampled and then filtered based on information such as our current

1 0

[torspec/master] dir-spec: Specify linewrapping behaviour for base64-encoded data.
by isis＠torproject.org 03 Aug '17

03 Aug '17

commit 8ebdf2e99f93628055c7d7a89327191cf4d161dc Author: Isis Lovecruft <isis(a)torproject.org> Date: Tue Aug 1 17:15:21 2017 +0000 dir-spec: Specify linewrapping behaviour for base64-encoded data. * THANKS TO Mr. George Costanza for pointing out the ambiguity. --- dir-spec.txt | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/dir-spec.txt b/dir-spec.txt index c3f9563..ec0b2ab 100644 --- a/dir-spec.txt +++ b/dir-spec.txt @@ -201,8 +201,12 @@ Objects. A KeywordLine begins with a Keyword, optionally followed by whitespace and more non-newline characters, and ends with a newline. A Keyword is a sequence of one or more characters in the set [A-Za-z0-9-]. - An Object is a block of encoded data in pseudo-Open-PGP-style - armor. (cf. RFC 2440) + An Object is a block of encoded data in pseudo-Privacy-Enhanced-Mail (PEM) + style format: that is, lines of encoded data MAY be wrapped by inserting + an ascii linefeed ("LF", also called newline, or "NL" here) character + (cf. RFC 4648 §3.1). When line wrapping, implementations MUST wrap lines + at 64 characters. Upon decoding, implementations MUST ignore and discard + all linefeed characters. More formally:

1 0

[torspec/master] prop271: Clarify when guards may be added to {CONFIRMED_GUARDS}.
by isis＠torproject.org 03 Aug '17

03 Aug '17

commit db17344f15a9e02d041e9d51ff325133cc07007e Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Aug 3 18:20:22 2017 +0000 prop271: Clarify when guards may be added to {CONFIRMED_GUARDS}. --- proposals/271-another-guard-selection.txt | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/proposals/271-another-guard-selection.txt b/proposals/271-another-guard-selection.txt index a0fba19..38da000 100644 --- a/proposals/271-another-guard-selection.txt +++ b/proposals/271-another-guard-selection.txt @@ -389,7 +389,10 @@ marking guards as confirmed. —isis] Randomized as RAND(now, {GUARD_LIFETIME}/10). We add new members to {CONFIRMED_GUARDS} when we mark a circuit - built through a guard as "for user traffic." + built through a guard as "for user traffic." That is, a circuit is + considered for use for client traffic when we have decided that we + could attach a stream to it; at that point the guard for that + circuit SHOULD be added to {CONFIRMED_GUARDS}. Whenever we remove a member from {SAMPLED_GUARDS}, we also remove it from {CONFIRMED_GUARDS}.

1 0

[sandboxed-tor-browser/master] fixup! Bug 22984: Force IDNs to be displayed as punycode to thwart homograph attacks.
by yawning＠torproject.org 03 Aug '17

03 Aug '17

commit 4cd7ad592166c8146c0f1493e69fea66cc057c19 Author: Yawning Angel <yawning(a)schwanenlied.me> Date: Thu Aug 3 17:32:37 2017 +0000 fixup! Bug 22984: Force IDNs to be displayed as punycode to thwart homograph attacks. --- ChangeLog | 1 + data/installer/mozilla.cfg | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index 89e69a5..fab4f9a 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,5 +1,6 @@ Changes in version 0.0.13 - UNRELEASED: * Bug 13170: Disable the rest of the Firefox experiments botnet prefs. + * Use lockPref for the IDN override done as part of #22984. Changes in version 0.0.12 - 2017-08-01: * Bug 22969: Disable the addon blocklist. diff --git a/data/installer/mozilla.cfg b/data/installer/mozilla.cfg index 71ea6f5..040d80e 100644 --- a/data/installer/mozilla.cfg +++ b/data/installer/mozilla.cfg @@ -41,7 +41,7 @@ lockPref("network.allow-experiments", false); lockPref("experiments.manifest.uri", ""); // Force IDNs to be displayed as punycode to thwart homograph attacks. -defaultPref("network.IDN_show_punycode", true); +lockPref("network.IDN_show_punycode", true); // Anything that tries to blur the line between CA signed HTTPS and Onion // Services is misguided at best.

1 0

[sandboxed-tor-browser/master] Bug 13170: Disable the rest of the Firefox experiments botnet prefs.
by yawning＠torproject.org 03 Aug '17

03 Aug '17

commit 3cb78c8c65ea220c01a9abbd484c8a0284beb5fd Author: Yawning Angel <yawning(a)schwanenlied.me> Date: Thu Aug 3 17:26:05 2017 +0000 Bug 13170: Disable the rest of the Firefox experiments botnet prefs. The Tor Browser developers have not took action on this, so I will. --- ChangeLog | 1 + data/installer/mozilla.cfg | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/ChangeLog b/ChangeLog index 274f4e2..89e69a5 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,4 +1,5 @@ Changes in version 0.0.13 - UNRELEASED: + * Bug 13170: Disable the rest of the Firefox experiments botnet prefs. Changes in version 0.0.12 - 2017-08-01: * Bug 22969: Disable the addon blocklist. diff --git a/data/installer/mozilla.cfg b/data/installer/mozilla.cfg index 1b8f97d..71ea6f5 100644 --- a/data/installer/mozilla.cfg +++ b/data/installer/mozilla.cfg @@ -35,6 +35,11 @@ lockPref("extensions.getAddons.showPane", false); // install date among other things to Mozilla. lockPref("extensions.blocklist.enabled", false); +// Disable experiments, because opt-out is anti-privacy. +lockPref("experiments.enabled", false); +lockPref("network.allow-experiments", false); +lockPref("experiments.manifest.uri", ""); + // Force IDNs to be displayed as punycode to thwart homograph attacks. defaultPref("network.IDN_show_punycode", true);

1 0

[tor-browser-build/master] Bug 22499: build obfsproxy and fteproxy for Windows
by boklm＠torproject.org 03 Aug '17

03 Aug '17

commit 6221fac747287361a0e4432c3ebf6590461bbbcf Author: Nicolas Vigier <boklm(a)torproject.org> Date: Thu Aug 3 17:56:39 2017 +0200 Bug 22499: build obfsproxy and fteproxy for Windows Also finish including the Linux build of obfsproxy and fteproxy. Based on gitian/descriptors/windows/gitian-pluggable-transports.yml from tor-browser-bundle.git. --- keyring/winpython.gpg | Bin 0 -> 1167 bytes projects/argparse/build | 20 +++++++--- projects/argparse/config | 7 ++++ projects/fteproxy/build | 21 +++++++++- projects/fteproxy/config | 9 +++++ projects/libfte/build | 23 +++++++++-- projects/libfte/config | 8 ++++ projects/obfsproxy/build | 32 ++++++++++++++-- projects/obfsproxy/config | 25 +++++++++++- projects/parsley/build | 21 +++++++--- projects/parsley/config | 7 ++++ projects/pycrypto/build | 29 +++++++++++--- projects/pycrypto/config | 7 ++++ projects/pyptlib/build | 21 +++++++--- projects/pyptlib/config | 7 ++++ projects/pyyaml/build | 21 +++++++--- projects/pyyaml/config | 7 ++++ projects/tor-browser/build | 4 ++ projects/tor-browser/config | 3 ++ projects/twisted/build | 32 +++++++++++++--- projects/twisted/config | 10 +++++ projects/txsocksx/build | 21 +++++++--- projects/txsocksx/config | 7 ++++ projects/winpython/build | 36 ++++++++++++++++++ projects/winpython/config | 55 +++++++++++++++++++++++++++ projects/winpython/pyc-timestamp.sh | 18 +++++++++ projects/winpython/wine-wrappers/common.py | 27 +++++++++++++ projects/winpython/wine-wrappers/dllwrap.py | 36 ++++++++++++++++++ projects/winpython/wine-wrappers/g++.py | 39 +++++++++++++++++++ projects/winpython/wine-wrappers/gcc.py | 36 ++++++++++++++++++ projects/winpython/wine-wrappers/settings.py | 2 + projects/winpython/wine-wrappers/setup.py | 7 ++++ projects/zope.interface/build | 27 ++++++++++--- projects/zope.interface/config | 9 +++++ rbm.conf | 29 ++++++++++++++ 35 files changed, 612 insertions(+), 51 deletions(-) diff --git a/keyring/winpython.gpg b/keyring/winpython.gpg new file mode 100644 index 0000000..97abdc1 Binary files /dev/null and b/keyring/winpython.gpg differ diff --git a/projects/argparse/build b/projects/argparse/build index 938c5e9..29e4221 100644 --- a/projects/argparse/build +++ b/projects/argparse/build @@ -1,14 +1,24 @@ #!/bin/bash [% c("var/set_default_env") -%] distdir="/var/tmp/dist/[% project %]" -[% c("var/set_PTDIR_DOCSDIR") -%] -mkdir -p $PTDIR +[% IF c("var/windows") -%] + [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] +[% ELSE -%] + [% c("var/set_PTDIR_DOCSDIR") -%] + mkdir -p $PTDIR +[% END -%] mkdir -p /var/tmp/build tar -C /var/tmp/build -xf $rootdir/[% project %]-[% c('version') %].tar.gz cd /var/tmp/build/[% project %]-[% c('version') %] -export PYTHON=python2 -$PYTHON setup.py build --build-lib build -cp -a build/argparse.py $PTDIR/ +[% IF c("var/windows") -%] + pydir="$distdir/python" + mkdir -p $pydir/Lib/site-packages + export PYTHONPATH="$(winepath -w $pydir)\\Lib\\site-packages" + $PYTHON setup.py install --prefix=$(winepath -w $pydir) +[% ELSE -%] + python2 setup.py build --build-lib build + cp -a build/argparse.py $PTDIR/ +[% END -%] cd $distdir [% c('tar', { tar_src => [ '.' ], diff --git a/projects/argparse/config b/projects/argparse/config index 87f80b0..c4d9dec 100644 --- a/projects/argparse/config +++ b/projects/argparse/config @@ -12,8 +12,15 @@ targets: arch_deps: - python-setuptools - python-dev + windows-i686: + var: + compiler: winpython + post_pkginst: '[% c("var/install_wine_ppa") %]' input_files: - project: container-image - URL: 'https://pypi.python.org/packages/source/a/argparse/argparse-[% c("version") %].tar.gz' sha256sum: ddaf4b0a618335a32b6664d4ae038a1de8fbada3b25033f9021510ed2b3941a4 + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + enable: '[% c("var/windows") %]' diff --git a/projects/fteproxy/build b/projects/fteproxy/build index b33c3a5..0d9baf1 100644 --- a/projects/fteproxy/build +++ b/projects/fteproxy/build @@ -3,11 +3,28 @@ distdir="/var/tmp/dist/[% project %]" [% c("var/set_PTDIR_DOCSDIR") -%] mkdir -p $PTDIR $DOCSDIR +[% IF c("var/windows") -%] + [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] +[% END -%] +tar -C $distdir -xf $rootdir/[% c('input_files_by_name/libfte') %] mkdir -p /var/tmp/build tar -C /var/tmp/build -xf $rootdir/[% project %]-[% c('version') %].tar.gz cd /var/tmp/build/[% project %]-[% c('version') %] -cp -a bin/fteproxy $PTDIR/fteproxy.bin -cp -ra fteproxy $PTDIR/ +[% IF c("var/windows") -%] + pydir="$distdir/python" + export PYTHONPATH="$(winepath -w $pydir)\\Lib\\site-packages" + $PYTHON setup_tbb.py py2exe + py2exe_zip_timestomp dist/fteproxy.zip + cp -an dist/{*.pyd,*.exe,*.zip} $PTDIR/ + mkdir -p $PTDIR/fteproxy + cp -a fteproxy/VERSION $PTDIR/fteproxy + mkdir -p $PTDIR/fteproxy/defs + cp -a fteproxy/defs/*.json $PTDIR/fteproxy/defs + rm -Rf $pydir +[% ELSE -%] + cp -a bin/fteproxy $PTDIR/fteproxy.bin + cp -ra fteproxy $PTDIR/ +[% END -%] cp -a {COPYING,README.md} $DOCSDIR/ cd $distdir [% c('tar', { diff --git a/projects/fteproxy/config b/projects/fteproxy/config index 059381b..6a13e0d 100644 --- a/projects/fteproxy/config +++ b/projects/fteproxy/config @@ -15,6 +15,15 @@ targets: arch_deps: - python-setuptools - python-dev + windows-i686: + var: + compiler: winpython + post_pkginst: '[% c("var/install_wine_ppa") %]' input_files: - project: container-image + - project: libfte + name: libfte + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + enable: '[% c("var/windows") %]' diff --git a/projects/libfte/build b/projects/libfte/build index fc7b9f6..62c8fce 100644 --- a/projects/libfte/build +++ b/projects/libfte/build @@ -10,9 +10,26 @@ tar -C /var/tmp/build -xf $rootdir/[% project %]-[% c('version') %].tar.gz cd /var/tmp/build/[% project %]-[% c('version') %] tar -C /var/tmp/dist -xf $rootdir/[% c('input_files_by_name/gmp') %] ln -s /var/tmp/dist/gmp thirdparty/gmp -export PYTHON=python2 -make -cp -ra fte $PTDIR/ +mkdir -p $distdir +tar -C $distdir -xf $rootdir/[% c('input_files_by_name/obfsproxy') %] +[% IF c("var/windows") -%] + pydir=$distdir/python + export FAKETIME="[% USE date; GET date.format(c('timestamp'), format = '%Y-%m-%d %H:%M:%S') %]" + export LD_PRELOAD= + export PYTHONPATH="$(winepath -w $pydir)\\Lib\\site-packages" + # FTE only needs libgmp-10.dll and no libgmpxx anymore. + cp -a /var/tmp/dist/gmp/bin/libgmp-10.dll . + cp -a /var/tmp/dist/gmp/bin/libgmp-10.dll $distdir/TorBrowser/Tor/ + WINDOWS_BUILD=1 CROSS_COMPILE=1 make + $PYTHON setup.py install --prefix=$(winepath -w "$pydir") + $PYTHON setup.py install +[% ELSE -%] + export PYTHON=python2 + make + cp -ra fte $PTDIR/ + mkdir -p $distdir/TorBrowser/Tor + cp /var/tmp/dist/gmp/lib/libgmp.so.10 $distdir/TorBrowser/Tor/ +[% END -%] cp -a {LICENSE,README.md} $DOCSDIR/ cp -a thirdparty/re2/LICENSE $DOCSDIR/LICENSE.re2 cd $distdir diff --git a/projects/libfte/config b/projects/libfte/config index e2219a7..4c34116 100644 --- a/projects/libfte/config +++ b/projects/libfte/config @@ -14,6 +14,12 @@ targets: arch_deps: - python-setuptools - python-dev + windows-i686: + var: + compiler: winpython + post_pkginst: '[% c("var/install_wine_ppa") %]' + arch_deps: + - faketime input_files: - project: container-image @@ -21,3 +27,5 @@ input_files: project: '[% c("var/compiler") %]' - project: gmp name: gmp + - project: obfsproxy + name: obfsproxy diff --git a/projects/obfsproxy/build b/projects/obfsproxy/build index 841c608..9d64aa9 100644 --- a/projects/obfsproxy/build +++ b/projects/obfsproxy/build @@ -3,13 +3,37 @@ distdir="/var/tmp/dist/[% project %]" [% c("var/set_PTDIR_DOCSDIR") -%] mkdir -p $PTDIR $DOCSDIR +[% IF c("var/windows") -%] + [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] +[% ELSE -%] +[% END -%] +mkdir -p $distdir +tar -C $distdir -xf $rootdir/[% c('input_files_by_name/pycrypto') %] +tar -C $distdir -xf $rootdir/[% c('input_files_by_name/twisted') %] +tar -C $distdir -xf $rootdir/[% c('input_files_by_name/argparse') %] +tar -C $distdir -xf $rootdir/[% c('input_files_by_name/parsley') %] +tar -C $distdir -xf $rootdir/[% c('input_files_by_name/pyptlib') %] +tar -C $distdir -xf $rootdir/[% c('input_files_by_name/pyyaml') %] +tar -C $distdir -xf $rootdir/[% c('input_files_by_name/txsocksx') %] +tar -C $distdir -xf $rootdir/[% c('input_files_by_name/zope.interface') %] mkdir -p /var/tmp/build tar -C /var/tmp/build -xf $rootdir/[% project %]-[% c('version') %].tar.gz cd /var/tmp/build/[% project %]-[% c('version') %] -export PYTHON=python2 -$PYTHON setup.py build --build-lib build -cp -a build/obfsproxy $PTDIR/ -cp -a bin/obfsproxy $PTDIR/obfsproxy.bin +[% IF c("var/windows") -%] + pydir="$distdir/python" + export PYTHONPATH="$(winepath -w $pydir)\\Lib\\site-packages" + $PYTHON setup_py2exe.py py2exe + $PYTHON setup.py install --prefix=$(winepath -w $pydir) + py2exe_zip_timestomp py2exe_bundle/dist/obfsproxy.zip + cp -an py2exe_bundle/dist/{*.pyd,*.exe,*.zip} $PTDIR/ + # http://bugs.winehq.org/show_bug.cgi?id=3591 + cp -a /var/tmp/dist/winpython/python27.dll $PTDIR +[% ELSE -%] + export PYTHON=python2 + $PYTHON setup.py build --build-lib build + cp -a build/obfsproxy $PTDIR/ + cp -a bin/obfsproxy $PTDIR/obfsproxy.bin +[% END -%] cp -a {LICENSE,README} $DOCSDIR cd $distdir [% c('tar', { diff --git a/projects/obfsproxy/config b/projects/obfsproxy/config index ad436dd..9134de1 100644 --- a/projects/obfsproxy/config +++ b/projects/obfsproxy/config @@ -7,7 +7,7 @@ gpg_keyring: obfsproxy.gpg filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.gz' var: - DOCSDIR_project: Obsfproxy + DOCSDIR_project: Obfsproxy container: use_container: 1 @@ -17,6 +17,29 @@ targets: arch_deps: - python-setuptools - python-dev + windows-i686: + var: + compiler: winpython + post_pkginst: '[% c("var/install_wine_ppa") %]' input_files: - project: container-image + - project: pycrypto + name: pycrypto + - project: twisted + name: twisted + - project: argparse + name: argparse + - project: parsley + name: parsley + - project: pyptlib + name: pyptlib + - project: pyyaml + name: pyyaml + - project: txsocksx + name: txsocksx + - project: zope.interface + name: zope.interface + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + enable: '[% c("var/windows") %]' diff --git a/projects/parsley/build b/projects/parsley/build index ea5f19a..1d52f9a 100644 --- a/projects/parsley/build +++ b/projects/parsley/build @@ -1,14 +1,25 @@ #!/bin/bash [% c("var/set_default_env") -%] distdir="/var/tmp/dist/[% project %]" -[% c("var/set_PTDIR_DOCSDIR") -%] -mkdir -p $PTDIR +[% IF c("var/windows") -%] + [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] +[% ELSE -%] + [% c("var/set_PTDIR_DOCSDIR") -%] + mkdir -p $PTDIR +[% END -%] mkdir -p /var/tmp/build tar -C /var/tmp/build -xf $rootdir/Parsley-[% c('version') %].tar.gz cd /var/tmp/build/Parsley-[% c('version') %] -export PYTHON=python2 -$PYTHON setup.py build --build-lib build -cp -a build/parsley.py build/ometa build/terml $PTDIR/ +[% IF c("var/windows") -%] + pydir="$distdir/python" + mkdir -p $pydir/Lib/site-packages + export PYTHONPATH="$(winepath -w $pydir)\\Lib\\site-packages" + $PYTHON setup.py install --prefix=$(winepath -w $pydir) +[% ELSE -%] + PYTHON=python2 + $PYTHON setup.py build --build-lib build + cp -a build/parsley.py build/ometa build/terml $PTDIR/ +[% END -%] cd $distdir [% c('tar', { tar_src => [ '.' ], diff --git a/projects/parsley/config b/projects/parsley/config index 72d5bfc..b788899 100644 --- a/projects/parsley/config +++ b/projects/parsley/config @@ -12,8 +12,15 @@ targets: arch_deps: - python-setuptools - python-dev + windows-i686: + var: + compiler: winpython + post_pkginst: '[% c("var/install_wine_ppa") %]' input_files: - project: container-image - URL: 'https://pypi.python.org/packages/source/P/Parsley/Parsley-[% c("version") %].tar.gz' sha256sum: 50d30cee70770fd44db7cea421cb2fb75af247c3a1cd54885c06b30a7c85dd23 + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + enable: '[% c("var/windows") %]' diff --git a/projects/pycrypto/build b/projects/pycrypto/build index 9b005ab..51ec5ce 100644 --- a/projects/pycrypto/build +++ b/projects/pycrypto/build @@ -1,22 +1,39 @@ #!/bin/bash +set -e [% c("var/set_default_env") -%] [% c("var/setarch") -%] [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] distdir="/var/tmp/dist/[% project %]" -[% c("var/set_PTDIR_DOCSDIR") -%] -mkdir -p $PTDIR +[% IF c("var/linux") -%] + [% c("var/set_PTDIR_DOCSDIR") -%] + mkdir -p $PTDIR +[% END -%] mkdir -p /var/tmp/build tar -C /var/tmp/build -xf $rootdir/[% project %]-[% c('version') %].tar.gz cd /var/tmp/build/[% project %]-[% c('version') %] -export PYTHON=python2 [% IF c("var/linux-i686") -%] export CFLAGS=-m32 export CXXFLAGS=-m32 export LDFLAGS=-m32 [% END -%] -./configure --build=i686-linux-gnu [% c("var/configure_opt") %] -$PYTHON setup.py build --build-lib build -cp -a build/Crypto $PTDIR/ +[% IF c("var/windows") -%] + pydir="$distdir/python" + mkdir -p "$pydir" + export FAKETIME="[% USE date; GET date.format(c('timestamp'), format = '%Y-%m-%d %H:%M:%S') %]" + export LD_PRELOAD=[% c("var/faketime_path") %] + # This is bogus, that we run the configure script in the build environment, + # but it seems to work. https://bugs.launchpad.net/pycrypto/+bug/1096207 for + # ac_cv_func_malloc_0_nonnull. + ac_cv_func_malloc_0_nonnull=yes sh configure --host=i686-w64-mingw32 + export LD_PRELOAD= + $PYTHON setup.py build_ext -c mingw32 + $PYTHON setup.py install --prefix=$(winepath -w $pydir) +[% ELSE -%] + export PYTHON=python2 + ./configure --build=i686-linux-gnu [% c("var/configure_opt") %] + python2 setup.py build --build-lib build + cp -a build/Crypto $PTDIR/ +[% END -%] cd $distdir [% c('tar', { tar_src => [ '.' ], diff --git a/projects/pycrypto/config b/projects/pycrypto/config index 3a9f24b..ca314c8 100644 --- a/projects/pycrypto/config +++ b/projects/pycrypto/config @@ -12,6 +12,13 @@ targets: - python-dev - python-setuptools + windows-i686: + var: + compiler: winpython + post_pkginst: '[% c("var/install_wine_ppa") %]' + arch_deps: + - faketime + input_files: - project: container-image - name: '[% c("var/compiler") %]' diff --git a/projects/pyptlib/build b/projects/pyptlib/build index 1dc0c07..de70c5b 100644 --- a/projects/pyptlib/build +++ b/projects/pyptlib/build @@ -1,14 +1,25 @@ #!/bin/bash [% c("var/set_default_env") -%] distdir="/var/tmp/dist/[% project %]" -[% c("var/set_PTDIR_DOCSDIR") -%] -mkdir -p $PTDIR +[% IF c("var/linux") -%] + [% c("var/set_PTDIR_DOCSDIR") -%] + mkdir -p $PTDIR +[% END -%] +[% IF c("var/windows") -%] + [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] +[% END -%] mkdir -p /var/tmp/build tar -C /var/tmp/build -xf $rootdir/[% project %]-[% c('version') %].tar.gz cd /var/tmp/build/[% project %]-[% c('version') %] -export PYTHON=python2 -$PYTHON setup.py build --build-lib build -cp -a build/pyptlib $PTDIR/ +[% IF c("var/windows") -%] + pydir="$distdir/python" + mkdir -p "$pydir" + $PYTHON setup.py install --single-version-externally-managed --record /dev/null --prefix=$(winepath -w $pydir) +[% ELSE -%] + export PYTHON=python2 + $PYTHON setup.py build --build-lib build + cp -a build/pyptlib $PTDIR/ +[% END -%] cd $distdir [% c('tar', { tar_src => [ '.' ], diff --git a/projects/pyptlib/config b/projects/pyptlib/config index e8ca100..7ae1d68 100644 --- a/projects/pyptlib/config +++ b/projects/pyptlib/config @@ -16,6 +16,13 @@ targets: arch_deps: - python-setuptools - python-dev + windows-i686: + var: + compiler: winpython + post_pkginst: '[% c("var/install_wine_ppa") %]' input_files: - project: container-image + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + enable: '[% c("var/windows") %]' diff --git a/projects/pyyaml/build b/projects/pyyaml/build index c10855c..d3d60d7 100644 --- a/projects/pyyaml/build +++ b/projects/pyyaml/build @@ -1,14 +1,25 @@ #!/bin/bash [% c("var/set_default_env") -%] distdir="/var/tmp/dist/[% project %]" -[% c("var/set_PTDIR_DOCSDIR") -%] -mkdir -p $PTDIR +[% IF c("var/windows") -%] + [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] +[% ELSE -%] + [% c("var/set_PTDIR_DOCSDIR") -%] + mkdir -p $PTDIR +[% END -%] mkdir -p /var/tmp/build tar -C /var/tmp/build -xf $rootdir/PyYAML-[% c('version') %].tar.gz cd /var/tmp/build/PyYAML-[% c('version') %] -export PYTHON=python2 -$PYTHON setup.py build --build-lib build -cp -a build/yaml $PTDIR/ +[% IF c("var/windows") -%] + pydir="$distdir/python" + mkdir -p $pydir/Lib/site-packages + export PYTHONPATH="$(winepath -w $pydir)\\Lib\\site-packages" + $PYTHON setup.py install --prefix=$(winepath -w $pydir) +[% ELSE -%] + export PYTHON=python2 + $PYTHON setup.py build --build-lib build + cp -a build/yaml $PTDIR/ +[% END -%] cd $distdir [% c('tar', { tar_src => [ '.' ], diff --git a/projects/pyyaml/config b/projects/pyyaml/config index 5324c10..8d43bf6 100644 --- a/projects/pyyaml/config +++ b/projects/pyyaml/config @@ -12,8 +12,15 @@ targets: arch_deps: - python-setuptools - python-dev + windows-i686: + var: + compiler: winpython + post_pkginst: '[% c("var/install_wine_ppa") %]' input_files: - project: container-image - URL: 'https://pypi.python.org/packages/source/P/PyYAML/PyYAML-[% c("version") %].tar.gz' sha256sum: c36c938a872e5ff494938b33b14aaa156cb439ec67548fcab3535bb78b0846e8 + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + enable: '[% c("var/windows") %]' diff --git a/projects/tor-browser/build b/projects/tor-browser/build index b9537a3..77a02ea 100644 --- a/projects/tor-browser/build +++ b/projects/tor-browser/build @@ -56,6 +56,10 @@ mv $TBDIR/meek-http-helper(a)bamsoftware.com.xpi $TBDIR/$MEEKPROFILEPATH/extension tar -C $TBDIR -xf [% c('input_files_by_name/snowflake') -%] [% END -%] +[% IF c("var/fteproxy") -%] + tar -C $TBDIR -xf [% c('input_files_by_name/fteproxy') %] +[% END -%] + tar -C $TBDIR[% IF c("var/osx") %]/Contents/Resources[% END %] -xf [% c('input_files_by_name/fonts') %] [% IF c("var/linux") %] diff --git a/projects/tor-browser/config b/projects/tor-browser/config index fef279b..4f708bc 100644 --- a/projects/tor-browser/config +++ b/projects/tor-browser/config @@ -50,6 +50,9 @@ input_files: name: meek - project: obfs4 name: obfs4 + - project: fteproxy + name: fteproxy + enable: '[% c("var/fteproxy") %]' - project: snowflake name: snowflake enable: '[% c("var/snowflake") %]' diff --git a/projects/twisted/build b/projects/twisted/build index acc5c9d..0d9154f 100644 --- a/projects/twisted/build +++ b/projects/twisted/build @@ -1,14 +1,36 @@ #!/bin/bash [% c("var/set_default_env") -%] distdir="/var/tmp/dist/[% project %]" -[% c("var/set_PTDIR_DOCSDIR") -%] -mkdir -p $PTDIR +[% IF c("var/windows") -%] + [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] +[% ELSE -%] + [% c("var/set_PTDIR_DOCSDIR") -%] + mkdir -p $PTDIR +[% END -%] mkdir -p /var/tmp/build tar -C /var/tmp/build -xf $rootdir/Twisted-[% c('version') %].tar.bz2 cd /var/tmp/build/Twisted-[% c('version') %] -export PYTHON=python2 -$PYTHON setup.py build --build-lib build -cp -a build/twisted $PTDIR/ +[% IF c("var/windows") -%] + pydir="$distdir/python" + export FAKETIME="[% USE date; GET date.format(c('timestamp'), format = '%Y-%m-%d %H:%M:%S') %]" + export LD_PRELOAD=[% c("var/faketime_path") %] + # twisted/internet/iocpreactor/iocpsupport/iocpsupport.c includes "python.h" + # rather than "Python.h". + ln -sf Python.h /var/tmp/dist/winpython/include/python.h + # We need to set the "mingw32" compiler to avoid an error in build_ext, but + # Twisted's "install" command calls build_ext unconditionally, whether the + # extensions have been built already or not, so we can't just call build_ext + # separately as with other packages. The "install" command doesn't recognize + # the -c option, so we set the compiler in a configuration file. + echo $'[build_ext]\ncompiler=mingw32' > setup.cfg + mkdir -p $pydir/Lib/site-packages + export PYTHONPATH="$(winepath -w $pydir)\\Lib\\site-packages" + LD_PRELOAD= $PYTHON setup.py install --single-version-externally-managed --record /dev/null --prefix=$(winepath -w $pydir) +[% ELSE -%] + export PYTHON=python2 + $PYTHON setup.py build --build-lib build + cp -a build/twisted $PTDIR/ +[% END -%] cd $distdir [% c('tar', { tar_src => [ '.' ], diff --git a/projects/twisted/config b/projects/twisted/config index f2f7de8..2f35166 100644 --- a/projects/twisted/config +++ b/projects/twisted/config @@ -12,8 +12,18 @@ targets: arch_deps: - python-setuptools - python-dev + windows-i686: + var: + compiler: winpython + post_pkginst: '[% c("var/install_wine_ppa") %]' + arch_deps: + - p7zip-full + - faketime input_files: - project: container-image - URL: 'https://pypi.python.org/packages/source/T/Twisted/Twisted-[% c("version") %].tar.bz2' sha256sum: 095175638c019ac7c0604f4c291724a16ff1acd062e181b01293bf4dcbc62cf3 + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + enable: '[% c("var/windows") %]' diff --git a/projects/txsocksx/build b/projects/txsocksx/build index 3491a26..31d7b18 100644 --- a/projects/txsocksx/build +++ b/projects/txsocksx/build @@ -1,8 +1,12 @@ #!/bin/bash [% c("var/set_default_env") -%] distdir="/var/tmp/dist/[% project %]" -[% c("var/set_PTDIR_DOCSDIR") -%] -mkdir -p $PTDIR +[% IF c("var/windows") -%] + [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] +[% ELSE -%] + [% c("var/set_PTDIR_DOCSDIR") -%] + mkdir -p $PTDIR +[% END -%] mkdir -p /var/tmp/build tar -C /var/tmp/build -xf $rootdir/[% project %]-[% c('version') %].tar.gz cd /var/tmp/build/[% project %]-[% c('version') %] @@ -11,9 +15,16 @@ cd /var/tmp/build/[% project %]-[% c('version') %] # dependency should be fine here as txsocksx catches the exception due to # missing __version__ and __sha__ . mkdir vcversioner-1.14.1.1-py2.7.egg -export PYTHON=python2 -$PYTHON setup.py build --build-lib build -cp -a build/txsocksx $PTDIR/ +[% IF c("var/windows") -%] + pydir="$distdir/python" + mkdir -p $pydir/Lib/site-packages + export PYTHONPATH="$(winepath -w $pydir)\\Lib\\site-packages" + $PYTHON setup.py install_lib --install-dir=$(winepath -w "$pydir/Lib/site-packages") +[% ELSE -%] + export PYTHON=python2 + $PYTHON setup.py build --build-lib build + cp -a build/txsocksx $PTDIR/ +[% END -%] cd $distdir [% c('tar', { tar_src => [ '.' ], diff --git a/projects/txsocksx/config b/projects/txsocksx/config index 6142943..18fc4a8 100644 --- a/projects/txsocksx/config +++ b/projects/txsocksx/config @@ -14,6 +14,13 @@ targets: arch_deps: - python-setuptools - python-dev + windows-i686: + var: + compiler: winpython + post_pkginst: '[% c("var/install_wine_ppa") %]' input_files: - project: container-image + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + enable: '[% c("var/windows") %]' diff --git a/projects/winpython/build b/projects/winpython/build new file mode 100644 index 0000000..2a30acc --- /dev/null +++ b/projects/winpython/build @@ -0,0 +1,36 @@ +#!/bin/bash +set -e +[% c("var/set_default_env") -%] +[% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] +distdir="/var/tmp/dist/[% project %]" +mkdir -p $distdir +WINEROOT=$HOME/.wine/drive_c +wineboot -i +msiexec /qn /i python-[% c("version") %].msi TARGETDIR=$distdir +sed -i 's/self.dll_libraries = get_msvcr()/pass#self.dll_libraries = get_msvcr()/g' $distdir/Lib/distutils/cygwinccompiler.py +tar xf setuptools-*.tar.gz +rm setuptools-*.tar.gz +cd setuptools-* +wine $distdir/python.exe setup.py install +cd $rootdir +7z x $rootdir/py2exe-*.exe +cp -a PLATLIB/* $distdir/Lib/site-packages/ +export FAKETIME="[% USE date; GET date.format(c('timestamp'), format = '%Y-%m-%d %H:%M:%S') %]" +export LD_PRELOAD=[% c("var/faketime_path") %] +cd wine-wrappers +# Push our config into wine-wrappers. +> settings.py +echo "LD_PRELOAD = \"$LD_PRELOAD\"" >> settings.py +echo "FAKETIME = \"$FAKETIME\"" >> settings.py +# Must pre-copy python27.dll into the build directory, or else py2exe can't find it. +mkdir -p build/bdist.win32/winexe/bundle-2.7/ +cp -a $distdir/python27.dll build/bdist.win32/winexe/bundle-2.7/ +LD_PRELOAD= $distdir/python.exe setup.py py2exe +mkdir -p $distdir/wineroot/windows/ +cp -a dist/gcc.exe dist/g++.exe dist/dllwrap.exe $distdir/wineroot/windows/ +cp $rootdir/pyc-timestamp.sh $distdir/ +cd $distdir/.. +[% c('tar', { + tar_src => [ project, 'mingw-w64' ], + tar_args => '-czf ' _ dest_dir _ '/' _ c('filename'), + }) %] diff --git a/projects/winpython/config b/projects/winpython/config new file mode 100644 index 0000000..dcd21f6 --- /dev/null +++ b/projects/winpython/config @@ -0,0 +1,55 @@ +# vim: filetype=yaml sw=2 +version: 2.7.5 +filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.gz' + +var: + compiler: mingw-w64 + post_pkginst: '[% c("var/install_wine_ppa") %]' + arch_deps: + - p7zip-full + - faketime + container: + use_container: 1 + setup: | + [% pc('mingw-w64', 'var/setup') %] + # wine path ($HOME/.wine) gets included in some binaries, so set HOME + # to a fixed path. + export HOME=/var/tmp/home + mkdir -p $HOME + WINEROOT=$HOME/.wine/drive_c + wineboot -i + cp -a /var/tmp/dist/winpython/wineroot/windows/* $WINEROOT/windows/ + export PYTHON="wine /var/tmp/dist/winpython/python.exe" + + # Set the timestamp on every .pyc file in a zip file, and re-dzip the zip file. + function py2exe_zip_timestomp { + ZIPFILE="$1" + local tmpdir="$(mktemp -d)" + local tmpzip="$(mktemp -u)" + unzip -d "$tmpdir" "$ZIPFILE" + cd "$tmpdir" + find . -name '*.pyc' -print0 | xargs -0 /var/tmp/dist/winpython/pyc-timestamp.sh "2000-01-01 00:00:00" + [% c('zip', { + zip_src => [ '.' ], + zip_args => '$tmpzip', + }) %] + cd - + mv -f "$tmpzip" "$ZIPFILE" + rm -rf "$tmpdir" + } + + +input_files: + - project: container-image + - URL: 'https://www.python.org/ftp/python/[% c("version") %]/python-[% c("version") %].msi' + file_gpg_id: 1 + sig_ext: asc + gpg_keyring: winpython.gpg + - URL: https://pypi.python.org/packages/source/s/setuptools/setuptools-1.4.tar.gz + sha256sum: 75d288687066ed124311d6ca5f40ffa92a0e81adcd7fff318c6e84082713cf39 + - URL: http://downloads.sourceforge.net/py2exe/0.6.9/py2exe-0.6.9.win32-py2.7.exe + sha256sum: 610a8800de3d973ed5ed4ac505ab42ad058add18a68609ac09e6cf3598ef056c + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + - filename: wine-wrappers + - filename: pyc-timestamp.sh diff --git a/projects/winpython/pyc-timestamp.sh b/projects/winpython/pyc-timestamp.sh new file mode 100755 index 0000000..e092268 --- /dev/null +++ b/projects/winpython/pyc-timestamp.sh @@ -0,0 +1,18 @@ +#!/bin/bash +# Usage: pyc-timestamp.sh "2001-01-01" FILENAMES... +# Overwrite (in place) the timestamp in .pyc Python bytecode files. +# +# http://hg.python.org/cpython/file/2.7/Lib/py_compile.py#l123 +# http://nedbatchelder.com/blog/200804/the_structure_of_pyc_files.html +# http://benno.id.au/blog/2013/01/15/python-determinism + +TIMESPEC="$1" +shift + +hex=$(printf 0x%08x $(date +%s --date="$TIMESPEC")) +# Write little-endian. +esc=$(printf "\\\\x%02x\\\\x%02x\\\\x%02x\\\\x%02x" $(($hex&0xff)) $((($hex>>8)&0xff)) $((($hex>>16)&0xff)) $((($hex>>24)&0xff))) +for filename in "$@"; do + echo $filename + echo -n -e "$esc" | dd of="$filename" bs=1 seek=4 conv=notrunc +done diff --git a/projects/winpython/wine-wrappers/common.py b/projects/winpython/wine-wrappers/common.py new file mode 100644 index 0000000..19e994b --- /dev/null +++ b/projects/winpython/wine-wrappers/common.py @@ -0,0 +1,27 @@ +import os +import subprocess + +import settings + +def winepath(windowspath): + """Convert a Windows path to a Unix path.""" + return subprocess.check_output(["winepath", "-u", windowspath])[:-1] + +def search_startswith(a, elems): + """Search for the first element of the array a that startswith any of the + members of elems, and return it. Return None if no match was found.""" + for e in elems: + if a.startswith(e): + return e + return None + +def popen_faketime(*args, **kwargs): + """Does subprocess.Popen after setting faketime environment variables.""" + ld_preload = os.getenv("LD_PRELOAD") + if ld_preload: + # ld.so(8): "The items of [LD_PRELOAD] can be separated by spaces or colons." + ld_preload += ":" + ld_preload += settings.LD_PRELOAD + os.putenv("LD_PRELOAD", ld_preload) + os.putenv("FAKETIME", settings.FAKETIME) + return subprocess.Popen(*args, **kwargs) diff --git a/projects/winpython/wine-wrappers/dllwrap.py b/projects/winpython/wine-wrappers/dllwrap.py new file mode 100755 index 0000000..300a3bd --- /dev/null +++ b/projects/winpython/wine-wrappers/dllwrap.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python + +# A wrapper for i686-w64-mingw32-dllwrap that removes -mno-cygwin and converts +# Windows paths to Unix paths, so that the w64-mingw32 dllwrap can be called by +# Python distutils. + +import os +import subprocess +import sys + +import common + +args = ["/var/tmp/dist/mingw-w64/bin/i686-w64-mingw32-dllwrap"] +sys.argv.pop(0) +while sys.argv: + a = sys.argv.pop(0) + if not a.startswith("-"): + args.append(common.winepath(a)) + continue + if a == "-mno-cygwin": + continue + if a in ("-I", "-L", "-s", "--def", "--output-lib"): + args.append(a) + args.append(common.winepath(sys.argv.pop(0))) + continue + o = common.search_startswith(a, ("-I", "-L", "--def=", "--output-lib=")) + if o is not None: + path = a[len(o):] + args.append("%s%s" % (o, common.winepath(path))) + continue + args.append(a) +p = common.popen_faketime(args, stderr=subprocess.PIPE) +stderr = p.stderr.read() +sys.stderr.write(stderr) +if " error: " in stderr: + sys.exit(1) diff --git a/projects/winpython/wine-wrappers/g++.py b/projects/winpython/wine-wrappers/g++.py new file mode 100755 index 0000000..a71c4e3 --- /dev/null +++ b/projects/winpython/wine-wrappers/g++.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python + +# A wrapper for i686-w64-mingw32-g++ that removes -mno-cygwin and converts +# Windows paths to Unix paths, so that the w64-mingw32 g++ can be called by +# Python distutils. + +import os +import subprocess +import sys + +import common + +args = ["/var/tmp/dist/mingw-w64/bin/i686-w64-mingw32-g++"] +sys.argv.pop(0) +while sys.argv: + a = sys.argv.pop(0) + if not a.startswith("-"): + args.append(common.winepath(a)) + continue + if a == "-mno-cygwin": + continue + if a == "--output-lib": + o = sys.argv.pop(0) + a = "-out-implib="+o + if a in ("-I", "-L"): + args.append(a) + args.append(common.winepath(sys.argv.pop(0))) + continue + o = common.search_startswith(a, ("-I", "-L")) + if o is not None: + path = a[len(o):] + args.append("%s%s" % (o, common.winepath(path))) + continue + args.append(a) +p = common.popen_faketime(args, stderr=subprocess.PIPE) +stderr = p.stderr.read() +sys.stderr.write(stderr) +if " error: " in stderr: + sys.exit(1) diff --git a/projects/winpython/wine-wrappers/gcc.py b/projects/winpython/wine-wrappers/gcc.py new file mode 100755 index 0000000..3db89b4 --- /dev/null +++ b/projects/winpython/wine-wrappers/gcc.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python + +# A wrapper for i686-w64-mingw32-gcc that removes -mno-cygwin and converts +# Windows paths to Unix paths, so that the w64-mingw32 gcc can be called by +# Python distutils. + +import os +import subprocess +import sys + +import common + +args = ["/var/tmp/dist/mingw-w64/bin/i686-w64-mingw32-gcc"] +sys.argv.pop(0) +while sys.argv: + a = sys.argv.pop(0) + if not a.startswith("-"): + args.append(common.winepath(a)) + continue + if a == "-mno-cygwin": + continue + if a in ("-I", "-L"): + args.append(a) + args.append(common.winepath(sys.argv.pop(0))) + continue + o = common.search_startswith(a, ("-I", "-L")) + if o is not None: + path = a[len(o):] + args.append("%s%s" % (o, common.winepath(path))) + continue + args.append(a) +p = common.popen_faketime(args, stderr=subprocess.PIPE) +stderr = p.stderr.read() +sys.stderr.write(stderr) +if " error: " in stderr: + sys.exit(1) diff --git a/projects/winpython/wine-wrappers/settings.py b/projects/winpython/wine-wrappers/settings.py new file mode 100644 index 0000000..f395433 --- /dev/null +++ b/projects/winpython/wine-wrappers/settings.py @@ -0,0 +1,2 @@ +LD_PRELOAD = "/usr/lib/faketime/libfaketime.so.1" +FAKETIME = "2000-01-01 00:00:00" diff --git a/projects/winpython/wine-wrappers/setup.py b/projects/winpython/wine-wrappers/setup.py new file mode 100644 index 0000000..f80f2c0 --- /dev/null +++ b/projects/winpython/wine-wrappers/setup.py @@ -0,0 +1,7 @@ +from distutils.core import setup +import py2exe +setup( + console=["gcc.py", "g++.py", "dllwrap.py"], + zipfile=None, + options={"py2exe": {"bundle_files": 1, "compressed": True}} +) diff --git a/projects/zope.interface/build b/projects/zope.interface/build index 23501a3..29406fb 100644 --- a/projects/zope.interface/build +++ b/projects/zope.interface/build @@ -1,14 +1,31 @@ #!/bin/bash [% c("var/set_default_env") -%] distdir="/var/tmp/dist/[% project %]" -[% c("var/set_PTDIR_DOCSDIR") -%] -mkdir -p $PTDIR +[% IF c("var/linux") -%] + [% c("var/set_PTDIR_DOCSDIR") -%] + mkdir -p $PTDIR +[% END -%] +[% IF c("var/windows") -%] + [% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %] +[% END -%] mkdir -p /var/tmp/build unzip -d /var/tmp/build $rootdir/[% project %]-[% c('version') %].zip cd /var/tmp/build/[% project %]-[% c('version') %] -export PYTHON=python2 -$PYTHON setup.py build --build-lib build -cp -a build/zope $PTDIR/ +[% IF c("var/windows") -%] + find -type f -print0 | xargs -0 [% c("var/touch") %] + pydir="$distdir/python" + mkdir -p $pydir/Lib/site-packages + export PYTHONPATH="$(winepath -w $pydir)\\Lib\\site-packages" + export FAKETIME="[% USE date; GET date.format(c('timestamp'), format = '%Y-%m-%d %H:%M:%S') %]" + LD_PRELOAD= $PYTHON setup.py build_ext -c mingw32 + $PYTHON setup.py install --single-version-externally-managed --record /dev/null --prefix=$(winepath -w $pydir) + # Must create this file in order for py2exe to find the package. + touch $pydir/Lib/site-packages/zope/__init__.py +[% ELSE -%] + export PYTHON=python2 + $PYTHON setup.py build --build-lib build + cp -a build/zope $PTDIR/ +[% END -%] cd $distdir [% c('tar', { tar_src => [ '.' ], diff --git a/projects/zope.interface/config b/projects/zope.interface/config index 9743b79..4ae3cff 100644 --- a/projects/zope.interface/config +++ b/projects/zope.interface/config @@ -12,8 +12,17 @@ targets: arch_deps: - python-setuptools - python-dev + windows-i686: + var: + compiler: winpython + post_pkginst: '[% c("var/install_wine_ppa") %]' + arch_deps: + - faketime input_files: - project: container-image - URL: 'https://pypi.python.org/packages/source/z/zope.interface/zope.interface-[% c("version") %].zip' sha256sum: 1a7c84716bbd9981915b64a81d8a3f076a5934a8c8df4224655469b3564940cc + - name: '[% c("var/compiler") %]' + project: '[% c("var/compiler") %]' + enable: '[% c("var/windows") %]' diff --git a/rbm.conf b/rbm.conf index ae8c5b2..c93019e 100644 --- a/rbm.conf +++ b/rbm.conf @@ -144,6 +144,7 @@ targets: compiler: gcc # We only build snowflake on linux and osx for now snowflake: 1 + fteproxy: 1 container: suite: wheezy deps: @@ -186,6 +187,34 @@ targets: export RBM_SETARCH=1 exec setarch i686 ./build fi + fteproxy: 1 + install_wine_ppa: | + # Install a Wine new enough to have a fix for + # http://bugs.winehq.org/show_bug.cgi?id=29764; otherwise Python run under + # Wine constantly crashes in _PyVerify_fd, which is called by such common + # operations as io.open and os.fstat (anything involving a file descriptor + # number). Ubuntu's main repository only has wine1.4, and the issue was fixed + # in 1.5.29. + echo 'deb http://ppa.launchpad.net/ubuntu-wine/ppa/ubuntu precise main' >> /etc/apt/sources.list + # This key is from https://launchpad.net/~ubuntu-wine/+archive/ppa and + # http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0x5A9A06AEF9CB8D…. + apt-key add - << EOF + -----BEGIN PGP PUBLIC KEY BLOCK----- + Version: GnuPG v1 + + mI0ESXVzlQEEAN1BxiR961SiFrJ7tacrAImCmDdxs4OSifgpBAp8q0fe0iLcSeAo + WdS7H+7Y4T+/2t3XEw7+3cD831WBu8c/Pv0ldx5TyDyOQmEtUBlMqy33WdKVxsci + rnag8ShrNM9PtG/vAQU/JtfQl68dXfD3BsCtrXVrjqcO2AxNYBSvd9hpABEBAAG0 + IkxhdW5jaHBhZCBQUEEgZm9yIFVidW50dSBXaW5lIFRlYW2ItgQTAQIAIAUCSXVz + lQIbAwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEFqaBq75y42wflcD/jLMihWM + zRCO60S/a7SqD0QNqV/nAYEOsma/Y2C/uhQ44j0np+iOB6+PDRbyJ8GVgIjpaIbt + l4sReXcf7bS9Dhhn5Gbe/n6VQr7xoMr1Io0PrXdmWpmsOfCuebWU4bV1w+YBvHFF + qTu5QF2Y0Fj9bRxQdQ1M2HcnXOiLq82hKlb+ + =Z9DY + -----END PGP PUBLIC KEY BLOCK----- + EOF + apt-get update + apt-get --no-install-recommends -y install wine torbrowser-osx-x86_64: - osx-x86_64

1 0

[tor-browser-build/master] meek: include *.1.txt instead of *.1 files on Windows
by boklm＠torproject.org 03 Aug '17

03 Aug '17

commit 9a392fd711a582e1c275c1c454deb766e07c5c6f Author: Nicolas Vigier <boklm(a)torproject.org> Date: Thu Aug 3 16:32:26 2017 +0200 meek: include *.1.txt instead of *.1 files on Windows --- projects/meek/build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/meek/build b/projects/meek/build index 81734be..57185b3 100644 --- a/projects/meek/build +++ b/projects/meek/build @@ -27,7 +27,7 @@ cp -a meek-client-torbrowser[% IF c("var/windows") %].exe[% END %] $PTDIR [% END %] cd .. -cp -a README doc/*.1 $DOCSDIR +cp -a README doc/*.1[% IF c("var/windows") %].txt[% END %] $DOCSDIR cd firefox [% c('zip', {

1 0

[tor/master] Try to improve documentation for DirAuthority's port field.
by nickm＠torproject.org 03 Aug '17

03 Aug '17

commit 42e787817cc78aba714bed683f8a7a9efd7d4f30 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Jul 5 12:58:51 2017 -0400 Try to improve documentation for DirAuthority's port field. Closes ticket 20152 --- changes/doc20152 | 3 +++ doc/tor.1.txt | 8 +++++--- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/changes/doc20152 b/changes/doc20152 new file mode 100644 index 000000000..8b044e56d --- /dev/null +++ b/changes/doc20152 @@ -0,0 +1,3 @@ + o Documentation: + - Improve the documentation for the directory port part of the + DirAuthority line. Closes ticket 20152. diff --git a/doc/tor.1.txt b/doc/tor.1.txt index f0b7fa8e4..2f4337630 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -436,9 +436,11 @@ GENERAL OPTIONS authority is listening for IPv6 connections on the indicated IPv6 address and OR Port. + + - Tor will contact the authority at __address__:__port__ (the DirPort) to - download directory documents. If an IPv6 address is supplied, Tor will - also download directory documents at the IPv6 address on the DirPort. + + Tor will contact the authority at __address__ to + download directory documents. The provided __port__ value is a dirport; + clients ignore this in favor of the specified "orport=" value. If an + IPv6 address is supplied, Tor will + also download directory documents at the IPv6 address on the ORPort. + + If no **DirAuthority** line is given, Tor will use the default directory authorities. NOTE: this option is intended for setting up a private Tor

1 0