Hi Matthew and Isis,
On Sun, 22 Dec 2013, isis agora lovecruft wrote:
If proper packaging is helpful for Jenkins, however, I can easily do so.
An idea could be to have a Debian package for bridgedb, and make Jenkins update the packages in a repository automatically when there are new commits.
For our purpose I think debian packages are a bit overkill, but I have nothing against creating them if it will make testing and deployment easier.
I am moderately to strongly against using Debian's packaging system for Python things, because it is perpetually outdated, combined with the Python Software Foundation's complete disregard for the standard packaging concept of backporting patches. When something breaks in Python, they fix it in an upcoming release. If you complain that something is broken which is fixed in a newer release than the Python version you're using, the Python devs will tell you to upgrade.
Debian sid's version is currently 2.7.5-5. Which is outdated. 2.7.6 was released two weeks ago. Wheezy is even worse; it's nearly two years outdated. Briefly skimming it, I can point at roughly 30 bugs in the Python release changelog [0] which BridgeDB will likely hit, if we use the wheezy version (which we are using). Several of those bugs due to using ancient, deprecated, OpenSSL API features, and other rather severe SSL bugs, one of which was a recent CVE. (CVE-2013-4238) [1]
What is more important, and what I would *really* prefer not to fight, is the inevitable slew of horrific glitches and hiccups which will occur from using Debian's extremely outdated Python modules. Here is a list of Python packages in Debian, compared to the version in Ubuntu, along with excuses for why the haven't been updated. [2] Twisted in wheezy is also about two years outdated. Jessie/sid aren't up-to-date either. This is madness.
I don't mean to start a holy war, and I love Debian… but their Python team is failing. I think the only reason Debian managed to get a somewhat safe version of Pip (v1.4.1) into jessie and sid is that one of the main organisers in Debian Security convinced me to file CVEs for vulnerabilities in pip-1.3.1, even though dstufft had already fixed the issues in 1.4.1. If I understood the politics correctly, they needed me to file those CVEs so that the security team could update the packages themselves ― without waiting for the lead of the Debian Python group to give the okay (the latter being the person who seems to be causing most of these problems; incidentally, that person works for Canonical).
Ok, I didn't know that, thanks for the details. So let's avoid Debian packages for BridgeDB for now.
If, somehow, this process involves staging a violent revolutionary upheaval of the current lead of the Debian Python team, and whichever other jerks are preventing Debian from having decent Python packaging… count me in! Otherwise I'd really rather steer clear of them.
We could have the following package repositories :
bridgedb-common-wheezy: backports of dependencies required to run bridgedb on wheezy
bridgedb-master-wheezy: packages produced from master (or development) branch on wheezy
bridgedb-stable-wheezy: packages produced from stable branch on wheezy
bridgedb-xxxx-wheezy: packages from xxxx branch, if needed
We also have a develop branch, right now it only resides in our personal repos, though. This will be useful to test in the future.
Develop is probably the branch to test. I don't ever make commits straight onto master branches in any repo.
When you push a new commit, Jenkins rebuilds the package and moves it to a temporary repository. Then it starts a new container or VM with that repository enabled, install the bridgedb package and run some tests. If the tests succeed, the package is moved to the repository corresponding to the branch.
Does "repository" here mean "Debian repository"? Or "directory of tarballs and sha256sums"?
I was thinking about a Debian repository when writting this, but this can be done as a directory of tarballs and sha256sums if that's better. Is it something that pip can use ?
But maybe that is not very useful as it seems pip can install from a git URL directly.
When you think that what is in the master branch is ready to be used, you merge it to the stable branch, wait for Jenkins to rebuild the packages
This is done! With the slight modification that my master = your stable, and my develop = your master, my fix/* and feature/* are all branches for specific tickets, and my testing/* branches are all staging branches (from fix/* and feature/*) which will get merged into my develop.
The main idea being that someone pulling from master should always get the latest stable release.
Ok. So to summarize the things that would be useful to do, we have:
- run tests from Jenkins, with different python versions, when there are new commits in the master branch. We do that by creating a virtualenv, installing BridgeDB in the virtualenv and runnig "bridgedb test". (#10417)
- do the same for the develop branch in user repositories
- install a staging instance on ponticum
- make jenkins build the docs from the master branch, and upload them somewhere like docs.tpo/bridgedb