TL;DR: upgrade is a little slower, snag on PostgreSQL and you may need to upgrade to Python3 earlier than expected.
On 2022-04-27 11:25:16, Antoine Beaupré wrote:
Reminder: the bullseye upgrade run is continuing in May.
A little update on the progress of the bullseye upgrades... As you can see here, progress has been a little slower than the first batch:
https://gitlab.torproject.org/tpo/tpa/team/-/issues/40692
During the first week of the first batch, we had most of the servers done ("only three left!"). It was pretty impressive. But in this first week, we were a little slower: we only did a third.
That was partly due to people's availability: I was away on Monday, and kez was also less available. Plus, we had kernel reboots to handle, which took a day off lavamind's.
But it was also due to the complexity of this second batch: most of the servers are managed by service admins and have more moving parts and legacy.
In particular, it seems we might be having unexpected trouble with the PostgreSQL 13 upgrade, which is a little disappointing. Materculae is showing signs of increase memory usage, including an OOM last night. That issue is tracked here and we welcome any input from PostgreSQL nerds:
https://gitlab.torproject.org/tpo/tpa/team/-/issues/40750
We also hit a few problems with the Python 2 deprecation. Normally, we haven't announced removing support for Python 2 just yet, and Debian bullseye did ship with Python 2.7, even though it's been dead since April 2020. But bullseye *does* ship with a bunch of Python 2 *modules* removed. So far, we have found that we needed to deal with those removals:
* python-dateutil * python-dnspython * python-psycopg2 * python-stem
In general, buster shipped with 3470 Python 2 packages, and bullseye brought that list down to *only* 766! So there are a *lot* more packages like this that may cause problem on our servers. The details of which packages those are is available here:
https://people.debian.org/~anarcat/python2-in-debian/
We don't actually *know* of any such packages left: all the ones that we had specified in Puppet are noted above and have been replaced with their Python 3 equivalent, and the service admins have fixed their service.
But we don't actually manage everything through Puppet, so it is perfectly possible that you are relying on a dependency that will be removed in the pending Python upgrade.
So if you rely on any Python script which relies on Debian packages, now is a good time to make sure it works with the following header:
#!/usr/bin/python3
... and you can do this right now, even before we upgrade your service to Python 3.
I plan on making a formal RFC to clarify this situation as well, today, so that this gets to a broader audience.
We plan to resume the bullseye upgrades next week, as we don't do major changes like this before the weekend.
Thank you for your attention!