[tor-project] TPA-RFC-20: bullseye upgrade schedule
anarcat at torproject.org
Thu Mar 24 20:35:34 UTC 2022
Note: this proposal is also visible in:
Summary: bullseye upgrades will roll out starting the first weeks of
April and May, and should complete before the end of August 2022. Let
us know if your service requires special handling.
Debian 11 [bullseye] was [released on August 14 2021]). Tor
started the upgrade to bullseye shortly after and hopes to complete
the process before the [buster] EOL, [one year after the stable
release], so normally around August 2022.
In other words, we have until this summer to upgrade *all* of TPA's
machine to the new release.
New machines that were setup recently have already been installed in
bullseye, as the installers were changed shortly after the release. A
few machines were upgraded manually without any ill effects and we do
not consider this upgrade to be risky or dangerous, in general.
This work is part of the [%Debian 11 bullseye upgrade milestone],
itself part of the [OKR 2022 Q1/Q2 plan].
The proposal, broadly speaking, is to upgrade all servers in three
batches. The first two are somewhat equally sized and spread over
April and May, and the rest will happen at some time that will be
announced later, individually, per server.
## Affected users
All service admins are affected by this change. If you have shell
access on any TPA server, you want to read this announcement.
## Upgrade schedule
The upgrade is split in multiple batches:
* low complexity (mostly TPA): April
* moderate complexity (service admins): May
* high complexity (hard stuff): to be announced separately
* to be retired or rebuilt servers: not upgraded
* already completed upgrades
The free time between the first two will also allow us to cover for
unplanned contingencies: upgrades that could drag on and other work
that will inevitably need to be performed.
The objective is to do the batches in collective "upgrade parties"
that should be "fun" for the team (and work parties *have* generally
been generally fun in the past).
### Low complexity, batch 1: April
A first batch of servers will be upgraded in the first week of April.
Those machines are considered to be somewhat trivial to upgrade as
they are mostly managed by TPA or that we evaluate that the upgrade
will have minimal impact on the service's users.
27 machines. At a worst case 45 minutes per machine, that is 20 hours
of work. At three people, this might be doable in a day.
Feedback and coordination of this batch happens in issue
### Moderate complexity, batch 2: May
The second batch of "moderate complexity servers" happens in the first
week of May. The main difference with the first batch is that the second
batch regroups services mostly managed by service admins, who are given
a longer heads up before the upgrades are done.
26 machines. If the worst case scenario holds, this is another day of
work, at three people.
Not mentioned here is the `gnt-fsn` Ganeti cluster upgrade, which is
covered by ticket [tpo/tpa/team#40689]. That alone could be a few
day-person of work.
Feedback and coordination of this batch happens in issue [tpo/tpa/team#40692]
### High complexity, individually done
Those machines are harder to upgrade, due to some major upgrades of
their core components, and will require individual attention, if not
major work to upgrade.
Each machine could take a week or two to upgrade, depending on the
situation and severity. To detail each server:
* `alberti`: `userdir-ldap` is, in general, risky and needs special
attention, but should be moderately safe to upgrade, see ticket
* `eugeni`: messy server, with lots of moving parts (e.g. Schleuder,
Mailman), Mailman 2 EOL, needs to decide whether to migrate to
Mailman 3 or replace with Discourse (and self-host), see
[tpo/tpa/team#40471], followup in [tpo/tpa/team#40694]
* `hetzner-hel1-01`: Nagios AKA Icinga 1 is end-of-life and needs to
be migrated to Icinga 2, which involves fixing our git hooks to
generate Icinga 2 configuration (unlikely), or rebuilding a Icinga
2 server, or replacing with Prometheus (see
[tpo/tpa/team#29864]), followup in [tpo/tpa/team#40695]
* `pauli`: Puppet packages are severely out of date in Debian, and
Puppet 5 is EOL (with Puppet 6 soon to be). doesn't necessarily
block the upgrade, but we should deal with this problem sooner than
later, see [tpo/tpa/team#33588], followup in [tpo/tpa/team#40696]
All of those require individual decision and design, and specific
announcements will be made for upgrades once a decision has been made
for each service.
### To retire
Those servers are possibly scheduled for removal and may not be
upgraded to bullseye at all. If we miss the summer deadline, they
might be upgraded as a last resort.
* cupani/vineale is covered by [tpo/tpa/team#40472]
* gayi is [TPA-RFC-11: SVN retirement], [tpo/tpa/team#17202]
* moly/peninsulare is [tpo/tpa/team#29974]
### To rebuild
Those machines are planned to be rebuilt and should therefore not be
Some of those machines are hosted at a Sunet and need to be migrated
elsewhere, see [tpo/tpa/team#40684] for details. `colchicifolium` will
is planned to be rebuilt in the `gnt-chi` cluster, no ticket created
They will be rebuilt in new bullseye machines which should allow for a
safer transition that shouldn't require specific coordination or
### Completed upgrades
Those machines have already been upgraded to (or installed as) Debian
### Other related work
There is other work related to the bullseye upgrade that is mentioned
in the [%Debian 11 bullseye upgrade milestone].
# Alternatives considered
We have not set aside time to automate the upgrade procedure any
further at this stage, as this is considered to be a too risky
development project, and the current procedure is fast enough for
We could also move to the cloud, Kubernetes, serverless, and Ethereum
and pretend none of those things exist, but so far we stay in the real
world of operating systems.
Also note that this doesn't cover Docker container images
upgrades. Each team is responsible for upgrading their image tags in
GitLab CI appropriately and is *strongly* encouraged to keep a close
eye on those in general. We may eventually consider enforcing stricter
control over container images if this proves to be too chaotic to
It is estimates this will take one or two person-month to complete, full
# Approvals required
This proposal needs approval from TPA team members, but service admins
can request additional delay if they are worried about their service
being affected by the upgrade.
Comments or feedback can be provided in issues linked above.
Upgrades will start in the first week of April 2022 (2022-04-04)
unless an objection is raised.
This proposal will be considered adopted by then unless an objection
is raised within TPA.
This proposal is currently in the `proposed` state.
* [TPA bullseye upgrade procedure]
* [%Debian 11 bullseye upgrade milestone]
[TPA bullseye upgrade procedure]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye/
[%Debian 11 bullseye upgrade milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/5
[released on August 14 2021]: https://www.debian.org/News/2021/20210814
[one year after the stable release]: https://www.debian.org/security/faq#lifespan
[OKR 2022 Q1/Q2 plan]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2022
[TPA-RFC-11: SVN retirement]: policy/tpa-rfc-11-svn-retirement
torproject.org system administration
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 487 bytes
Desc: not available
More information about the tor-project