[tor-project] TPA-RFC-27: Python 2 end of life

Mon May 9 14:35:56 UTC 2022

Reminder: this proposal will be considered adopted unless there are any
objections tomorrow, at which point we will resume upgrades.

a.

On 2022-05-05 15:19:47, Antoine Beaupré wrote:
> Summary: Python 2 is officially unsupported by TPA. Major breakage to
> unfixed code is expected after the Debian bullseye upgrade completes
> (May-July 2022), and definite breakage will occur when Python 2
> support is completely dropped in Debian bookworm (some time in 2023).
>
> # Background
>
> [Python 2.7.18][] was released on April 20th 2020. It was the last
> Python 2 release that will ever happen, and Python 2 is now
> unsupported, end of life, [dead][].
>
> [Python 2.7.18]: https://www.python.org/downloads/release/python-2718/
> [dead]: https://www.enricozini.org/blog/2020/python/python-2-is-dead/
>
> ## Status of Python 2 in Debian
>
> It was originally thought that the Debian 11 "bullseye" [release][]
> (on August 14th 2021) would not support Python 2 at all, but it was
> actually released with *some* Python 2 support.
>
> [release]: https://www.debian.org/News/2021/20210814
>
> However, an analysis from anarcat about the [Python 2 modules shipped
> in bullseye][] shows that a large number of Python 2 modules were
> actually removed from Debian 11. Out of the 2699 "python 2" packages
> in Debian buster (packages starting with `python2?-`, excluding `-doc`
> and `-dbg`), 2616 were removed. Therefore, only 90 such packages
> remain in Debian bullseye, a 97% reduction.
>
> [Python 2 modules shipped in bullseye]: https://people.debian.org/~anarcat/python2-in-debian/
>
> As a consequence, odds are that your Python 2 code will just stop
> working after the bullseye upgrade, if it uses one of the [modules
> missing from bullseye][]. Which, really, means if it uses anything
> outside the standard library that is not vendored with your code
> (e.g. in a "virtualenv"), because the odds of that module being one of
> the few 90 modules are pretty low.
>
> [modules missing from bullseye]: https://people.debian.org/~anarcat/python2-in-debian/python2-removed-bullseye.txt
>
> The next Debian stable release (12, code name "bookworm") doesn't yet
> have a clear plan to remove Python 2, but it's likely to shrink the
> list of Python 2 modules even further. It is currently down to 79
> packages.
>
> Bookworm also does not currently ship the magic [python-is-python2][]
> package, which ensures the existence of `/usr/bin/python`. This means
> any script with such a header will start breaking in Debian bookworm:
>
>     #!/usr/bin/python
>
> [python-is-python2]: https://packages.debian.org/bullseye/python-is-python2
>
> # Status of Python 2 in TPA
>
> We currently are in the middle of a the [Debian 11 bullseye
> upgrade][], so we have both Debian 10 and Debian 11 machines, which
> means we have actually Python 2.7.16 and 3.7.3 (buster) and Python
> 2.7.18 and 3.9.2 (bullseye) currently deployed.
>
> In any case, we have two "reasonable" versions of Python 2 (2.7+) and
> Python 3 (3.5+) available everywhere, it should be fairly easy to
> target Python 3 for ports, without having to concern ourselves with
> Python 2 support any longer.
>
> [Debian 11 bullseye upgrade]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/5
>
> We do not currently knowingly deploy any Python 2 module in the above
> list, although it's possible some packages are manually installed on
> some host.
>
> The TPA code base still has a lot of Python 2 code, particularly on
> the LDAP server, but there's a lot of Python 2 code floating around
> the infrastructure. We haven't performed an audit of the code and are
> fixing issues as they come up as part of the Python 2 upgrade
> procedure.
>
> Other services have not been examined. So far, most services actually
> run under Python 3, or have been found to be already ported and just
> needing a redeployment (see
> [tpo/network-health/metrics/exit-scanner#40004][] for an example).
>
> [tpo/network-health/metrics/exit-scanner#40004]: https://gitlab.torproject.org/tpo/network-health/metrics/exit-scanner/-/issues/40004
>
> # Proposal
>
> After the Debian 11 "bullseye" upgrade, TPA will *not* support Python
> 2 modules that were removed from Debian. Any program using such a
> module will need to be ported to Python 3, as the packages shipping
> those modules will be removed as part of the upgrade procedure. The
> `/usr/bin/python` binary will remain, for now, as the 2.7.18
> executable.
>
> After the Debian 12 "bookworm" upgrade, Python 2 will be completely
> removed from servers. Any program using Python 2 will likely stop
> working completely as the `/usr/bin/python` command will be removed.
>
> The `/usr/bin/python` command *may* eventually be replaced by the
> Python 3 interpreter, but that will not be before the bookworm upgrade
> procedure begins, and only if the lack of a `python` binary is too
> problematic for users.
>
> ## Timeline
>
> Debian 11 bullseye upgrades should complete by July 1st 2022, but most
> upgrades should complete by the second week of May 2022, that is next
> week, starting on May 9th 2022 and continuing during the week.
>
> A grace period *may* be given to certain projects that cannot
> immediately port their code to Python 3, by keeping Python 2 modules
> from Debian buster installed, even after the bullseye upgrade. Those
> modules will *definitely* be removed by July 1st 2022, however.
>
> Debian 12 bookworm upgrades are currently scheduled to begin some time
> in 2023 and should be completed before July 2024. An actual schedule
> will be proposed in a future announcement. When this change will be
> deployed, Python 2 will be gone from TPA servers.
>
> ## Affected users
>
> This affects any service admin who deploys Python code on TPA
> infrastructure.
>
> # Alternatives considered
>
> We have considered just ignoring this problem, and in fact that was
> the approach with the [original Debian 11 bullseye upgrade
> proposal][]. Although it didn't state it explicitly, it didn't have
> any plan for the Python 2 upgrade.
>
> And indeed, the issue about the [Python end of life][] was postponed
> to the [Debian 12 bookworm upgrade milestone][], because it was
> believed Python 2 would just keep working in Debian 11. Unfortunately,
> the [second batch of upgrades][] showed the situation was much more
> severe than we expected, and required a more radical approach.
>
> [second batch of upgrades]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40692
> [Python end of life]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/33949
> [original Debian 11 bullseye upgrade proposal]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-20-bullseye-upgrades
>
> Another alternative to porting your code to Python 3 is actually to
> use the PyPy interpreter, which still supports Python 2 (and is
> actually still struggling with its Python 3 port). However, we
> strongly discourage this approach, and `pypy` is not currently
> installed on any TPA server.
>
> GitLab CI users may be able to ignore this issue by using containers
> that do ship Python 2. Note that we *may*, in the future, implement
> controls on the container images deployed from GitLab CI to avoid
> using old, unsupported software in this way, exactly for this kind of
> situation. But for now there are no such controls. We strongly
> discourage the use of outdated software, including containers, inside
> your tool chain, in general.
>
> # Costs
>
> Staff. 
>
> There is no estimate on the volume of Python 2 code left to upgrade. A
> study of this should probably be performed at some point, but so far
> we have assumed this wasn't a problem, so we are dealing with this on
> a case-by-case basis.
>
> # Approval
>
> TPA.
>
> # Deadline
>
> This proposal will welcome comments until Tuesday May 10th, at which
> point it will be considered adopted and the Debian bullseye upgrades
> will resume.
>
> We acknowledge this is an extremely short deadline (~5 days), but we
> have actually planned those Debian bullseye upgrade for a while, and
> somehow expected there wouldn't be much Python 2 code lying around. We
> hope that the exception for Python 2 modules (until July 1st) will be
> sufficient mitigation for us to continue with the bullseye upgrades in
> a timely manner.
>
> # Status
>
> This proposal is currently in the `proposed` state.
>
> # References
>
>  * [Discussions about this proposal][], comments welcome here!
>  * [Debian 11 bullseye upgrade milestone][]
>  * [Debian 12 bookworm upgrade milestone][]
>
> [Discussions about this proposal]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/33949
> [Debian 11 bullseye upgrade milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/5
> [Debian 12 bookworm upgrade milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/6
>
> -- 
> Antoine Beaupré
> torproject.org system administration
> _______________________________________________
> tor-project mailing list
> tor-project at lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-project

-- 
Antoine Beaupré
torproject.org system administration
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20220509/8eed093a/attachment.sig>