Summary: Python 2 is officially unsupported by TPA. Major breakage to unfixed code is expected after the Debian bullseye upgrade completes (May-July 2022), and definite breakage will occur when Python 2 support is completely dropped in Debian bookworm (some time in 2023).
# Background
[Python 2.7.18][] was released on April 20th 2020. It was the last Python 2 release that will ever happen, and Python 2 is now unsupported, end of life, [dead][].
[Python 2.7.18]: https://www.python.org/downloads/release/python-2718/ [dead]: https://www.enricozini.org/blog/2020/python/python-2-is-dead/
## Status of Python 2 in Debian
It was originally thought that the Debian 11 "bullseye" [release][] (on August 14th 2021) would not support Python 2 at all, but it was actually released with *some* Python 2 support.
[release]: https://www.debian.org/News/2021/20210814
However, an analysis from anarcat about the [Python 2 modules shipped in bullseye][] shows that a large number of Python 2 modules were actually removed from Debian 11. Out of the 2699 "python 2" packages in Debian buster (packages starting with `python2?-`, excluding `-doc` and `-dbg`), 2616 were removed. Therefore, only 90 such packages remain in Debian bullseye, a 97% reduction.
[Python 2 modules shipped in bullseye]: https://people.debian.org/~anarcat/python2-in-debian/
As a consequence, odds are that your Python 2 code will just stop working after the bullseye upgrade, if it uses one of the [modules missing from bullseye][]. Which, really, means if it uses anything outside the standard library that is not vendored with your code (e.g. in a "virtualenv"), because the odds of that module being one of the few 90 modules are pretty low.
[modules missing from bullseye]: https://people.debian.org/~anarcat/python2-in-debian/python2-removed-bullsey...
The next Debian stable release (12, code name "bookworm") doesn't yet have a clear plan to remove Python 2, but it's likely to shrink the list of Python 2 modules even further. It is currently down to 79 packages.
Bookworm also does not currently ship the magic [python-is-python2][] package, which ensures the existence of `/usr/bin/python`. This means any script with such a header will start breaking in Debian bookworm:
#!/usr/bin/python
[python-is-python2]: https://packages.debian.org/bullseye/python-is-python2
# Status of Python 2 in TPA
We currently are in the middle of a the [Debian 11 bullseye upgrade][], so we have both Debian 10 and Debian 11 machines, which means we have actually Python 2.7.16 and 3.7.3 (buster) and Python 2.7.18 and 3.9.2 (bullseye) currently deployed.
In any case, we have two "reasonable" versions of Python 2 (2.7+) and Python 3 (3.5+) available everywhere, it should be fairly easy to target Python 3 for ports, without having to concern ourselves with Python 2 support any longer.
[Debian 11 bullseye upgrade]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/5
We do not currently knowingly deploy any Python 2 module in the above list, although it's possible some packages are manually installed on some host.
The TPA code base still has a lot of Python 2 code, particularly on the LDAP server, but there's a lot of Python 2 code floating around the infrastructure. We haven't performed an audit of the code and are fixing issues as they come up as part of the Python 2 upgrade procedure.
Other services have not been examined. So far, most services actually run under Python 3, or have been found to be already ported and just needing a redeployment (see [tpo/network-health/metrics/exit-scanner#40004][] for an example).
[tpo/network-health/metrics/exit-scanner#40004]: https://gitlab.torproject.org/tpo/network-health/metrics/exit-scanner/-/issu...
# Proposal
After the Debian 11 "bullseye" upgrade, TPA will *not* support Python 2 modules that were removed from Debian. Any program using such a module will need to be ported to Python 3, as the packages shipping those modules will be removed as part of the upgrade procedure. The `/usr/bin/python` binary will remain, for now, as the 2.7.18 executable.
After the Debian 12 "bookworm" upgrade, Python 2 will be completely removed from servers. Any program using Python 2 will likely stop working completely as the `/usr/bin/python` command will be removed.
The `/usr/bin/python` command *may* eventually be replaced by the Python 3 interpreter, but that will not be before the bookworm upgrade procedure begins, and only if the lack of a `python` binary is too problematic for users.
## Timeline
Debian 11 bullseye upgrades should complete by July 1st 2022, but most upgrades should complete by the second week of May 2022, that is next week, starting on May 9th 2022 and continuing during the week.
A grace period *may* be given to certain projects that cannot immediately port their code to Python 3, by keeping Python 2 modules from Debian buster installed, even after the bullseye upgrade. Those modules will *definitely* be removed by July 1st 2022, however.
Debian 12 bookworm upgrades are currently scheduled to begin some time in 2023 and should be completed before July 2024. An actual schedule will be proposed in a future announcement. When this change will be deployed, Python 2 will be gone from TPA servers.
## Affected users
This affects any service admin who deploys Python code on TPA infrastructure.
# Alternatives considered
We have considered just ignoring this problem, and in fact that was the approach with the [original Debian 11 bullseye upgrade proposal][]. Although it didn't state it explicitly, it didn't have any plan for the Python 2 upgrade.
And indeed, the issue about the [Python end of life][] was postponed to the [Debian 12 bookworm upgrade milestone][], because it was believed Python 2 would just keep working in Debian 11. Unfortunately, the [second batch of upgrades][] showed the situation was much more severe than we expected, and required a more radical approach.
[second batch of upgrades]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40692 [Python end of life]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/33949 [original Debian 11 bullseye upgrade proposal]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-20-bullsey...
Another alternative to porting your code to Python 3 is actually to use the PyPy interpreter, which still supports Python 2 (and is actually still struggling with its Python 3 port). However, we strongly discourage this approach, and `pypy` is not currently installed on any TPA server.
GitLab CI users may be able to ignore this issue by using containers that do ship Python 2. Note that we *may*, in the future, implement controls on the container images deployed from GitLab CI to avoid using old, unsupported software in this way, exactly for this kind of situation. But for now there are no such controls. We strongly discourage the use of outdated software, including containers, inside your tool chain, in general.
# Costs
Staff.
There is no estimate on the volume of Python 2 code left to upgrade. A study of this should probably be performed at some point, but so far we have assumed this wasn't a problem, so we are dealing with this on a case-by-case basis.
# Approval
TPA.
# Deadline
This proposal will welcome comments until Tuesday May 10th, at which point it will be considered adopted and the Debian bullseye upgrades will resume.
We acknowledge this is an extremely short deadline (~5 days), but we have actually planned those Debian bullseye upgrade for a while, and somehow expected there wouldn't be much Python 2 code lying around. We hope that the exception for Python 2 modules (until July 1st) will be sufficient mitigation for us to continue with the bullseye upgrades in a timely manner.
# Status
This proposal is currently in the `proposed` state.
# References
* [Discussions about this proposal][], comments welcome here! * [Debian 11 bullseye upgrade milestone][] * [Debian 12 bookworm upgrade milestone][]
[Discussions about this proposal]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/33949 [Debian 11 bullseye upgrade milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/5 [Debian 12 bookworm upgrade milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/6
Reminder: this proposal will be considered adopted unless there are any objections tomorrow, at which point we will resume upgrades.
a.
On 2022-05-05 15:19:47, Antoine Beaupré wrote:
Summary: Python 2 is officially unsupported by TPA. Major breakage to unfixed code is expected after the Debian bullseye upgrade completes (May-July 2022), and definite breakage will occur when Python 2 support is completely dropped in Debian bookworm (some time in 2023).
# Background
[Python 2.7.18][] was released on April 20th 2020. It was the last Python 2 release that will ever happen, and Python 2 is now unsupported, end of life, [dead][].
## Status of Python 2 in Debian
It was originally thought that the Debian 11 "bullseye" [release][] (on August 14th 2021) would not support Python 2 at all, but it was actually released with *some* Python 2 support.
However, an analysis from anarcat about the [Python 2 modules shipped in bullseye][] shows that a large number of Python 2 modules were actually removed from Debian 11. Out of the 2699 "python 2" packages in Debian buster (packages starting with `python2?-`, excluding `-doc` and `-dbg`), 2616 were removed. Therefore, only 90 such packages remain in Debian bullseye, a 97% reduction.
As a consequence, odds are that your Python 2 code will just stop working after the bullseye upgrade, if it uses one of the [modules missing from bullseye][]. Which, really, means if it uses anything outside the standard library that is not vendored with your code (e.g. in a "virtualenv"), because the odds of that module being one of the few 90 modules are pretty low.
The next Debian stable release (12, code name "bookworm") doesn't yet have a clear plan to remove Python 2, but it's likely to shrink the list of Python 2 modules even further. It is currently down to 79 packages.
Bookworm also does not currently ship the magic [python-is-python2][] package, which ensures the existence of `/usr/bin/python`. This means any script with such a header will start breaking in Debian bookworm:
#!/usr/bin/python# Status of Python 2 in TPA
We currently are in the middle of a the [Debian 11 bullseye upgrade][], so we have both Debian 10 and Debian 11 machines, which means we have actually Python 2.7.16 and 3.7.3 (buster) and Python 2.7.18 and 3.9.2 (bullseye) currently deployed.
In any case, we have two "reasonable" versions of Python 2 (2.7+) and Python 3 (3.5+) available everywhere, it should be fairly easy to target Python 3 for ports, without having to concern ourselves with Python 2 support any longer.
We do not currently knowingly deploy any Python 2 module in the above list, although it's possible some packages are manually installed on some host.
The TPA code base still has a lot of Python 2 code, particularly on the LDAP server, but there's a lot of Python 2 code floating around the infrastructure. We haven't performed an audit of the code and are fixing issues as they come up as part of the Python 2 upgrade procedure.
Other services have not been examined. So far, most services actually run under Python 3, or have been found to be already ported and just needing a redeployment (see [tpo/network-health/metrics/exit-scanner#40004][] for an example).
# Proposal
After the Debian 11 "bullseye" upgrade, TPA will *not* support Python 2 modules that were removed from Debian. Any program using such a module will need to be ported to Python 3, as the packages shipping those modules will be removed as part of the upgrade procedure. The `/usr/bin/python` binary will remain, for now, as the 2.7.18 executable.
After the Debian 12 "bookworm" upgrade, Python 2 will be completely removed from servers. Any program using Python 2 will likely stop working completely as the `/usr/bin/python` command will be removed.
The `/usr/bin/python` command *may* eventually be replaced by the Python 3 interpreter, but that will not be before the bookworm upgrade procedure begins, and only if the lack of a `python` binary is too problematic for users.
## Timeline
Debian 11 bullseye upgrades should complete by July 1st 2022, but most upgrades should complete by the second week of May 2022, that is next week, starting on May 9th 2022 and continuing during the week.
A grace period *may* be given to certain projects that cannot immediately port their code to Python 3, by keeping Python 2 modules from Debian buster installed, even after the bullseye upgrade. Those modules will *definitely* be removed by July 1st 2022, however.
Debian 12 bookworm upgrades are currently scheduled to begin some time in 2023 and should be completed before July 2024. An actual schedule will be proposed in a future announcement. When this change will be deployed, Python 2 will be gone from TPA servers.
## Affected users
This affects any service admin who deploys Python code on TPA infrastructure.
# Alternatives considered
We have considered just ignoring this problem, and in fact that was the approach with the [original Debian 11 bullseye upgrade proposal][]. Although it didn't state it explicitly, it didn't have any plan for the Python 2 upgrade.
And indeed, the issue about the [Python end of life][] was postponed to the [Debian 12 bookworm upgrade milestone][], because it was believed Python 2 would just keep working in Debian 11. Unfortunately, the [second batch of upgrades][] showed the situation was much more severe than we expected, and required a more radical approach.
Another alternative to porting your code to Python 3 is actually to use the PyPy interpreter, which still supports Python 2 (and is actually still struggling with its Python 3 port). However, we strongly discourage this approach, and `pypy` is not currently installed on any TPA server.
GitLab CI users may be able to ignore this issue by using containers that do ship Python 2. Note that we *may*, in the future, implement controls on the container images deployed from GitLab CI to avoid using old, unsupported software in this way, exactly for this kind of situation. But for now there are no such controls. We strongly discourage the use of outdated software, including containers, inside your tool chain, in general.
# Costs
Staff.
There is no estimate on the volume of Python 2 code left to upgrade. A study of this should probably be performed at some point, but so far we have assumed this wasn't a problem, so we are dealing with this on a case-by-case basis.
# Approval
TPA.
# Deadline
This proposal will welcome comments until Tuesday May 10th, at which point it will be considered adopted and the Debian bullseye upgrades will resume.
We acknowledge this is an extremely short deadline (~5 days), but we have actually planned those Debian bullseye upgrade for a while, and somehow expected there wouldn't be much Python 2 code lying around. We hope that the exception for Python 2 modules (until July 1st) will be sufficient mitigation for us to continue with the bullseye upgrades in a timely manner.
# Status
This proposal is currently in the `proposed` state.
# References
- [Discussions about this proposal][], comments welcome here!
- [Debian 11 bullseye upgrade milestone][]
- [Debian 12 bookworm upgrade milestone][]
-- Antoine Beaupré torproject.org system administration _______________________________________________ tor-project mailing list tor-project@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-project
tor-project@lists.torproject.org
-
Antoine Beaupré