[Impact]
Certbot users who first used < 0.26.0 have their configurations locked into using ACMEv1. This is a deprecated protocol. Let's Encrypt brownouts for ACMEv1 are scheduled to begin at the beginning of 2021, and Let's Encrypt will stop serving ACMEv1 in June 2021.
Based on Let's Encrypt's metrics, 23,847 users were counted as being locked into ACMEv1 in this way. These users will start receiving certification renewal failures unless they are patched.
Users affected are users who first used Certbot on Xenial or first used Certbot on the release pocket version of Certbot in Bionic.
Users who first used Certbot >= 0.26.0 are not affected. This includes users who used Certbot on Bionic after 0.27.0-1~ubuntu18.04.1 (published 2019-10-29) and users who first used Certbot on Focal or above.
[Test Case]
Get Focal/Bionic systems that you can get a public IP and DNSname on
comment #17 shows how to do so with Canonistack, but any other method is fine as well
1. get certbot (I'll use apache for testing)
$ sudo apt install apache2 python3-certbot python3-certbot-apache
2. Get a ACMEv2 cert (current default)
$ sudo certbot --apache
# go along the questions and use the DNS you have set up
3. modify the server endpoint to v1 manually
$ sudo sed -i -e 's/acme-v02/acme-v01/g' /etc/letsencrypt/renewal/*
4. renew Cert (will try to use the patched v1 sever)
$ sudo certbot renew --force-renewal
This will fail without the fix and "simulate" what will happen to old installs (which had a v1 config) after the upgrade.
Comment #18 has sample output of good/bad case.
[Regression Potential]
Since the endpoint is being changed, users who are controlling reachable endpoints (such as with egress firewalls or proxies) may not be able to reach the new endpoint until they have adjusted their configurations. However as the old endpoint will stop functioning soon, deliberately making this change appears to be the least worst option.
Renewal configuration parsing of the server URL is being modified. Users with unusual configurations such as those that have different server URLs defined may find themselves on untested paths.
Users trying to debug a problem configuration will find it surprising that a configuration that specifies the LE ACMEv1 endpoint goes to the LE ACMEv2 endpoint instead. However, again this seems to be the least worst option.
[Further Details]
Let’s Encrypt is in the process of shutting down ACMEv1. The full shutdown process will be completed in June 2021 with temporary brown-outs starting at the beginning of the year; more specific details are available at https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430.
When ACMEv1 is shut down, many older versions of Certbot will be unable to get new certificates. ACMEv2 support was first made default in 0.26.0 for new certificates, but it wasn’t until 1.6.0 that certificates which had originally been issued using ACMEv1 were transitioned to ACMEv2. The original update was supposed to move people off of ACMEv1, but due to some old configuration management code, we missed a small group of early Certbot users.
Based on recent counts, there are a total of 23,847 distinct non-EOL Ubuntu users still using ACMEv1 who use the version of Certbot packaged in their system’s package manager (the versions available in 16.04 universe, 16.04 universe updates, 18.04 universe, 18.04 universe updates, and 20.04). These users will no longer receive certs in June, but would be automatically upgraded to ACMEv2 if the package for their system were updated.
The commit that switches ACMEv1 users to ACMEv2 is here: https://github.com/certbot/certbot/commit/340a4280eacc3eac8915996d89ff0c0a0cd023f9
One option to address the upcoming shutdown is to backport the commit into older versions of Certbot.
Another option to address the shutdown, which is preferable from our perspective, would be to update Certbot to 1.6.0+. First, there’s the inherent risk in backporting an individual change, especially onto much older code. Released versions are tested extensively both on our systems and by our users, so we’re much more sure of their stability than a backported patch. Additionally, Certbot continues to improve over time, closing up bugs, supporting more edge cases, improving usability, and offering more robust and modern security practices.
Since we made backwards incompatible changes in 0.40.0 and 1.0.0, to update Certbot to a newer version, our other components will have to be updated as well. Certbot relies on our other libraries `acme` and `josepy`, and we have a series of plugins which will need to be updated as well, including the `certbot-nginx` and `certbot-apache` plugins, as well as our `certbot-dns-*` plugins. Certbot 1.0.0 in particular contained significant API changes, and if any of our packages are updated to 1.0.0 or newer, it will probably be easiest to update all of them. josepy may be fine depending on the version of certbot, as certbot 1.0.0 relies on `josepy>=1.1.0`, which is already available packaged on all relevant systems. But Certbot 1.0.0 also requires `acme>=0.40.0`, which is only one release behind 1.0.0, so it would probably be easier to update it to a matching version. Basically, I would recommend choosing a certbot version, then updating `acme`, `certbot-nginx`, `certbot-apache`, and `certbot-dns-*` to that version. None of our 3rd party dependencies should need to be updated.
One thing to note when choosing a version is that Certbot 1.7.0 deprecated Python 3.5 support, which may be necessary on older systems, so 1.6.0 may be a better choice than later versions on older systems.
Updating to anything past 0.38.0 will require the `distro` dependency, which is not currently packaged on Xenial. It is in Bionic and it has no transitive dependencies that aren't in Xenial: https://packages.ubuntu.com/bionic/python-distro
Certbot 0.40.0 and 1.0.0 introduced backwards incompatible changes; these include:
* CLI flags --tls-sni-01-port and --tls-sni-01-address have been removed.
* The values tls-sni and tls-sni-01 for the --preferred-challenges flag are no
longer accepted.
* Removed the flags: `--agree-dev-preview`, `--dialog`, and `--apache-init-script`
* Certbot's `config_changes` subcommand has been removed
* `certbot.plugins.common.TLSSNI01` has been removed.
* Deprecated attributes related to the TLS-SNI-01 challenge in `acme.challenges` and `acme.standalone` have been removed.
* The functions `certbot.client.view_config_changes`, `certbot.main.config_changes`, `certbot.plugins.common.Installer.view_config_changes`, `certbot.reverter.Reverter.view_config_changes`, and `certbot.util.get_systemd_os_info` have been removed
* Certbot's `register --update-registration` subcommand has been removed
* When possible, default to automatically configuring the webserver so all requests
redirect to secure HTTPS access. This is mostly relevant when running Certbot
in non-interactive mode. Previously, the default was to not redirect all requests.
Hi Erica,
Thank you for looking after the certbot packages in Ubuntu and being proactive about managing service deprecations as always.
My feeling on this one is to cherry-pick the commit you identified only, if that is all that is required? Could you confirm which Ubuntu releases require this please? Is it all of 16.04, 18.04 and 20.04? Is the version in Ubuntu Groovy (1.7.0-1 currently, not yet released) affected?
I appreciate your preference to update our existing releases to 1.6.0+, especially considering your confidence in your own newer releases as opposed to cherry-picking that you have not tested. However I think the other side of the trade-off is in the complexity that you describe in the number of dependencies that would need to be updated or introduced. While we might have confidence that certbot 1.6.0+ when correctly presented with the necessary dependencies will work correctly, there's a risk that packaging will get it wrong and certbot don't correctly get that - for example in specific upgrade paths. This already happened to us in our first attempt to update certbot like this (that we realized and thus didn't release, but that did delay us). Also, distribution package consumers prefer stability in the "doesn't change behaviour" sense.
Given that historically we find it difficult to find volunteers to work on this, the triviality of this particular fix, and my points in the previous paragraph, I think we should focus on the cherry-pick.
If you could confirm the affected release list please, I (or someone else on my team) can drive the SRU process for this update based on a cherry-pick.
Going forwards, I suggest that the policy we adopt in making a decision on whether to update distribution certbot packaging in Ubuntu should be to prefer cherry-picks if they are reasonably simple to achieve, but permit major version updates when cherry-picks aren't practical to solve an "Internet deprecation". Users could then expect distribution certbot packaging to avoid changing behaviour when possible, but still change behaviour where that is required to keep it working. Users who specifically want to upgrade to newer certbot behaviour but remain on an old distribution release now have the option of using the snap.
What changed my opinion from before, when we set up the certbot exception, is that the complexity of the necessary changes to certbot needed to keep it working as Let's Encrypt and ACME have changed seem to me to have reduced considerably over time. I think this is a sign of maturity of the project. I think users expect the churn in stable distribution release packages to reduce accordingly. However I appreciate that we might yet need major changes in the future and so I don't rule out using the standing exception again should that become necessary.
What do you think?