Remove X509Extension, which has been deprecated for a year

[alex]

No description provided.

[alex]

I'm a bit nervous about the impact of this. Notwithstanding that it's been deprecated for a while, it effectively ends using pyOpenSSL to generate certs, and I don't have a sense of how widely that's relied upon.

[alex]

From the failing twisted tests, I guess we have our answer.

[alex]

This is waiting on certbot/certbot#9909

[Copilot]

Copilot reviewed 2 out of 4 changed files in this pull request and generated no comments.

Files not reviewed (2)

• CHANGELOG.rst: Language not supported
• doc/api/crypto.rst: Language not supported

Comments suppressed due to low confidence (3)

tests/test_crypto.py:36

• The removal of the X509Extension import is consistent with the deprecation, but ensure this change does not break any dependent tests or references to deprecated functionality.

    X509Extension,

tests/test_crypto.py:875

• [nitpick] Ensure that the removal of the X509Extension tests is accompanied by alternative tests covering the new recommended cryptography-based implementations.

class TestX509Ext:

src/OpenSSL/crypto.py:63

• Confirm that the removal of the X509Extension export does not leave behind any outdated or dangling references in the module documentation or related code.

    "X509Extension",

[alex]

certbot 4.0 which removed the usage of this has been released.

@wgreenberg, if we merge this and do a release, will that break older installs of certbot, or do they bound their pyopenssl versions?

Assuming no bounds, is there any way for us to track what versions of certbot people use? (Is it in the user-agent? Happy to chat with our friendly neighborhood Lets Encrypt SREs :-))

[wgreenberg]

ack, unfortunately we don't have an upper bound on pyopenssl versions, so this would indeed break older, pip-based certbot/acme installations (but not snap). not that that's a blocker necessarily, but should be noted! theoretically we could backport version bounds for older versions; i'll discuss that w/ the team.

i'll get back to you on version metrics

[alex]

Thanks.

Unclear how much a backport would do. (We've seen a lot of people who pin their deps, but not transitive ones, and those still break.)

We'd like to avoid breaking too many folks, so I think tracking version is the most important thing we could do.

[wgreenberg]

so the bad news is we don't have stats on whether people have installed their certbot via snap, a distro-managed package, or pip (the only method that would be affected by the breakage), but some basic back of the envelope estimates puts the number of pip installations anywhere from 50k-2M users. of these, any version <4.0 might be affected

given this, i think it'd be reasonable to post an announcement on the letsencrypt community forum to give folks a heads-up -- i'd be happy to do that, since the call to action is a little nuanced based on people's certbot installation method

[mhils]

@wgreenberg: Do we have stats on certbot version distribution?

[alex]

Sure -- I think the action item is basically "if you're installing with pip and going to pin to a certbot<4, make sure to also pin your pyopenssl".

50k-2M is a lot of people, so we're going to either need to build confidence most people are on distro versions, or go slow here.

[Copilot]

Pull Request Overview

This PR removes the deprecated X509Extension functionality that has been unused for over a year. The changes include the removal of the X509Extension class and its associated tests, as well as related extension-handling methods in the X509Req and X509Store APIs.

• Removed the entire TestX509Ext test suite and related tests in tests/test_crypto.py.
• Removed the X509Extension class and any references in src/OpenSSL/crypto.py.

Reviewed Changes

Copilot reviewed 2 out of 4 changed files in this pull request and generated no comments.

File	Description
tests/test_crypto.py	Removed tests for the deprecated X509Extension functionality
src/OpenSSL/crypto.py	Removed the deprecated X509Extension class and associated deprecation APIs

Files not reviewed (2)

• CHANGELOG.rst: Language not supported
• doc/api/crypto.rst: Language not supported

Comments suppressed due to low confidence (1)

tests/test_crypto.py:875

• The removal of the X509Extension tests significantly reduces the test coverage for certificate extension handling. Consider adding tests for the new cryptography-based APIs to ensure that extension-related functionality remains fully verified.

class TestX509Ext:

[jvanasco]

so the bad news is we don't have stats on whether people have installed their certbot via snap, a distro-managed package, or pip (the only method that would be affected by the breakage), but some basic back of the envelope estimates puts the number of pip installations anywhere from 50k-2M users. of these, any version <4.0 might be affected

How were those numbers calculated? IIRC, PyPi stats include downloads by mirrors, but not those using mirrors... so many CI systems do not hit this. They're showing 450k monthly downloads - which is pretty high (https://pypistats.org/packages/certbot). The most popular certbot environments are py3.10 and py3.11; same with pyopenssl. pyopenssl is currently supporting py3.7+ ; certbot is supporting py3.9+. The potentially affected platforms are a minority of "actively maintained" installs ("actively maintaned" as in people occasionally update them; most people who hit the letsencrypt forums with a pip install have never updated it).

A safe approach might be for Certbot to create a pin release for the versions that last supported py3.7 and py3.8 - an update or install on earlier systems won't pull in this release, and anything py3.9+ shouldn't care about this because it would use the newer certbots that don't reference this code. Looking at the changelog, Certbot dropped py3.8 in v3.1 , and py3.7 in v2.8.

[alex]

An important fact here is that nothing we do in a new certbot release matters, the backwards compatibility issues here exist for people doing pip install certbot<4 (for example), which installs an old certbot but will install a new pyopenssl.

[jvanasco]

Sorry if I was not clear with this: What I meant to convey is that a v3.1.1 release and a v2.8.1 release with pins should cover most legacy platforms, because of how the minimum python versions of both PyOpenSSL and Certbot align across python environments. Anything py36 or under will not be affected, because pip won't pull in an affected PyOpenssl due to the minimum python version; so we would just need to handle py3.7 and py3.8.

For anyone running py3.9 above – which seems to cover >80% of actively maintained installations based on pypi stats of 3.10 and 3.11 alone – pip install certbot<4 should pull in Certbot 3.3.0. Certbot >= 3.2.0 shouldn't ever call this code – it switched to make_self_signed_cert, though gen_ss_cert is still in there as deprecated for compatibility through 3.3.0. It is possible some third party extension or in-house tech is still leveraging this, but those numbers should be negligible. I handled some of this deprecation and removal in Certbot/Acme, which is why all this stuff is still fresh in my mind.

There will of course be issues for people trying to install a specific version that wouldn't be caught by the 2 suggested pins that will cause issues. Perhaps people are running specific pins like this for various reasons, such as a popular plugin. I don't have any insight into that.

Based on my understanding of Certbot/PyOpenSSL code and the platform/library version distributions though, I don't think pip install certbot<4 is likely to affect anyone on py3.9 or above due to the previous api migrations - and patch version bumps against 3.1.0 and 2.8.0 with only a pyopenssl pin should address most legacy Certbot installs.