OpenSSL 3.0 support in OpenVPN 2.5

Bug #1975574 reported by Arne Schwabe
40
This bug affects 7 people
Affects Status Importance Assigned to Milestone
openvpn (Ubuntu)
Fix Released
High
Unassigned
Jammy
Fix Released
High
Sergio Durigan Junior
Kinetic
Fix Released
High
Unassigned

Bug Description

[Impact]

Users running Ubuntu Jammy have been having a poor experience because of the not fully support of OpenSSL 3. One of the upstream maintainers gladly provided a set of commits that should be backported to Jammy in order to improve this situation (they are listed in the "Original Message" section).

[Test Plan]

** Testcase for the SHA1 certificate support

One of the main problems this SRU addresses is the current inability to handle certificates that use legacy cryptographic algorithms (like SHA1). So let's reproduce this problem.

We will be using two VMs, one acting as the server and the other as the client. The server will be running Focal, and the client will be running Jammy.

== Server setup ==

$ lxc launch ubuntu:focal ovpn-server --vm
$ lxc shell ovpn-server
# apt update
# apt install -y openvpn easy-rsa
# cd /etc/openvpn
# make-cadir easy-rsa
# cd easy-rsa
# sed -i 's/^#set_var EASYRSA_DIGEST.*/set_var EASYRSA_DIGEST\t\t"SHA1"/' vars
# ./easyrsa init-pki
# ./easyrsa build-ca
... when asked for the Common Name, provide "ovpn-server" ...
# ./easyrsa build-server-full ovpn-server nopass
# ./easyrsa build-client-full ovpn-client nopass
# ./easyrsa gen-dh
# cp pki/ca.crt pki/issued/ovpn-server.crt pki/private/ovpn-server.key pki/dh.pem /etc/openvpn/
# cd /etc/openvpn
# openvpn --genkey --secret ta.key
# cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz .
# gunzip server.conf.gz
# mv server.conf ovpn-server.conf
# sed -i -e 's/^cert .*/cert ovpn-server.crt/' -e 's/^key .*/key ovpn-server.key/' -e 's/^dh .*/dh dh.pem/' ovpn-server.conf
# systemctl start openvpn@ovpn-server

Make sure that the service has successfully started by checking the output of "systemctl status openvpn@ovpn-server" and verifying that there's a new "tun0" interface when you invoke "ip a".

The server setup is done. As you can see above, we've instructed easy-rsa to generate SHA1 digests for our certificates. This is what will trigger the issue on the client side.

Make sure you get the server's IP address (not the VPN one!); you will need it below.

== Client setup ==

$ lxc launch ubuntu:jammy ovpn-client --vm
$ lxc shell ovpn-client
# apt update
# apt install -y openvpn

Now you have to transfer the following files from the server to the client:

- /etc/openvpn/ca.crt -> /etc/openvpn/ca.crt
- /etc/openvpn/easy-rsa/pki/issued/ovpn-client.crt -> /etc/openvpn/ovpn-client.crt
- /etc/openvpn/easy-rsa/pki/private/ovpn-client.key -> /etc/openvpn/ovpn-client.key
- /etc/openvpn/ta.key -> /etc/openvpn/ta.key

# cd /etc/openvpn
# cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf .
# mv client.conf ovpn-client.conf
# sed -i -e 's/^cert .*/cert ovpn-client.crt/' -e 's/^key .*/key ovpn-client.key/' ovpn-client.conf

Assuming you have the server's IP address, you can:

# sed -i 's/^remote .*/remote SERVER_IP_ADDRESS 1194/' ovpn-client.conf

Now, we will try to start the client. This should fail:

# systemctl start openvpn@ovpn-client
# systemctl status openvpn@ovpn-client
...
     Status: "Pre-connection initialization successful"
        CPU: 19ms

Oct 04 02:11:29 ovpn-client systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
Oct 04 02:11:29 ovpn-client systemd[1]: <email address hidden>: Failed with result 'exit-code'.
...

If you inspect journalctl, you will see the following failure:

# journalctl -xeu openvpn@ovpn-client
...
Oct 04 02:12:32 ovpn-client ovpn-ovpn-client[1755]: OpenSSL: error:0A00018E:SSL routines::ca md too weak
Oct 04 02:12:32 ovpn-client ovpn-ovpn-client[1755]: Cannot load certificate file ovpn-client.crt
Oct 04 02:12:32 ovpn-client ovpn-ovpn-client[1755]: Exiting due to fatal error
...

** Testcase for the new "tls-cert-profile insecure" option

Aside from regenerating the certificates (which is something that not everyone can do), upstream has implemented a new parameter that can be specified to enable the use of legacy crypto algorithms: "tls-cert-profile insecure". If we add this option to the configuration file and try to start the service again, we will then see the following error on journalctl:

...
Oct 04 02:16:46 ovpn-client ovpn-ovpn-client[3165]: ERROR: Invalid cert profile: insecure
Oct 04 02:16:46 ovpn-client ovpn-ovpn-client[3165]: Exiting due to fatal error
Oct 04 02:16:46 ovpn-client systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
...

** Testcase for the new "providers" option

The Jammy OpenVPN package already carries a patch that loads the "legacy" OpenSSL 3 providers by default. Arguably, the new "providers" option that's being added with in this SRU simply gives the user a choice to decide whether to enable the legacy providers or not. However, since we've already shipped a version of OpenVPN with these providers enabled, it's now unlikely that we will revert this change and potentially break scripts out there.

An easy (but superficial) way to test this new option is to include the following line in the configuration file:

"providers legacy default"

Upon restarting the OpenVPN service with the current version in Jammy, an error will be triggered because the option is not recognized.

[Where problems could occur]

The new options being added ("tls-cert-profile legacy" and "providers legacy") don't introduce a possibility to causing regressions, but they still could cause breakage if they're used. However, the fact that the OpenVPN package on Jammy already loads legacy OpenSSL providers by default makes these options no-ops in terms of functionality change. The risk for regression is minimal; if it exists, it will likely be associated with the code responsible for parsing the new parameters.

Arguably, the "new" feature this patchset adds is the ability to use the legacy SHA1 crypto algorithm for certificates, which is something that has been broken ever since we backported the OpenSSL 3 support to the package. The code this "new" feature exercises, however, has been part of OpenVPN for quite a while; we're merely re-enabling it. Therefore, it is very unlikely that it will cause any regressions.

Finally, there are cosmetic changes that are being backported, like new error messages, improvements to cipher listings or simple code refactoring. The regression potential with these changes is minimal.

[Other Info]

- All the patches applied in this SRU were provided by one of the upstream maintainers. This is not a random selection of things.

- We have pre-discussed with the SRU team that in this case we can not verify each patch/change individually. We have worked on trying to get some covered as we have been able to come up with a setup, but would ask for it to not be blocked on "but it needs a test for each line changed" kind of thinking. Instead we'd like to ask to accept it as discussed and in addition maybe let it be in -proposed for a bit longer than usual.

--- ---

[Original Message]

Upstream developer of OpenVPN here. We basically got caught off guard by distributions like Ubuntu already bundling OpenVPN with OpenSSL 3.0 and had hoped to release OpenVPN 2.6 which has proper OpenSSL 3.0 earlier. So far OpenVPN 2.5.x has a number of shortcoming/bugs when used with OpenSSL 3.0. We backported/fixed most of them for 2.5.7.

As much as we as upstream would prefer using 2.5.7, I think Ubuntu policy is not to update to new upstream version. But some of the bugs might be considered bugs worthy still fixing in Ubuntu 22. So I am listing here the bug/fixes that you might consider:

The individual fixes/bugs are (all from the release/2.5 branch):

- sending "new" OpenSSL digest names and causing auth mismatch warnings:
   https://github.com/OpenVPN/openvpn/commit/b158125f67b49149ffd3e2617479fbd27860713c

- Add message when decoding PKCS12 file fails.
  https://github.com/OpenVPN/openvpn/commit/1f54811e92c89fe07d7cea8339e928980bfe0536

  Several old OpenSSL version default to RC2-40-CBC when encoding pkcs12 which OpenSSL 3.0 does not
  like anymore and this at least gives a better error in these cases

- Fix allowing/showing unsupported ciphers and digests
   https://github.com/OpenVPN/openvpn/commit/3690939126cf84b166157bad96e724caea61346d

  Without this patch OpenVPN will error out much much later when choosing a cipher like BF-CBC that
  is only provided by the legacy provider.

- Allow loading of non default providers
  https://github.com/OpenVPN/openvpn/commit/aef0e595132bd75b0a089e0536c7f910667f1c07

  Even though insecure a lot of people still run OpenVPN config with the bf-cbc cipher. This commit
  allows using it again when using --providers legacy default.
  (Needs https://github.com/OpenVPN/openvpn/commit/3f25bf7f7c1f32c2d3ef5b52443c97553a6c8977 to apply)

- Add insecure tls-cert-profile options
  https://github.com/OpenVPN/openvpn/commit/7b1b100557608db8a311d06f7578ceb7c4d33aa6

  This one is already picked up.

Related branches

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openvpn (Ubuntu):
status: New → Confirmed
Revision history for this message
Sebastien Bacher (seb128) wrote :

Thanks Arne for the details, we will want to cherry pick some of those changes indeed, I will let the server team comment on the details

Changed in openvpn (Ubuntu):
importance: Undecided → High
tags: added: server-todo
Changed in openvpn (Ubuntu Jammy):
assignee: nobody → Lucas Kanashiro (lucaskanashiro)
Changed in openvpn (Ubuntu Kinetic):
assignee: nobody → Lucas Kanashiro (lucaskanashiro)
Simon Chopin (schopin)
tags: added: transition-openssl3-jj
Revision history for this message
Arne Schwabe (arneschwabe) wrote :

There yet another commit that allows better compatibility with OpenSSL 3, which allows configuration that have no explict --cipher in them to work without loading the legacy provider:

https://github.com/OpenVPN/openvpn/commit/cf5864f5922e4f40357d9f75a35cd448e671dddf

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Thanks for all the info Arne. I am checking whether this is all fixed in Kinetic before backporting those patches to Jammy, and I think all the patches are applied in the snapshot we have from 2022-05-28 (package version 2.6.0~git20220518+dco-2ubuntu1) but in some of them the changes are not exactly the same but very similar. Except the last commit you mentioned in comment #3, that one I think we need to also add to Kinetic. Could you take a quick look and confirm that?

Revision history for this message
Arne Schwabe (arneschwabe) wrote :

Yes. The last commit is also needed for OpenSSL 3.0 compatibility. The commit is for the release 2.5 branch. The version in Karmic (2.6.0~git) is a snapshot of OpenVPN master, it should include the patches already. Note that for development branches, we do not generally support them or look at issues that might need to be later resolved. I would advise against using a development branch in a stable Ubuntu release.

Revision history for this message
Arne Schwabe (arneschwabe) wrote :

On a further note, it seems that version of the package includes experimental OpenVPN dco code from May this year. That patch set has seen a few revisions since then and issues with DCO and non-DCO operation have been found/fixed.

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

The decision to ship a snapshot from the upstream master branch was made by the Debian maintainer. I'll try to get in contact with them so we can at least update the snapshot. Do you have a timeline to release version 2.6? All this is to get Kinetic and also the next debian release in a good shape.

For Jammy, it seems that the commits you mentioned should apply cleanly. I'll be working on that.

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Regarding using an upstream master branch snapshot, there is a discussion in Debian:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1014376

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote (last edit ):

I needed to apply some small changes in some upstream patches because they were trying to modify part of the code that was added after 2.5.5 release. But I believe everything is good. I am sharing this debdiff to let others take a look before we move forward with the upload. In order to do that I also need to find a way to write a test plan for this.

Revision history for this message
Arne Schwabe (arneschwabe) wrote :

You might want to change the Changes.rst part and maybe add (Debian backport or something similar to it, otherwise users might be confused why their 2.5.5 package contains a changelog that mentions 2.5.7

tags: added: patch
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Thanks for spotting that Arne, I just applied the change you suggested in the linked merge proposal.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvpn - 2.6.0~git20220518+dco-2ubuntu3

---------------
openvpn (2.6.0~git20220518+dco-2ubuntu3) kinetic; urgency=medium

  * d/t/control: add allow-stderr restriction. With 'set -x' in place some
    messages are printed out in stderr.

 -- Lucas Kanashiro <email address hidden> Thu, 14 Jul 2022 11:47:23 -0300

Changed in openvpn (Ubuntu Kinetic):
status: Confirmed → Fix Released
Revision history for this message
Alexandru Tol (alexdatacare) wrote :

Any update on the fix for Jammy?

description: updated
description: updated
Robie Basak (racb)
Changed in openvpn (Ubuntu Jammy):
assignee: Lucas Kanashiro (lucaskanashiro) → nobody
Paride Legovini (paride)
Changed in openvpn (Ubuntu Jammy):
status: New → Confirmed
Changed in openvpn (Ubuntu Jammy):
importance: Undecided → High
Changed in openvpn (Ubuntu Kinetic):
assignee: Lucas Kanashiro (lucaskanashiro) → nobody
Changed in openvpn (Ubuntu Jammy):
assignee: nobody → Sergio Durigan Junior (sergiodj)
description: updated
description: updated
description: updated
description: updated
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks Sergio for identifying a test for this.
By the former state of discussions (I have also clarified that in an update to the SRU template) I think we could upload that to jammy-unapproved for re-evaluation.

@Lucas / @Sergio does one of you have a branch at hand representing the debdiff shown here that we could upload?

description: updated
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Thanks Sergio and Christian. Package uploaded:

Uploading openvpn_2.5.5-1ubuntu3.1.dsc
Uploading openvpn_2.5.5-1ubuntu3.1.debian.tar.xz
Uploading openvpn_2.5.5-1ubuntu3.1_source.changes

Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Arne, or anyone else affected,

Accepted openvpn into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.5.5-1ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openvpn (Ubuntu Jammy):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :
Download full text (4.2 KiB)

Performing the verification on Jammy.

First, reproducing the issue.

... after configuring the server and the client as instructed in the Test Plan section ...

# systemctl start openvpn@ovpn-client
# journalctl -xeu openvpn@ovpn-client
...
Oct 22 01:14:52 ovpn-client ovpn-ovpn-client[1678]: OpenSSL: error:0A00018E:SSL routines::ca md too weak
Oct 22 01:14:52 ovpn-client ovpn-ovpn-client[1678]: Cannot load certificate file ovpn-client.crt
Oct 22 01:14:52 ovpn-client ovpn-ovpn-client[1678]: Exiting due to fatal error
Oct 22 01:14:52 ovpn-client systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE

# apt policy openvpn
openvpn:
  Installed: 2.5.5-1ubuntu3
  Candidate: 2.5.5-1ubuntu3
  Version table:
 *** 2.5.5-1ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status

Now, verifying that the new package fixes the problem.

# systemctl start openvpn@ovpn-client
# systemctl status openvpn@ovpn-client
● <email address hidden> - OpenVPN connection to ovpn-client
     Loaded: loaded (/lib/systemd/system/openvpn@.service; enabled-runtime; vendor preset: enabled)
     Active: active (running) since Sat 2022-10-22 01:17:55 UTC; 3s ago
       Docs: man:openvpn(8)
             https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
             https://community.openvpn.net/openvpn/wiki/HOWTO
   Main PID: 2437 (openvpn)
     Status: "Initialization Sequence Completed"
      Tasks: 1 (limit: 1173)
     Memory: 2.0M
        CPU: 23ms
     CGroup: /<email address hidden>
             └─2437 /usr/sbin/openvpn --daemon ovpn-ovpn-client --status /run/openvpn/ovpn-client.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/ovpn-client.conf -->
...

# ip a
...
3: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN...

Read more...

tags: added: verification-done verification-done-jammy
removed: verification-needed verification-needed-jammy
description: updated
Revision history for this message
Alexandru Tol (alexdatacare) wrote :

Hi All, I'd like to report that the proposed package does not work for me.
When attempting to 'systemctl start openvpn@Server' I get the following for the systemctl status:

● <email address hidden> - OpenVPN connection to server
     Loaded: loaded (/lib/systemd/system/openvpn@.service; enabled-runtime; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Tue 2022-10-25 22:03:36 PDT; 2s ago
       Docs: man:openvpn(8)
             https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
             https://community.openvpn.net/openvpn/wiki/HOWTO
    Process: 39664 ExecStart=/usr/sbin/openvpn --daemon ovpn-server --status /run/openvpn/server.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/server.conf --writepid /run/openvpn/server.pid (code=exited, status=1/FA>
   Main PID: 39664 (code=exited, status=1/FAILURE)
     Status: "Pre-connection initialization successful"
        CPU: 34ms

Oct 25 22:03:36 sjvpn systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
Oct 25 22:03:36 sjvpn systemd[1]: <email address hidden>: Failed with result 'exit-code'.

I also get the same message from the logs:

2022-10-25 22:01:46 OpenSSL: error:0A00018E:SSL routines::ca md too weak

Revision history for this message
Arne Schwabe (arneschwabe) wrote :

@Alexandru Your certifccates singing hashes are too weak. OpenSSL does generally not allow them anymore. However, one of the patches adds tls-cert-seclevel insecure which will help you for now. But you should regnerate your certificates anyway.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvpn - 2.5.5-1ubuntu3.1

---------------
openvpn (2.5.5-1ubuntu3.1) jammy; urgency=medium

  * d/p/openssl-3/*.patch: backport upstream patch set to better support
    OpenSSL 3 (LP: #1975574)

 -- Lucas Kanashiro <email address hidden> Thu, 14 Jul 2022 11:21:14 -0300

Changed in openvpn (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for openvpn has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.