[SRU] OpenVPN will not reload due to misconfigured .service file

(PS: issue also exists in 2.4.7-1ubuntu2)

The "ExecReload=+/bin/kill" way of reloading without needing extra caps seems sensible. That said, I'm wondering what's the use case for a reload instead of a restart as man openvpn(8) describes what happens on SIGHUP:

SIGNALS
  SIGHUP Cause OpenVPN to close all TUN/TAP and network connections, restart, re-read the
  configuration file (if any), and reopen TUN/TAP and network connections.

I'm wondering how it is possible when OpenVPN downgraded UID/GID? Maybe it works when using the openvpn-plugin-down-root.so plugin?

That is a very good point. I never noticed it but it looks like my OpenVPN is actually failing to reload, exiting, and being restarted by systemd. It looks like I'll need to modify my 'overlay' service to use SIGUSR1 instead.

Executing the ExecReload= command with full privileges (adding '+') indeed fixes the reload failure (I ran my tests in a Bionic and Focal VMs but it should apply to the other releases):

May 26 10:01:41 openvpn-reload kill[1764]: kill: (1738): Operation not permitted

I mean the command 'systemctl reload openvpn@<server>' does not fail (returns 0). However, after checking the journal log I found the following error messages:

May 26 10:30:57 openvpn-reload ovpn-server[10626]: SIGHUP[hard,] received, process restarting
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --dh fails with 'dh.pem': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --ca fails with 'ca.crt': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --cert fails with 'server.crt': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --key fails with 'server.key': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --tls-auth fails with 'ta.key': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --writepid fails with '/run/openvpn/server.pid': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --status fails with '/var/log/openvpn/openvpn-status.log': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: Please correct these errors.
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Use --help for more information.

As discussed above I am not quite sure about the practical difference between restarting the service and sending the SIGHUP signal to the process. The upstream message when handling this signal is "process restarting". Since defining a reload command is optional and the difference between restart and reload is not clear, we may want to remove the ExecReload= from this unit file. Maybe run the ExecReload= with full privileges ('+') in the supported releases to not remove this feature and avoid the failure, and remove the ExecReload= from the development release (stop supporting reload)?

I'm not sure there's even any point to adding the '+': an error will occur
either way, whether it's in kill or in OpenVPN. At least when it appears
from kill it's obvious something went wrong (even if it's not obvious
what...).

Personally I think removing reload is the right call going forward - having
perused the man page more thoroughly, neither SIGHUP nor SIGUSR1 accomplish
what I would call a 'reload'. But that's just my opinion.

On Tue, May 26, 2020, 14:00 Lucas Kanashiro <email address hidden>
wrote:

> Executing the ExecReload= command with full privileges (adding '+')
> indeed fixes the reload failure (I ran my tests in a Bionic and Focal
> VMs but it should apply to the other releases):
>
> May 26 10:01:41 openvpn-reload kill[1764]: kill: (1738): Operation not
> permitted
>
> I mean the command 'systemctl reload openvpn@<server>' does not fail
> (returns 0). However, after checking the journal log I found the
> following error messages:
>
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: SIGHUP[hard,] received,
> process restarting
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --dh
> fails with 'dh.pem': Permission denied (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --ca
> fails with 'ca.crt': Permission denied (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --cert
> fails with 'server.crt': Permission denied (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --key
> fails with 'server.key': Permission denied (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error:
> --tls-auth fails with 'ta.key': Permission denied (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error:
> --writepid fails with '/run/openvpn/server.pid': Permission denied
> (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --status
> fails with '/var/log/openvpn/openvpn-status.log': Permission denied
> (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: Please
> correct these errors.
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Use --help for more
> information.
>
> As discussed above I am not quite sure about the practical difference
> between restarting the service and sending the SIGHUP signal to the
> process. The upstream message when handling this signal is "process
> restarting". Since defining a reload command is optional and the
> difference between restart and reload is not clear, we may want to
> remove the ExecReload= from this unit file. Maybe run the ExecReload=
> with full privileges ('+') in the supported releases to not remove this
> feature and avoid the failure, and remove the ExecReload= from the
> development release (stop supporting reload)?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1868127
>
> Title:
> OpenVPN will not reload due to misconfigured .service file
>
> Status in openvpn package in Ubuntu:
> Triaged
> Status in openvpn source package in Bionic:
> Triaged
> Status in openvpn source package in Eoan:
> Triaged
> Sta...

I proposed to add the '+' in the supported releases to avoid a failure while users try to reload the service:

$ sudo systemctl reload openvpn@server
Job for <email address hidden> failed because the control process exited with error code.
See "systemctl status <email address hidden>" and "journalctl -xe" for details.
$ echo $?
1
$ sudo systemctl status openvpn@server
● <email address hidden> - OpenVPN connection to server
   Loaded: loaded (/lib/systemd/system/openvpn@.service; indirect; vendor preset: enabled)
   Active: active (running) (Result: exit-code) since Tue 2020-05-26 17:11:33 -03; 24s ago
     Docs: man:openvpn(8)
           https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
           https://community.openvpn.net/openvpn/wiki/HOWTO
  Process: 9548 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=1/FAILURE)
 Main PID: 9528 (openvpn)
   Status: "Initialization Sequence Completed"
    Tasks: 1 (limit: 1152)
   CGroup: /<email address hidden>
           └─9528 /usr/sbin/openvpn --daemon ovpn-server --status /run/openvpn/server.status 10

May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: GID set to nogroup
May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: UID set to nobody
May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: MULTI: multi_init called, r=256 v=256
May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: IFCONFIG POOL: base=10.8.0.4 size=62,
May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: IFCONFIG POOL LIST
May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: Initialization Sequence Completed
May 26 17:11:49 openvpn-reload-bionic systemd[1]: Reloading OpenVPN connection to server.
May 26 17:11:49 openvpn-reload-bionic kill[9548]: kill: (9528): Operation not permitted
May 26 17:11:49 openvpn-reload-bionic systemd[1]: <email address hidden>: Control process exite
May 26 17:11:49 openvpn-reload-bionic systemd[1]: Reload failed for OpenVPN connection to serve

This error (it returns 1) might confuse users. However, as you can see above, even when the reload command fails the service itself remains active and running. The same happens if we add the '+', but the reload command succeeds (returns 0).

Anyway, I'll prepare a MP to drop the reload support in Groovy while we decide if it worth to SRU a "fix" for this to the supported releases.

After a discussion with a colleague I think we should leave it as is in the supported/stable releases (I already fixed it in Groovy, it should land in the archive soon). The rationale is: even if the reload command fails the service keeps active and running, the proposed solutions would not fix the problem but would hide it from users. Adding the CAP_KILL capability or the '+' prefix would allow the service to send the SIGHUP signal and make 'systemctl reload openvpn@<server>' return 0, however, it would still face some permission denied errors, making the reload fails silently. In short, with or without the proposed fixes the reload process will not succeed.

I'd like to mention that `systemctl $action openvpn@$foo` is using the legacy systemd unit.

On 18.04 and newer, you are better off moving your config to /etc/openvpn/client or /etc/openvpn/server and use corresponding and modern systemd units: openvpn-client@$foo and openvpn-server@$foo

This bug was fixed in the package openvpn - 2.4.9-2ubuntu2

---------------
openvpn (2.4.9-2ubuntu2) groovy; urgency=medium

  * Drop reload support from systemd unit files (LP: #1868127)

 -- Lucas Kanashiro <email address hidden> Tue, 26 May 2020 19:04:33 -0300

I brought more people into this discussion and we decided to drop the reload support from the stable/supported releases since it is already broken and the restart does basically the same thing.

Hi Lucas,

> ...we decided to drop the reload support from the stable/supported releases since it is already broken and the restart does basically the same thing.

I'm puzzled by this. What effect will dropping the reload support have?

Is it that it previously failed, and with this SRU it will continue to fail but for a different reason? If so, what's the benefit to users for landing this SRU?

Is it that it previously failed, and with this SRU it will now succeed but with a more appropriate behaviour? If so, please could you document this?

Hi Robie,

With the proposed change users will not get the error mentioned in the description while trying to reload the service and get a better error (reload not supported). The current status is:

1 - User tries to reload the service
2 - Reload fails
3 - User gets an error

The problem here is that the service keeps active and running, so some users might infer that everything is fine and the reload process might have worked. But actually it did not.

With the proposed change we will drop the reload support so users will know that it did not do what they were expecting. I hope this make sense for you.

OK, thanks. Let's accept these into the proposed pockets, but defer updating users until there a more significant update needed - then this change can be bundled in.

Hello John, or anyone else affected,

Accepted openvpn into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.4.7-1ubuntu2.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello John, or anyone else affected,

Accepted openvpn into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.4.7-1ubuntu2.19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello John, or anyone else affected,

Accepted openvpn into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.4.4-2ubuntu1.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Anyone willing to do the verification work?

Reproduction procedure followed for all verification:

1) initial problem:
 apt install --no-install-recommends -y openvpn

 touch /etc/openvpn/lp1868127.conf
 chmod 0600 /etc/openvpn/lp1868127.conf
 cat << EOF > /etc/openvpn/lp1868127.conf
 ...
 EOF
 systemctl start openvpn@lp1868127
 systemctl reload openvpn@lp1868127

 # observe reload failing

2) update:

 # deploy -proposed update
 echo "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main" >> /etc/apt/sources.list
 apt update && apt install openvpn

3) confirm fix:
 systemctl reload openvpn@lp1868127

This last step must print:

> Failed to reload <email address hidden>: Job type reload is not applicable for unit <email address hidden>.

Focal:

1) initial problem:

root@focal-openvpn-lp1868127:~# systemctl reload openvpn@lp1868127
Job for <email address hidden> failed.
See "systemctl status <email address hidden>" and "journalctl -xe" for details.

2) update:

root@focal-openvpn-lp1868127:~# apt update && apt install openvpn
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [111 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [107 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-proposed InRelease [265 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [310 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [116 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [28.9 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [7,664 B]
Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [142 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [71.7 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [11.6 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [3,892 B]
Get:13 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages [37.3 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-proposed/main Translation-en [17.6 kB]
Get:15 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [147 kB]
Get:16 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [51.1 kB]
Get:17 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [28.9 kB]
Get:18 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [7,664 B]
Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [42.8 kB]
Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [22.6 kB]
Fetched 1,530 kB in 1s (1,311 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
23 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  resolvconf openvpn-systemd-resolved easy-rsa
The following packages will be upgraded:
  openvpn
1 upgraded, 0 newly installed, 0 to...

The verification of the Stable Release Update for openvpn has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package openvpn - 2.4.7-1ubuntu2.19.10.1

---------------
openvpn (2.4.7-1ubuntu2.19.10.1) eoan; urgency=medium

  * Drop reload support from systemd unit files (LP: #1868127)

 -- Lucas Kanashiro <email address hidden> Wed, 27 May 2020 19:25:32 -0300

Untagged server-next since this bug is now in its goal state (fixes staged in stable releases)

This bug was fixed in the package openvpn - 2.4.4-2ubuntu1.5

---------------
openvpn (2.4.4-2ubuntu1.5) bionic-security; urgency=medium

  * SECURITY UPDATE: data channel v2 packet injection
    - debian/patches/CVE-2020-11810.patch: fix illegal client float in
      src/openvpn/multi.c.
    - CVE-2020-11810
  * SECURITY UPDATE: Authentication bypass with deferred authentication
    - debian/patches/CVE-2020-15078.patch: ensure key state is
      authenticated before sending push reply in src/openvpn/push.c.
    - CVE-2020-15078

 -- Marc Deslauriers <email address hidden> Tue, 27 Apr 2021 10:54:29 -0400

It looks like the fix for this in Focal landed together with the security update in 2.4.7-1ubuntu2.20.04.2 (2.4.7-1ubuntu2.20.04.1 doesn't look like it ever got copied to focal-updates). I confirmed the patch is still present in the current version in Focal (2.4.7-1ubuntu2.20.04.3).