OpenVPN server does not start properly on boot

Thank you for reporting this bug. I will add it to the server team backlog, as it does appear to be a real issue.

I finally got to confirm this. I had to install ntp to see the same problem in my vm. Before that, openvpn was always running after a reboot.

The moment I installed ntp and rebooted, I got the issue:
Mon Nov 20 19:17:41 2017 TCP/UDP: Socket bind failed on local address [AF_INET]10.0.6.10:1194: Cannot assign requested address
Mon Nov 20 19:17:41 2017 Exiting due to fatal error

root@04-57:~# systemctl status <email address hidden>
● <email address hidden> - OpenVPN connection to server
   Loaded: loaded (/lib/systemd/system/openvpn@.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2017-11-20 19:17:41 UTC; 1min 30s ago
     Docs: man:openvpn(8)
           https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
           https://community.openvpn.net/openvpn/wiki/HOWTO
  Process: 577 ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid (code=exited, sta
 Main PID: 612 (code=exited, status=1/FAILURE)

Nov 20 19:17:41 04-57 systemd[1]: Starting OpenVPN connection to server...
Nov 20 19:17:41 04-57 systemd[1]: Started OpenVPN connection to server.
Nov 20 19:17:41 04-57 systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
Nov 20 19:17:41 04-57 systemd[1]: <email address hidden>: Unit entered failed state.
Nov 20 19:17:41 04-57 systemd[1]: <email address hidden>: Failed with result 'exit-code'.
lines 1-14/14 (END)

A possible workaround would be to add "Restart=on-failure" in the "[Service]" section of the systemd unit.

Found these other bugs which look related:

https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1598522

https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356 (marked as dup of the above)

> My guess is that systemd starts OpenVPN too early before the network is brought
> up sufficiently. Running 'sudo systemctl edit --full openvpn' and adding
> 'Wants=network-online.target' does not change that behaviour.

I tried this to test:
[Unit]
(...)
After=syslog.target network-online.target
Wants=network-online.target

It then worked. Can you give that a shot?

Upstream adopted a mixed approach and split the services into a server and client part, and adopted different after/wants values for each (see https://github.com/OpenVPN/openvpn/commit/28bd79ac980488dbfce2e8136287e38c6f35a043)

I don't think we can take an approach as the above for a stable release update, though.

In fact, our current openvpn package in bionic is using just these After and Wants values:
After=network-online.target
Wants=network-online.target

https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ is a good read about this

I tried your suggested settings on fresh installs including all updates but this still regularly fails to bind the address. I have those servers running as virtual machines with Qemu/KVM.
If I do the same thing on bare metal, the binding will succeed in like 95% of boots.

/var/log/openvpn.log:
Tue Apr 10 11:40:44 2018 OpenVPN 2.3.10 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Jun 22 2017
Tue Apr 10 11:40:44 2018 library versions: OpenSSL 1.0.2g 1 Mar 2016, LZO 2.08
Tue Apr 10 11:40:44 2018 Diffie-Hellman initialized with 2048 bit key
Tue Apr 10 11:40:44 2018 Control Channel Authentication: using './easy-rsa/keys/ta.key' as a OpenVPN static key file
Tue Apr 10 11:40:44 2018 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Tue Apr 10 11:40:44 2018 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Tue Apr 10 11:40:44 2018 Socket Buffers: R=[212992->212992] S=[212992->212992]
Tue Apr 10 11:40:44 2018 TCP/UDP: Socket bind failed on local address [AF_INET]192.168.11.254:1194: Cannot assign requested address
Tue Apr 10 11:40:44 2018 Exiting due to fatal error

@jnx, the 192.168.11.254 address you are trying to bind to, is that the normal nic of that machine? Or something else, like a wifi interface, or something that would only come up after the user logs in, assuming this is a desktop?

It's the normal builtin NIC on a headless server. Tried with different hardware as well:

 *-network
       description: Ethernet interface
       product: 82574L Gigabit Network Connection
       vendor: Intel Corporation

 *-network
       description: Ethernet interface
       product: NetXtreme BCM5720 Gigabit Ethernet PCIe
       vendor: Broadcom Corporation

 *-network
       description: Ethernet interface
       product: Virtio network device
       vendor: Red Hat, Inc

Also affected by this bug, fresh Ubuntu Ubuntu 16.04.6 LTS minimal server install + openvpn

/var/log/openvpn-XXXX.log:

Sun Mar 17 14:45:59 2019 us=746742 TCP/UDP: Socket bind failed on local address [AF_INET]XXX.XXX.XXX.131:1094: Cannot assign requested address
Sun Mar 17 14:45:59 2019 us=746750 Exiting due to fatal error

Where XXX.XXX.XXX.131 is an IP aliased to eth0 (aliased eth0:1) in /etc/network/interfaces:

auto eth0:1
allow-hotplug eth0:1
iface eth0:1 inet static
  address XXX.XXX.XXX.131
  netmask 255.255.255.224

None of the workarounds listed in this bug or here https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ work - occurs 100% every boot.

I can repeat this reliably on a fresh minimal install + openvpn + alias on primary interface

Seems upstream kindly solved this bug, the solution is to:

In /etc/default/openvpn

AUTOSTART="none"

In /etc/network/interfaces

iface ethX inet static
  address XXX.XXX.XXX.XX
  netmask 255.255.255.0
  openvpn my_vpn_name

Where ethX is your interface (ex, eth0 for a regular interface, eth0:1 for an alias of an interface), my_vpn_name is the base filename without the .conf suffix of your /etc/openvpn/my_vpn_name.conf file.

To explain, the above causes the script /etc/network/if-up.d/openvpn to get executed, this in turn executes systemctl --no-block start openvpn@my_vpn_name which starts the openvpn after the interface or alias is brought up.

Do you have a pointer to the upstream discussion?

Note: the if the solution needs an ifupdown hook like shown in comment #12 in >=Bionic I'd ask you to take a look for networkd-dispatch (the final solution for almost all of these bugs is that the upstream project starts to listen o netlink to pick up late ready IP addresses)

FYI: There seem to be a whole class of issues like this so Rbasak added it also to a card that tracks related work.