netplan network-manager plugin tries to save temporary connections

Bug #1998207 reported by Alfonso Sanchez-Beato
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Netplan
Invalid
Medium
Unassigned
network-manager (Ubuntu)
Fix Released
Medium
Unassigned

Bug Description

*** Note: This bug is mostly about comment #10, now ***

When creating an OpenVPN connection, a temporal connection called tunN is created. For instance, after activating a connection called vpntest, I have:

NAME UUID TYPE DEVICE
vpntest 458856e6-8f0f-4dc6-82f2-dd72868252a0 vpn ens3
tun0 1eb1dbe8-5678-4818-9adf-fb2dc01ed132 tun tun0

tun0 is created/removed after activating/deactivating vpntest and should not really be saved, but I see netplan adding it in /etc/netplan. And while doing so the plugin also reports some errors (I see these when stopping the connection):

Nov 28 16:16:57 ubuntu NetworkManager[11752]: <error> [1669652217.2920] BUG: the profile cannot be stored in keyfile format without becoming unusable: cannot access file: No such file or directory
Nov 28 16:16:57 ubuntu NetworkManager[11752]: ((src/libnm-core-impl/nm-connection.c:342)): assertion '<dropped>' failed
Nov 28 16:16:57 ubuntu NetworkManager[11752]: <warn> [1669652217.2920] keyfile: commit: failure to write 1eb1dbe8-5678-4818-9adf-fb2dc01ed132 ((null)) to "/run/NetworkManager/system-connections/tun0-1eb1dbe8-5678-4818-9adf-fb2dc01ed132.nmconnection": keyfile writer produces an invalid connection: cannot access file: No such file or directory

Tags: nm-netplan
Revision history for this message
Lukas Märdian (slyon) wrote :

ACK. IIRC there's a "nm-generated" flag inside the NM connection profile, which we could check to avoid calling into the libnetplan integration if exists.

So this actually affects NetworkManager's libnetplan integration patch more than Netplan itself, IMO: https://github.com/slyon/NetworkManager/tree/slyon/backend-1.36.6

Changed in netplan:
status: New → Triaged
importance: Undecided → Medium
importance: Medium → High
Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

Right, it is more about the plugin, was not fully sure where to put this, but now probably we can put under network-manager deb.

Revision history for this message
Lukas Märdian (slyon) wrote :
Changed in network-manager (Ubuntu):
status: New → Fix Committed
Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

Unfortunately this is still happening with the patch from comment #3. In fact I do not even need to create a VPN connection for this. If I install easy-openvpn-server:

$ snap install easy-openvpn-server

it creates two tun devices (tun0 and tun1) that can be seen with "ip address" command. That makes NM create the corresponding devices and connections:

/ssh:ubuntu@localhost#8022: $ /snap/bin/network-manager.nmcli c
NAME UUID TYPE DEVICE
netplan-ens3 bec3d02a-c9e5-3283-92ab-ee43a4246c85 ethernet ens3
tun0 edc2d2e1-b126-475d-8d5b-1a5b3bfd43d3 tun tun0
tun1 2945f4f4-2d44-4a61-83b4-cee73fd81129 tun tun1
/ssh:ubuntu@localhost#8022: $ /snap/bin/network-manager.nmcli d
DEVICE TYPE STATE CONNECTION
ens3 ethernet connected netplan-ens3
tun0 tun connected (externally) tun0
tun1 tun connected (externally) tun1
lo loopback unmanaged

but unfortunately these connections are still written as static connections in /etc/netplan/. And when I reboot, I see 4 of them:

/ssh:ubuntu@localhost#8022: $ /snap/bin/network-manager.nmcli c
NAME UUID TYPE DEVICE
netplan-ens3 bec3d02a-c9e5-3283-92ab-ee43a4246c85 ethernet ens3
tun0 864e0de2-e5c6-4c9b-aa7f-20aef970067c tun tun0
tun1 ddfc7db2-2baa-488f-95da-c8e1e66e458b tun tun1
default be2bc22c-fc98-4b2b-9484-fd4d5c838d5f vpn --
tun0 3c1e12f9-1a08-4a6f-aef3-5ea22d9e9bb2 tun --
tun1 eb8ed73b-3312-401f-a4c4-c26aadb3a014 tun --

The files look like https://paste.ubuntu.com/p/KvTVXPwK75/

Lukas Märdian (slyon)
Changed in network-manager (Ubuntu):
status: Fix Committed → Confirmed
Revision history for this message
Lukas Märdian (slyon) wrote :

Hmm.. OK. So I assume those connections would normally (not using the netplan integration) be created in /run/NetworkManager/system-connections/...?

So I think we need to find a way to detect NMS_KEYFILE_STORAGE_TYPE_RUN, or respectively ignore any connections that are not of NMS_KEYFILE_STORAGE_TYPE_ETC, but that data is not directly available in the nms-keyfile-writer.c code. So it needs some more digging.

tags: added: netplan-nm
tags: added: nm-netplan
removed: netplan-nm
Revision history for this message
Lukas Märdian (slyon) wrote :

Actually, https://developer-old.gnome.org/NetworkManager/stable/nm-dbus-types.html#NMSettingsConnectionFlags suggests this could possibly be avoided, by checking for `!is_external`, too.

Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote (last edit ):

Changing the check as suggested to:

if (!is_volatile && !is_nm_generated && !is_external) {
...

does help and netplan files for tun0 and tun1 are not written anymore. But, when I create a VPN connection I still have problems.

$ network-manager.nmcli c import type openvpn file <open_vpn_cfg>
$ network-manager.nmcli c up <vpn_conn>
$ network-manager.nmcli c
NAME UUID TYPE DEVICE
<vpn_conn> 7b7fd99d-2651-4796-ba50-beda0890aab9 vpn ens3
tun0 1d7b454c-4897-4d3d-899a-619165a996bf tun tun0
tun1 f7b9ba1f-496b-4cca-ae79-31db28a64c29 tun tun1
tun2 46935370-3662-4aac-b563-877214b48cd8 tun tun2
netplan-ens3 bec3d02a-c9e5-3283-92ab-ee43a4246c85 ethernet ens3
$ sudo /snap/bin/network-manager.nmcli d
DEVICE TYPE STATE CONNECTION
tun0 tun connected (externally) tun0
tun1 tun connected (externally) tun1
tun2 tun connected (externally) tun2
ens3 ethernet connected netplan-ens3
lo loopback unmanaged --
$ network-manager.nmcli c down default
$ network-manager.nmcli c
NAME UUID TYPE DEVICE
netplan-ens3 bec3d02a-c9e5-3283-92ab-ee43a4246c85 ethernet ens3
tun0 8bc8d5b1-6000-4b78-9988-72d2d811bfb7 tun tun0
tun1 9d614be8-dac1-44ef-acfd-f9c60005b56f tun tun1
default 7b7fd99d-2651-4796-ba50-beda0890aab9 vpn --
$ ls /etc/netplan/
00-default-nm-renderer.yaml
50-cloud-init.yaml
90-NM-7b7fd99d-2651-4796-ba50-beda0890aab9.yaml
90-NM-46935370-3662-4aac-b563-877214b48cd8

Note that when you create an OpenVPN connection both a normal NM connection and a new tunnel device (tun2) are created. NM creates a external device for tun2. The interesting thing here is that the file for tun2 is written after setting down the connection, and the NM daemon actually forgets it (until you restart it and reads the netplan file). Maybe this storing is in a different code path.

Revision history for this message
Lukas Märdian (slyon) wrote :

Thanks for your testing and detailed analysis!

So IIUC the connection works as expected, but only after reloading the connection profiles; it doesn't show up at the time it is expected, right?

Does running 'nmcli connection reload' after the connection profile for the VPN connection (tun2?) got written, make it show up in the NM daemon?

Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

> So IIUC the connection works as expected, but only after reloading the connection profiles; it doesn't show up at the time it is expected, right?

Hm, not sure if we mean the same. That two connections (tun2 and <vpn_conn>) appear after connecting to the VPN is expected. NM recognizes tun2 as "external" as it is an auxiliary device/connection created by the OpenVPN plugin. The problem that is happening is that that connection is being stored in /etc/netplan/, which should not be the case.

Note that tun0 and tun1 are slightly different as those are created by the OpenVPN server also running on the device. The netplan files for them are not written anymore after changing the patch.

> Does running 'nmcli connection reload' after the connection profile for the VPN connection (tun2?) got written, make it show up in the NM daemon?

No, that does not happen. But it happens if I reboot the system. I think that is because there is some file in /run that prevents this from happening if just reloading things.

Before rebooting:
$ sudo ls run/NetworkManager/system-connections/ -l
total 20
lrwxrwxrwx 1 root root 9 Jan 10 11:22 891164a5-1fed-42b8-8f6e-903db6396d5e.nmmeta -> /dev/null
-rw------- 1 root root 403 Jan 10 11:24 netplan-NM-891164a5-1fed-42b8-8f6e-903db6396d5e.nmconnection
-rw------- 1 root root 1248 Jan 10 11:24 netplan-NM-af486148-2495-48a9-9704-2a1230e97e32.nmconnection
-rw------- 1 root root 131 Jan 10 11:24 netplan-ens3.nmconnection
-rw------- 1 root root 336 Jan 10 09:37 tun0.nmconnection
-rw------- 1 root root 336 Jan 10 09:37 tun1.nmconnection

After rebooting:
$ sudo ls run/NetworkManager/system-connections/ -l
total 20
-rw------- 1 root root 403 Jan 10 11:28 netplan-NM-891164a5-1fed-42b8-8f6e-903db6396d5e.nmconnection
-rw------- 1 root root 1248 Jan 10 11:28 netplan-NM-af486148-2495-48a9-9704-2a1230e97e32.nmconnection
-rw------- 1 root root 131 Jan 10 11:28 netplan-ens3.nmconnection
-rw------- 1 root root 336 Jan 10 11:28 tun0.nmconnection
-rw------- 1 root root 336 Jan 10 11:28 tun1.nmconnection

891164a5 was tun2 and af486148 the VPN connection.

Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

To reproduce the problem, this should be enough:

1. Create a connection to an OpenVPN server
2. Start the connection. The OpenVPN plugin will create a tunnel interface.
3. "nmcli c" should show a tunX connection and the VPN connection. "nmcli d"
   should show tunX as a external connection.
4. Disconnect the VPN connection (nmcli c down <vpn_conn>)
5. A file in /etc/netplan/ for the tunX which should not be there is created

Lukas Märdian (slyon)
description: updated
Revision history for this message
Lukas Märdian (slyon) wrote (last edit ):

Thanks! I've added the "!is_external" check to my latest development patch revision at: https://github.com/slyon/NetworkManager/commits/netplan-nm-1.40

So this bug report is now mostly about comment #10, where the openvpn client/plugin is creating a temporary tunX device, which is displayed as "external" in "nmcli d", but still a Netplan YAML file is created in /etc/netplan on "nmcli con down <VPN>". I can easily reproduce this issue locally, using Canonical VPN and see tunX connection definitions piling up in /etc/netplan/ when bringing it up&down a few times in a row.

This doesn't seem to affect the functionality, though. The VPN is connection is working when up in each iteration.

Changed in network-manager (Ubuntu):
status: Confirmed → Triaged
importance: Undecided → Medium
Changed in netplan:
importance: High → Medium
Revision history for this message
Danilo Egea Gondolfo (danilogondolfo) wrote :

The same problem happens if you spin up an OpenVPN server and kill it. An yaml file will be created for the tun interface used by the server.

Revision history for this message
Danilo Egea Gondolfo (danilogondolfo) wrote :

After taking another look at this I noticed that the function where we create the Netplan YAML is called when the connection is created AND when the connection is going down as though it was being updated.

The first time it's called the parameters is_volatile, is_nm_generated and is_external are all set to true so we skip the libnetplan calls. Although when it's called when the connection is going down they are all false. And that's when the Netplan file is created and network-manager gets to a point in the code it shouldn't reach, so it crashes and the files are never removed.

Not sure if calling the plugin writer in this case is a bug or not as the connection is volatile and is going to be deleted anyway. Maybe lost tracking of the connection nature (volatile, nm_generated and external) is the actual bug, not sure yet.

One way to work around this is generating Netplan YAMLs regardless if the connection is volatile. If network-manager will create a file for it we could create a file for Netplan too. In fact, after removing the conditions to call libnetplan, the files are created and deleted when the OpenVPN client and/or server is stopped.

Note that from network-manager 1.42.0 on, even the loopback interface can be represented as a connection, so we will have a Netplan file for it as well.

Revision history for this message
Lukas Märdian (slyon) wrote :

I think this should probably fix the duplicated temporary connections issue. Netplan should only ever try to handle non-temporary connections, nothing that's being written to NMS_KEYFILE_STORAGE_TYPE_RUN (like those external OpenVPN connections). More testing is needed to investigate the side-effects of this change.

```
diff --git a/src/core/settings/plugins/keyfile/nms-keyfile-writer.c b/src/core/settings/plugins/keyfile/nms-keyfile-writer.c
index b893717c20..65e4e857ec 100644
--- a/src/core/settings/plugins/keyfile/nms-keyfile-writer.c
+++ b/src/core/settings/plugins/keyfile/nms-keyfile-writer.c
@@ -418,7 +418,8 @@ _internal_write_connection(NMConnection *connection,
         unlink(existing_path);

     /* NETPLAN: write only non-temporary files to /etc/netplan/... */
- if (!is_volatile && !is_nm_generated && !is_external) {
+ if (!is_volatile && !is_nm_generated && !is_external &&
+ strstr(keyfile_dir, "/etc/NetworkManager/system-connections")) {
         g_autofree gchar *ssid = g_key_file_get_string(kf_file, "wifi", "ssid", NULL);
         g_autofree gchar *escaped_ssid = ssid ?
                                          g_uri_escape_string(ssid, NULL, TRUE) : NULL;
```

Lukas Märdian (slyon)
Changed in netplan:
status: Triaged → Invalid
Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

@slyon thanks, I have manually tested and seems to work. I've created https://github.com/snapcore/network-manager-snap/pull/15, let's see how tests go.

Revision history for this message
Lukas Märdian (slyon) wrote :

Thanks for your testing and confirmation! Looks like the spread test (GitHub action passed, too).

I've also confirmed our autopkgtest is now passing, using this source package: https://git.launchpad.net/network-manager/log/?h=netplan/lunar-gu

I confirmed our (previously failing) autopkgtest passes now for the OpenVPN duplicate connections fix: https://paste.ubuntu.com/p/FWTVxbjzC6/ (especially the TestNetplan.test_openvpn_connection case)

So this will soon land in the "Netplan Everywhere" PPA: https://launchpad.net/~canonical-foundations/+archive/ubuntu/networkmanager-netplan

Changed in network-manager (Ubuntu):
status: Triaged → Fix Committed
Lukas Märdian (slyon)
Changed in network-manager (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.