[SRU] SEGFAULT on upgrade to 0.102-0ubuntu1~20.04.1

Hello Carl,

thank you very much for reporting this issue. Do you have a crash report available for this by any chance (in /var/crash/), which you could provide by using 'ubuntu-bug netplan.io'?

Hey Carl, Lukas!

I pulled netplan.io 0.102 from -updates back to -proposed until we get a better understanding of what's going on.

Hello Carl,

we have not been able to reproduce the problem locally and on AWS. Did this problem happen on all of your machines, or only a few?

Could you please provide the /etc/netplan/*.yaml configuration of an affected machine? And possibly describe a way of how to reproduce the crash?

Thanks!

Attaching a crash dump. Should I still submit a new bug report with `ubuntu-bug`?

The only file in /etc/netplan:

$ cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        ens5:
            dhcp4: true
            dhcp6: false
            match:
                macaddress: 06:f8:32:e5:34:28
            set-name: ens5
    version: 2

It happens to hundreds of our servers, wrecking quite a havoc this morning, we `apt-mark hold netplan.io` on the rest before they crashed too.

Thank you very much for that information.
Yes, please also submit a new bug using 'ubuntu-bug' this should collect quite some contextual information about the system, so we can better understand what is going on.
We can mark the new bug as a duplicate of this one then.

Ok, filed here: https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/1922935

From what i can see all servers that updated netplan, netplan crashed with SEGFAULT, but not all lost their IP and had to be rebooted. Looks like systemd was reloaded as part of the unattened-upgrade, and both systemd-networkd and netplan (among other) was restarted as part of that, but it seems to vary which restarted first. If netplan restarted first and thus seg faulted, systemd-networkd then released the IP but didn't aquire a new one. But on some servers netplan was restarted after systemd-networkd and then the network connectivity wasn't affected.

re Comment #3 - at least for s390x, 0.101-0ubuntu3~20.04.2 was not restored to focal-updates. This leaves only 0.99-0ubuntu1 in focal ... a major regression.

netplan segfaulted on all our GCP servers too, but there the IP wasn't release immediately, but when the DHCP lease expired, which is starts to happen right now, they don't acquire a new one and loose connectivity too and have to be rebooted.

This morning it starts to happen again, even though netplan.io is on hold.

What's even worse this time is that not even a reboot seems to make the machine acquire a new IP!

Apr 08 06:00:55 spirited-ibis-01 unattended-upgrade Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security, origin=*
Apr 08 06:00:55 spirited-ibis-01 unattended-upgrade Initial blacklist: erlang-.* rabbitmq-server
Apr 08 06:00:55 spirited-ibis-01 unattended-upgrade Initial whitelist (not strict):
Apr 08 06:01:08 spirited-ibis-01 unattended-upgrade Packages that will be upgraded: alsa-ucm-conf apt apt-utils libapt-pkg6.0 libnetplan0 libnss-systemd libpam-systemd libsystemd0 libudev1 open-vm-tools systemd systemd-sysv systemd-timesyncd udev
Apr 08 06:01:08 spirited-ibis-01 unattended-upgrade Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
Apr 08 06:01:11 spirited-ibis-01 dbus-daemon [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.5936' (uid=0 pid=2154232 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Apr 08 06:01:11 spirited-ibis-01 systemd Starting PackageKit Daemon...
Apr 08 06:01:11 spirited-ibis-01 PackageKit daemon start
Apr 08 06:01:12 spirited-ibis-01 dbus-daemon [system] Successfully activated service 'org.freedesktop.PackageKit'
Apr 08 06:01:12 spirited-ibis-01 systemd Started PackageKit Daemon.
Apr 08 06:01:21 spirited-ibis-01 systemd Reloading.
Apr 08 06:01:21 spirited-ibis-01 dbus-daemon [system] Reloaded configuration
Apr 08 06:01:21 spirited-ibis-01 kernel [25635194.817292] netplan[2154341]: segfault at 100000000 ip 00007f2924b06675 sp 00007ffcae2f4a28 error 4 in libc-2.31.so[7f29249a0000+178000]
Apr 08 06:01:21 spirited-ibis-01 kernel [25635194.817299] Code: 00 00 0f 1f 00 31 c0 c5 f8 77 c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 89 f9 48 89 fa c5 f9 ef c0 83 e1 3f 83 f9 20 77 2b <c5> fd 74 0f c5 fd d7 c1 85 c0 0f 85 eb 00 00 00 48 83 c7 20 83 e1
Apr 08 06:01:21 spirited-ibis-01 systemd /usr/lib/systemd/system-generators/netplan terminated by signal SEGV.
Apr 08 06:01:21 spirited-ibis-01 systemd /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
Apr 08 06:01:21 spirited-ibis-01 systemd /lib/systemd/system/nginx.service:9: PIDFile= references a path below legacy directory /var/run/, updating /var/run/nginx.pid → /run/nginx.pid; please update the unit file accordingly.
Apr 08 06:01:21 spirited-ibis-01 systemd lvm2-lvmpolld.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.
Apr 08 06:01:21 spirited-ibis-01 systemd iscsid.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.
Apr 08 06:01:21 spirited-ibis-01 dbus-daemon [system] Reloaded configuration
Apr 08 06:01:23 sp...

Here's an strace log from when netplan generate seg faults

Oh, it seems to happen when there's a version mismatch between libnetplan0 and netplan.io. And unattended-upgrade normally only upgrade one package at the time:

Log started: 2021-04-07 06:58:28
Preparing to unpack .../libnetplan0_0.102-0ubuntu1~20.04.1_arm64.deb ...
Unpacking libnetplan0:arm64 (0.102-0ubuntu1~20.04.1) over (0.101-0ubuntu3~20.04.2) ...
Setting up libnetplan0:arm64 (0.102-0ubuntu1~20.04.1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Log ended: 2021-04-07 06:58:34

Apr 7 06:58:38 smart-gold-dragonfly-02 systemd[2226429]: /usr/lib/systemd/system-generators/netplan terminated by signal SEGV.

Log started: 2021-04-07 06:59:00
Preparing to unpack .../netplan.io_0.102-0ubuntu1~20.04.1_arm64.deb ...
Unpacking netplan.io (0.102-0ubuntu1~20.04.1) over (0.101-0ubuntu3~20.04.2) ...
Setting up netplan.io (0.102-0ubuntu1~20.04.1) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for dbus (1.12.16-2ubuntu2.1) ...
Log ended: 2021-04-07 06:59:05

Unattended-Upgrade::MinimalSteps "false";

will probably avoid this situation, but "true" is the default.

And right now it seems that because you pulled 0.102 but not libnetplan0 0.102 we can't get out of the version mismatch, without manually copying and installing the deb packages of course.

That's interesting!
Having libnetplan0 and netplan.io upgraded independently is definitely something I can dig deeper into, which could also cause a segfault.

But if you upgrade the system completely (e.g. via 'apt upgrade') netplan 0.102 still works, right?
Like, you can execute 'netplan generate' or more specifically (as a systemd-generator):
/usr/lib/systemd/system-generators/netplan /run/systemd/generator /run/systemd/generator.early /run/systemd/generator.late

(after removing '/run/systemd/generator/netplan.stamp')
As this is basically what happens when the upgrade executes 'systemctl daemon-reload' or the system reboots.

There might still be some propagation going on on the mirrors, but we've rolled back focal-updates to: 0.101-0ubuntu3~20.04.2 (for netplan.io and libnetplan0) and 0.102-0ubuntu1~20.04.1 is available in focal-proposed (for both packages).

Okay, I've now been able to reproduce the problem in a Focal LXD container.

Steps to reproduce:
1) Have netplan 0.101 installed
2) Upgrade ONLY libnetplan0 to 0.102
3) remove
4) Execute '/usr/lib/systemd/system-generators/netplan /run/systemd/generator /run/systemd/generator.early /run/systemd/generator.late'

root@ff:~# dpkg -l | grep netplan
ii libnetplan0:amd64 0.102-0ubuntu1~20.04.1 amd64 YAML network configuration abstraction runtime library
ii netplan.io 0.101-0ubuntu3~20.04.2 amd64 YAML network configuration abstraction for various backends
root@ff:~# rm /run/systemd/generator/netplan.stamp
root@ff:~# /usr/lib/systemd/system-generators/netplan /run/systemd/generator /run/systemd/generator.early /run/systemd/generator.late
Segmentation fault (core dumped)

Ok, so to summarize what happened here was that we've modified unattended-upgrade with:

Unattended-Upgrade::Origins-Pattern {
    "origin=*";
};

instead of the default:

Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}";
        "${distro_id}:${distro_codename}-security";
        "${distro_id}ESMApps:${distro_codename}-apps-security";
        "${distro_id}ESM:${distro_codename}-infra-security";
};

So we also unattendently upgrade packages from focal-packages.

And because unattended-upgrade default to Unattended-Upgrade::MinimalSteps "true"; then libnetplan0 was upgraded before and independetly from netplan.io, and the upgrade process was running systemctl daemon-reload in between, which caused a segfault in netplan generate, which caused our network interface to loose its IP and didn't acquire a new one.

correction: "So we also unattendently upgrade packages from focal-updates."

So is that not recommended or even advised against? Or was this a bug? Should libnetplan0 and netplan.io have a harder dependencies so that they're upgraded together even when `Unattended-Upgrade::MinimalSteps "true";`? Or this something we should expect can happen?

netplan.io should depend on libnetplan0 with the same binary version until library versioning is not maintained (kept at 0.0.0).
https://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html

Status changed to 'Confirmed' because the bug affects multiple users.

Same here:
Many machines with unattended-upgrade (UU). Some Focals upgraded both netplan.io & libnetplan0 in tandem to 0.102, others left both packages at 0.101, probably depending on when they ran UU and what state the APT mirrors where at at that time. All those machines were fine.

One machine however upgraded only libnetplan0 to 0.102 and left netplan.io at 0.101, which broke netplan (segfault) and rendered the machine offline. Installing netplan.io 0.102 manually from focal-proposed fixed it.

I wonder whether retracting 0.102 from focal-updates might have actually worsened the situation and left even more machines with different versions of netplan.io & libnetplan0. It definitely made it harder to rectify.

Is it adviced to use

   Unattended-Upgrade::MinimalSteps "false";

or would that have its own pitfalls?

This bug was fixed in the package netplan.io - 0.102-0ubuntu2

---------------
netplan.io (0.102-0ubuntu2) hirsute; urgency=medium

  * gbp.conf: use ubuntu/<version> tags
  * control: strict libnetplan dependency on binary version (LP: #1922898)

 -- Lukas Märdian <email address hidden> Wed, 14 Apr 2021 11:35:12 +0200

Defining the netplan.io -> libnetplan0 dependency as "libnetplan0 (= ${binary:Version})" makes it work for this specific "unattended-upgrade case", but unfortunately the regression is more generic than that... There are some other consumers of libnetplan0, such as the NetworkManager snap on Ubuntu Core 20, which will fail in the same way if the new libnetplan0 v0.102 is in place.

I've used git-bisect to find the relevant commit and even found TWO of them:
1) https://github.com/canonical/netplan/commit/6c8ed65df7c7f31280d5d27b67195a1e9a746e7a
2) https://github.com/canonical/netplan/pull/181/commits/4275ade922b63ddacc2db2aba9b413ba71836a04

Commit (2) is part of PR#181, itself part of the huge PR#193: https://github.com/canonical/netplan/pull/193

If both of them are reverted, the old (v0.101) "generate" binary can be executed nicely, using the newer (v0.102 + two reverts) library. Unfortunately, we cannot just revert those commits, as they provide some of the core functionality of netplan v0.102...

Both commits have in common that they add new data to the "NetplanNetDefinition struct", making it grow in size. So we're probably crossing some kind of memory boundary here, breaking the ABI.

I need to dig deeper, in order to find a proper way to solve this.

Previous fix in Hirsute is not enough, so I'm resetting it to "Triaged".

Hello Carl, or anyone else affected,

Accepted netplan.io into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.102-0ubuntu1~20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Thank you very much Brian, for quickly accepting the regression fix into focal-proposed!

I verified the Focal SRU (for 0.102-0ubuntu1~20.04.2) and attached the corresponding test logs to the bug description.

Furthermore, I've verified the regression case, where libnetplan0 v0.102 is used with the netplan.io "generate" binary v0.101, which also passes (i.e. does not crash anymore):

root@ff:~# dpkg -l | grep netplan
ii libnetplan0:amd64 0.102-0ubuntu1~20.04.2 amd64
ii netplan.io 0.101-0ubuntu3~20.04.2 amd64
root@ff:~# cat /etc/netplan/00-config.yaml
network:
    ethernets:
        ens5:
            dhcp4: true
            dhcp6: false
            match:
                macaddress: 06:f8:32:e5:34:28
            set-name: ens5
    version: 2
root@ff:~# /usr/lib/netplan/generate
root@ff:~# echo $?
0

As you can see the "generate" binary does not crash with "Segmentation fault (core dumped)" anymore when using libnetplan0 0.102-0ubuntu1~20.04.2

This bug was fixed in the package netplan.io - 0.102-0ubuntu3

---------------
netplan.io (0.102-0ubuntu3) hirsute; urgency=medium

  * Fix regression (LP: #1922898) for good, by avoiding to break the ABI
    This reverts the "Added ttl option for tunnels" feature
  * Revert previous fix of strict libnetplan dependency

 -- Lukas Märdian <email address hidden> Mon, 19 Apr 2021 13:33:19 +0200

Hello Carl, or anyone else affected,

Accepted netplan.io into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.102-0ubuntu1~20.10.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I verified the Groovy SRU (for 0.102-0ubuntu1~20.10.3) and attached the corresponding test logs to the bug description.

Furthermore, I've verified the regression case, where libnetplan0 v0.102 is used with the netplan.io "generate" binary v0.101, which also passes (i.e. does not crash anymore):

root@gg:~# dpkg -l | grep netplan
ii libnetplan0:amd64 0.102-0ubuntu1~20.10.3
ii netplan.io 0.101-0ubuntu3~20.10.1
root@gg:~# cat /etc/netplan/00-config.yaml
network:
    ethernets:
        ens5:
            dhcp4: true
            dhcp6: false
            match:
                macaddress: 06:f8:32:e5:34:28
            set-name: ens5
    version: 2
root@gg:~# /usr/lib/netplan/generate
root@gg:~# echo $?
0

As you can see the "generate" binary does not crash with "Segmentation fault (core dumped)" anymore when using libnetplan0 0.102-0ubuntu1~20.10.3

It's a day early for groovy but considering the fact that this got enough testing already, let me release those today. Thanks for the verification!

The verification of the Stable Release Update for netplan.io has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package netplan.io - 0.102-0ubuntu1~20.04.2

---------------
netplan.io (0.102-0ubuntu1~20.04.2) focal; urgency=medium

  * Backport netplan.io 0.102-0ubuntu1 to 20.04 (LP: #1919453)
    - Includes NetworkManager YAML backend API
    - Includes 'congestion-window' & 'advertised-receive-window' keys
    - Includes 'netplan set' improvements
  * Keep riscv64 build-time tests disabled
  * Add d/p/0002-tests-tunnels-improve-flaky-wireguard-test-with-wait.patch
  * Fix regression (LP: #1922898), by avoiding to break the ABI
    This reverts the "Added ttl option for tunnels" feature

 -- Lukas Märdian <email address hidden> Mon, 19 Apr 2021 15:08:37 +0200

This bug was fixed in the package netplan.io - 0.102-0ubuntu1~20.10.3

---------------
netplan.io (0.102-0ubuntu1~20.10.3) groovy; urgency=medium

  * Backport netplan.io 0.102-0ubuntu1 to 20.10 (LP: #1919453)
    - Includes NetworkManager YAML backend API
    - Includes 'congestion-window' & 'advertised-receive-window' keys
    - Includes 'netplan set' improvements
  * Fix regression (LP: #1922898), by avoiding to break the ABI
    This reverts the "Added ttl option for tunnels" feature
  * Improve flaky autopkgtests:
    + Skip flaky 'regressions' test on armhf/LXD
    + Add d/p/0002-tests-tunnels-improve-flaky-wireguard-test-with-wait.patch
    + Add d/p/fix-autopkgtest-waiting-logic.patch + prepare.patch

 -- Lukas Märdian <email address hidden> Fri, 07 May 2021 14:03:19 +0200