Update 0.105-0ubuntu2~22.04.1 seems to cause regressions with snapd integration

As a reproducer on 22.04:
# cat > /etc/netplan/0-snapd-defaults.yaml <<EOF
network:
  version: 2
  bridges:
    br54:
      dhcp4: true
EOF
# busctl call io.netplan.Netplan /io/netplan/Netplan io.netplan.Netplan Config
# busctl call io.netplan.Netplan /io/netplan/Netplan/config/OY6LV1 io.netplan.Netplan.Config Set ss "network.bridges.br54.dhcp4=false" "90-snapd-config"
# busctl call io.netplan.Netplan /io/netplan/Netplan/config/OY6LV1 io.netplan.Netplan.Config Apply
# netplan get
network:
  version: 2
  renderer: NetworkManager
  bridges:
    br54:
      dhcp4: true

Fwiw, runing the same in a 20.04 xenial lxc container:
"""
...
# root@focal:~# netplan get
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
  bridges:
    br54:
      dhcp4: false
root@focal:~# apt list netplan.io
Listing... Done
netplan.io/focal-updates,now 0.104-0ubuntu2~20.04.2 amd64 [installed]
"""

I can confirm this is a regression and happens with an even simpler reproducer like this:

# cat > /etc/netplan/0-snapd-defaults.yaml <<EOF
network:
  version: 2
  bridges:
    br54:
      dhcp4: true
EOF
# netplan set --origin-hint 90-snapd-config "network.bridges.br54.dhcp4=false"

Seems to happen whenever the setting is about to be overridden (i.e. already exists in another file, like 0-snapd-defaults.yaml here).

Most probably related to switching the backend for `netplan set` to libnetplan: https://github.com/canonical/netplan/pull/254

Needs further debugging.

Here's a test case reproducer, which can be put into netplan's test_cli_get_set.py suite:

    def test_set_origin_hint_override(self):
        defaults = os.path.join(self.workdir.name, 'etc', 'netplan', '0-snapd-defaults.yaml')
        with open(defaults, 'w') as f:
            f.write('''network: {bridges: {br54: {dhcp4: true}}}''')
        self._set(['bridges.br54.dhcp4=false', '--origin-hint=90-snapd-config'])
        p = os.path.join(self.workdir.name, 'etc', 'netplan', '90-snapd-config.yaml')
        self.assertTrue(os.path.isfile(p))
        with open(p, 'r') as f:
            self.assertIs(False, yaml.safe_load(f)['network']['bridges']['br54']['dhcp4'])

I've come up with an initial draft for a patch.

Not sure if this is the full solution already, but at least it fixes the above mentioned test case.

@mvo This is all inside the python code. could you try to modify that file inside your test environment and see if that fixes the snapd tests?

I've prepared something here: https://github.com/canonical/netplan/pull/299

Status changed to 'Confirmed' because the bug affects multiple users.

I feel like I'm pretty close to a solution now. Although, my work from PR#299 is not yet fully reviewed and merged, I prepared a fix for this in the following PPA: https://launchpad.net/~slyon/+archive/ubuntu/netplan-set-lp1997467

It could be copied to the UC22 build PPA, if needed, while we work on refining and merging the upstream PR and making it fit for SRU.

Alternatively, we could revert the switch to the libnetplan YAML parser based "netplan set" implementation for the time being (basically everything from https://github.com/canonical/netplan/pull/254 -> netplan/cli/commands/set.py)

Thanks for working on the fix! Sadly I had not yet managed to find time to build a core22 that uses the PPA to test it.

However sadly this is hitting customers now and because of this I uploaded a revert of netplan to the 0.104-0ubuntu2.1 into ppa:snappy-dev/image to unblock our customers.

With the most recent changes from https://github.com/canonical/netplan/pull/299 this seems to be fixed. Both reproducers pass for me, using a PPA build including the PR#299 commits: https://launchpad.net/~slyon/+archive/ubuntu/netplan-set-lp1997467

This bug was fixed in the package netplan.io - 0.105-0ubuntu4

---------------
netplan.io (0.105-0ubuntu4) lunar; urgency=medium

  * d/p/lp1997467: set only specific origin-hint if given (LP: #1997467)
  * d/libnetplan0.symbols: Add netplan_parser_load_nullable_overrides() API

 -- Lukas Märdian <email address hidden> Thu, 26 Jan 2023 14:34:37 +0100

A quick review on the debdiffs:
* SRU version should be "0.105-0ubuntu2.1" / "0.105-0ubuntu2~22.04.2" (see https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging)
* improve d/changelog body formatting
* Adopt .symbols version for new API function according to the SRU version
* We might want to cherry-pick/backport https://github.com/canonical/netplan/commit/102f189e8f3a1828a3beddf686dcc96c15a3b9b5 too, as this is a fixup for PR#299

Those are all small things and it lgtm otherwise.
Overall, I prefer git Merge-Proposals over debdiffs, but that's a personal preference :)

Thanks, merged and sponsored into kinetic and jammy!

Hello Michael, or anyone else affected,

Accepted netplan.io into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.105-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Michael, or anyone else affected,

Accepted netplan.io into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.105-0ubuntu2~22.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted netplan.io (0.105-0ubuntu2.1) for kinetic have finished running.
The following regressions have been reported in tests triggered by the package:

netplan.io/0.105-0ubuntu2.1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#netplan.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

hm the test from the regression above passed for all the other architectures and it passes locally. Probably some transient issue that a retry will fix.

The kinetic/amd64 autopkgtest regression was fixed by re-try.

All autopkgtests for the newly accepted netplan.io (0.105-0ubuntu2~22.04.2) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.140ubuntu13.1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#netplan.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

initramfs-tools failing only for amd64, it seems to be related with qemu timeouts. It will probably work after retrying it.

So, it seems like we can't get the initramfs-tools tests to pass.

But it seems to be the normal: https://autopkgtest.ubuntu.com/packages/initramfs-tools/jammy/amd64
it fails multiple times with the same trigger but then eventually it works.

I tried it locally a bunch of times (using netplan.io from jammy-proposed) and it never failed.

ACK. I feel like this should be hinted in https://git.launchpad.net/~ubuntu-release/britney/+git/hints-ubuntu

The last retry triggered with the new netplan passed all the tests: https://autopkgtest.ubuntu.com/packages/initramfs-tools/jammy/amd64

I am trying to test this on core22 with:
"""
$ git diff
diff --git a/tests/nested/manual/core20-early-config/task.yaml b/tests/nested/manual/core20-early-config/task.yaml
index 797232fa26..fbb92d4240 100644
--- a/tests/nested/manual/core20-early-config/task.yaml
+++ b/tests/nested/manual/core20-early-config/task.yaml
@@ -28,6 +28,17 @@ prepare: |

     rm -f "$SNAKEOIL_KEY" "$SNAKEOIL_CERT"

+ snap download --basename=core22 core22
+ unsquashfs -d core22 core22.snap
+ wget https://launchpad.net/ubuntu/+source/netplan.io/0.105-0ubuntu2~22.04.2/+build/25592898/+files/libnetplan0_0.105-0ubuntu2~22.04.2_amd64.deb
+ dpkg-deb -x libnetplan0_0.105-0ubuntu2~22.04.2_amd64.deb ./core22
+ wget https://launchpad.net/ubuntu/+source/netplan.io/0.105-0ubuntu2~22.04.2/+build/25592898/+files/netplan.io_0.105-0ubuntu2~22.04.2_amd64.deb
+ dpkg-deb -x netplan.io_0.105-0ubuntu2~22.04.2_amd64.deb ./netplan
+ # netplan ships stuff in /lib instead of /usr/lib, fix this
+ mv netplan/lib/ netplan/usr/
+ rsync -a ./netplan/* ./core22
+ snap pack ./core22 "$(tests.nested get extra-snaps-path)"
+
     # Note that the early config is configured to set port to 8023 instead
     # of the default 8022 (which maps to 22 on the VM).
     remote.setup config --host localhost --port 8023 --user user1 --pass ubuntu

$ spread -debug -v qemu-nested:ubuntu-22.04-64:tests/nested/manual/core20-early-config
"""

With that the "core20-early-config" test that was also used to find the original issue fails in strange ways. The netplan config itself looks fine:
"""
# remote.exec "sudo netplan get "
network:
  version: 2
  ethernets:
    ens3:
      addresses:
      - "10.0.2.15/24"
      dhcp4: false
      routes:
      - to: "default"
        via: "10.0.2.2"
  bridges:
    br54:
      dhcp4: true
"""

But netplan seems to be no longer able to give dbus results:
"""
qemu-nested:ubuntu-22.04-64 .../tests/nested/manual/core20-early-config# remote.exec 'sudo dbus-send --system --print-reply --type=method_call --dest=io.netplan.Netplan /io/netplan/Netplan io.netplan.Netplan.Config'
method return time=1677251025.338293 sender=:1.27 -> destination=:1.232 serial=65 reply_serial=2
   object path "/io/netplan/Netplan/config/Q6S201"
qemu-nested:ubuntu-22.04-64 .../tests/nested/manual/core20-early-config# remote.exec 'sudo dbus-send --system --print-reply --type=method_call --dest=io.netplan.Netplan /io/netplan/Netplan/config/Q6S201 io.netplan.Netplan.Config.Get'
method return time=1677251055.826055 sender=:1.27 -> destination=:1.237 serial=68 reply_serial=2
   string ""
"""
and the same in snapd (which uses the dbus API).

I also tried to reproduce this on my desktop system and see similar results, i.e. the dbus call for a config dbus object is now giving me an empty string on my desktop. So maybe more than just a test setup issue?

Fwiw, on my 22.10 machine I get something like "'network:\n version: 2\n renderer: NetworkManager\n'" from the "Get()" call.

Just for reference, my test commit that exercises the new netplan is under https://github.com/snapcore/snapd/compare/master...mvo5:snappy:test-netplan-sru-1997467?expand=1

I noticed the error below when trying to use netplan-dbus:

sudo dbus-send --system --print-reply --type=method_call --dest=io.netplan.Netplan /io/netplan/Netplan io.netplan.Netplan.Config

This is from dbus-monitor --system:

string "Failed to copy file /etc/netplan/01-network-manager-all.yaml -> /run/netplan/config-1YXM01etc/netplan/01-network-manager-all.yaml: Error opening file “/run/netplan/config-1YXM01etc/netplan/01-network-manager-all.yaml”: No such file or directory

Apparently the path is malformed: /run/netplan/config-1YXM01etc/netplan/01-network-manager-all.yaml

This will fix the dbus issue https://github.com/canonical/netplan/pull/331

$ sudo dbus-send --system --print-reply --type=method_call --dest=io.netplan.Netplan /io/netplan/Netplan io.netplan.Netplan.Config
method return time=1677605377.300772 sender=:1.9462 -> destination=:1.9463 serial=4 reply_serial=2
   object path "/io/netplan/Netplan/config/WM6X01"

$ sudo dbus-send --system --print-reply --type=method_call --dest=io.netplan.Netplan /io/netplan/Netplan/config/WM6X01 io.netplan.Netplan.Config.Get
method return time=1677605387.117285 sender=:1.9462 -> destination=:1.9464 serial=6 reply_serial=2
   string "network:
  version: 2
  renderer: NetworkManager
  bridges:
    br123:
      dhcp4: false
"

I suppose we need to include it in the SRU as well.

I've created two new merge requests including patches for the dbus issue

Kinetic: https://code.launchpad.net/~danilogondolfo/netplan/+git/ubuntu/+merge/438307

Jammy: https://code.launchpad.net/~danilogondolfo/netplan/+git/ubuntu/+merge/438310

A PPA with packages for both are available here https://launchpad.net/~danilogondolfo/+archive/ubuntu/netplan.io

netplan.io in lunar-proposed already has the fix.

Hello Michael, or anyone else affected,

Accepted netplan.io into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.105-0ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Michael, or anyone else affected,

Accepted netplan.io into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.105-0ubuntu2~22.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted netplan.io (0.105-0ubuntu2~22.04.3) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.140ubuntu13.1 (amd64)
netplan.io/0.105-0ubuntu2~22.04.3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#netplan.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Thanks for the updated version.

I tested it with https://github.com/snapcore/snapd/compare/master...mvo5:snappy:test-netplan-sru-1997467?expand=1 (updated to test exactly this version) and with that our spread test works:
"""
$ spread -debug -v qemu-nested:ubuntu-22.04-64:tests/nested/manual/core20-early-config
...
2023-03-09 12:16:57 Executing qemu-nested:ubuntu-22.04-64:tests/nested/manual/core20-early-config (qemu-nested:ubuntu-22.04-64) (1/1)...
2023-03-09 12:18:11 Restoring qemu-nested:ubuntu-22.04-64:tests/nested/manual/core20-early-config (qemu-nested:ubuntu-22.04-64)...
2023-03-09 12:18:29 Restoring qemu-nested:ubuntu-22.04-64:tests/nested/manual/ (qemu-nested:ubuntu-22.04-64)...
2023-03-09 12:18:31 Restoring qemu-nested:ubuntu-22.04-64 (qemu-nested:ubuntu-22.04-64)...
2023-03-09 12:18:32 Discarding qemu-nested:ubuntu-22.04-64...
2023-03-09 12:18:32 Successful tasks: 1
2023-03-09 12:18:32 Aborted tasks: 0
"""
so +1 from me.

Thanks, Michael!

So I believe we can proceed with the SRU.

I did another batch of tests today. The results are described below:

The steps below were used to validate the fix on Jammy and Kinetic:

1) Launch a LXD Jammy instance and access it

lxc launch ubuntu:22.04 jammy-netplan
lxc shell jammy-netplan

2) Create the base config

cat <<EOF > /etc/netplan/0-snapd-defaults.yaml
network:
  version: 2
  ethernets:
    ens3:
      addresses:
      - "10.0.2.15/24"
      dhcp4: false
      routes:
      - to: "default"
        via: "10.0.2.2"
  bridges:
    br54:
      dhcp4: true
EOF

3) Try to change it with netplan set

netplan set --origin-hint 90-snapd-config network.bridges.br54.dhcp4=false

4) Check the the new file WASN'T created

cat /etc/netplan/90-snapd-config.yaml

output: cat: /etc/netplan/90-snapd-config.yaml: No such file or directory

5) Check that the DBus interface IS working properly:

busctl call io.netplan.Netplan /io/netplan/Netplan io.netplan.Netplan Config

output: o "/io/netplan/Netplan/config/88ZI11" <<---- USE THIS PATH IN THE COMMANDS BELOW

busctl call io.netplan.Netplan /io/netplan/Netplan/config/88ZI11 io.netplan.Netplan.Config Set ss "network.bridges.br54.dhcp4=false" "90-snapd-config"

output: b true

busctl call io.netplan.Netplan /io/netplan/Netplan/config/88ZI11 io.netplan.Netplan.Config Get

output: s "network:\n version: 2\n ethernets:\n ens3:\n addresses:\n - \"10.0.2.15/24\"\n dhcp4: false\n routes:\n - to: \"default\"\n via: \"10.0.2.2\"\n eth0:\n dhcp4: true\n bridges:\n br54:\n dhcp4: true\n"

6) Enable -proposed and update your netplan.io packages

echo "deb http://archive.ubuntu.com/ubuntu jammy-proposed main" >> /etc/apt/sources.list.d/proposed.list
apt update && apt -y upgrade

# kill netplan-dbus to make sure the new one will run when you try busctl again

killall netplan-dbus

7) Use netplan set again to change the configuration created in the second step

netplan set --origin-hint 90-snapd-config network.bridges.br54.dhcp4=false

8) Check the the new file WAS created this time

cat /etc/netplan/90-snapd-config.yaml

network:
  version: 2
  bridges:
    br54:
      dhcp4: false

9) Check that the DBus interface IS still working properly:

busctl call io.netplan.Netplan /io/netplan/Netplan io.netplan.Netplan Config

output: o "/io/netplan/Netplan/config/PMTC11" <<---- USE THIS PATH IN THE COMMANDS BELOW

busctl call io.netplan.Netplan /io/netplan/Netplan/config/PMTC11 io.netplan.Netplan.Config Set ss "network.bridges.br54.dhcp4=false" "90-snapd-config"

output: b true

busctl call io.netplan.Netplan /io/netplan/Netplan/config/PMTC11 io.netplan.Netplan.Config Get

output: s "network:\n version: 2\n ethernets:\n ens3:\n addresses:\n - \"10.0.2.15/24\"\n dhcp4: false\n routes:\n - to: \"default\"\n via: \"10.0.2.2\"\n eth0:\n dhcp4: true\n bridges:\n br54:\n dhcp4: false\n"

---------

Validation on Kinetic

The same steps can be performed to valid netplan.io on kinetic

1) Launch a LXD Kinetic instance and access it

lxc launch ubuntu:22.10 kinetic-netplan
lxc shell kinetic-net...

This bug was fixed in the package netplan.io - 0.105-0ubuntu2.2

---------------
netplan.io (0.105-0ubuntu2.2) kinetic; urgency=medium

  * Fix and improvements for the DBus integration (LP: #1997467)
    Cherry-picked from upstream: https://github.com/canonical/netplan/pull/331
    - d/p/lp1997467/0009-dbus-Build-the-copy-path-correctly.patch
      Properly build the destination path before copying files in the dbus
      integration and improve error handling
    - d/p/lp1997467/0010-tests-Add-an-integration-test-for-netplan-dbus.patch
      Add an integration test to exercise the code path where the issue was
      addressed.

netplan.io (0.105-0ubuntu2.1) kinetic; urgency=medium

  * d/p/lp1997467: set only specific origin-hint if given (LP: #1997467)
    Cherry-picked from upstream: https://github.com/canonical/netplan/pull/299
    - d/libnetplan0.symbols: Add netplan_parser_load_nullable_overrides() API
    - d/p/0008-src-parse-plug-memory-leaks-in-nullable-handling.patch backport
      upstream commit 40c53bb (memory leak fixup of PR#299)

 -- Danilo Egea Gondolfo <email address hidden> Fri, 03 Mar 2023 12:35:15 +0000

The verification of the Stable Release Update for netplan.io has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package netplan.io - 0.105-0ubuntu2~22.04.3

---------------
netplan.io (0.105-0ubuntu2~22.04.3) jammy; urgency=medium

  * Fix and improvements for the DBus integration (LP: #1997467)
    Cherry-picked from upstream: https://github.com/canonical/netplan/pull/331
    - d/p/lp1997467/0009-dbus-Build-the-copy-path-correctly.patch
      Properly build the destination path before copying files in the dbus
      integration and improve error handling
    - d/p/lp1997467/0010-tests-Add-an-integration-test-for-netplan-dbus.patch
      Add an integration test to exercise the code path where the issue was
      addressed.

netplan.io (0.105-0ubuntu2~22.04.2) jammy; urgency=medium

  * d/p/lp1997467: set only specific origin-hint if given (LP: #1997467)
    Cherry-picked from upstream: https://github.com/canonical/netplan/pull/299
    - d/libnetplan0.symbols: Add netplan_parser_load_nullable_overrides() API
    - d/p/0008-src-parse-plug-memory-leaks-in-nullable-handling.patch backport
      upstream commit 40c53bb (memory leak fixup of PR#299)

 -- Danilo Egea Gondolfo <email address hidden> Fri, 03 Mar 2023 13:14:22 +0000