netplan reparses wifi when bridge member is listed before definition

This impacts bonded interfaces on a fresh install of Ubuntu 18.04.3 LTS

# cat /etc/netplan/50-cloud-init.yaml
network:
    wifis:
        wlx00XXXXXXXXXXXX:
           access-points:
               "guest":
                   mode: ap
                   password: "lol netplan"
           addresses: [192.168.42.0/24]
           optional: true
           match:
              macaddress: xx:xx:xx:xx:xx:xx
    bonds:
        bond0:
            dhcp4: false
            dhcp6: false
            addresses: [192.168.0.2/24]
            nameservers:
               # search: [lab, home]
               addresses: [8.8.8.8, 1.1.1.1]
            interfaces:
            - enp3s0
            - enp5s0
            parameters:
                mode: broadcast
    ethernets:
        enp3s0: {}
        enp5s0: {}
    version: 2

# netplan --debug generate
DEBUG:command generate: running ['/lib/netplan/generate']
** (generate:1388): DEBUG: 17:40:48.166: Processing input file /etc/netplan/50-cloud-init.yaml..
** (generate:1388): DEBUG: 17:40:48.166: starting new processing pass
** (generate:1388): DEBUG: 17:40:48.166: wlx00XXXXXXXXXXXX: adding wifi AP 'guest'
** (generate:1388): DEBUG: 17:40:48.166: recording missing yaml_node_t enp3s0
** (generate:1388): DEBUG: 17:40:48.166: recording missing yaml_node_t enp5s0
** (generate:1388): DEBUG: 17:40:48.166: starting new processing pass
** (generate:1388): DEBUG: 17:40:48.166: wlx00XXXXXXXXXXXX: adding wifi AP 'guest'
/etc/netplan/50-cloud-init.yaml:12:16: Error in network definition: wlx00XXXXXXXXXXXX: Duplicate access point SSID 'guest'
               "guest":
               ^

With bridges or vlans interfaces same problem here

I can confirm that bug from my Chef cookbook: https://gitlab.com/pharmony/netplan.io-cookbook.

I fixed it using the given trick of moving the 'bridges' device to the end of the config file.

cloud-init uses "netplan generate" command to convert cloud-init network configuration from data-source to /etc/cloudinit/50-cloud-init.yaml file. Keys in generated file are ordered in alphabetical order (bonds/bridgeas are at the top pf the file and wifis are at the end).
This issues make impossible to deploy devices required to set-up bridges, bonds and wifi interfaces using cloud-init. This is serious blocker.

I can confirm it's still an issue.

I prepared a fix for this issue https://github.com/canonical/netplan/pull/413

Hello Ryan, or anyone else affected,

Accepted netplan.io into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.107-5ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted netplan.io (0.107-5ubuntu0.2) for mantic have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.142ubuntu15 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/mantic/update_excuses.html#netplan.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

This bug was fixed in the package netplan.io - 0.107-5ubuntu2

---------------
netplan.io (0.107-5ubuntu2) noble; urgency=medium

  * d/p/lp1809994/0010-parse-improve-the-parsing-of-access-points-LP-180999.patch.
    Improve access-points parsing. Properly handle access-points if a second
    pass if required in the parser and not report them as duplicates. (LP: #1809994)
  * d/p/lp1809994/0011-wifi-replace-the-previously-defined-AP-with-the-new-.patch.
    Overwrite the access-point object if another access-point with the same
    name is found instead of ignoring the new one. This behavior is closer
    to the merging process Netplan does.

 -- Danilo Egea Gondolfo <email address hidden> Tue, 14 Nov 2023 14:13:31 +0000

SRU validation on Mantic

netplan.io installed is 0.107-5ubuntu0.1

---
root@mantic:~# dpkg --list | grep netplan
ii libnetplan0:amd64 0.107-5ubuntu0.1 amd64 YAML network configuration abstraction runtime library
ii netplan-generator 0.107-5ubuntu0.1 amd64 YAML network configuration abstraction systemd-generator
ii netplan.io 0.107-5ubuntu0.1 amd64 YAML network configuration abstraction for various backends
ii python3-netplan 0.107-5ubuntu0.1 amd64 YAML network configuration abstraction Python bindings
---

After adding the reproducer from the description, netplan will fail to generate configuration:

cat > /etc/netplan/10-config.yaml <<EOF
network:
  bridges:
    mybridge:
      interfaces: [enp5s0]
  ethernets:
    enp5s0: {}
  wifis:
    wlp6s0:
      access-points:
        "mywifi":
          password: "mypassword"
EOF

---
root@mantic:~# netplan generate

** (generate:2912): WARNING **: 11:31:27.671: Permissions for /etc/netplan/10-config.yaml are too open. Netplan configuration should NOT be accessible by others.
/etc/netplan/10-config.yaml:10:9: Error in network definition: wlp6s0: Duplicate access point SSID 'mywifi'
        "mywifi":
        ^
---

At this point, Network Manager will not work anymore (because it will also call netplan generate):

---
apt install network-manager
...

root@mantic:~# nmcli con add type dummy ifname dummy0
Error: Failed to add 'dummy-dummy0' connection: Message recipient disconnected from message bus without replying

root@mantic:~# journalctl -u NetworkManager
...
Nov 24 11:31:57 mantic NetworkManager[3389]: /etc/netplan/10-config.yaml:10:9: Error in network definition: wlp6s0: Duplicate access point SSID 'mywifi'
Nov 24 11:31:57 mantic NetworkManager[3389]: "mywifi":
Nov 24 11:31:57 mantic NetworkManager[3389]: ^
...
---

Enabling -proposed and upgrading to netplan.io 0.107-5ubuntu0.2:

cat <<EOF >/etc/apt/preferences.d/proposed-updates
Package: src:netplan.io
Pin: release a=$(lsb_release -cs)-proposed
Pin-Priority: 501
EOF

echo "deb http://archive.ubuntu.com/ubuntu mantic-proposed main" >> /etc/apt/sources.list

apt update
apt upgrade

---
root@mantic:~# dpkg --list | grep netplan
ii libnetplan0:amd64 0.107-5ubuntu0.2 amd64 YAML network configuration abstraction runtime library
ii netplan-generator 0.107-5ubuntu0.2 amd64 YAML network configuration abstraction systemd-generator
ii netplan.io 0.107-5ubuntu0.2 amd64 YAML network configuration abstraction for various backends
ii python3-netplan 0.107-5ubuntu0.2 amd64 YAML network configuration abstraction Python bindings
---

Check that netplan generate and nmcli will work fine

---
root@mantic:~# netplan generate

root@mantic:~# nmcli con add type dummy ifname dummy0
Connection 'dummy-dummy0' (2c97024c-35d3-4f2d-932c-33baccae43fb) successfully added.

root@mantic:...

I think this SRU got enough validation to be considered early.

This bug was fixed in the package netplan.io - 0.107-5ubuntu0.2

---------------
netplan.io (0.107-5ubuntu0.2) mantic; urgency=medium

  * lp1809994/0010-parse-improve-the-parsing-of-access-points-LP-180999.patch:
    Improve access-points parsing. Properly handle access-points if a second
    pass if required in the parser and not report them as duplicates
    (LP: #1809994).
  * lp1809994/0011-wifi-replace-the-previously-defined-AP-with-the-new-.patch:
    Overwrite the access-point object if another access-point with the same
    name is found instead of ignoring the new one. This behavior is closer
    to the merging process Netplan does.

 -- Danilo Egea Gondolfo <email address hidden> Wed, 15 Nov 2023 10:58:49 +0000

The verification of the Stable Release Update for netplan.io has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.