Unset of a subtree crashes with invalid (non existing) origin-hint

Okay, I think this is actually a combination of invalid usage and non-optimal error handling. I can reproduce the error locally on Impish:

# busctl call io.netplan.Netplan /io/netplan/Netplan io.netplan.Netplan Config
o "/io/netplan/Netplan/config/DS7XA1"
# busctl call io.netplan.Netplan /io/netplan/Netplan/config/DS7XA1 io.netplan.Netplan.Config Set ss "network.ethernets=null" "90-snapd-conf"
Call failed: netplan set failed: Child process exited with code 1
stdout: ''
stderr: 'Traceback (most recent call last):
  File "/usr/sbin/netplan", line 23, in <module>
    netplan.main()
  File "/usr/share/netplan/netplan/cli/core.py", line 50, in main
    self.run_command()
  File "/usr/share/netplan/netplan/cli/utils.py", line 310, in run_command
    self.func()
  File "/usr/share/netplan/netplan/cli/commands/set.py", line 53, in run
    self.run_command()
  File "/usr/share/netplan/netplan/cli/utils.py", line 310, in run_command
    self.func()
  File "/usr/share/netplan/netplan/cli/commands/set.py", line 106, in command_set
    self.write_file(subtree, hint + '.yaml', self.root_dir)
  File "/usr/share/netplan/netplan/cli/commands/set.py", line 176, in write_file
    raise Exception('Invalid input: {}'.format(set_tree))
Exception: Invalid input: {'network': {'ethernets': None}}

BUT: That only seems to happen if the file hint (here: 90-snapd-conf.yaml) does not exist. If 90-snapd-conf.yaml is filled with some config before, it works as expected:

# busctl call io.netplan.Netplan /io/netplan/Netplan/config/DS7XA1 io.netplan.Netplan.Config Set ss "network.ethernets.eth99.dhcp4=true" "90-snapd-conf"
b true
# busctl call io.netplan.Netplan /io/netplan/Netplan/config/DS7XA1 io.netplan.Netplan.Config Set ss "network.ethernets=null" "90-snapd-conf"
b true

Also, if you want to clear any ethernet definition (not just the ones from 90-snapd-conf.yaml) you could pass an empty origin hint, to make it work:

# busctl call io.netplan.Netplan /io/netplan/Netplan/config/DS7XA1 io.netplan.Netplan.Config Set ss "network.ethernets=null" ""
b true

BTW: the same issue happens with the CLI, if you pass a non existing origin-hint and we should probably improve the error handling and error message in cli/commands/set.py -> write_file():

# netplan set network.ethernets=null --origin-hint NON_EXISTING_FILE
Traceback (most recent call last):
  File "/usr/sbin/netplan", line 23, in <module>
    netplan.main()
  File "/usr/share/netplan/netplan/cli/core.py", line 50, in main
    self.run_command()
  File "/usr/share/netplan/netplan/cli/utils.py", line 310, in run_command
    self.func()
  File "/usr/share/netplan/netplan/cli/commands/set.py", line 53, in run
    self.run_command()
  File "/usr/share/netplan/netplan/cli/utils.py", line 310, in run_command
    self.func()
  File "/usr/share/netplan/netplan/cli/commands/set.py", line 106, in command_set
    self.write_file(subtree, hint + '.yaml', self.root_dir)
  File "/usr/share/netplan/netplan/cli/commands/set.py", line 176, in write_file
    raise Exception('Invalid input: {}'.format(set_tree))
Exception: Invalid input: {'network': {'ethernets': None}}

This is being worked on here https://github.com/canonical/netplan/pull/246 (as it is the same class of errors as described in LP: #1949884). I guess we could even mark it a duplicate.

Adding another reproducer script to check for all the different cases reported (also from the duplicate bugs).

This bug was fixed in the package netplan.io - 0.103-0ubuntu10

---------------
netplan.io (0.103-0ubuntu10) jammy; urgency=medium

  * update gbp.conf branch
  * Drop d/p/0002-Revert-cli-apply-reload-reconfigure-networkd-instead.patch
    This is not needed anymore with systemd v249
  * Refresh d/p/0006-netplan-set-make-it-possible-to-unset-a-whole-devtyp.patch
  * Add d/p/0012-test-bridge-base-give-bridge-some-more-time-to-reach.patch
    To fix flaky test_bridge_anonymous autopkgtest (upstream c6ad8e6)
  * Upstream cherry-picks for snapd dbus config set-try-apply integration fixes
    - dbus-wait-for-netplan-try-to-be-ready-LP-1949893-245.patch (LP: #1949893)
    - get-set-ignore-empty-YAML-hints-and-delete-files-on-.patch (LP: #1946957)

 -- Lukas Märdian <email address hidden> Mon, 29 Nov 2021 17:14:32 +0100

Hello Michael, or anyone else affected,

Accepted netplan.io into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.103-0ubuntu7.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Michael, or anyone else affected,

Accepted netplan.io into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.103-0ubuntu5~21.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Michael, or anyone else affected,

Accepted netplan.io into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.103-0ubuntu5~20.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted netplan.io (0.103-0ubuntu5~21.04.4) for hirsute have finished running.
The following regressions have been reported in tests triggered by the package:

netplan.io/0.103-0ubuntu5~21.04.4 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/hirsute/update_excuses.html#netplan.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

The netplan.io/0.103-0ubuntu5~21.04.4 (arm64) autopkgtest regression was resolved by retry.

I tested netplan.io 0.103-0ubuntu7.2 from impish-proposed and attached the official autopkgtest logs, all passed.

root@ii:~# dpkg -l | grep netplan
ii libnetplan0:amd64 0.103-0ubuntu7.2
ii netplan.io 0.103-0ubuntu7.2

root@ii:~# ./netplan-dbus-perculiar2.sh
method return time=1638359097.746850 sender=:1.19 -> destination=:1.20 serial=6 reply_serial=2
   boolean true
method return time=1638359098.468015 sender=:1.19 -> destination=:1.21 serial=8 reply_serial=2
   boolean true
method return time=1638359099.532484 sender=:1.19 -> destination=:1.24 serial=11 reply_serial=2
   boolean true

root@ii:~# ./repro.sh
=== Case 1 ===
    br54:
OK (br45 exists)
=== Case 2 ===
network:
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 3 ===
    br54:
OK (br45 exists)
=== Case 4 ===
network:
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 5 ===
    br54:
OK (br45 exists)
=== Case 6 ===
network:
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 7 ===
{}
cat: '/etc/netplan/*.yaml': No such file or directory
OK (all YAML gone)
=== Case 8 ===
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
network:
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK 90-hint.yaml already gone
=== Case 9 ===
cat: /etc/netplan/00-not-exist.yaml: No such file or directory
rm: cannot remove '/etc/netplan/00-not-exist.yaml': No such file or directory
./repro.sh: 57: OK file already gone: not found

=> No netplan CLI crash can be observed. All good.

I tested netplan.io 0.103-0ubuntu5~21.04.4 from hirsute-proposed and attached the official autopkgtest logs, all passed.

root@hh:~# dpkg -l | grep netplan
ii libnetplan0:amd64 0.103-0ubuntu5~21.04.4
ii netplan.io 0.103-0ubuntu5~21.04.4

root@hh:~# ./netplan-dbus-perculiar2.sh
method return time=1638359822.189644 sender=:1.20 -> destination=:1.21 serial=6 reply_serial=2
   boolean true
method return time=1638359822.725752 sender=:1.20 -> destination=:1.22 serial=8 reply_serial=2
   boolean true
method return time=1638359823.743116 sender=:1.20 -> destination=:1.25 serial=11 reply_serial=2
   boolean true
root@hh:~#
root@hh:~#
root@hh:~#
root@hh:~# ./repro.sh
=== Case 1 ===
    br54:
OK (br45 exists)
=== Case 2 ===
network:
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 3 ===
    br54:
OK (br45 exists)
=== Case 4 ===
network:
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 5 ===
    br54:
OK (br45 exists)
=== Case 6 ===
network:
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 7 ===
{}
cat: '/etc/netplan/*.yaml': No such file or directory
OK (all YAML gone)
=== Case 8 ===
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
network:
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK 90-hint.yaml already gone
=== Case 9 ===
cat: /etc/netplan/00-not-exist.yaml: No such file or directory
rm: cannot remove '/etc/netplan/00-not-exist.yaml': No such file or directory
./repro.sh: 57: OK file already gone: not found

=> No netplan CLI crash can be observed. All good.

I tested netplan.io 0.103-0ubuntu5~20.04.4 from focal-proposed and attached the official autopkgtest logs, all passed.

root@ff:~# dpkg -l | grep netplan
ii libnetplan0:amd64 0.103-0ubuntu5~20.04.4
ii netplan.io 0.103-0ubuntu5~20.04.4

root@ff:~# ./netplan-dbus-perculiar2.sh
method return time=1638363145.040627 sender=:1.19 -> destination=:1.26 serial=15 reply_serial=2
   boolean true
method return time=1638363145.681561 sender=:1.19 -> destination=:1.27 serial=17 reply_serial=2
   boolean true
method return time=1638363146.731549 sender=:1.19 -> destination=:1.30 serial=20 reply_serial=2
   boolean true
root@ff:~#
root@ff:~#
root@ff:~#
root@ff:~# ./repro.sh
=== Case 1 ===
    br54:
OK (br45 exists)
=== Case 2 ===
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 3 ===
    br54:
OK (br45 exists)
=== Case 4 ===
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 5 ===
    br54:
OK (br45 exists)
=== Case 6 ===
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 7 ===
{}
cat: '/etc/netplan/*.yaml': No such file or directory
OK (all YAML gone)
=== Case 8 ===
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK 90-hint.yaml already gone
=== Case 9 ===
cat: /etc/netplan/00-not-exist.yaml: No such file or directory
rm: cannot remove '/etc/netplan/00-not-exist.yaml': No such file or directory
./repro.sh: 57: OK file already gone: not found

=> No netplan CLI crash can be observed. All good.

Hello Michael, or anyone else affected,

Accepted netplan.io into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.103-0ubuntu5~20.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I tested netplan.io 0.103-0ubuntu5~20.04.5 from focal-proposed and attached the official autopkgtest logs, all passed.

root@ff:~# dpkg -l | grep netplan
ii libnetplan0:amd64 0.103-0ubuntu5~20.04.5
ii netplan.io 0.103-0ubuntu5~20.04.5

root@ff:~# ./netplan-dbus-perculiar2.sh
method return time=1638878208.426246 sender=:1.19 -> destination=:1.26 serial=15 reply_serial=2
   boolean true
method return time=1638878208.938657 sender=:1.19 -> destination=:1.27 serial=17 reply_serial=2
   boolean true
method return time=1638878209.981659 sender=:1.19 -> destination=:1.30 serial=20 reply_serial=2
   boolean true
root@ff:~#
root@ff:~#
root@ff:~#
root@ff:~# ./repro.sh
=== Case 1 ===
    br54:
OK (br45 exists)
=== Case 2 ===
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 3 ===
    br54:
OK (br45 exists)
=== Case 4 ===
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 5 ===
    br54:
OK (br45 exists)
=== Case 6 ===
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK (file already gone)
=== Case 7 ===
{}
cat: '/etc/netplan/*.yaml': No such file or directory
OK (all YAML gone)
=== Case 8 ===
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
network:
  bridges:
    br54:
      dhcp4: true
  ethernets:
    eth0:
      dhcp4: true
  version: 2
rm: cannot remove '/etc/netplan/90-hint.yaml': No such file or directory
OK 90-hint.yaml already gone
=== Case 9 ===
cat: /etc/netplan/00-not-exist.yaml: No such file or directory
rm: cannot remove '/etc/netplan/00-not-exist.yaml': No such file or directory
./repro.sh: 57: OK file already gone: not found

=> No netplan CLI crash can be observed. All good.

This bug was fixed in the package netplan.io - 0.103-0ubuntu7.2

---------------
netplan.io (0.103-0ubuntu7.2) impish; urgency=medium

  * Add d/p/0012-test-bridge-base-give-bridge-some-more-time-to-reach.patch
    To fix flaky test_bridge_anonymous autopkgtest (upstream c6ad8e6)
  * Upstream cherry-picks for snapd dbus config set-try-apply integration fixes
    - dbus-wait-for-netplan-try-to-be-ready-LP-1949893-245.patch (LP: #1949893)
    - get-set-ignore-empty-YAML-hints-and-delete-files-on-.patch (LP: #1946957)

 -- Lukas Märdian <email address hidden> Tue, 30 Nov 2021 17:14:05 +0100

The verification of the Stable Release Update for netplan.io has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package netplan.io - 0.103-0ubuntu5~21.04.4

---------------
netplan.io (0.103-0ubuntu5~21.04.4) hirsute; urgency=medium

  * Add d/p/0012-test-bridge-base-give-bridge-some-more-time-to-reach.patch
    To fix flaky test_bridge_anonymous autopkgtest (upstream c6ad8e6)
  * Upstream cherry-picks for snapd dbus config set-try-apply integration fixes
    - dbus-wait-for-netplan-try-to-be-ready-LP-1949893-245.patch (LP: #1949893)
    - get-set-ignore-empty-YAML-hints-and-delete-files-on-.patch (LP: #1946957)

 -- Lukas Märdian <email address hidden> Tue, 30 Nov 2021 17:12:25 +0100

This bug was fixed in the package netplan.io - 0.103-0ubuntu5~20.04.5

---------------
netplan.io (0.103-0ubuntu5~20.04.5) focal; urgency=medium

  * Add d/p/give-some-more-time-for-dbus-test-on-focal-riscv64.diff
    To make test_netplan_dbus_config_set_uninvalidate_timeout() pass in slow
    environments (looking at you, riscv64)
  * Add d/p/0012-test-bridge-base-give-bridge-some-more-time-to-reach.patch
    To fix flaky test_bridge_anonymous autopkgtest (upstream c6ad8e6)
  * Upstream cherry-picks for snapd dbus config set-try-apply integration fixes
    - dbus-wait-for-netplan-try-to-be-ready-LP-1949893-245.patch (LP: #1949893)
    - get-set-ignore-empty-YAML-hints-and-delete-files-on-.patch (LP: #1946957)

 -- Lukas Märdian <email address hidden> Fri, 03 Dec 2021 11:12:19 +0100