subiquity fails to see autoinstall data conveyed by cloud-init 23.2

cloud-init component of this tracked in Github Issues as https://github.com/canonical/cloud-init/issues/4158

This bug is seen on Mantic Daily live installer images. While specifically related to differing versions of cloud-init in snap environment versus the host system in the daily live ISOs, cloud-init will surface a flat file /run/cloud-init/merged-cloud-config.json which will allow subiquity to source a flat file instead of invoking stages.Init()..fetch[1] which has to deal with unpickling any cached datasource.

This cloud-init feature work will be tracked in SC-1452

[1] https://github.com/canonical/subiquity/blob/main/subiquity/server/server.py#L549-L553

An older cloud-init deserializing a pickled Datasource cached /var/lib/cloud/instance/obj.pkl from a newer verion of cloud-init is an unsupported use case.

That said, there is work in progress that will allow to sourcing /run/cloud-init/merged-cloud-config which will allow subiquity to consume user-data without reading from a cached pickle of instance data.

https://github.com/canonical/cloud-init/compare/main...blackboxsw:cloud-init:SC-1452-persist-run-combined-cloud-config-json. An update will be provided here when that feature has landed.

Did a retest today with the copy of cloud-init from ppa proposed staged in the snap (23.2.1-0ubuntu0~22.04.1). See a different variant of the same problem:

2023-06-29 17:01:35,900 DEBUG cloudinit.util:948 Attempting to load yaml from string of length 272 with allowed root types (<class 'dict'>,)
2023-06-29 17:01:35,900 DEBUG cloudinit.util:1536 Reading from /var/lib/cloud/instance/obj.pkl (quiet=False)
2023-06-29 17:01:35,900 DEBUG cloudinit.util:1547 Read 8272 bytes from /var/lib/cloud/instance/obj.pkl
2023-06-29 17:01:35,911 WARNING cloudinit.sources:1727 Failed loading pickled blob from /var/lib/cloud/instance/obj.pkl
2023-06-29 17:01:35,911 DEBUG cloudinit.sources:1736 Failed loading pickled blob from /var/lib/cloud/instance/obj.pkl
Traceback (most recent call last):
  File "/snap/subiquity/x1/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 1170, in pkl_load
    return pickle.loads(pickle_contents)
ModuleNotFoundError: No module named 'cloudinit.net.netops'

Thanks for the efforts on the combined-cloud-config.json.

Two different version of the fix are in place:

1) Mantic+ - read combined-cloud-config.json, but this requires that cloud-init 23.3 or later is present in the HOST os
2) Jammy - use the legacy method of reading autoinstall data. Requires that compatible versions of cloud-init are int the snap and in the host OS.

"2) Jammy - use the legacy method of reading autoinstall data. Requires that compatible versions of cloud-init are in the snap and in the host OS."

What exactly does this mean? Cloud-init recently had two new bugs pop up:
https://github.com/canonical/cloud-init/issues/4302
https://github.com/canonical/cloud-init/issues/4304

and they are both reproducable on https://cdimage.ubuntu.com/ubuntu-server/jammy/daily-live/20230730/ . They symptom is ds-identify related, not unpickle related, but the cause is the same: that the cloud-init version in the snap is less than the cloud-init version on the host.

Does "compatible versions of cloud-init are in the snap and in the host OS" mean the same version? If the answer is yes, is there a build somewhere that can be used to test that these bugs are truly fixed? If the answer is no, then I believe that there is still an issue here to resolve.

Marking this fix released for Subiquity:

* general fix: on main branch, read the combined-cloud-config to be able to get the autoinstall
* mantic: build containing above fix promoted to stable/ubuntu-23.10
* jammy: rebuilt of stable branch promoted to stable/22.04.3 (what snap refresh will look at), and stable/22.04 (what the ISO build will look at)