Mantic Cloud-Images on amd64 Fail To Preseed Snaps

In my test, with a revert of livecd-rootfs to before my CHANNEL changes (equivalent of ver 2.849), I see the same results on amd64 (no snaps).

A fix in livecd-rootfs may well be necessary, but I wonder if there isn't also something else going on.

https://launchpadlibrarian.net/677155499/buildlog_ubuntu_mantic_amd64_test_BUILDING.txt.gz

@John - would you expect lxd to be part of the cloud-image.snaps germinate output?

In this log, with some set -x, https://launchpadlibrarian.net/677345865/buildlog_ubuntu_mantic_amd64_test_BUILDING.txt.gz

config/seeded-snaps appears to be empty. But looking at the seed git history I don't see why this would have changed? I'm confused.

I'm not sure what's going on with the build failures when using bartender, I wasn't testing in that way.

As for why the cpc builds stopped on the 19th, I do suspect it was my changes, but Łukasz followed up shortly after with a fix, so I'm not sure why the builds stopped since then.

For LXD being seeded or not, I traced this to livecd-rootfs 2.837.

* passing build log - lxd seeded - with livecd-rootfs 2.836 equivalent + some "set -x"
https://launchpad.net/~dbungert/+livefs/ubuntu/mantic/test/+build/467075/+files/buildlog_ubuntu_mantic_amd64_test_BUILDING.txt.gz

* failing build log - lxd not seeded - with livecd-rootfs 2.837 equivalent + "set -x"
https://launchpad.net/~dbungert/+livefs/ubuntu/mantic/test/+build/467080/+files/buildlog_ubuntu_mantic_amd64_test_BUILDING.txt.gz

As a result of that change, there is a difference in which seed is considered (old considered server, new considered cloud-image), but that difference sounds more correct so I don't want to just revert 2.837.

So for the lxd part of this, I suspect the answer is to seed the lxd snap in cloud-image, but that sounds like a cpc team call.

On Sun, 16 Jul 2023 at 03:15, Dan Bungert <email address hidden>
wrote:

> As a result of that change, there is a difference in which seed is
> considered (old considered server, new considered cloud-image), but that
> difference sounds more correct so I don't want to just revert 2.837.
>

Yeah no let's not revert that change please.

> So for the lxd part of this, I suspect the answer is to seed the lxd
> snap in cloud-image, but that sounds like a cpc team call.
>

Alternatively the ubuntu-cpc project could add_task both server and
cloud-image (the cloud-image seed includes ubuntu-server so cloud images
get all the _packages_ from the server seed, but not of course the snaps.
This is a mess, yes).

Status changed to 'Confirmed' because the bug affects multiple users.

So it sounds like the "correct" move may be to add snaps directly to `cloud-image` seed. It seems the breaking issue is that while a snap can be included in a seed, and germinate does the proper thing, a _metapackage_ produced from a seed does not reference any snaps.

so any snap listed in `server` should also be in `cloud-image` seed, otherwise the snaps will not get populated by germinate.

On Mon, Jul 24, 2023 at 03:06:15PM -0000, John Chittum wrote:
> So it sounds like the "correct" move may be to add snaps directly to
> `cloud-image` seed.

I think calling 'add_task install cloud-image server' from livecd-rootfs is
better than duplicating the list of snaps between the seeds.

> It seems the breaking issue is that while a snap can
> be included in a seed, and germinate does the proper thing, a
> _metapackage_ produced from a seed does not reference any snaps.

Correct, although by definition "add_task" does not refer to a metapackage,
it refers to a "task". And there is currently no cloud-image metapackage in
the archive.

The cloud-image seed currently refers to the ubuntu-server package as part
of the seed, which is how add_task pulls it in. If we changed the seed pod
to declare the cloud-image seed has a dependency on the server seed, would
that be enough to make the current livecd-rootfs implementation follow it
for snaps?

> The cloud-image seed currently refers to the ubuntu-server package as part
> of the seed, which is how add_task pulls it in. If we changed the seed pod
> to declare the cloud-image seed has a dependency on the server seed, would
> that be enough to make the current livecd-rootfs implementation follow it
> for snaps?

pardon my ignorance, but is that done in the STRUCTURE file? right now we have (snippet)

```
server: standard
server-cloud-minimal:
server-minimal: server-cloud-minimal
server-raspi: server raspi-common
server-ship-live: server-minimal
cloud-image: standard
cloud-minimal: server-cloud-minimal
```

So we'd add the dependency thusly?

```
cloud-image: standard server
```

daily lxd images are using a removed kernel.

daily cpc images are not updated because of this bug.

yet livecd-rootfs changes continue to be uploaded and released.

This is not ok, setting block-proposed-mantic on this bug report, to prevent livecd-rootfs migrating with more changes piled onto it, unless this bug is resolved.

> pardon my ignorance, but is that done in the STRUCTURE file?

That was what I had in mind, but the implementation says:

> def getTaskSeeds(task_headers, seedname):
> """Return the list of seeds used to generate a task from this seed.
>
> The list of packages in this task comes from this seed plus any
> other seeds listed in a Task-Seeds header.

So I've committed a change to add Task-Seeds: server to the cloud-image seed.

> setting block-proposed-mantic on this bug report, to prevent livecd-rootfs
> migrating with more changes piled onto it, unless this bug is resolved.

Not an appropriate use of the tag. The tag is for preventing regressions from entering the release pocket. It is not a "stop the line" marker when you want to escalate that a team should prioritize fixing a particular issue, and it also does not mean we will revert any of the already-staged changes in mantic-proposed.

Anyway, the proper fix for this looks to not be in the livecd-rootfs code at all. I'm starting a test build now with the seed change and will report back on the bug.

oh wait, I can't start a test build. Asking CPC to.

> pardon my ignorance, but is that done in the STRUCTURE file?
> So we'd add the dependency thusly?
>
> cloud-image: standard server

negative. The structure from germinate output appears completely ignored (except checking its presence) by add_task, and seeds appear to only be expanded if added explicitly. The cloud-image seed depends on ubuntu-server so happens to have most of the seed (minus the snaps) without previously having it added explicitly.

For fixing the current issue, adding the server task where we add cloud-image would do the trick (we almost do it for all archs; see linked branch)

For the longer term, I think it would make sense to actually walk the germinate structure and recursively add tasks. I've tested that but I'm unsure of the implications of such a change, apart from also having minimized point to cloud-minimal instead of cloud-image.

ACK the seed change does fix the issue for server images, but also breaks minimal (which now has snaps). We'll need to fix those separately.

On Thu, 27 Jul 2023 at 03:55, Steve Langasek <email address hidden>
wrote:

> > pardon my ignorance, but is that done in the STRUCTURE file?
>
> That was what I had in mind, but the implementation says:
>

Yeah, the STRUCTURE file is actually kind of the opposite of a dependency:
if STRUCTURE says seed2: seed1 then the package list for seed2 will not
include any packages that are in seed1! The use case for this is package
pools on ISOs, for example the declaration

server-ship-live: server-minimal

Means that we do not end up with any packages in the pool that are always
going to be installed in the target system anyway.

I had forgotten / failed to find the task-seed header which was one of the
reasons I hadn't gotten around to replying to this bug!

This is all a big mess yes.

It looks like the change in -proposed is failing autopkgtest which may be preventing other packages from releasing right now[0].

Seems related to minimal builds (as Simon pointed out). Is this being worked on? I don't see a bug for it.

[0] - https://autopkgtest.ubuntu.com/packages/l/livecd-rootfs/mantic/amd64

I uploaded a change that might have helped but apparently it didn't (see the 23.10.5 upload). I'm looking at it now (and am quite confused!)

Thanks for taking a look at the new failures! I don't know much about livecd-rootfs right now, but just wanted to make sure the impact on packages in -proposed is known.