failed to install packages that are not in pool, flawed network detection

Hey Ken. This is the part that failed.

        Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/target', 'eatmydata', 'apt-get', '--quiet', '--assume-yes', '--option=Dpkg::options::=--force-unsafe-io', '--option=Dpkg::Options::=--force-confold', 'install', 'efibootmgr', 'grub-efi-amd64', 'grub-efi-amd64-signed', 'shim-signed'] with allowed return codes [0] (capture=False)
        Reading package lists...
        Building dependency tree...
        Reading state information...
        Package efibootmgr is not available, but is referred to by another package.
        This may mean that the package is missing, has been obsoleted, or
        is only available from another source

        E: Package 'efibootmgr' has no installation candidate
        E: Unable to locate package grub-efi-amd64-signed
        E: Unable to locate package shim-signed

For u-d-i, please add the needed packages to the correct ship-live seed.
For curtin, curthooks are producing a flawed sources.list and that should be resolved.

I confirm the issue on hardware with EFI + SB enabled.

Summary of problem on Subiquity side:

TLDR: we want to patch subiquity to read the list of default routes when we receive a route_change event

* when we believe that we are offline, we intentionally limit the scope of where we look for packages to just the cdrom, this part is working fine
* when we are actually online, if we are acting as if we are offline (has_network == False), then that is incorrect behavior
* has_network is derived from the existence of default routes - if the default_route info is not correct, we risk the wrong has_network state
* subiquity receives route_change events from probert
* probert builds route_change events from libnl-route-3
* per https://github.com/thom311/libnl/issues/226#issuecomment-527888667, libln-route-3 route events are hashed using certain keys, and metric is not among those keys
* on live-server, which is not using network manager, adjusting the link state to active / inactive will cause a single route event to be generated, and subiquity can rely on that to know if there is a default route or not
* on desktop, which is using network manager, a different sequence of network events is generated. `ip monitor route|grep default` is helpful here. Using `ip monitor`, we can see the following sequence:

link goes active - ip monitor route reports 3 events
1) default route added at metric 20100 for interface X for family Y
2) default route added at metric 100 for interface X for family Y
3) default route deleted at metric 20100 for interface X for family Y

libnl-route-3 reports only 2 events
- default route added for interface X for family Y
- default route deleted for interface X for family Y

* subiquity sees only that interface X has no default route (more correctly, it sees a default route for a few milliseconds) - the unintended coalescing of the two events with different metrics means the events aren't enough info for subiquity to know conclusively that there is a default route.

In summary we need to use these events as a trigger to read the default routes instead of using the events directly to maintain the list.

Some notes from testing
* why do VMs work better - I believe these VMs usually or always have ipv6 in them, which is causing a different set of route events, which happens to produce a has_network state more out of luck than anything
* why does server work better - simpler set of route events with no coalesing hiding one of the ones we need
* why does physical installs of desktop work only sometimes on some hardware - different ordering of events from libnl-route3

Wow amazing debugging. What a mess.

A little further digging shows that the crazy high metric is related to connectivity checking: networkmanager adds a 20000 "penalty" to interfaces until they have demonstrated connectivity by accessing http://connectivity-check.ubuntu.com./ successfully. It's unfortunate (and maybe slightly hilarious) that this approach runs straight into the libnl-route issue you found.

Your proposed change sounds reasonable. I guess we should ignore routes that have a metric over 20000.

I wonder how all this interacts with the changes ogayot is making around mirror checking. After all, when that is done we'll know if the configured archive is accessible, which is the most consequential decision hooked on the value of has_network today. I guess we probably don't even try to configure a mirror if has_network is false so we do want to get this right.

https://github.com/canonical/subiquity/pull/1569

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
https://iso.qa.ubuntu.com/qatracker/reports/bugs/2004659

We believe this issue has been resolved in Subiquity 23.04.2.

If you had tested this with a pre-final version of Ubuntu 23.04, it's
recommended to download the final install media.

For testing with Ubuntu Server 22.04.x or 20.04.x, when running
Subiquity, you should offered a new version of the installer. Please
take that update to version 23.04.2 or later to get the fix.

If this is still a problem for you, please make a comment and set the state
back to New. Thank you for the bug report.