subiquity

Mantic autoinstall.yaml fails on late commands

Bug #2040654 reported by Mike Ferreira on 2023-10-25

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	subiquity	Fix Released	Critical	Unassigned
	ubuntu-desktop-installer	Fix Released	Undecided	Unassigned

Bug Description

Description:
Late commands in the autoinstall.yaml file worked for 23.04 installer. "As Far As We Know": Nothing has been changed for that process between 23.04 and 23.10. I have verified that from the example yaml files in /snap/ubuntu-desktop-installer/1[0-9][0-9][0-9]/usr/share/doc/*/examples/

Yet the autoinstall scripts we have working for 23.04, fail on late commands for 23.10.They seem to work for internal bash commands, but not for starting installer bash scripts used for installing packages and other environment setup.

This worked for 23.04, no longer with 23.10 or 23.04.1.

RE: https://ubuntuforums.org/showthread.php?t=2491776

Expected Behavior:
Late Command to run custom late-install scripts works.

Steps to Reproduce:

Use this yaml, or sililar file --+v
###
version: 1
  refresh-installer:
          update: yes
  keyboard:
    layout: se
  locale: en_US.UTF-8
  timezone: Europe/Stockholm
  identity:
    hostname: unassigned-hostname
    username: ubuntu
    password: ubuntu (not the real one)
  source:
    search_drivers: true
    id: ubuntu-desktop-minimal
  storage:
    layout:
      name: direct
  updates: all
  late-commands:
    - mkdir /target/opt/packages
    - mkdir /target/opt/scripts
    - cp -r /cdrom/packages /target/opt
    - cp -r /cdrom/nocloud/scripts /target/opt
    - curtin in-target -- bash /opt/scripts/post-install1.sh
    - curtin in-target -- bash /opt/scripts/post-install2.sh
    - curtin in-target -- bash /opt/scripts/post-install3.sh
###
Use logger in those scripts to log that they were called an ran to verify that it is working...

If no log entries, then it failed.

*** The scripts and everything is getting copied over. The scripts are not running
All those scripts can run manually after the install. Just not from within the installer.

Alternate work-arounds:
- We have tried adding the full path /usr/bin/bash to the late command,. That also fails...
- We have tried chmod'ing the scripts to make them executable and calling them directly. That also fails

Has something changed?

The Noble repo's are up now. We start testing that as soon as the packages are really ready for that. Right now they are still the Mantic packages, so this release is also affected.

ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: subiquity (unknown)
ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
Uname: Linux 6.5.0-9-generic x86_64
NonfreeKernelModules: zfs
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: pass
CasperVersion: 1.486
CloudArchitecture: x86_64
CloudID: nocloud
CloudName: unknown
CloudPlatform: nocloud
CloudSubPlatform: seed-dir (/var/lib/cloud/seed/nocloud)
CurrentDesktop: ubuntu:GNOME
Date: Wed Oct 25 19:40:17 2023
DesktopInstallerRev: 1267
LiveMediaBuild: Ubuntu 23.10.1 "Mantic Minotaur" - Release amd64 (20231016.1)
ProcEnviron:
LANG=C.UTF-8
PATH=(custom, no user)
SHELL=/bin/bash
TERM=xterm-256color
XDG_RUNTIME_DIR=<set>
SourcePackage: subiquity
UpgradeStatus: No upgrade log present (probably fresh install)

See original description

Tags:

Revision history for this message

Mike Ferreira (mafoelffen) wrote on 2023-10-25:

CrashDB.txt Edit (162 bytes, text/plain; charset="utf-8")
DesktopInstallerLog.txt Edit (3.9 KiB, text/plain; charset="utf-8")
ProbeData.txt Edit (43.3 KiB, text/plain; charset="utf-8")
ProcCpuinfoMinimal.txt Edit (1.4 KiB, text/plain; charset="utf-8")

description:

updated

Dan Bungert (dbungert) on 2023-10-25

Changed in subiquity:
status:	New → Triaged
importance:	Undecided → Critical
tags:	added: foundations-todo

Revision history for this message

Dan Bungert (dbungert) wrote on 2023-10-25:

Thanks for the report.

This problem was created while attempting to fix LP: #2032961, however it has caused the problem you see here.

The next steps are to revert the changes for LP: #2032961, and then issue new stable builds when possible.

Changed in subiquity:
status:	Triaged → In Progress

Revision history for this message

Mike Ferreira (mafoelffen) wrote on 2023-10-25 (last edit on 2023-10-26):

Thank you Dan. So after that is reverted, then a new Mantic ISO image (forthcoming) will be released (first) in Daily's for us to test. Then to the public in cdrom releases?

Revision history for this message

Dan Bungert (dbungert) wrote on 2023-10-26:

Hi Mike,

> a new Mantic ISO image

Snaps, plus the ability for the installers to self-refresh, mean that in the overwhelming number of situations, we should deliver the fix as a snap update. No ISO refresh should be expected for this bug.

I expect a beta channel snap with this revert inside of 24 hours, at which point autoinstall users (the affected users) can pick up this proposed fix with:

```
  refresh-installer:
    update: yes
    channel: beta
```

Revision history for this message

Mike Ferreira (mafoelffen) wrote on 2023-10-28:

I wasn't sure if that would work for autoinstall. But happy that is does. That makes a lot more sense as a long range answer to updating changes than an ISO respin.

By adding that refresh-installer section to the autoinstall.yaml file...
autoinstall.yaml...

It is ready yet?

Revision history for this message

Mike Ferreira (mafoelffen) wrote on 2023-10-31:

@Dan Bungert (dbungert) -- Still waiting on word when to test this..

Revision history for this message

Dan Bungert (dbungert) wrote on 2023-11-03:

A beta of the server version is ready for test by the above instructions. Forgot to update the bug about this and I've been off for a few days, sorry for not updating sooner.

We'll continue to work on an equivalent for Desktop.

Revision history for this message

madjax (madjax) wrote on 2023-11-09:

curtin-install.log Edit (86.5 KiB, text/plain)

Hello
Still doesn't work. See attached curtin-install.log. Also ubuntu_desktop_installer.log shows "2023-11-09 08:19:39.965118 WARNING subiquity_server: Unable to find the subiquity_client package. Falling back to the current working dir: /home/ubuntu"

Revision history for this message

Mike Ferreira (mafoelffen) wrote on 2023-11-10:

Yup. And... We don't have a working Noble iso yet, so I cannot test it on that yet. Checking every morning though. Because I want to test all the example autoinstall.yamls in ubuntu-desktop-installer to verify them.

Revision history for this message

madjax (madjax) wrote on 2023-11-13:

#10

subiquity-server-info.log Edit (4.9 KiB, text/plain)

When I looked into /var/log/installer/subiquity-server-info.log I found more info. Makes it makes sense to you devs :) Se attached file

Revision history for this message

Dan Bungert (dbungert) wrote on 2023-11-14:

#11

@madjax - you are testing desktop. The beta listed here is for server.

Revision history for this message

Mike Ferreira (mafoelffen) wrote on 2023-11-15:

#12

LOL. Dang.

Revision history for this message

madjax (madjax) wrote on 2023-11-15:

#13

@dbungert ok I understand, but isn't desktop also important that it works? :) How will the issues on desktop be addressed? It stopped working after 23.04. I know it was new installer for the desktop version then, but it is very important to me that it works.
My university wishes to have a fully functional Ubuntu 24.04 Desktop LTS available with autoinstall by next year (no pressure on me...haha). That is why I have been experimenting with 23.04 and 23.10. We have 22.04 Desktop LTS automated with preseed on the older installer, but it has its limitations.
If I can show a automated install with 23.10/24.04 I can get a real project going with real funding for Ubuntu Pro licenses (Landscape)...and build a support team around it.
I hope you can help me :)

Revision history for this message

Mike Ferreira (mafoelffen) wrote on 2023-12-04 (last edit on 2023-12-04):

#14

@Dan Bungert-- Tested with the Beta channel and it now works. Can we get those changes moved to Stable?

Once that is done, this bug should be resolved.

Revision history for this message

James Paton-Smith (jamesps) wrote on 2024-01-10:

#15

Any movement on this? I encountered this bug in the new Ubuntu 23.10 desktop installer. If we can get the changes pushed to Stable, the desktop installer could be updated too.

Revision history for this message

Dan Bungert (dbungert) wrote on 2024-02-22:

#16

A fix for this issue has been released in Subiquity 24.02.1 for Ubuntu live-server, and also in the latest build for Ubuntu 23.10.1 desktop. You can find this build now on the stable channel for both those cases.

To pick up this fix, please allow the installer snap to update, or you can use the directive `refresh-installer: { update: yes }` in autoinstall to achieve the same result.