"overlayfs" no longer exists

Thanks for taking the time to report this bug and helping to make Ubuntu better. We appreciate the difficulties you are facing, but this appears to be a "regular" (non-security) bug. I have unmarked it as a security issue since this bug does not show evidence of allowing attackers to cross privilege boundaries nor directly cause loss of data/privacy. Please feel free to report any other bugs you may find.

Oh, sorry, I don't mean to set it as a 'security' bug.
Thanks!

I've marked cloud-utils as relevant here tto fix mount-image-callback as well.

for some specific information on different ubuntu kernels:

-- latest kernel before 4.8 entered yakkety --
$ uname -r
4.4.0-9136-generic

$ grep overlay /lib/modules/4.4.0-9136-generic/modules.alias
alias overlayfs overlay
alias fs-overlayfs overlay
alias fs-overlay overlay

$ sudo modprobe -v overlayfs
insmod /lib/modules/4.4.0-9136-generic/kernel/fs/overlayfs/overlay.ko

$ lsmod | grep overlay
overlay

$ grep overlay /proc/filesystems
nodev overlayfs
nodev overlay

-- 4.8 kernel in yakkety --
$ uname -r
4.8.0-15-generic

$ grep overlay /lib/modules/4.8.0-15-generic/modules.alias
alias fs-overlay overlay

$ modprobe -v overlayfs
modprobe: FATAL: Module overlayfs not found in directory /lib/modules/4.8.0-15-generic

$ modprobe -v overlay
insmod /lib/modules/4.8.0-15-generic/kernel/fs/overlayfs/overlay.ko

$ grep overlay /proc/filesystems
nodev overlay

-- 3.13 kernel (trusty GA) --
$ uname -r
3.13.0-96-generic

$ grep overlay /lib/modules/3.13.0-96-generic/modules.alias
alias fs-overlayfs overlayfs

$ sudo modprobe -v overlay
modprobe: FATAL: Module overlay not found.

$ sudo modprobe -v overlayfs
insmod /lib/modules/3.13.0-96-generic/kernel/fs/overlayfs/overlayfs.ko

$ grep overlay /proc/filesystems
nodev overlayfs

Hello 宋文武, or anyone else affected,

Accepted cloud-initramfs-tools into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-initramfs-tools/0.29ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I can test this too if we build a copy for xenial for arm (raspi3).

I've verified functional in yakkety in uvt-kvm.
Below 'ussh' is a wrapper arond uvt-kvm
  https://gist.github.com/smoser/88a5a77ab0debf268b945d46314ea447
for reader purposes, it can be assumed that it operates like 'lxc exec'. ie,
   ussh name command arguments
executed command and arguments on 'name'.

# make a new system.
name=sm-y1
uvt-kvm create $name release=yakkety

# enable proposed
$ ussh $name sudo sh -c 'echo "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main" >/etc/apt/sources.list.d/proposed.list && apt-get update -qy'
# install a package
ussh $name sudo apt-get install -qy overlayroot

# show.
$ ussh $name dpkg-query --show overlayroot
overlayroot 0.29ubuntu1

# enable overlayroot without recurse
$ ussh "$name" sh -c 'echo overlayroot=tmpfs:recurse=0 | sudo tee /etc/overlayroot.local.conf'

# reboot
$ ussh $name reboot

# show
$ ussh $name grep overlay /proc/mounts
overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0

# turn on recurse
$ ussh "$name" sh -c 'echo overlayroot=tmpfs:recurse=1 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf'

# reboot
$ ussh $name reboot

# show /proc/mounts
$ ussh "$name" grep overlay /proc/mounts
overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0
/media/root-ro/mnt /mnt overlay rw,relatime,lowerdir=/media/root-ro/mnt,upperdir=/media/root-rw/overlay/mnt,workdir=/media/root-rw/overlay-workdir/mnt 0 0

$ ussh $name grep overlay /etc/fstab
# This fstab is in an overlay. The real one can be found at
# sudo overlayroot-chroot
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
/media/root-ro/mnt /mnt overlay lowerdir=/media/root-ro/mnt,upperdir=/media/root-rw/overlay/mnt,workdir=/media/root-rw/overlay-workdir/mnt 0 0

# show kernel version
$ ussh $name uname -r
4.8.0-17-generic

# boot into normal root
$ ussh sm-y1 sudo sh -c 'overlayroot-chroot rm /etc/overlayroot.local.conf && reboot'

This bug was fixed in the package cloud-initramfs-tools - 0.29ubuntu1

---------------
cloud-initramfs-tools (0.29ubuntu1) yakkety; urgency=medium

  * overlayroot: support 'overlay' filesystem explicitly rather than
    relying on ubuntu specific kernel module 'overlayfs'. (LP: #1493188)

 -- Scott Moser <email address hidden> Fri, 23 Sep 2016 17:00:37 -0400

fix committed in cloud-utils trunk at revno 303.

This bug was fixed in the package cloud-utils - 0.29-0ubuntu4

---------------
cloud-utils (0.29-0ubuntu4) yakkety; urgency=medium

  * sync to trunk at revno 303
    * mount-image-callback: support 'overlay' filesystem type rather
      than ubuntu specific 'overlayfs' (LP: #1493188)

 -- Scott Moser <email address hidden> Tue, 27 Sep 2016 21:20:58 -0400

Hello 宋文武, or anyone else affected,

Accepted cloud-initramfs-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-initramfs-tools/0.27ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I've verified this on xenial using the steps provided in the test case above.
$ uname -r
4.8.0-28-generic
$ dpkg -S /boot/vmlinuz-$(uname -r)
linux-image-4.8.0-28-generic: /boot/vmlinuz-4.8.0-28-generic
$ dpkg-query --show overlayroot
overlayroot 0.27ubuntu1.3

This bug was fixed in the package cloud-initramfs-tools - 0.27ubuntu1.3

---------------
cloud-initramfs-tools (0.27ubuntu1.3) xenial; urgency=medium

  * sync with upstream at 0.32ubuntu1 (revno 129)
    * overlayroot: fix overlayroot=crypt with newer initramfs-tools
      (LP: #1634310)
    * overlayroot: support random seed from systemd systemd-random-seed.service
    * cloud-initramfs-dyn-netconf: Further adjustments to support
      IPV6 and the new DEVICE6 and net6-DEVICE.conf files. (LP: #1621615)
    * overlayroot: support 'overlay' filesystem explicitly rather than
      relying on ubuntu specific kernel module 'overlayfs'. (LP: #1493188)
    * overlayroot: write debug to /run/initramfs not /dev/.initramfs but
      support writing to /dev/.initramfs if that is all there is. (LP: #1485752)
    * overlayroot: fix overlayroot if recurse=0 was not provided.
      This fuctionality was lost since workdir support was added. (LP: #1619459)
    * whitespace cleanup.

 -- Scott Moser <email address hidden> Mon, 28 Nov 2016 20:12:51 -0500

The verification of the Stable Release Update for cloud-initramfs-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

this fix is present in cloud-utils 0.30

With the move to Bionic host kernels on the builders, the Xenial MAAS images fail to build. This is because cloud-utils mount-image-callback remains unfixed in Xenial while the Bionic kernel does not have the overlayfs kernel module. The following patch at minimum is required in Xenial:

https://bazaar.launchpad.net/~smoser/cloud-utils/trunk.lp1493188/revision/303

Could we have this fix SRUed to Xenial please?

Proposed branch is up for xenial handling overlayfs issues.

https://code.launchpad.net/~chad.smith/ubuntu/+source/cloud-utils/+git/cloud-utils/+ref/upload-xenial-handle-missing-overlayfs

 The review has been approved and a test ppa is setup with cloud-utils version 0.27-0ubuntu25.2 on Xenial which should resolve this issue.

If there is an opportunity to this this to confirm the fix, that would be great:

Verification procedure:
sudo add-apt-repository ppa:chad.smith/cloud-utils-uploads -y
sudo apt-get update
sudo apt-get install cloud-utils=0.27-0ubuntu25.2

@chad.smith I have used the cloud-utils in your ppa to build Xenial maas-images on a bionic host to confirm that things are working now.

Hello 宋文武, or anyone else affected,

Accepted cloud-utils into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-utils/0.27-0ubuntu25.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted cloud-utils (0.27-0ubuntu25.2) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

lxc/2.0.11-0ubuntu1~16.04.3 (ppc64el, amd64, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#cloud-utils

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

While the test case run on xenial did show overlay properly mounted across reboot. The generated deb doesn't contain the applied patches to mount-image-callback per the upload changelog. Needs investigation before claiming victory here.

Extended the SRU verification test case to validate that expected
changes are present in mount-image-callback which loop over "fstype in overlay overlayfs".

I can confirm both that proper bits are uploaded in any overlay fs is functional using cloud-utils (0.27-0ubuntu25.2).

#### Updated test script

#!/bin/bash

set -x

cat > install_hwe_kernel_and_overlay_root.sh <<EOF
#!/bin/bash

echo "3.) Enable proposed, install HWE kernel and install overlayroot to show fix."
mirror=http://archive.ubuntu.com/ubuntu
echo deb \$mirror xenial-proposed main | tee /etc/apt/sources.list.d/proposed.list
apt-get update
apt-get install -qy cloud-utils overlayroot linux-generic-hwe-16.04 | egrep 'linux-image|cloud-utils'
EOF

expect_overlayfs_mounts(){
  expected_mounts=$1
  overlay_mounts=$(multipass exec $VM mount | grep overlay -c)
  [[ "$expected_mounts" = "$overlay_mounts" ]] && echo "SUCCESS: found $overlay_mounts overlay mounts" || echo "FAILURE: expected $expected_mounts overlay mounts found $overlay_mounts"
}

VM=sru-1493188
RELEASE=xenial
multipass launch daily:$RELEASE -n $VM
multipass exec $VM -- cloud-init status --wait --long > /dev/null
echo --- No overlayfs mounts on initial boot
expect_overlayfs_mounts 0
multipass transfer install_hwe_kernel_and_overlay_root.sh $VM:.
multipass exec $VM -- sudo bash ./install_hwe_kernel_and_overlay_root.sh
echo "--- confirm overlay overlayfs patch present in mount-image-callback"
fs_types=$(multipass exec $VM -- grep -c "for fstype in overlay overlayfs" /usr/bin/mount-image-callback)
[[ "1" = "$fs_types" ]] || echo "FAILURE: didn't find 'for fstype in overlay overlayfs'"
multipass exec $VM -- sudo sh -c "echo overlayroot=tmpfs:recurse=0 | sudo tee -a /etc/overlayroot.local.conf"
multipass restart $VM
echo "--- expect 1 overlayfs mounts because using HWE kernel"
multipass exec $VM -- cloud-init status --wait --long
expect_overlayfs_mounts 1
multipass exec $VM -- sudo sh -c "echo overlayroot=tmpfs:recurse=1 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf"
multipass restart $VM
expect_overlayfs_mounts 1
multipass exec $VM -- grep overlay /etc/fstab || echo "FAIL: didn't find overlayroot"
multipass exec $VM -- grep success /run/initramfs/overlayroot.log || echo "FAIL: didn't find overlayroot"

### Xenial verification results
+ cat
+ VM=sru-1493188
+ RELEASE=xenial
+ multipass launch daily:xenial -n sru-1493188
Launched: sru-1493188
+ multipass exec sru-1493188 -- cloud-init status --wait --long
+ echo --- No overlayfs mounts on initial boot
--- No overlayfs mounts on initial boot
+ expect_overlayfs_mounts 0
+ expected_mounts=0
++ multipass exec sru-1493188 mount
++ grep overlay -c
+ overlay_mounts=0
+ [[ 0 = \0 ]]
+ echo 'SUCCESS: found 0 overlay mounts'
SUCCESS: found 0 overlay mounts
+ multipass transfer install_hwe_kernel_and_overlay_root.sh sru-1493188:.
+ multipass exec sru-1493188 -- sudo bash ./install_hwe_kernel_and_overlay_root.sh
3.) Enable proposed, install HWE kernel and install overlayroot to show fix.
d...

This bug was fixed in the package cloud-utils - 0.27-0ubuntu25.2

---------------
cloud-utils (0.27-0ubuntu25.2) xenial; urgency=medium

  * debian/patches/lp-1493188-support-overlay-filesystem:
    mount-image-callback: support 'overlay' filesystem type rather
    than ubuntu specific 'overlayfs' (LP: #1493188)
  * debian/patches/lp-1630274-mount-overlay-first:
    mount-image-callback: try mounting overlay rather than checking first
    (LP: #1630274)

 -- Chad Smith <email address hidden> Fri, 04 Sep 2020 08:47:08 -0600