Ubuntu
linux package

mount-image-callback cannot mount partitioned disk image

Bug #1628336 reported by Scott Moser on 2016-09-28

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	linux (Ubuntu)	Fix Released	Medium	Stefan Bader
	Yakkety	Fix Released	Medium	Stefan Bader

Bug Description

This fails on 4.8.0-17-generic and passes on 4.4.0-9136-generic

$ sudo apt-get update
$ sudo apt-get install -qy cloud-image-utils
$ wget http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img
$ cp xenial-server-cloudimg-amd64-disk1.img disk.img
$ sudo mount-image-callback xenial-server-cloudimg-amd64-disk1.img -v ls _MOUNTPOINT_
replaced string _MOUNTPOINT_ in arguments arg 1
waiting on pidfile for /dev/nbd0 in /sys/block/nbd0/pid
connected xenial-server-cloudimg-amd64-disk1.img (qcow2) to /dev/nbd0. waiting for device.
partitioned disk.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
gave up on waiting for /dev/nbd0p1
$ echo $?
1

On an identical system other than 'apt-get install linux-virtual' to get the new kernel, you see:

$ uname -r
4.4.0-9136-generic

$ sudo mount-image-callback disk.img -v ls _MOUNTPOINT_
replaced string _MOUNTPOINT_ in arguments arg 1
waiting on pidfile for /dev/nbd0 in /sys/block/nbd0/pid
connected disk.img (qcow2) to /dev/nbd0. waiting for device.
partitioned disk.
mounted /dev/nbd0p1 via qemu-nbd /dev/nbd0
invoking: MOUNTPOINT=/tmp/mount-image-callback.Y8D4lk/mp ls /tmp/mount-image-callback.Y8D4lk/mp
bin etc lib media proc sbin sys var
boot home lib64 mnt root snap tmp vmlinuz
dev initrd.img lost+found opt run srv usr
cmd returned 0. unmounting /tmp/mount-image-callback.Y8D4lk/mp

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: linux-image-4.8.0-17-generic 4.8.0-17.19
ProcVersionSignature: User Name 4.8.0-17.19-generic 4.8.0-rc7
Uname: Linux 4.8.0-17-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Sep 27 19:04 seq
crw-rw---- 1 root audio 116, 33 Sep 27 19:04 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.3-0ubuntu7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
Date: Wed Sep 28 00:58:49 2016
Ec2AMI: ami-00000438
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
PciMultimedia:

ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.8.0-17-generic root=UUID=072136f3-5666-4381-83bd-8d7175059be3 ro console=tty1 console=ttyS0
RelatedPackageVersions:
linux-restricted-modules-4.8.0-17-generic N/A
linux-backports-modules-4.8.0-17-generic N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: Ubuntu-1.8.2-1ubuntu1~cloud0
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-vivid
dmi.modalias: dmi:bvnSeaBIOS:bvrUbuntu-1.8.2-1ubuntu1~cloud0:bd04/01/2014:svnOpenStackFoundation:pnOpenStackNova:pvr12.0.3:cvnQEMU:ct1:cvrpc-i440fx-vivid:
dmi.product.name: OpenStack Nova
dmi.product.version: 12.0.3
dmi.sys.vendor: OpenStack Foundation

See original description

Tags:

Revision history for this message

Scott Moser (smoser) wrote on 2016-09-28:

CurrentDmesg.txt Edit (37.2 KiB, text/plain; charset="utf-8")
Dependencies.txt Edit (2.8 KiB, text/plain; charset="utf-8")
JournalErrors.txt Edit (2.7 KiB, text/plain; charset="utf-8")
Lspci.txt Edit (4.5 KiB, text/plain; charset="utf-8")
ProcCpuinfo.txt Edit (791 bytes, text/plain; charset="utf-8")
ProcInterrupts.txt Edit (1.6 KiB, text/plain; charset="utf-8")
ProcModules.txt Edit (2.5 KiB, text/plain; charset="utf-8")
UdevDb.txt Edit (69.4 KiB, text/plain; charset="utf-8")
WifiSyslog.txt Edit (55.3 KiB, text/plain; charset="utf-8")

Revision history for this message

Scott Moser (smoser) wrote on 2016-09-28:

script to show failure Edit (3.7 KiB, text/x-sh)

Attaching a script that shows the failure without downloading an image.

Revision history for this message

Brad Figg (brad-figg) wrote on 2016-09-28: Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status:	New → Confirmed

Leann Ogasawara (leannogasawara) on 2016-09-28

tags:

added: kernel-4.8

Revision history for this message

Stefan Bader (smb) wrote on 2016-09-28:

I can recreate the issue following the steps in the initial description. Interestingly there are some (rare) cases where things works on 4.8 as well. There is a difference in udev messages:

Not working:
KERNEL[3512.687794] change /devices/virtual/block/loop0 (block)
KERNEL[3512.704230] change /devices/virtual/block/loop0 (block)
KERNEL[3512.705301] change /devices/virtual/block/loop0 (block)
UDEV [3512.732209] change /devices/virtual/block/loop0 (block)
UDEV [3512.748186] change /devices/virtual/block/loop0 (block)
KERNEL[3512.752751] change /devices/virtual/block/nbd0 (block)
UDEV [3512.764683] change /devices/virtual/block/nbd0 (block)
UDEV [3512.772212] change /devices/virtual/block/loop0 (block)
KERNEL[3522.912384] change /devices/virtual/block/nbd0 (block)
UDEV [3522.919200] change /devices/virtual/block/nbd0 (block)

Working:
KERNEL[3528.095845] change /devices/virtual/block/loop0 (block)
KERNEL[3528.124184] change /devices/virtual/block/loop0 (block)
UDEV [3528.124410] change /devices/virtual/block/loop0 (block)
UDEV [3528.144352] change /devices/virtual/block/loop0 (block)
KERNEL[3528.148718] change /devices/virtual/block/nbd0 (block)
KERNEL[3528.151216] change /devices/virtual/block/nbd0 (block)
KERNEL[3528.151302] add /devices/virtual/block/nbd0/nbd0p1 (block)
UDEV [3528.163222] change /devices/virtual/block/nbd0 (block)
UDEV [3528.167442] change /devices/virtual/block/nbd0 (block)
UDEV [3528.173677] add /devices/virtual/block/nbd0/nbd0p1 (block)
KERNEL[3528.267889] change /devices/virtual/block/nbd0 (block)
KERNEL[3528.269940] remove /devices/virtual/block/nbd0/nbd0p1 (block)
UDEV [3528.275148] change /devices/virtual/block/nbd0 (block)
UDEV [3528.276144] remove /devices/virtual/block/nbd0/nbd0p1 (block)

Revision history for this message

Stefan Bader (smb) wrote on 2016-09-28:

Somehow looks like the creation of the nbd device does not reliably trigger the partition scanning in kernel. The following work-around would at least here get past the problem:

--- /usr/bin/mount-image-callback 2016-09-01 22:07:51.000000000 +0200
+++ mount-image-callback 2016-09-28 11:46:29.404822620 +0200
@@ -321,6 +321,7 @@
     debug 1 "unpartitioned disk."
    else
     debug 1 "partitioned disk."
+ blockdev --rereadpt $nbd
    fi
    ptnum=$nptnum
   fi

Stefan Bader (smb) on 2016-09-28

Changed in linux (Ubuntu):
assignee:	nobody → Stefan Bader (smb)
importance:	Undecided → Medium

Scott Moser (smoser) on 2016-09-28

description:

updated

Tim Gardner (timg-tpi) on 2016-09-29

Changed in linux (Ubuntu Yakkety):
status:	Confirmed → Fix Committed

Revision history for this message

Stefan Bader (smb) wrote on 2016-09-29:

Looking further into it, this is somehow caused by changes to the nbd module. Before 4.8 it was actually not creating any uevents on its own. Still there were events to be seen. With 4.8 they code changed to generate a change event, but also will delay the capacity change until the moment the socket is connected. And this seems to prevent whatever is otherwise producing the change event and triggering a partition scan. The nbd driver itself does not do that (like for example the loop driver does). So it could be that this had ever worked only because of the mysterious helper.

It seems that reverting a bit back to change the capacity as soon as it is set by ioctl (there are three ioctls which do this, so the values likely get touched multiple times, too) will again get the partition detected reliably. Not sure this can be guaranteed always...

Revision history for this message

Launchpad Janitor (janitor) wrote on 2016-10-05:

This bug was fixed in the package linux - 4.8.0-19.21

---------------
linux (4.8.0-19.21) yakkety; urgency=low

[ Tim Gardner ]

* Release Tracking Bug
- LP: #1629057

* 4.8.0 kernels do not complete boot process on VM (LP: #1627198)
- [Config] CONFIG_HARDENED_USERCOPY_PAGESPAN=n

* mount-image-callback cannot mount partitioned disk image (LP: #1628336)
- SAUCE: nbd: Only delay uevent until connected

  * Support snaps inside of lxd containers (LP: #1611078)
    - apparmor: add interface to be able to grab loaded policy
    - securityfs: update interface to allow inode_ops, and setup from vfs fns
    - apparmor: refactor aa_prepare_ns into prepare_ns and create_ns routines
    - apparmor: add __aa_find_ns fn
    - apparmor: add mkdir/rmdir interface to manage policy namespaces
    - apparmor: fix oops in pivot_root mediation
    - apparmor: fix warning that fn build_pivotroot discards const
    - apparmor: add interface to advertise status of current task stacking
    - apparmor: update policy permissions to consider ns being viewed/managed
    - apparmor: add per ns policy management interface
    - apparmor: bump domain stacking version to 1.2

  * linux-image-extra-4.8.0-17-generic does not provide many sound card modules
    (LP: #1628523)
    - [Config] CONFIG_ZONE_DMA=y for generic

* Yakkety - disable ARCH_ZX (LP: #1628503)
- [Config] armhf: disable ARCH_ZX

* Enable switchdev config parameter for Yakkety (LP: #1628241)
- [Config] CONFIG_NET_SWITCHDEV=y for amd64/arm64

  * Ubuntu 16.10 kernel v4.8: Installation failing on Habanero with Shiner card
    (LP: #1628009)
    - firmware: Update bnx2x to 7.13.1.0

* vNIC driver missing in 4.8 kernel package (LP: #1628187)
- [Config] Enable CONFIG_IBMVNIC=m

  * Yakkety - armhf: MFD_TPS65217 and REGULATOR_TPS65217 are boot essential
    (LP: #1628112)
    - [Config] armhf: MFD_TPS65217=y && REGULATOR_TPS65217=y

  * Miscellaneous Ubuntu changes
    - Rebase to v4.8-rc8
    - [Config] skip Ubuntu-4.8.0-18.20
    - [Config] missing modules in armhf/s390x

* Miscellaneous Ubuntu changes
- rebase to v4.8-rc8

-- Leann Ogasawara <email address hidden> Sun, 25 Sep 2016 12:13:35 -0700

This bug was fixed in the package linux - 4.8.0-19.21

---------------
linux (4.8.0-19.21) yakkety; urgency=low

[ Tim Gardner ]

* Release Tracking Bug
    - LP: #1629057

* 4.8.0 kernels do not complete boot process on VM (LP: #1627198)
    - [Config] CONFIG_HARDENED_USERCOPY_PAGESPAN=n

* mount-image-callback cannot mount partitioned disk image (LP: #1628336)
    - SAUCE: nbd: Only delay uevent until connected

* linux-image-extra-4.8.0-17-generic does not provide many sound card modules
    (LP: #1628523)
    - [Config] CONFIG_ZONE_DMA=y for generic

* Yakkety - disable ARCH_ZX (LP: #1628503)
    - [Config] armhf: disable ARCH_ZX

* Enable switchdev config parameter for Yakkety (LP: #1628241)
    - [Config] CONFIG_NET_SWITCHDEV=y for amd64/arm64

* Ubuntu 16.10 kernel v4.8: Installation failing on Habanero with Shiner card
    (LP: #1628009)
    - firmware: Update bnx2x to 7.13.1.0

* vNIC driver missing in 4.8 kernel package (LP: #1628187)
    - [Config] Enable CONFIG_IBMVNIC=m

* Yakkety - armhf: MFD_TPS65217 and REGULATOR_TPS65217 are boot essential
    (LP: #1628112)
    - [Config] armhf: MFD_TPS65217=y && REGULATOR_TPS65217=y

* Miscellaneous Ubuntu changes
    - Rebase to v4.8-rc8
    - [Config] skip Ubuntu-4.8.0-18.20
    - [Config] missing modules in armhf/s390x

* Miscellaneous Ubuntu changes
    - rebase to v4.8-rc8

-- Leann Ogasawara <leann.ogasawara@canonical.com>  Sun, 25 Sep 2016 12:13:35 -0700

Changed in linux (Ubuntu Yakkety):
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1728109

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntulinux package

mount-image-callback cannot mount partitioned disk image

Bug Description

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
linux package