stale file handle when restarting ntp on overlayfs (maas)

Bug #1719733 reported by Scott Moser
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Critical
Seth Forshee
Artful
Fix Released
Critical
Seth Forshee

Bug Description

MAAS is currently failing to deploy artful.

[ 0.000000] Kernel command line: nomodeset iscsi_target_name=iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-ppc64el-ga-17.10-artful-daily iscsi_target_ip=10.245.71.3 iscsi_target_port=3260 iscsi_initiator=wichita ip=::::wichita:BOOTIF ip6=off ro root=/dev/disk/by-path/ip-10.245.71.3:3260-iscsi-iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-ppc64el-ga-17.10-artful-daily-lun-1 overlayroot=tmpfs overlayroot_cfgdisk=disabled cc:{'datasource_list': ['MAAS']}end_cc cloud-config-url=http://10.245.71.3:5240/MAAS/metadata/latest/by-id/node-45f9009a-0bc4-11e6-9882-00163e5b6994/?op=get_preseed apparmor=0 log_host=10.245.71.3 log_port=514 --- console=hvc0 BOOTIF=01-6c-ae-8b-01-e9-00^M

...

[ 60.064994] cloud-init[4805]: ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.^M
[ 60.216965] cloud-init[4805]: Created symlink /etc/systemd/system/multi-user.target.wants/ntp.service → /lib/systemd/system/ntp.service.^M
[ 60.837682] cloud-init[4805]: Processing triggers for libc-bin (2.26-0ubuntu1) ...^M
[ 60.851733] cloud-init[4805]: Processing triggers for ureadahead (0.100.0-20) ...^M
[ 61.003403] cloud-init[4805]: Processing triggers for systemd (234-2ubuntu10) ...^M
[ 61.183944] cloud-init[4805]: dpkg: error: error removing old backup file '/var/lib/dpkg/status-old': Stale file handle^M
[ 62.202113] cloud-init[4805]: E: Sub-process /usr/bin/dpkg returned an error code (2)^M
@

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: linux-image-4.13.0-11-generic 4.13.0-11.12
ProcVersionSignature: User Name 4.13.0-11.12-generic 4.13.1
Uname: Linux 4.13.0-11-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 26 20:32 seq
 crw-rw---- 1 root audio 116, 33 Sep 26 20:32 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.7-0ubuntu1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
Date: Tue Sep 26 20:33:41 2017
Ec2AMI: ami-00000181
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.13.0-11-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
RelatedPackageVersions:
 linux-restricted-modules-4.13.0-11-generic N/A
 linux-backports-modules-4.13.0-11-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.1-1ubuntu1~cloud0
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-zesty
dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.1-1ubuntu1~cloud0:bd04/01/2014:svnOpenStackFoundation:pnOpenStackNova:pvr15.0.2:cvnQEMU:ct1:cvrpc-i440fx-zesty:
dmi.product.family: Virtual Machine
dmi.product.name: OpenStack Nova
dmi.product.version: 15.0.2
dmi.sys.vendor: OpenStack Foundation

Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :

Seems likely related to:
  https://github.com/coreos/bugs/issues/2152

There are three overlayfs commits that add ESTALE returns in 4.13. Two of them only affect mount time, and the third is torvalds/linux@b9ac5c2. Other 4.13 overlayfs changes may have introduced new calls to existing functions returning ESTALE, but I haven't checked.
aka, might be an issue *only* in 4.13
https://patchwork.kernel.org/patch/9955803/

tags: added: kernel-key
Changed in linux (Ubuntu Artful):
status: Confirmed → Triaged
Revision history for this message
Seth Forshee (sforshee) wrote :

4.13.0-12.13 was promoted to artful-release today and has a fix for overlayfs erroneously returning ESTALE, can you try this and see if it fixes the issue?

Changed in linux (Ubuntu Artful):
assignee: nobody → Seth Forshee (sforshee)
status: Triaged → Incomplete
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

I can still see this error while deploying ARM64 Artful today

[ 121.632228] cloud-init[2022]: ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
[ 122.172457] cloud-init[2022]: Created symlink /etc/systemd/system/multi-user.target.wants/ntp.service → /lib/systemd/system/ntp.service.
[ 123.362260] cloud-init[2022]: Processing triggers for libc-bin (2.26-0ubuntu1) ...
[ 123.396725] cloud-init[2022]: Processing triggers for ureadahead (0.100.0-20) ...
[ 123.929345] cloud-init[2022]: Processing triggers for systemd (234-2ubuntu10) ...
[ 124.235965] cloud-init[2022]: dpkg: error: error removing old backup file '/var/lib/dpkg/status-old': Stale file handle
[ 128.358899] cloud-init[2022]: E: Sub-process /usr/bin/dpkg returned an error code (2)

The interesting thing is that this issue seem to be only affecting the ARM64 nodes, as I can deploy it on a amd64/i386 node (affecting arm64 image maybe?)

Revision history for this message
Witold Krecicki (wpk) wrote :

I hit the same problem with current (http://images.maas.io/ephemeral-v3/daily/artful/amd64/20170926/) version.

Revision history for this message
Witold Krecicki (wpk) wrote :

Version 20170927 seems to have fixed it.

Revision history for this message
Seth Forshee (sforshee) wrote :

Thanks for the test results!

Changed in linux (Ubuntu Artful):
status: Incomplete → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.