[Ubuntu][Bionic] systemd caused kernel to hang on fsnotify wait-on-completion

Can you try using the hwe-18.04-edge kernel?

https://maas.io/docs/ubuntu-kernels

Hi,

It still gives the same behavior.
I ssh'd into the machine and syslog shows the following:

Apr 10 09:00:34 machine-name kernel: [ 78.597093] x86/PAT: ipmi-locate:2583 map pfn expected mapping type uncached-minus for [mem 0xbde1d000-0xbde1dfff], got write-back
Apr 10 09:00:34 machine-name kernel: [ 78.758144] x86/PAT: ipmi-locate:2583 map pfn expected mapping type uncached-minus for [mem 0xbde1c000-0xbde1cfff], got write-back
Apr 10 09:01:08 machine-name systemd[1]: Starting Stop ureadahead data collection...
Apr 10 09:01:08 machine-name systemd[1]: Stopping Read required files in advance...
Apr 10 09:01:08 machine-name systemd[1]: Started Stop ureadahead data collection.
Apr 10 09:01:25 machine-name kernel: [ 129.353796] INFO: rcu_sched detected stalls on CPUs/tasks:
Apr 10 09:01:25 machine-name kernel: [ 129.429260] 0-...!: (0 ticks this GP) idle=c18/0/0 softirq=2098/2098 fqs=0
Apr 10 09:01:25 machine-name kernel: [ 129.524289] (detected by 6, t=15044 jiffies, g=1622, c=1621, q=11120)
Apr 10 09:01:25 machine-name kernel: [ 129.613039] Sending NMI from CPU 6 to CPUs 0:
Apr 10 09:01:25 machine-name kernel: [ 129.613089] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x12/0x20
Apr 10 09:01:25 machine-name kernel: [ 129.614057] rcu_sched kthread starved for 15067 jiffies! g1622 c1621 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=0
Apr 10 09:01:25 machine-name kernel: [ 129.755325] rcu_sched I 0 9 2 0x80000000
Apr 10 09:01:25 machine-name kernel: [ 129.755332] Call Trace:
Apr 10 09:01:25 machine-name kernel: [ 129.755350] __schedule+0x24e/0x880
Apr 10 09:01:25 machine-name kernel: [ 129.755357] ? __switch_to_asm+0x41/0x70
Apr 10 09:01:25 machine-name kernel: [ 129.755363] schedule+0x2c/0x80
Apr 10 09:01:25 machine-name kernel: [ 129.755368] schedule_timeout+0x15d/0x350
Apr 10 09:01:25 machine-name kernel: [ 129.755375] ? __next_timer_interrupt+0xe0/0xe0
Apr 10 09:01:25 machine-name kernel: [ 129.755382] rcu_gp_kthread+0x53a/0x980
Apr 10 09:01:25 machine-name kernel: [ 129.755390] kthread+0x121/0x140
Apr 10 09:01:25 machine-name kernel: [ 129.755394] ? rcu_note_context_switch+0x150/0x150
Apr 10 09:01:25 machine-name kernel: [ 129.755399] ? kthread_create_worker_on_cpu+0x70/0x70
Apr 10 09:01:25 machine-name kernel: [ 129.755404] ret_from_fork+0x22/0x40

@maas team can the hang in impitool be isolated somehow for debugging?

MAAS instructs cloud-init to install ipmitool during commissioning. If you select "Skip configuring supported BMC controllers with a MAAS generated username and password" and "Allow SSH access and prevent machine powering off" ipmitool won't be installed and the machine will be left on after commissioning/testing to allow for debug. You can also boot into rescue mode on a failed host.

Thanks, I could commission and deploy the machine after disabling the BMC configuration step.
After the machine booted, I manually installed impitool and ran 'ipmitool lan print' successfully.

After that, I released the machine, and commissioned again with the options "Skip configuring supported BMC controllers with a MAAS generated username and password" and "Allow SSH access and prevent machine powering off".
I ssh'ed into the machine, and ran 'apt-get install ipmitools'.
The command hanged for 2 minutes, and then finally finished: https://pastebin.canonical.com/p/n2YFPFt5nY/
I opened another SSH session, which took ~30s to open, and took the output from dmesg:
https://pastebin.canonical.com/p/PkQXCy4gP4/

This was using the following config:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
$ uname -r
4.15.0-96-generic

The machine is still in this state and I can perform more tests.

Just re-deployed the machine to check that it worked with the same kernel, and it does:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
$ uname -r
4.15.0-96-generic

Summary...

For Ubuntu Bionic, dpkg triggers for systemd (237-3ubuntu10.39) might have caused systemd to hang:

[ 363.776878] wait_for_completion+0xba/0x140
[ 363.776890] __flush_work+0x15b/0x210
[ 363.776901] flush_delayed_work+0x41/0x50
[ 363.776908] fsnotify_wait_marks_destroyed+0x15/0x20
[ 363.776912] fsnotify_destroy_group+0x48/0xd0
[ 363.776917] inotify_release+0x1e/0x50
[ 363.776923] __fput+0xea/0x220
[ 363.776929] ____fput+0xe/0x10
[ 363.776935] task_work_run+0x9d/0xc0
[ 363.776942] exit_to_usermode_loop+0xc0/0xd0
[ 363.776947] do_syscall_64+0x121/0x130
[ 363.776954] entry_SYSCALL_64_after_hwframe+0x3d/0xa2

and

[ 364.050206] wait_for_completion+0xba/0x140
[ 364.050238] __synchronize_srcu.part.13+0x85/0xb0
[ 364.050248] synchronize_srcu+0x66/0xe0
[ 364.050256] fsnotify_mark_destroy_workfn+0x7b/0xe0
[ 364.050262] process_one_work+0x1de/0x420
[ 364.050267] worker_thread+0x228/0x410
[ 364.050272] kthread+0x121/0x140

and

[ 364.326985] wait_for_completion+0xba/0x140
[ 364.326988] __synchronize_srcu.part.13+0x85/0xb0
[ 364.326993] synchronize_srcu+0x66/0xe0
[ 364.326995] ? synchronize_srcu+0x66/0xe0
[ 364.326996] fsnotify_connector_destroy_workfn+0x4a/0x80
[ 364.326998] process_one_work+0x1de/0x420
[ 364.326999] worker_thread+0x253/0x410
[ 364.327001] kthread+0x121/0x140

All stack traces seem to come from "fsnotify" subsystem and waiting on delayed work (completion) for fsnotify marks destruction after a inotify_release() was called. Completion did not happen for the past 2 minutes. Without a kernel dump it is hard to tell if completion was still ok - due to kthread being overloaded doing scheduled work and/or the marks group destruction - or there was a dead lock for the completion due to a kernel bug.

If this is reproducible, I think that having a kernel dump would help identifying the issue. I'm letting the kernel team to handle this and marking all other issues as dealt per previous comments.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1872021

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

The Eoan Ermine has reached end of life, so this bug will not be fixed for that release

For documentation purposes,
(after working on items related to bug 1802021 / 1872021 / this bug)

The stack traces in this issue would suggest a duplicate of bug 1802021, fixed in kernel 4.15.0-47.50, but this happens in 4.15.0-96, which has that fix.

So, looking at comments #5 and #6, the issue seems particular to the MAAS deploy boot/environment, as it doesn't happen on the deployed system with the same kernel (i.e., same system and kernel, different boot/environment), even when running similar commands.

As such, this issue doesn't seem to be that kernel bug, but a manifestation with similar symptoms at the kernel level, due to what seems to be an underlying system/platform issue.

Supporting 'evidence' seems to be the lack of additional comments on those bugs after Fix Released, and later on this bug, perhaps due to some system/firmware upgrade, for example.

[Expired for linux (Ubuntu Bionic) because there has been no activity for 60 days.]

[Expired for linux (Ubuntu Focal) because there has been no activity for 60 days.]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]