nvme multipath does not report path relationships

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

------- Comment From <email address hidden> 2018-07-09 06:18 EDT-------
Canonical, any update on this..?

Kernel team, looks like the root cause identified by IBM (taken from description of the bug) is:

== Comment: #8 - Hari Krishna Bathini <> - 2018-06-26 06:06:13 ==
The dump target (/var/crash) is on NVMe device (also, the root disk).
But the kdump initrd is not being built with nvme driver modules.
Eventually, nvme disk is not found and kdump kernel is hitting the
initramfs shell. Using the default initrd, which has the nvme driver
modules included, dump was captured successfully.

Can someone from Canonical take a look at this and comment on why
nvme modules are not included in kdump initrd despite it being the
root disk..

Thanks
Hari

I will look at this later today. Will send updates on it.

As we use MODULES=dep for kdump initrd, that might fail to include the proper modules. It would be a bug on initramfs-tools-core.

Can you change /etc/initramfs-tools/initramfs.conf to have MODULES=dep instead of MODULES=most and run mkinitramfs -o /tmp/initrd.img `uname -r`, then send me output?

Also, I need the data usually collected for kernel bugs. /sys/ at least comes to mind, but also complete dmesg after normal boot and some other important files to detect root mountpoint, like /etc/fstab, /proc/cmdline and /proc/mounts.

Thank you.
Cascardo.

I have managed to test kdump and initramfs with MODULES=dep on an emulated nvme, using qemu. Both worked just fine. The sysfs tree seems reasonable too, in respect to how initramfs finds the necessary modules.

So, I would really need the specific details on that failure case, that is, the sysfs tree and all the rest I requested in the previous message.

Thank you.
Cascardo.

------- Comment on attachment From <email address hidden> 2018-07-23 07:12 EDT-------

Attached woo normal boot logs & kdump console logs with modules=dep

------- Comment on attachment From <email address hidden> 2018-07-23 07:13 EDT-------

Attached dmesg logs from initramfs state from woo

------- Comment From <email address hidden> 2018-07-23 07:30 EDT-------
Indira, I think Cascardo is looking for /tmp/initrd.img-`uname -r` file
whne MODULES=dep is used. Also, sosreport?

(In reply to comment #26)
> Created attachment 128855 [details]
> woo_kdump_nvmedisk_console logs

btw, looks like PHB initialization is failing. So, whether nvme driver
is present or not in the initrd is secondary here, the device initialization
itself could be at fault. What is the f/w level. Is it latest?

Thanks
Hari

------- Comment From <email address hidden> 2018-07-23 07:35 EDT-------
(In reply to comment #28)
> Indira, I think Cascardo is looking for /tmp/initrd.img-`uname -r` file
> whne MODULES=dep is used. Also, sosreport?
>
> (In reply to comment #26)
> > Created attachment 128855 [details]
> > woo_kdump_nvmedisk_console logs
>
> btw, looks like PHB initialization is failing. So, whether nvme driver
> is present or not in the initrd is secondary here, the device initialization
> itself could be at fault. What is the f/w level. Is it latest?

also, the initial console log is different from this.. Was the f/w on the system
updated as well..

Thanks
Hari

------- Comment From <email address hidden> 2018-07-23 08:14 EDT-------
Also both logs were placed in

banner@9.3.117.11 (don2rry) : /home/banner

banner@banner:~> ls -l sosreport-woo-20180723065534.tar.xz
-rw------- 1 banner banner 7171548 Jul 23 07:05 sosreport-woo-20180723065534.tar.xz
banner@banner:~> ls -l initrd.img
-rw-r--r-- 1 banner banner 24119847 Jul 23 07:04 initrd.img
banner@banner:~>

------- Comment on attachment From <email address hidden> 2018-07-23 08:11 EDT-------

Attached /tmpinitrd.imng file

------- Comment on attachment From <email address hidden> 2018-07-23 08:12 EDT-------

Attached sosreport when MODULES=set used

Can you run apport-collect -p linux 1778844 ?

Thank you.
Cascardo.

------- Comment From <email address hidden> 2018-07-24 06:36 EDT-------
(In reply to comment #36)
> Can you run apport-collect -p linux 1778844 ?
>
> Thank you.
> Cascardo.

HI Cascardo,

I have tried to collect , but it fails as below

root@woo:~# apport-collect -p linux 1778844
ERROR: connecting to Launchpad failed: [Errno 110] Connection timed out
You can reset the credentials by removing the file "/root/.cache/apport/launchpad.credentials"
root@woo:~# export http_proxy="http://10.33.11.31:3128"
root@woo:~# apport-collect -p linux 1778844
ERROR: connecting to Launchpad failed: [Errno 110] Connection timed out
You can reset the credentials by removing the file "/root/.cache/apport/launchpad.credentials"
root@woo:~#

Regards,
Indira

------- Comment From <email address hidden> 2018-07-25 01:28 EDT-------
(In reply to comment #34)
> Created attachment 128858 [details]
> sosreport_moduels=set
>
> Attached sosreport when MODULES=set used

Cascardo, is the sosreport no good?

Thanks
Hari

Hi, Hari.

The sosreport should be enough. I sent that message without an updated view of the bug, so did not notice the attachments. They came on the same day I asked. Just realized now that they are present. Let me look at them.

Now, if kdump works with MODULES=most, then the PHB initialization could require a module that is not in the initramfs. I know some Intel systems require their vmd module that adds a new PCI domain. Is there some new pci host module these days on these new Power systems?

Regards.
Cascardo.

Ah, I see what you meant about the PHB. The two new attached logs are not at all similar to the original ones, maybe pointing to an unrelated problem.

As for the sosreport, the /sys/ directory is missing almost everything that I would need to debug this. So, I need any other form of collecting /sys/. Maybe just tar czf lp1778844_sys.tar.gz /sys/.

Cascardo.

------- Comment From <email address hidden> 2018-09-20 07:06 EDT-------
Hi,

We lost the old environment as jmet got returned.
Got a new Witherspoon jmet and retried the kdump scenario & issue recreated on latest ubuntu180401 kernel (4.15.0-34-generic) kernel.

> Triggered kdump with default crashkernel parameter and it went to initramfs state
root@wax:~# cat /proc/cmdline
root=UUID=38d0124f-14d8-4098-a746-03f9bda5c22e ro crashkernel=8192M splash quiet crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@128M

> Collected console & dmesg logs from same state
> Rebooted system & collected sosreport, tar czf lp1778844_sys.tar.gz /sys/
> Also tried comment#23 steps & collected /tmp/initrd.img

root@wax:~# cat /etc/initramfs-tools/initramfs.conf | grep MODULES
# MODULES: [ most | netboot | dep | list ]
MODULES=dep
root@wax:~# mkinitramfs -o /tmp/initrd.img
W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast
root@wax:~# uname -r
4.15.0-34-generic
root@wax:~# ls -l /tmp/initrd.img
-rw-r--r-- 1 root root 24128581 Sep 20 04:59 /tmp/initrd.img

> Attaching all above logs & if i am unable to attach any of the logs because of size issue. Logs are placed in the path

banner.isst.aus.stglabs.ibm.com [banner/don2rry ] : /home/banner/169067_sep20

> Attaching console logs from first step till last step ( starting from system booting with latest kernel level, trigger crash , initramfs state etc...)

Thanks & Regards,
Indira

------- Comment on attachment From <email address hidden> 2018-09-20 07:31 EDT-------

Attached IPMI console logs from start of the tests till end.

------- Comment on attachment From <email address hidden> 2018-09-20 07:34 EDT-------

Attached sosreport from wax_kdump_initramfs state

------- Comment on attachment From <email address hidden> 2018-09-20 07:38 EDT-------

Attached initrd.img file from wax

------- Comment From <email address hidden> 2018-09-20 07:42 EDT-------
Unable to attach "lp1778844_sys.tar.gz" file because of size, please refer to logs path .Also test machine is taken back for other release testing.

banner.isst.aus.stglabs.ibm.com [banner/don2rry ] : /home/banner/169067_sep20

Regards,
Indira

------- Comment From <email address hidden> 2018-09-25 04:34 EDT-------
Hello Canonical

Please download the files using Anonymous FTP and download the files from following path

ftp://testcase.software.ibm.com/fromibm/linux/lp1778844_sys.tar.gz

Note:
Files will remain in /fromibm/ for 3 business days before being deleted.

------- Comment From <email address hidden> 2018-09-27 02:42 EDT-------
Cascardo/Canonical, Did you get a chance to look at the logs attached?

I managed to download the lp1778844_sys.tar.gz from FTP in time. It has some huge resource files, so I have to play a little with the tarball. It looks like some symlinks were tarred as empty files, so maybe I'll miss something, but I am investigating right now. I'll let you know if I need any other information. I will look at the other files as well.

So, this is a multipath disk, whose parent is the nvme-subsys0 device, not nvme0 device. The latter is a PCI device, but the former is a virtual one. Some code to add slaves/holders relationships was added, but reverted. I will investigate any further discussion about this upstream, or look into how to relate those nodes in a way we can just fix this in initramfs-tools.

So, the hidden disk was introduced by hch as commit 8ddcd653257c18a669fcb75ee42c37054908e0d6 just to support nvme multipath. This thing prevented lsblk from working when there were slaves under the multipath nvme, as lsblk would postpone processing those, and lsblk could not process the paths themselves, because they have no dev_t.

Some discussion about reverting that slaves/holders patch is under the revert submission, and they decided to go on with the revert and think about how to fix that later. A link for one of the discussion messages is below.

https://www.spinics.net/lists/stable/msg222846.html

Going forward, we could either use nvme cli, embed the assumption that kernel won't change and some of the numbers are kept (like the subsys instance is embedded in both the multipath device and the underlying devices), or introduce this new paths subdir in the kernel. Any of those solutions would require changes to initramfs-tools. The latter would require a kernel change.

The only way of not changing initramfs-tools is going back to using slaves relationships. That means we would need to get rid of the hidden disks thing. This always worked for dm-multipath, but I believe this required some use of blkdev and that is one thing hch wanted to avoid, to have a very fast path for doing multipath.

I don't like the idea of using nvme cli that much, as it would be a hard dependency on initramfs-tools. I don't like assuming some of the implementation details on the kernel numbering/naming of those devices. So, I will take a stab at providing the paths interface.

Cascardo.

As a workaround, I suggest setting nvme-core.ko multipath=0 parameter. Can you try that, rebooting, recreating the kdump-tools initrd, then crashing?

Cascardo.

i downloaded that, and tried to poke it.
First of all, i've never seen a 77MB tarball of /sys alone.
Also unpacking that, is dangerous as it does explode in size. However, is debugging that, please only _list_ the files present in the tarball and only extract things you need/want.

Reuploaded to http://people.canonical.com/~xnox/lp1778844_sys.tar.gz for ease of access by canonical engineers.

------- Comment From <email address hidden> 2018-09-27 22:06 EDT-------
(In reply to comment #61)
> As a workaround, I suggest setting nvme-core.ko multipath=0 parameter. Can
> you try that, rebooting, recreating the kdump-tools initrd, then crashing?

Hi Cascardo,

Where/How to set this parameters? You mean, passing them as boot parameters?

Thanks
Hari

Just create a /etc/modprobe.d/nvme.conf file with the following contents, recreate initrd, remove kdump initrds so they are recreated in the next boot, then reboot.

# cat /etc/modprobe.d/nvme.conf
options nvme-core multipath=0
# update-initramfs -u
# rm /var/lib/kdump/initrd.img*
# reboot

First piece of the puzzle submitted:
http://lists.infradead.org/pipermail/linux-nvme/2018-September/020174.html

Tested with a small change on initramfs-tools, though I will wait for comments on the first one before submitting that to initramfs-tools upstream.

I have been working upstream to get this fixed on the nvme driver, but it might still take some time, I already have done a back and forth with hch. Will send a v3 later this month.

Meanwhile, there are multiple workarounds we documented so far in the bug, namely:

1) Disable nvme multipath, which is done by setting the multipath module
parameter for the nvme module on modprobe.d config files, regenerate
initrd, reboot, then regenerate kdump initrd.

2) Use MODULES=most instead of MODULES=dep on
/etc/kernel/postinst.d/kdump-tools and regenerate kdump initrd.

3) Force insert nvme modules into kdump initrd. Namely, just copy the
/boot/ initrd file to replace the one in /var/lib/kdump/.

Upstream discussion is progressing. I have tested some patches from hch, will send him some feedback later today and post the links here.

https://lore.kernel.org/linux-block/20181213113549.GA7543@calabresa/T/#t

Upstream discussion in progress.

Upstream is a little reluctant to accept my changes, and seems to suggest that we find out the layout of the paths some other way (which is possible, but would be such a special case on initramfs-tools, only for nvme).

Cascardo.

On 01/15/2019 09:26 AM, Thadeu Lima de Souza Cascardo wrote:
> Upstream is a little reluctant to accept my changes, and seems to
> suggest that we find out the layout of the paths some other way (which
> is possible, but would be such a special case on initramfs-tools, only
> for nvme).
>
> Cascardo.
>
Hmm, I am really sorry about that - sux to do all that work and get that
kind of response.

So making the suggested changes seems like real work to me. Do you have
a rough estimate?

Terry

I have pushed a version of initramfs-tools to my ppa that seems to fix the problem. Can IBM test it?

add-apt-repository ppa:cascardo/kdump2

It's a version for bionic. Only initramfs-tools should come from this source. kdump-tools and makedumpfile should be from bionic-updates or bionic.

After that, remove the initrd images from /var/lib/kdump/, reboot, then test the crash.

So:

add-apt-repository ppa:cascardo/kdump2
apt install initramfs-tools
rm /var/lib/kdump/initr*
reboot

After reboot:

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

------- Comment From <email address hidden> 2019-02-06 01:54 EDT-------
Have installed below patch and retried kdump scenario with root disk as nvme disk. Now system is not dropping to initramfs state its moving further but not saving dump file (core) properly.

initramfs-tools_0.130ubuntu3.6+nvme1_all.deb
initramfs-tools-bin_0.130ubuntu3.6+nvme1_ppc64el.deb
initramfs-tools-core_0.130ubuntu3.6+nvme1_all.deb

root@woo:~# dpkg -l | grep initramfs
ii busybox-initramfs 1:1.27.2-2ubuntu3 ppc64el Standalone shell setup for initramfs
ii initramfs-tools 0.130ubuntu3.6+nvme1 all generic modular initramfs generator (automation)
ii initramfs-tools-bin 0.130ubuntu3.6+nvme1 ppc64el binaries used by initramfs-tools
ii initramfs-tools-core 0.130ubuntu3.6+nvme1 all generic modular initramfs generator (core tools)
ii libklibc 2.0.4-9ubuntu2 ppc64el minimal libc subset for use with initramfs
root@woo:~#

rm /var/lib/kdump/initr*
reboot
After reboot triggered crash

> Attached console logs

Regards,
Indira

------- Comment on attachment From <email address hidden> 2019-02-06 01:56 EDT-------

Attached woo kdump scenario console logs

The system has too many devices, so too many drivers are using too much memory. So, the system goes out-of-memory before makedumpfile is complete. Please, reserve more memory on crashkernel, reboot the system and repeat the crash test.

To reserve more memory on crashkernel, edit file /etc/default/grub.d/kdump-tools.cfg and change crashkernel value to crashkernel=384M, for example, then run update-grub.

Thanks.
Cascardo.

------- Comment From <email address hidden> 2019-02-06 09:27 EDT-------
(In reply to comment #82)
> The system has too many devices, so too many drivers are using too much
> memory. So, the system goes out-of-memory before makedumpfile is complete.
> Please, reserve more memory on crashkernel, reboot the system and repeat the
> crash test.
>
> To reserve more memory on crashkernel, edit file
> /etc/default/grub.d/kdump-tools.cfg and change crashkernel value to
> crashkernel=384M, for example, then run update-grub.
>
> Thanks.
> Cascardo.

Hi Cascardo

As per SUSE document recommendation used 4096M crash kernel and with this value i have seen OOM traces during kdump.

root@woo:~# free -h
total used free shared buff/cache available
Mem: 251G 7.2G 243G 16M 567M 243G
Swap: 2.0G 0B 2.0G
root@woo:~# cat /proc/cmdline
root=UUID=df4c0331-9e6c-4592-9e0c-84f431ecb1f7 ro splash quiet crashkernel=4096M

Will try to increase to some more above 4096M and retry it.

Regards,
Indira

------- Comment From <email address hidden> 2019-02-06 09:45 EDT-------
Triggered crash with patch using crashkernel=8192M and did not see OOM traces and dump copied properly as below

root@woo:/var/crash/201902060928# pwd
/var/crash/201902060928
root@woo:/var/crash/201902060928# ls -l
total 469212
-rw------- 1 root root 114308 Feb 6 09:29 dmesg.201902060928
-rw------- 1 root root 480474149 Feb 6 09:29 dump.201902060928
root@woo:/var/crash/201902060928# date
Wed Feb 6 09:44:42 EST 2019
root@woo:/var/crash/201902060928#

Regards,
Indira

------- Comment From <email address hidden> 2019-02-12 01:11 EDT-------
(In reply to comment #83)
[...]
> Will try to increase to some more above 4096M and retry it.

The recommendations are based on a few assumptions that don't apply to all machines.
So, hitting OOM in this case is not surprising in that sense. As increasing crashkernel
value helped, we can consider the original issue reported here resolved..

Thanks. Closing out the bug.

Hi, Andrew and Hari.

The original issue is only resolved by using an initramfs-tools from my ppa. We still need to get this fix included in our releases. I will start working to get this in disco.

Cascardo.

------- Comment From <email address hidden> 2019-02-12 06:15 EDT-------
(In reply to comment #87)
> Hi, Andrew and Hari.
>
> The original issue is only resolved by using an initramfs-tools from my ppa.
> We still need to get this fix included in our releases. I will start working
> to get this in disco.
>
> Cascardo.

Right, Cascardo.
Waiting for the fix to be released in an official build..

That's an OOM problem. Just increase the reserved crashkernel memory size and test again, please.

Next step: Thadeu to propose the diff described in comment #53 into the archive.
(If this incorrect, please flag.)

Hi, Andrew, with the debdiff attached and ubuntu sponsors team subscribed, now we should wait for someone on that team to sponsor that update. With the SRU template attached, the only thing blocking that sponsorship now is that disco is now on freeze, so the sponsor is the one who would decide whether this warrants an upload during the freeze.

Hi Thadeu, thanks for the update.

------- Comment From <email address hidden> 2019-04-12 05:41 EDT-------
Hi ,

Any update on this bug, when this fix is expected in official build.

Please update the approximate date, as issue exists with latest Ubuntu180401 kernel - 4.15.0-47-generic.

Regards,
Indira

initramfs-tools uploaded for disco. Thadeu, do you want to prepare a separate debdiff for bionic and cosmic?

This bug was fixed in the package initramfs-tools - 0.131ubuntu19

---------------
initramfs-tools (0.131ubuntu19) disco; urgency=medium

  * Add modules for nvme path components on multipath nvme. LP: #1778844

 -- Thadeu Lima de Souza Cascardo <email address hidden> Tue, 26 Mar 2019 14:49:29 -0300

Thadeu, the test case for the SRU needs to describe exactly the steps that will be followed to verify the correctness of this bugfix, so that anyone (with the necessary hardware) can follow them.

@vorlon, test case updated.

Thanks.
Cascardo.

Added to the SRU queue on 22nd April. Thanks!

Hello bugproxy, or anyone else affected,

Accepted initramfs-tools into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.131ubuntu15.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello bugproxy, or anyone else affected,

Accepted initramfs-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.130ubuntu3.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

------- Comment From <email address hidden> 2019-05-21 08:26 EDT-------
Installed latest initramfs-tools & kdump-tools packages from -proposed kernel as below

root@woo:~# dpkg -l initramfs*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================================-=====================-=====================-=======================================================================
ii initramfs-tools 0.130ubuntu3.8 all generic modular initramfs generator (automation)
ii initramfs-tools-bin 0.130ubuntu3.8 ppc64el binaries used by initramfs-tools
ii initramfs-tools-core 0.130ubuntu3.8 all generic modular initramfs generator (core tools)

root@woo:~# dpkg -l kdump-tools*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================================-=====================-=====================-=======================================================================
ii kdump-tools 1:1.6.5-1ubuntu1~18.0 ppc64el scripts and tools for automating kdump (Linux crash dumps)

Triggered crash with nvme disk as root disk ,crash triggered properly and dump file saved in /var/crash path

root@woo:~# uname -a
Linux woo 4.15.0-51-generic #55-Ubuntu SMP Wed May 15 14:26:12 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

crash logs:
***********
root@woo:~# cd /var/crash
root@woo:/var/crash# ls -l
total 580K
drwxrwxrwt 4 root root 4.0K May 21 07:51 ./
drwxr-xr-x 12 root root 4.0K May 18 12:23 ../
drwxr-xr-x 2 root root 4.0K May 20 13:12 201905201311/
drwxr-xr-x 2 root root 4.0K May 21 07:41 201905210740/
-rw-r--r-- 1 root root 228 May 21 07:51 kexec_cmd
-rw-r----- 1 root root 31K May 20 13:16 linux-image-4.15.0-51-generic-201905201311.crash
-rw-r----- 1 root root 177K May 21 07:51 linux-image-4.15.0-51-generic-201905210740.crash
-rw------- 1 root root 348K May 18 13:11 ssl-cert.0.crash

root@woo:/var/crash/201905210740# ls
total 538M
drwxr-xr-x 2 root root 4.0K May 21 07:41 ./
drwxrwxrwt 4 root root 4.0K May 21 07:51 ../
-rw------- 1 root root 994K May 21 07:41 dmesg.201905210740
-rw------- 1 root root 537M May 21 07:41 dump.201905210740
root@woo:/var/crash/201905210740# pwd
/var/crash/201905210740
root@woo:/var/crash/201905210740#

Issue is fixed with latest initramfs-tools package.

Regards,
Indira

Thank you for taking the time to verify this stable release fix. We have noticed that you have used the verification-done tag for marking the bug as verified and would like to point out that due to a recent change in SRU bug verification policy fixes now have to be marked with per-release tags (i.e. verification-done-$RELEASE). Please remove the verification-done tag and add one for the release you have tested the package in. Thank you!

https://wiki.ubuntu.com/StableReleaseUpdates#Verification

Thanks for the verification in comment #67. From the comment, I believe this was verified with bionic. Could you also verify with Cosmic and post your results?
Thanks.

This bug was fixed in the package initramfs-tools - 0.130ubuntu3.8

---------------
initramfs-tools (0.130ubuntu3.8) bionic; urgency=medium

  * Add modules for nvme path components on multipath nvme. LP: #1778844

 -- Thadeu Lima de Souza Cascardo <email address hidden> Tue, 16 Apr 2019 21:25:56 -0300

The verification of the Stable Release Update for initramfs-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

I installed cosmic on power9 (boston) a non-nvme system, installed initramfs-tools from -proposed and install kdump-tools. Triggered a dump, but I got a kernel panic. Please let me know if I am doing something wrong here.

ubuntu@tiselius:~$ uname -a
Linux tiselius 4.18.0-21-generic #22-Ubuntu SMP Wed May 15 13:12:45 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
ubuntu@tiselius:~$

ubuntu@tiselius:~$ apt policy initramfs-tools
initramfs-tools:
  Installed: 0.131ubuntu15.2
  Candidate: 0.131ubuntu15.2
  Version table:
 *** 0.131ubuntu15.2 500
        500 http://ports.ubuntu.com/ubuntu-ports cosmic-proposed/main ppc64el Packages
ubuntu@tiselius:~$

ubuntu@tiselius:~$ apt policy kdump-tools
kdump-tools:
  Installed: 1:1.6.5-1ubuntu1~18.10.1
  Candidate: 1:1.6.5-1ubuntu1~18.10.1
  Version table:
 *** 1:1.6.5-1ubuntu1~18.10.1 500
        500 http://ports.ubuntu.com/ubuntu-ports cosmic-updates/main ppc64el Packages
        100 /var/lib/dpkg/status
     1:1.6.4-2ubuntu1 500
        500 http://ports.ubuntu.com/ubuntu-ports cosmic/main ppc64el Packages
ubuntu@tiselius:~$

ubuntu@tiselius:~$ sudo kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.18.0-21-generic
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.18.0-21-generic
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="root=UUID=295f571b-b731-4ebb-b752-60aadc80fc1b ro console=hvc0 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll noirqdistrib nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
ubuntu@tiselius:~$

ubuntu@tiselius:~$ cat /proc/cmdline
root=UUID=295f571b-b731-4ebb-b752-60aadc80fc1b ro console=hvc0 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@128M
ubuntu@tiselius:~$

ubuntu@tiselius:~$ dmesg | grep Reser
[ 0.000000] Reserving 4096MB of memory at 128MB for crashkernel (System RAM: 131072MB)
[ 0.000000] cma: Reserved 6560 MiB at 0x0000200e62000000
ubuntu@tiselius:~$

root@tiselius:/home/ubuntu# echo 1 > /proc/sys/kernel/sysrq
root@tiselius:/home/ubuntu# echo c > /proc/sysrq-trigger

tiselius login: [ 150.167288] cloud-init[4529]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.10.1 running 'modules:final' at Mon, 10 Jun 2019 19:53:40 +0000. Up 149.80 seconds.
[ 150.167687] cloud-init[4529]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.10.1 finished at Mon, 10 Jun 2019 19:53:40 +0000. Datasource DataSourceMAAS [http://10-245-64-0--21.maas-internal:5248/MAAS/metadata/]. Up 150.11 seconds
[ 360.313029] kdump-tools[4915]: Stopping kdump-tools: * unloaded kdump kernel
[ 376.552452] kdump-tools[10449]: Starting kdump-tools: * Creating symlink /var/lib/kdump/vmlinuz
[ 376.553743] kdump-tools[10449]: * Creating symlink /var/lib/kdump/initrd.img
[ 376.585085] kdump-tools[10449]: Modified cmdline:root=UUID=295f571b-b731-4ebb-b752-60aadc80fc1b ro console=hvc0 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll noirqdistrib nousb elfcorehdr=158784K
[ 376.953223] kdump-tools[10449]: * loaded kdump kerne...

Hi, Manoj.

There is nothing wrong in what you are doing here. This seems to be an unrelated bug. I am looking at any known problems on cpuidle for power9. Can you, please, open a different bug report, and try a different system for regression testing? From the log message, it doesn't seem there is anything related to initramfs changes, but a real kernel bug, also, apparently, unrelated to kdump.

Thanks.
Cascardo.

@Thadeu, here is a new bug report that tracks this issue: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1832388 We dont have a system in house with NVme, we will need IBM to verify -proposed with cosmic and report back here.

verification-needed-cosmic is blocking publishing the next initramfs-tools update in cosmic.

Am I ok to trump the current initramfs-tools in cosmic-proposed with a new upload?

Do we want to back out this change?

Following call with IBM, verification testing is in progress. Hoping to have test results posted in the next 24 hours (approx).

------- Comment From <email address hidden> 2019-06-25 10:14 EDT-------
I tried this on a witherspoon with NVMe, but I believe I also hit LP 1778844:

ubuntu@tempbmc1-p1:~$ [ 116.427517] sysrq: SysRq : Trigger a crash
[ 116.427566] Unable to handle kernel paging request for data at address 0x00000000
[ 116.427619] Faulting instruction address: 0xc0000000008205b8
[ 116.427668] Oops: Kernel access of bad area, sig: 11 [#1]
[ 116.427762] LE SMP NR_CPUS=2048 NUMA PowerNV
[ 116.427838] Modules linked in: vmx_crypto crct10dif_vpmsum nouveau ast ttm drm_kms_helper drm at24 ofpart drm_panel_orientation_quirks fb_sys_fops syscopyarea cmdlinepart sysfillrect ipmi_powernv sysimgblt powernv_flash ipmi_devintf i2c_algo_bit ibmpowernv mtd ipmi_msghandler opal_prd uio_pdrv_genirq uio sch_fq_codel ip_tables x_tables autofs4 nvme crc32c_vpmsum nvme_core ahci tg3 libahci
[ 116.428241] CPU: 80 PID: 5322 Comm: bash Kdump: loaded Not tainted 4.18.0-24-generic #25-Ubuntu
[ 116.428366] NIP: c0000000008205b8 LR: c000000000821444 CTR: c000000000820590
[ 116.428464] REGS: c00020393dc5fa00 TRAP: 0300 Not tainted (4.18.0-24-generic)
[ 116.428581] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28422228 XER: 20040000
[ 116.428675] CFAR: c000000000821440 DAR: 0000000000000000 DSISR: 42000000 IRQMASK: 0
[ 116.428675] GPR00: c000000000821444 c00020393dc5fc80 c00000000177ca00 0000000000000063
[ 116.428675] GPR04: 0000000000000001 00000000000004f1 9000000000009033 0000000035c80060
[ 116.428675] GPR08: 0000000000000007 0000000000000001 0000000000000000 9000000000001003
[ 116.428675] GPR12: c000000000820590 c000203fff652480 00000dda71119760 0000000000000000
[ 116.428675] GPR16: 00000dda7bd4d910 00000dda710b4a48 00000dda71119760 00000dda71038b00
[ 116.428675] GPR20: 0000000000000000 0000000000000001 00000dda710c70b8 00007fffd6124ea4
[ 116.428675] GPR24: 00007fffd6124ea0 00000dda7111afe4 0000718ed556bfa8 c0000000016736b0
[ 116.428675] GPR28: 0000000000000004 0000000000000002 0000000000000063 c0000000015f2e38
[ 116.429616] NIP [c0000000008205b8] sysrq_handle_crash+0x28/0x30
[ 116.429708] LR [c000000000821444] __handle_sysrq+0xf4/0x230
[ 116.429792] Call Trace:
[ 116.429843] [c00020393dc5fc80] [c000000000821428] __handle_sysrq+0xd8/0x230 (unreliable)
[ 116.429976] [c00020393dc5fd10] [c000000000821bc8] write_sysrq_trigger+0x68/0x90
[ 116.430099] [c00020393dc5fd40] [c0000000004a99a4] proc_reg_write+0x84/0xc0
[ 116.430202] [c00020393dc5fd70] [c0000000003f992c] __vfs_write+0x3c/0x70
[ 116.430305] [c00020393dc5fd90] [c0000000003f9bc8] vfs_write+0xd8/0x220
[ 116.430407] [c00020393dc5fde0] [c0000000003f9f44] ksys_write+0x64/0x110
[ 116.430512] [c00020393dc5fe30] [c00000000000b288] system_call+0x5c/0x70
[ 116.430605] Instruction dump:
[ 116.430649] 4bfff9e1 4bfffe44 3c4c00f6 3842c470 7c0802a6 60000000 39200001 3d42001d
[ 116.430760] 394a90f0 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00f6 3842c440
[ 116.430867] ---[ end trace 4f8cfae97c893934 ]---
[ 116.534209]
[ 116.534308] Sending IPI to othe

Looks like that might be blocking this one, I will get that mirrored over, too.

------- Comment From <email address hidden> 2019-06-25 10:16 EDT----...

------- Comment From <email address hidden> 2019-06-25 12:13 EDT-------
Michael, actually this sounds like LP 1828597
It is worth trying with maxcpus=1 instead of nr_cpus=1

------- Comment From <email address hidden> 2019-06-25 13:53 EDT-------
Yeah. The issue was indeed with "nr_cpus=1".
Not hitting initramfs shell after updating to initramfs-tools version in -proposed
and using "maxcpus=1" (LP bug 1828597)

------- Comment (attachment only) From <email address hidden> 2019-06-25 13:54 EDT-------

This bug was fixed in the package initramfs-tools - 0.131ubuntu15.2

---------------
initramfs-tools (0.131ubuntu15.2) cosmic; urgency=medium

  * Add modules for nvme path components on multipath nvme. LP: #1778844

 -- Thadeu Lima de Souza Cascardo <email address hidden> Tue, 16 Apr 2019 21:19:25 -0300