[arm64] compute nodes unstable after upgrading from 4.2 to 4.4 kernel

perf data is at : https://private-fileshare.canonical.com/~axino/swirlix01.perf.data.LP1602577.xz

I'll run apport from this host, note that it's now running 4.2.

Version string of the unstable kernel was :
swirlix01 kernel: [ 0.000000] Linux version 4.4.0-30-generic (buildd@bos01-arm64-006) (gcc version 4.8.4 (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.3) ) #49~14.04.1-Ubuntu SMP Thu Jun 30 22:20:09 UTC 2016 (Ubuntu 4.4.0-30.49~14.04.1-generic 4.4.13)

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

This change was made by a bot.

Do you have a machine that could be used to test some kernels? If so, we can perform a kernel bisect to identify the commit that caused the regression.

Yes we can do that. If you have documentation for this process, please let us know.

Thanks

It could basically just require that you install and test kernels the we build.

cking mentioned that bisecting between 4.2 <-> 4.4 on this same platform caused various non-bootable issues in 4.3. Because of this, we may want to first try the mainline kernel to see if we can perform a "Reverse" bisect. That is if the bug is actually fixed in mainline.

Can you test the following mainline kernel:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.7-rc7/

I can't find an arm64 build in http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.7-rc7/

Sorry, the arm64 kernels are not built automatically like the other arches. I'll build it an post a link shortly.

Can you test out the latest 4.6 based yakkety kernel while I get an arm64 mainline kernel built?

The yakkety master-next kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1602577/yakkety/

Note that you need to install both the linux-image and linux-image-extra .deb packages.

There is now a 4.7 arm64 mainline kernel available here:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.7/

Can you test this kernel when you have a chance?

Hi Joseph,

I tried the 4.7, but it made the compute node unbootable : https://paste.ubuntu.com/21865889/
From the initramfs shell, I had to manually load a few modules (ahci_xgene, sd_mod and ext4) so I could mount the root filesystem and get back to the previous working version (4.2).

Was that expected ? Will the same thing happen with the 4.6 based yakkety kernel you pasted above ?

Thanks

It wasn't expected that the system would not boot. The 4.8-rc1 kernel is now available, can you give it a try:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8-rc1/

4.8-rc1 had the same behaviour (landed into initramfs), so I figure I'd dig a bit. Long story short, this was entirely due to the following error :

/scripts/init-top/udev: line 14: can't create /sys/kernel/uevent_helper: Permission denied

CONFIG_UEVENT_HELPER was y in 4.2, but got switched to "not set" in at least 4.7 and 4.8.

I modified /usr/share/initramfs-tools/scripts/init-top/udev to not error out on the "echo" to /sys/kernel/uevent_helper, and then the node booted properly \o/

I'll let you know if it's more stable or not.

Well, this 4.8-rc1 doesn't have CONFIG_VIRTUALIZATION so it's not very useful for a compute node kernel ...

I reverted to 4.2. Can you please build a kernel with CONFIG_VIRTUALIZATION and CONFIG_KVM ?

Thanks

I build a v4.8-rc1 kernel using the Xenial 4.4 kernel configs. This set of configs set CONFIG_VIRTUALIZATION and CONFIG_KVM.

This test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1602577/v4.8-rc1With4.4Configs/

Can you give this test kernel a test?

Kernel from #24 fails to boot with the following, and keeps looping. Unclear if it's due to a problem with our system or with the kernel. I've sought advice from someone more familiar with the hardware we're using and will update with any further info.

306 bytes read in 18 ms (16.6 KiB/s)
## Executing script at 4004000000
19403840 bytes read in 499 ms (37.1 MiB/s)
28340037 bytes read in 723 ms (37.4 MiB/s)
## Booting kernel from Legacy Image at 4002000000 ...
   Image Name: kernel 4.8.0-040800rc1-generic
   Created: 2016-08-11 4:06:14 UTC
   Image Type: ARM Linux Kernel Image (uncompressed)
   Data Size: 19403776 Bytes = 18.5 MiB
   Load Address: 00080000
   Entry Point: 00080000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 4005000000 ...
   Image Name: ramdisk 4.8.0-040800rc1-generic
   Created: 2016-08-11 4:06:15 UTC
   Image Type: ARM Linux RAMDisk Image (gzip compressed)
   Data Size: 28339973 Bytes = 27 MiB
   Load Address: 00000000
   Entry Point: 00000000
   Verifying Checksum ... OK
ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree

Can you give the 4.8-rc8 kernel a try? It can be downloaded from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8-rc8/

It's booting \o/ We're trying to replicate the instability, we'll let you know.

My two instances have been up and idle for 1:30 hours by now. They don't have any actual lxd workload due to bug 1628946 (juju deploy currently fails), but the original hang bug actually happened on idle boxes. Thus, so far so good :-)

I'll check again tomorrow.

(For those playing at home, the boot failure in comment #25 was because the m400 firmware preloads the dtb at 0x4003000000, which was clobbered by the >16MiB kernel. The latest 4.8 builds are gzipped, so they're short enough to leave it intact.)

My two arm64 instances had been idle for 16 hours, and after that fully busy with running tests for about 5 hours. So from my POV, the 4.8 kernel does not have the RCU hang (bug 1531768) any more, or at least much less noticeable. And apparently the host has survived about 24 hours as well now. I don't know what "after some time" in the bug description translates to (minutes? hours? days?), but so far this is looking pretty good \o/

Both of my instances have been working happily for three days now. SHIP IT! :-)

Many thanks to the kernel team and Junien!

@Martin Pitt, is this with the upstream v4.8-rc8 kernel, or with an Ubuntu kernel? If it is with 44.8-rc8, we may still have to perform a reverse bisect to identify the commit that fixes the bug, so we can SRU it to the stable kernels. Yakkety will pick up this fix when we rebase, but kernel prior to Yakkety may not have the fix, if it was not sent to upstream stable.

@Joseph: I don't know what is running on the compute hosts, I don't have access to those. I suppose that Junien used the kernel you offered in comment #26, though?

Yes, this is with the kernel from #26

Thanks for the update, Junien. Now we know the fix is in the upstream v4.8-rc8 kernel. Then next step is to identify the commit that fixes this bug, and then SRU it to Xenial.

It would be best to next step the latest upstream 4.4 kernel. That will tell us if the fix was already sent to upstream stable. If it was not, we will need to test some other upstream kernels to narrow down further when the fix landed in mainline.

Can you test the upstream 4.4.23 kernel? It can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.23

upstream 4.4.23 has been installed on a compute node, we'll let you know how it goes

FAOD

$ cat /proc/version
Linux version 4.4.23-040423-generic (kernel@tangerine) (gcc version 5.3.1 20160413 (Ubuntu/Linaro 5.3.1-14ubuntu2) ) #201609300709 SMP Fri Sep 30 11:52:40 UTC 2016

My lxd-armhf1 node that is supposedly running on 4.4.23-040423-generic (on the compute host) has worked fine for the last 13 hours.

Compute node is still up. I think we can consider upstream 4.4.23 to be good.

That is great news. That means the fix in mainline was already cc'd to upstream stable. Xenial will get the fix through the normal upstream stable update process.

If you want to, we can still perform a kernel bisect to identify the fix for this bug, but that will require testing about 10 test kernels. Or we can just wait for the fix to come down via updates.

OK cool. I think we can wait. Do you have an ETA for the fix to come down ? And will trusty also get the fix ?

Great news! until that happens, is there any harm in leaving 4.8 or 4.4.23 running on the current compute nodes?