(regression?) openipmi fails to start on kernels 3.13.0-11-generic and 3.13.0-12-generic

This is the kernel that worked for me:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.14-rc4-trusty/

I can also go back and test the others until I find one that breaks.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1284334

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

I've uploaded the .apport file, since the server can't get out to the internet very easily. Let me know if you need anything else.

The Trusty 3.13.0-12 kernel has been rebased to upstream 3.13.4. However, 3.13.5 is already available. Can you see if this bug is already fixed in 3.13.5:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13.5-trusty/

Can you provide the apport output from 3.13.0-8 for comparison?

The ipmi_si driver is built into the kernel in 3.13.0-11 and -12. It could be the driver gets started but user-space expects it always to be a module.

Here's the apport when booted to 3.13.0-8. I'm wondering if that is indeed the case. I saw a changelog for the -11 kernel stating IPMI modules were set to 'y'.

Also just tested the 3.13.5 from mainline and noticed that openipmi does not start there, either. Same symptoms.

Any updates on this?

Yes, you are correct. The latest Saucy kernel does have IPMI as a module. Here is the config entries for Saucy 3.11.0-18.32:

config.common.ubuntu:CONFIG_ACPI_IPMI=m
config.common.ubuntu:CONFIG_IPMI_DEVICE_INTERFACE=m
config.common.ubuntu:CONFIG_IPMI_HANDLER=m
config.common.ubuntu:# CONFIG_IPMI_PANIC_EVENT is not set
config.common.ubuntu:CONFIG_IPMI_POWEROFF=m
config.common.ubuntu:CONFIG_IPMI_SI=m
config.common.ubuntu:CONFIG_IPMI_WATCHDOG=m

Since you confirmed this is fixed in 3.14-rc4, it is probably best for us to perform a reverse bisect to identify the commit that fixes this.

Can you test the v3.14-rc1 kernel to see if that's when this was fixed? It can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.14-rc1-trusty/

If the bug still exists there, we should test rc2, then rc3, etc. Once we find the last bad version and first good version, we can bisect between those two versions.

In trusty, the CONFIG_IPMI_HANDLER option is set to Y:

config.common.ubuntu:CONFIG_ACPI_IPMI=m
config.common.ubuntu:CONFIG_IPMI_DEVICE_INTERFACE=m
config.common.ubuntu:CONFIG_IPMI_HANDLER=y
config.common.ubuntu:# CONFIG_IPMI_PANIC_EVENT is not set
config.common.ubuntu:CONFIG_IPMI_POWEROFF=m
config.common.ubuntu:CONFIG_IPMI_SI=m
config.common.ubuntu:CONFIG_IPMI_WATCHDOG=m

I can build a Trusty kernel with CONFIG_IPMI_HANDLER set back to m and see if that changes things.

Hi Joseph,

Thanks. I can check the 3.14rc1 kernel and see. I'll also be glad to try out a test kernel from you with the handler set back to module.

I'll let you know as soon as I find something out with 3.14rc1

I built a Trusty test kernel with CONFIG_IPMI_HANDLER=m. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1284334/

Can you test this kernel to see if it exhibits the bug or not? You will need to install both the linux-image and linux-image-extra .deb packages.

Also, you can confirm the IPMI config options on a running system by executing:

cat /boot/config-$(uname -r) | grep IPMI

Trusty 3.13.0-14-generic config shows IPMI_HANDLER option set to Y:

CONFIG_ACPI_IPMI=m
CONFIG_IPMI_HANDLER=y
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m

..with the 3.13.0-8 kernel it is set to "m", and this was the last kernel that I ran where openipmi didn't have issues

Joseph's test kernel shows this and it works as expected with openipmi in that the service starts properly. IPMI_HANDLER is set to 'm':

CONFIG_ACPI_IPMI=m
CONFIG_IPMI_HANDLER=y
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m

I will test rc1 tomorrow as I ran out of time today.

It sounds like this bug didn't exists in Ubuntu 3.13.0-8. That kernel was based off of upstream 3.13.2. That may indicate that this bug was introduced by upstream stable 3.13.3 or 3.13.4. If you have time, can you also test the following two kernels:

3.13.3: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13.3-trusty/
3.13.4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13.4-trusty/

If 3.13.3 has the bug, there is no need to test 3.13.4. However, the priority should be on the reverse bisect, so we can identify what fixes this in 3.14.

All of the mainline kernels are built with CONFIG_IPMI_HANDLER set to m, so I assume testing will not reproduce the bug for any of the upstream kernels. We now know that the change of CONFIG_IPMI_HANDLER from m to y in Trusty is what causes this bug.

Ok. I'll not test those if that's the case.

CONFIG_IPMI_HANDLER is still y in3.13.0-20, so this is still broken. This is the workaround I'm using at the moment:

# diff /etc/init.d/openipmi.orig /etc/init.d/openipmi
324,326c324,326
< modprobe ipmi_msghandler > /dev/null 2>&1
< modules_loaded ipmi_msghandler
< [ ${OnePlusLoaded} -ne 1 ] && unload_all_ipmi_modules && RETVAL=$((RETVAL | 1)) && return
---
> # modprobe ipmi_msghandler > /dev/null 2>&1
> # modules_loaded ipmi_msghandler
> # [ ${OnePlusLoaded} -ne 1 ] && unload_all_ipmi_modules && RETVAL=$((RETVAL | 1)) && return

This is why we set it =y in the first place: http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-trusty.git;a=commit;h=b0457d402649381ab0dd9a40b54f5bf1d35931b9

The OpenIPMI portion of this issue now seems to be fixed in Utopic & Trusty and I will try and verify the fix for Precise:

https://bugs.launchpad.net/ubuntu/+source/openipmi/+bug/1309860