Write performance regression severely affecting hpsa controllers

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1668557

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Here is some iostat and command lines from the slow mode (xenial kernel).

Last fast mainline kernel iostat.

And iostat from the first slow kernel.

Thanks maswan for the later logs as discussed.

TL;DR: the kernel now no more knows/considers this as a raid

It now merged write requests: wrqm/s 0 -> ~500
The writes are bigger due to that x8: avgrq-sz ~1000 -> ~8000
But we know it is a raid, it has to split it on stripe size
And we can see x8 in time would be ok, but we see: w_await ~10 -> ~600

I asked on IRC for the following on good/bad case while checking source.
 $ for i in /sys/class/block/sda/queue/iosched/* /sys/class/block/sda/queue/*; do echo $i $(cat $i); done

sys/block output for the slow case

Fast style sys/block output

Only diff is max_sectors_kb, 512->4096

That matches the iostat, 8k there is in 512b sectors which matched 4k max sectors.
I don't see where the patch is that breaks that in linux-stable, but it is worth a try to boot a bad kernel and set 512 in /sys.

Later to tune for your raid case with so many disks you might consider setting also
/sys/class/block/sdb/queue/rotational = 0
Locality doesn't mean a lot to your setup and might safe some cpu cycles.

I think I found the change in linux stable: http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=20d74bf29cfae86649bf1ec75038c79a9bc5010f

It is correct to "fix" it, the maximum size is bigger.

Yet in your case with the drives very likely not having 4k formatting and the strip-size/offset being different it appears as an anti-tuning which you need to overcome now.

Let me know how the results with setting 512 (and maybe also rotational=0) in a bad kernel work for you.

Theory confirmed, due to the fix the real "capable" max kb is set, that lets the block device layer merge and hold back requests. Only to later be split by the raid controller.

Following the IRC discussion using that as a tuning to "fix" is ok.

Closing as as it turned out not to be a bug.

The kernel has the proper tunable already (the one we used), but having a knob in the HW controller might be worth to avoid everybody finding that the hard way.

Last time I checked the hpsa driver is part of the kernel shipped with Ubuntu...

Regardless of whether the core kernel is doing it right or not, the hpsa driver should not advertise a larger max_sectors_kb than it can handle with good performance.

FYI, the hpsa driver in the xenial default kernel advertise the stripe size so this breaks performance for any setup with a stripe size more than approx 1 MiB.

As a workaround this udev rule can be used.

Caveat: It works as intended for us, but I'm not an udev expert.

We realized that this likely affects older controllers using the cciss driver as well, updated udev rule for completeness.

Johan Guldmyr pointed out that my udev rule attempted to apply the attribute changes to partitions as well, which causes errors in system log files.

Uploading a fixed version.