[multipath] failed to get sysfs information

Bug #1032550 reported by vincent
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
multipath-tools (Ubuntu)
Fix Released
Undecided
Unassigned
Trusty
Won't Fix
Medium
Unassigned

Bug Description

when shutdown switch port of host HBA, multippath-tool can't get correct information of subpath. by check the "multipath" output, some storage device type info disapppear and the failed path always stay in path group and don't be clear out.

mpath2 (3600601601c102900944737e4a73fe011) dm-51 ,
size=6.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| |- #:#:#:# - #:# failed faulty running
| `- 5:0:2:5 sdcu 70:32 active ready running
`-+- policy='round-robin 0' prio=0 status=enabled
  |- 5:0:3:5 sdfa 129:192 active ready running
  `- #:#:#:# - #:# failed faulty running
mpath38 (3600601601c1029008eb6dbe8ae3fe011) dm-59 DGC,VRAID
size=5.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 5:0:2:13 sddf 70:208 active ready running
`-+- policy='round-robin 0' prio=0 status=enabled
  `- 5:0:3:13 sdfk 130:96 active ready running
mpath63 (360000970000198700131533030303932) dm-13 EMC,SYMMETRIX
size=5.6G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 5:0:0:8 sdl 8:176 active ready running
  `- 5:0:1:8 sdbd 67:112 active ready running
mpath95 (360000970000198700131533030323445) dm-43 ,
size=898M features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- #:#:#:# - #:# failed faulty running
  |- #:#:#:# - #:# failed faulty running
  |- 5:0:0:38 sdas 66:192 active ready running
  `- 5:0:1:38 sdck 69:128 active ready running

Same time, the syslog show many

---------------
Aug 2 18:25:16 Linux51 multipathd: sdht: failed to get sysfs information
Aug 2 18:25:16 Linux51 multipathd: sdht: unusable path
... ...
---------------

After path recover, all failed path come back without problem. there is no IP blocked and error happend during fail/recover period.

Revision history for this message
vincent (vincent-y-chen) wrote :

Here is OS and Packge info also attach log and configuration.

root@Linux51:/var/tmp# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04 LTS
Release: 12.04
Codename: precise

root@Linux51:/var/tmp# dpkg -s multipath-tools
Package: multipath-tools
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 637
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Version: 0.4.9-3ubuntu5

Changed in multipath-tools (Ubuntu):
assignee: nobody → Peter Petrakis (peter-petrakis)
Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

@Vincent

Could you please attach the syslog associated with this fault?
How reproducible is this?

Also I noticed that your lvm.conf is a little broken.
    scan = [ "/dev" ]
...
    filter = [ "r/block/", "r/disk/", "r|/dev/sd.*|", "a/.*/" ]

Since the scan starts in /dev, that's the root of the search, specifying
"r|/dev/sd.*|" won't match anything, instead you want "r|sd.*|".

vgscan -vvv will show you the regexp working.
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/comments/23

Changed in multipath-tools (Ubuntu):
status: New → Incomplete
Revision history for this message
vincent (vincent-y-chen) wrote :

@peter
I have change the filter and update into new init ram disk, thanks.
For powerpath issue, i reproduce by shutdown FC switch port. Here is new complete syslog of whole prodefure and multipath result.
1)multipath_before_port_down --> multipath status before shutdown port
2)multipath_port_down --> status when shutdown one switch port which connect to host HBA
3)multipath_enable_port

Changed in multipath-tools (Ubuntu):
status: Incomplete → In Progress
Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

@Vincent

I have a test package for you to try.

Add this ppa, https://launchpad.net/~peter-petrakis/+archive/storage

The install incantation should be:
apt-get install multipath-tools=0.4.9-3ubuntu5+lp1032250dbg1

It includes a fix to the state checker which *should* eliminate the sysfs coherency issue.
Based on RH BZ 680480: https://bugzilla.redhat.com/show_bug.cgi?id=680480

Revision history for this message
vincent (vincent-y-chen) wrote :

i have applied the fix, however the issue still happened :-(

ii kpartx 0.4.9-3ubuntu5+lp1032250dbg1 create device mappings for partitions
ii multipath-tools 0.4.9-3ubuntu5+lp1032250dbg1 maintain multipath block device access

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

So you restarted multipath-tools or rebooted before testing again? I was afraid of this,
the RH patch isn't even upstream and was applied against a different code base. There
may be additional patches that have a cumulative effect that we're missing.

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

I'm seeing the same issue with an EMC clariion, see attached syslog. On my systems this problem leads to hanging processes and a server crash that can only be fixed with a reboot ..

Revision history for this message
Ronald Moesbergen (intercommit) wrote :
Revision history for this message
vincent (vincent-y-chen) wrote : RE: [Bug 1032550] Re: [multipath] failed to get sysfs information

It maybe a kernel bug
Dec 9 08:16:07 ealxs00195 kernel: [1723012.226196] BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:5:1283]

Vincent Chen
EMC2 | E-Lab Linux team

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Ronald Moesbergen
Sent: Monday, December 10, 2012 5:07 PM
To: Chen, Vincent
Subject: [Bug 1032550] Re: [multipath] failed to get sysfs information

** Attachment added: "My multipath.conf"
   https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+attachment/3456042/+files/multipath.conf

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1032550

Title:
  [multipath] failed to get sysfs information

Status in “multipath-tools” package in Ubuntu:
  In Progress

Bug description:
  when shutdown switch port of host HBA, multippath-tool can't get
  correct information of subpath. by check the "multipath" output,
  some storage device type info disapppear and the failed path always
  stay in path group and don't be clear out.

  mpath2 (3600601601c102900944737e4a73fe011) dm-51 ,
  size=6.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
  |-+- policy='round-robin 0' prio=1 status=active
  | |- #:#:#:# - #:# failed faulty running
  | `- 5:0:2:5 sdcu 70:32 active ready running
  `-+- policy='round-robin 0' prio=0 status=enabled
    |- 5:0:3:5 sdfa 129:192 active ready running
    `- #:#:#:# - #:# failed faulty running
  mpath38 (3600601601c1029008eb6dbe8ae3fe011) dm-59 DGC,VRAID
  size=5.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
  |-+- policy='round-robin 0' prio=1 status=active
  | `- 5:0:2:13 sddf 70:208 active ready running
  `-+- policy='round-robin 0' prio=0 status=enabled
    `- 5:0:3:13 sdfk 130:96 active ready running
  mpath63 (360000970000198700131533030303932) dm-13 EMC,SYMMETRIX
  size=5.6G features='0' hwhandler='0' wp=rw
  `-+- policy='round-robin 0' prio=1 status=active
    |- 5:0:0:8 sdl 8:176 active ready running
    `- 5:0:1:8 sdbd 67:112 active ready running
  mpath95 (360000970000198700131533030323445) dm-43 ,
  size=898M features='0' hwhandler='0' wp=rw
  `-+- policy='round-robin 0' prio=1 status=active
    |- #:#:#:# - #:# failed faulty running
    |- #:#:#:# - #:# failed faulty running
    |- 5:0:0:38 sdas 66:192 active ready running
    `- 5:0:1:38 sdck 69:128 active ready running

  Same time, the syslog show many

  ---------------
  Aug 2 18:25:16 Linux51 multipathd: sdht: failed to get sysfs information
  Aug 2 18:25:16 Linux51 multipathd: sdht: unusable path
  ... ...
  ---------------

  After path recover, all failed path come back without problem. there
  is no IP blocked and error happend during fail/recover period.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

What I really need is a crashdump. I'm simply short on time to reproduce this myself.

https://wiki.ubuntu.com/Kernel/CrashdumpRecipe

Configure sysctl to panic on oops, that should do it.

This stack section is interesting.
Dec 9 09:18:27 ealxs00195 kernel: [1726974.078795] RIP: 0010:[<ffffffff8165b049>] [<ffffffff8165b049>] _raw_spin_unlock_irqrestore+0x19/0x30
...
Dec 9 09:18:27 ealxs00195 kernel: [1726974.091694] [<ffffffff8142f764>] __scsi_remove_target+0xd4/0xf0
Dec 9 09:18:27 ealxs00195 kernel: [1726974.091696] [<ffffffff8142f841>] scsi_remove_target+0xc1/0xe0
Dec 9 09:18:27 ealxs00195 kernel: [1726974.091701] [<ffffffffa00deb96>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
Dec 9 09:18:27 ealxs00195 kernel: [1726974.091706] [<ffffffff81084b1a>] process_one_work+0x11a/0x480

It's required to be able to sleep in a workq context, we're grabbing spinlocks here, which when done quickly
is fine, but holding them too long and well, you see this. I doubt this is the root cause, something else is likely
holding the locks due to an error handling condition and the removal threads are just victims.

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

@Peter: I'll try do reproduce on our acceptance systems with crashdump installed and report back.

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

I just retested with the multipath-tools version '0.4.9-3ubuntu5+lp1032250dbg1', on 2 systems simulatiously. The test consisted of repeatedly shutting down the fibre port in the switch for one of the paths. One system survived (see syslog.test2.survived.gz), one system did not (syslog.test2.broken.gz). The crashdump stuff was installled and panic_on_oops set to '1', but there were no kernel 'BUG' or 'OOPS'-en going on this time... The 'failed to get sysfs information' error is gone, however. It might have been replaced with the following though:

Dec 11 16:04:44 ealxs00162 udevd[8828]: rename '/dev/disk/by-id/wwn-0x6006016061e02e008a2d4fa5b307e211.udev-tmp' '/dev/disk/by-id/wwn-0x6006016061e02e008a2d4fa5b307e211' failed: No such file or directory

Hope this helps.

Revision history for this message
Ronald Moesbergen (intercommit) wrote :
Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

I should have been more specific, please set kernel.hung_task_panic as well, which
should trigger on the "blocked more than N sec" events.

Also, if you still have the system in this state. Does ps | aux show a series of kpartx
processes blocked?

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

I've been trying to get the crashdump stuff to work, but it just won't generate a dump, or do anything at all when the kernel crashes... Is there another way to get the info you want? Would a console screenshot with a high res console work?

About the hanging processes: The only processes I saw in 'D' state were these 2 processes:
/sbin/multipath -v0 /dev/sdl' [14785]
'/sbin/multipath -v0 /dev/sdj' [14739]

And ofcourse anything accessing the disks on top of the multipath devices. When I do another run, I'll make sure to create a dump of the process list as well.

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

hmm, try this.

kernel.panic_on_oops = 0
kernel.softlockup_panic = 0
kernel.unknown_nmi_panic = 0
kernel.panic_on_unrecovered_nmi = 0
kernel.panic_on_io_nmi = 0
kernel.hung_task_panic = 0

Make sure these are all set to 1 in /etc/sysctl.conf and then run sysctl -p

Also, lets configure ftrace, see if we get lucky and can catch the deadlock
as it happens.

[as root]
cd /sys/kernel/debug/tracing

echo function > current_tracer
echo 16000 > buffer_size_kb
echo "scsi_*" >> set_ftrace_filter
echo "fc_*" >> set_ftrace_filter

When the kernel crashes, I'll be able to retrieve this buffer from the dump

Should none of the dump triggers work, then just watch it and trigger it yourself.
# echo c > /proc/sysrq-trigger

The hung multipath -v0 processes are usually spawned on behalf of udev events.
Before you begin the test, please record the current dm map (dmsetup table -v)
and the output of (lsscsi -lv) so I have a point of reference. Thanks.

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

The problem is not making the kernel crash, that's easy :) The issue is that no crash dump is ever generated. The whole kexec dump stuff just doesn't seem to work. I already went through all the troubleshooting tips and tricks, but it the kernel just crashed and hangs forever and only a reset works. Causing a crash with echo c > /proc/sysrq-trigger: same effect. Crash, hang, nothing...

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

Peter, is there another way to get you a kernel crash-dump? Whatever I try, the crashdump tools don't produce a crashdump.

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

Hi Ronald,

Sorry I haven't been timely, this is the best I can do with community level support
 If kdump isn't launching even in the most trivial case then you have to start from zero.

is crashkernel even configured?
 * grep crash /proc/crashkernel

How much memory do you have, could you assign more memory to the crash kernel?
 * http://lxr.linux.no/linux+v3.7.1/Documentation/kdump/kdump.txt#L270
 * 256MB would be preferable

Can you even kexec at all?
 * kexec -p # loads the panic kernel, man kexec

If you boot your system with maxcpus=1 (I think that's it) and pretend you're
a uniprocessor system, will kexec load?

Can you attach a serial console to your machine and post the output?

In /etc/init.d/kdump
        # Append kdump_needed for initramfs to know what to do, and add
        # maxcpus=1 to keep things sane.
        APPEND="$APPEND kdump_needed maxcpus=1 irqpoll reset_devices"

Start adjusting these variables, like remove 'reset_devices', reload the
kexec kernel (service kdump restart), and systematically remove variables
(except kdump_needed) noting the change in the kernel output.

Is this an enterprise server with an NMI button? If you configure "panic on nmi"
pressing that button, that will definitely change the base variables used to
launch kexec.

Folks thought Stratus was a bit overkill, having a complete mirror of CPU/Memory
operating in lockstep for HA. The nice thing about it is if the primary ever did crash,
we would literally hold that unit in stasis, reboot on the other unit, and reap the
dump from it's preserved memory, works 100% and automatic. Be nice to have
right about now.

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

Correction:
is crashkernel even configured?
 * grep crash /proc/cmdline

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

Removing the 'reset_devices' and 'irq_poll' in /etc/init.d/kdump did the trick, both machines now generate a nice vmcore file when they crash. I'll re-test with all your settings applied and report back. Thanks.

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

^5!

Make sure you enable ftrace per:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/comments/16

That'll save me a lot of work, should crash with a nice long call graph and the CPUs
caught executing the code.

Well done!

Revision history for this message
Ronald Moesbergen (intercommit) wrote :
Download full text (16.7 KiB)

Peter,

I got it to crash again, this time with a nice kernel dump. The dump can be fetched here:

http://www.rmoesbergen.nl/linux-image-3.2.0-34-generic.0.crash.gz

The crash itself looked like this:

Dec 21 11:07:32 ealxs00161 kernel: [63272.392812] sd 4:0:1:1: emc: ALUA failover mode detected
Dec 21 11:07:32 ealxs00161 kernel: [63272.392820] sd 4:0:1:1: emc: at SP B Port 1 (owned, default SP B)
Dec 21 11:07:32 ealxs00161 kernel: [63272.393180] sd 3:0:0:1: emc: ALUA failover mode detected
Dec 21 11:07:32 ealxs00161 kernel: [63272.393187] sd 3:0:0:1: emc: at SP B Port 0 (owned, default SP B)
Dec 21 11:10:36 ealxs00161 kernel: [63455.641431] qla2xxx [0000:07:00.0]-500b:3: LOOP DOWN detected (2 3 0 0).
Dec 21 11:10:52 ealxs00161 multipathd: sdf: remove path (uevent)
Dec 21 11:10:52 ealxs00161 kernel: [63471.548255] rport-3:0-1: blocked FC remote port time out: removing target and saving binding
Dec 21 11:10:52 ealxs00161 kernel: [63471.676065] rport-3:0-0: blocked FC remote port time out: removing target and saving binding
Dec 21 11:11:08 ealxs00161 cimserver[2079]: Authentication failed for user=root.
Dec 21 11:11:10 ealxs00161 cimserver[2079]: Authentication failed for user=root.
Dec 21 11:13:28 ealxs00161 kernel: [63627.745648] INFO: task jbd2/dm-1-8:1530 blocked for more than 120 seconds.
Dec 21 11:13:28 ealxs00161 kernel: [63627.746025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 21 11:13:28 ealxs00161 kernel: [63627.756371] jbd2/dm-1-8 D ffff8803aa11a620 0 1530 2 0x00000000
Dec 21 11:13:28 ealxs00161 kernel: [63627.756380] ffff880416141ac0 0000000000000046 ffff880416141a60 ffff88042ee137c0
Dec 21 11:13:28 ealxs00161 kernel: [63627.756388] ffff880416141fd8 ffff880416141fd8 ffff880416141fd8 00000000000137c0
Dec 21 11:13:28 ealxs00161 kernel: [63627.756395] ffffffff81c0d020 ffff880415ef9700 ffff880416141a90 ffff88042ee14080
Dec 21 11:13:28 ealxs00161 kernel: [63627.756403] Call Trace:
Dec 21 11:13:28 ealxs00161 kernel: [63627.756416] [<ffffffff81117230>] ? __lock_page+0x70/0x70
Dec 21 11:13:28 ealxs00161 kernel: [63627.756431] [<ffffffff81659ebf>] schedule+0x3f/0x60
Dec 21 11:13:28 ealxs00161 kernel: [63627.756441] [<ffffffff81659f6f>] io_schedule+0x8f/0xd0
Dec 21 11:13:28 ealxs00161 kernel: [63627.756451] [<ffffffff8111723e>] sleep_on_page+0xe/0x20
Dec 21 11:13:28 ealxs00161 kernel: [63627.756460] [<ffffffff8165a78f>] __wait_on_bit+0x5f/0x90
Dec 21 11:13:28 ealxs00161 kernel: [63627.756470] [<ffffffff811173a8>] wait_on_page_bit+0x78/0x80
Dec 21 11:13:28 ealxs00161 kernel: [63627.756481] [<ffffffff8108ad60>] ? autoremove_wake_function+0x40/0x40
Dec 21 11:13:28 ealxs00161 kernel: [63627.756492] [<ffffffff811174bc>] filemap_fdatawait_range+0x10c/0x1a0
Dec 21 11:13:28 ealxs00161 kernel: [63627.756503] [<ffffffff8111757b>] filemap_fdatawait+0x2b/0x30
Dec 21 11:13:28 ealxs00161 kernel: [63627.756516] [<ffffffff81260ea0>] journal_finish_inode_data_buffers+0x70/0x170
Dec 21 11:13:28 ealxs00161 kernel: [63627.756528] [<ffffffff81261795>] jbd2_journal_commit_transaction+0x665/0x1240
Dec 21 11:13:28 ealxs00161 kernel: [63627.756538] [<ffffffff8108ad20>] ? add_wait_queue+0x60/0x60
Dec 21...

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

I've uploaded the vmcore file separately, because I have some doubts about the dumpfile created by apport beging complete. Please find it here: http://www.rmoesbergen.nl/vmcore-crash.tgz

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

@Ronald

First, please attach http://www.rmoesbergen.nl/vmcore-crash.tgz to the bug, launchpad
can handle it just fine. Also, this is going to take awhile. We're off all next week so don't
expect any movement on this until early-mid Jan. Feel free to ping me if I forget.

Also, at what time did your testing start? I'm seeing this everywhere almost immediately
         emc: ALUA failover mode detected

Could you also illustrate what the steady state target distribution should be?

I see targets like this:
        sd 3:0:0:0: [sdb] 41943040 512-byte logical blocks: (21.4 GB/20.0 GiB)

in the minority compared to
        sd 3:0:0:1: [sdc] 419430400 512-byte logical blocks: (214 GB/200 GiB)

Wondering if your SAN is misreporting READ CAPACITY.

The dump looks good. Immediately I can tell you that all the scsi hosts
are still RUNNING and not in error handling. It looks like I'll have examine
the scsi target states and the dm tables.

So there are these stuck processes

crash> ps | grep UN
   1530 2 0 ffff880415ef9700 UN 0.0 0 0 [jbd2/dm-1-8]
   2180 2 1 ffff88040613ae00 UN 0.0 0 0 [flush-252:1]
   4739 1 2 ffff880418e70000 UN 5.8 16426520 1029488 mysqld

Which adds up, you can't write back.

This also looks really suspicious.

[62856.457650] end_request: I/O error, dev sdf, sector 21272960
[62856.457966] device-mapper: multipath: Failing path 8:80.
[62856.462495] scsi 3:0:0:0: emc: Detached
[62856.462730] device-mapper: multipath: Failing path 8:80.
[62856.462798] sd 4:0:0:0: emc: ALUA failover mode detected
[62856.462806] sd 4:0:0:0: emc: at SP A Port 0 (owned, default SP A)
# sketchy
[62856.462814] device-mapper: multipath: Could not failover the device: Handler scsi_dh_emc Error 15.

# it looks like it's retrying

[63122.241178] sd 3:0:1:0: [sdf] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[63122.241185] sd 3:0:1:0: [sdf] CDB: Write(10): 2a 00 01 44 b4 d8 00 00 20 00
[63122.241198] end_request: I/O error, dev sdf, sector 21279960
[63122.241513] device-mapper: multipath: Failing path 8:80.
[63122.244865] scsi 3:0:0:0: emc: Detached
[63122.245045] sd 4:0:0:0: emc: ALUA failover mode detected
[63122.245053] sd 4:0:0:0: emc: at SP A Port 0 (owned, default SP A)
# sketchy
[63122.245062] device-mapper: multipath: Could not failover the device: Handler scsi_dh_emc Error 15.

...
which comes from: [drivers/md/dm-mpath.c]
        case SCSI_DH_NOSYS:
                if (!m->hw_handler_name) {
                        errors = 0;
                        break;
                }
                DMERR("Could not failover the device: Handler scsi_dh_%s "
                      "Error %d.", m->hw_handler_name, errors);
                /*
                 * Fail path for now, so we do not ping pong
                 */
                fail_path(pgpath);
                break;

Hey, was this intentional?

[ 0.018792] Hardware name: ProLiant DL380p Gen8
[ 0.018794] Your BIOS is broken and requested that x2apic be disabled
[ 0.018795] This will leave your machine vulnerable to irq-injection attacks
[ 0.018796] Use 'intremap=no_x2apic_optout' to override BIOS request

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

vmcore file of crashed kernel.

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

The normal state of the disks is as follows:

LUN-DATABASE (36006016061e02e003cf1aca4ae07e211) dm-2 DGC,VRAID
size=200G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| |- 4:0:0:1 sdg 8:96 active ready running
| `- 3:0:0:1 sdc 8:32 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 4:0:1:1 sdi 8:128 active ready running
  `- 3:0:1:1 sde 8:64 active ready running
LUN-LOGGING (36006016061e02e000286c1adae07e211) dm-1 DGC,VRAID
size=20G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| |- 4:0:1:0 sdh 8:112 active ready running
| `- 3:0:1:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 4:0:0:0 sdf 8:80 active ready running
  `- 3:0:0:0 sdb 8:16 active ready running

So one LUN of 20G, another of 200G, both having 4 path's to the SAN.

Today's test started at Dec 21 10:53:44, with this:
ealxs00161 kernel: [62445.130300] qla2xxx [0000:07:00.0]-500b:3: LOOP DOWN detected (2 3 0 0).

It took several up/down sequences to reproduce the problem.

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

Peter,

First: happy new year!

I've been doing some more tests to track down the cause of this bug. Since it looks like a kernel bug, I tried reproducing this with kernel 3.5.0, version 3.5.0-21.32~precise1. I could reproduce the faulty paths that multipathd was unable to remove, however: there were no hanging processes this time and thus no kernel crash.. which is an improvement. During the test I did see this happening:

LUN-DATABASE (36006016061e02e003cf1aca4ae07e211) dm-1 DGC,VRAID
size=200G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| |- 4:0:1:1 sdi 8:128 active ready running
| `- #:#:#:# - #:# active faulty running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 4:0:0:1 sdg 8:96 active ready running
  `- #:#:#:# - #:# active faulty running
LUN-LOGGING (36006016061e02e000286c1adae07e211) dm-2 ,
size=20G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| |- #:#:#:# - #:# failed faulty running
| `- 4:0:0:0 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- #:#:#:# - #:# active faulty running
  `- 4:0:1:0 sdh 8:112 active ready running

As you can see, multipathd fails to remove the 'faulty' paths from the device-mapping again. However, for some reason this didn't lead to processes stuck in 'D' state this time. During this, the following message was logged repeatedly:

Jan 3 10:24:14 ealxs00161 multipathd: sdd: failed to get sysfs information
Jan 3 10:24:14 ealxs00161 multipathd: sdd: unusable path

So multipathd was retrying the removal, but it failed every time. After bringing the path back up, it restored OK and everything was fine again:

LUN-DATABASE (36006016061e02e003cf1aca4ae07e211) dm-1 DGC,VRAID
size=200G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| |- 4:0:1:1 sdi 8:128 active ready running
| `- 3:0:0:1 sdc 8:32 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 4:0:0:1 sdg 8:96 active ready running
  `- 3:0:1:1 sdf 8:80 active ready running
LUN-LOGGING (36006016061e02e000286c1adae07e211) dm-2 DGC,VRAID
size=20G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| |- 3:0:1:0 sdd 8:48 active ready running
| `- 4:0:0:0 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 3:0:0:0 sdb 8:16 active ready running
  `- 4:0:1:0 sdh 8:112 active ready running

After this, failing over again worked just fine, the paths that failed to be removed the last time were now removed without problems... Both machines survived about 10 up/down testruns. I'll attach the syslog of this run shortly.

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

Syslog of the last testrun on kernel 3.5.0. Test started @ 10:15

Revision history for this message
Ronald Moesbergen (intercommit) wrote :

Syslog for server2, the same testrun also on kernel 3.5.0

Revision history for this message
Jakub Pavlik (pavlk-jakub) wrote :

I can confirm this issue at

root@cpt151:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty

root@cpt151:~# dpkg -s multipath-tools
Package: multipath-tools
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 626
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Version: 0.4.9-3ubuntu7.2
Depends: libaio1 (>= 0.3.106-8), libc6 (>= 2.14), libdevmapper1.02.1 (>= 2:1.02.36), libreadline6 (>= 6.0), udev (>> 136-1), initscripts (>= 2.85-16), kpartx (>= 0.4.9-3ubuntu7.2), lsb-base (>= 3)
Suggests: multipath-tools-boot
Conflicts: multipath-tools-boot (<= 0.4.8+git0.761c66f-2~), multipath-tools-initramfs (<= 1.0.1)
Conffiles:
 /etc/init.d/multipath-tools-boot e3b1a4949a282ceeda254c185f780002
 /etc/init.d/multipath-tools b0f5ff36184c823ec91ffd8b36133949
Description: maintain multipath block device access
 These tools are in charge of maintaining the disk multipath device maps and
 react to path and map events.
 .
 If you install this package you may have to change the way you address block
 devices. See README.Debian for details.
Homepage: http://christophe.varoqui.free.fr/
Original-Maintainer: Debian LVM Team <email address hidden>

Revision history for this message
Ronny Trommer (r-trommer) wrote :

I can confirm this issue at

root@myhost:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty

root@myhost:~# dpkg -s multipath-tools
Package: multipath-tools
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 626
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Version: 0.4.9-3ubuntu7.2
Depends: libaio1 (>= 0.3.106-8), libc6 (>= 2.14), libdevmapper1.02.1 (>= 2:1.02.36), libreadline6 (>= 6.0), udev (>> 136-1), initscripts (>= 2.85-16), kpartx (>= 0.4.9-3ubuntu7.2), lsb-base (>= 3)
Suggests: multipath-tools-boot
Conflicts: multipath-tools-boot (<= 0.4.8+git0.761c66f-2~), multipath-tools-initramfs (<= 1.0.1)
Conffiles:
 /etc/init.d/multipath-tools-boot e3b1a4949a282ceeda254c185f780002
 /etc/init.d/multipath-tools b0f5ff36184c823ec91ffd8b36133949
Description: maintain multipath block device access
 These tools are in charge of maintaining the disk multipath device maps and
 react to path and map events.
 .
 If you install this package you may have to change the way you address block
 devices. See README.Debian for details.
Homepage: http://christophe.varoqui.free.fr/
Original-Maintainer: Debian LVM Team <email address hidden>

Error message in syslog:

Jan 8 12:03:59 myhost multipathd: sde: failed to get sysfs information
Jan 8 12:03:59 myhost multipathd: sde: unusable path

Changed in multipath-tools (Ubuntu):
assignee: Peter Petrakis (peter-petrakis) → nobody
Changed in multipath-tools (Ubuntu):
status: In Progress → Fix Released
Changed in multipath-tools (Ubuntu Trusty):
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

While it is sad that no one found the time to prepare an upload for this fix while clearing old bugs I have to state that Trusty is (for a while already) in ESM support and this isn't a security issue. Therefore to reflect the most correct state I sadly have to set this to Won't Fix :-/

Changed in multipath-tools (Ubuntu Trusty):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.