Bug#821049: multipath-tools: reloading devmap results in block device being created incorrectly

[Andrew Patterson]

On Fri, 15 Apr 2016 16:07:48 -0600 Andrew Patterson
<andrew.patterson at hpe.com> wrote:
> On Sat, 16 Apr 2016 00:42:33 +0530 Ritesh Raj Sarraf <rrs at debian.org> wrote: > > On Sat, 2016-04-16 at 00:35 +0530, Ritesh Raj Sarraf wrote: > >
Hello Mike, > > > > All I can say is that I am able to see the >
behavior that you've mentioned (So > > marking the bug as confirmed). >
THis is seen on my LIO iSCSI Setup running on 2 > > Guest VMs. > > > > >
In doing that, I was surprised that those dev/mapper/ entries are now >
> > symlinks. > > My storage knowledge has rusted for some time now, but
> from what I recollect, > > it > > used to by direct block node names,
> not symlinks. > > Maybe this could be of some help to you in debugging
> this further. > > Apr 16 00:34:11 debian-sanboot systemd[1]: Started >
Device-Mapper Multipath > Device Controller. > Apr 16 00:34:11 >
debian-sanboot systemd-udevd[949]: conflicting device node > >
'/dev/mapper/sanroot' found, link to '/dev/dm-0' will not be created > >
> > So I think my understanding was correct about Device Mapper's >
behavior. Now I'm > not sure what 'systemd-udevd[949]' is reflecting >
here as. I doubt if that is the > ID for the rules processing because >
then I'd have seen the respective rule. > > I hope this is of some use >
to you. My Debian time for today is running out :-) > > > Hi Ritesh, > >
I am working with Mike on this problem. > > We have seen this issue
using multiple transports (FCoE, FC, and > iSCSI), although most of our
testing has been using QLogic FC cards > with 3Par FC storage. I don't
think this is a transport problem, but > is instead, some sort of
udev/multipath interaction problem. > > Here is a test I ran with
multipath-tools_0.5.0-7 and > multipath-tools_0.5.0+git0.770e6d0d-1 on
stretch using the same > hardware and configuration that Mike has been
using: > > Here is the storage: > > 1:0:0:1] disk 3PARdata VV 3212
/dev/sdb > [1:0:0:2] disk 3PARdata VV 3212 /dev/sdc > [1:0:0:254]
enclosu 3PARdata SES 3212 - > [1:0:1:1] disk 3PARdata VV 3212 /dev/sdd >
[1:0:1:2] disk 3PARdata VV 3212 /dev/sde > [1:0:1:254] enclosu 3PARdata
SES 3212 - > > And udev monitor output (udevadm monitor -u -k): > >
multipath-tools_0.5.0-7 > > # udevadm monitor -k -u > monitor will print
the received events for: > UDEV - the event which udev sends out after
rule processing > KERNEL - the kernel uevent > > KERNEL[90540.185998]
change /devices/virtual/block/dm-0 (block) > KERNEL[90540.187857] change
/devices/virtual/block/dm-1 (block) > UDEV [90540.265654] change
/devices/virtual/block/dm-1 (block) > UDEV [90540.266138] change
/devices/virtual/block/dm-0 (block) > > # ls -l /dev/mapper > total 0 >
crw------- 1 root root 10, 236 Apr 14 14:31 control > lrwxrwxrwx 1 root
root 7 Apr 15 15:40 mpatha -> ../dm-0 > lrwxrwxrwx 1 root root 7 Apr 15
15:40 mpathb -> ../dm-1 >

My git bisection resulted in this commit causing the change in behavior:

commit 4a2431aa944eb2e5b6f3ccd2d4fe1df67f9e5679
Author: Hannes Reinecke <hare at suse.de>
Date:   Tue Jul 29 15:44:46 2014 +0200

    Fixup device-mapper 'cookie' handling
   
    device-mapper has a 'cookie', which is inserted with the ioctl
    for modifying device-mapper devices.
    It is used as a synchronization point between udev and any other
    applications to notify the latter when udev has finished
    processing the event.
    Originally multipath would only use a single cookie for every
    transaction, and wait for that cookie at the end of the program.
    Which works well if you only have one transaction, but for several
    (like calling 'multipath') it will actually overwrite the cookie
    and fail to wait for earlier events.
    This causes libdevmapper to create the device nodes on its own,
    and the device nodes not being handled by udev.
   
    Signed-off-by: Hannes Reinecke <hare at suse.de>

So it looks like this is the designed behavior.

-- 
Andrew Patterson
Hewlett-Packard Enterprise