Bug#821049: multipath-tools: reloading devmap results in block device being created incorrectly

[Andrew Patterson]

On Sat, 16 Apr 2016 00:42:33 +0530 Ritesh Raj Sarraf <rrs at debian.org> wrote:
> On Sat, 2016-04-16 at 00:35 +0530, Ritesh Raj Sarraf wrote: > > Hello Mike, > > > > All I can say is that I am able to see the
behavior that you've mentioned (So > > marking the bug as confirmed).
THis is seen on my LIO iSCSI Setup running on 2 > > Guest VMs. > > > >
In doing that, I was surprised that those dev/mapper/ entries are now >
> symlinks. > > My storage knowledge has rusted for some time now, but
from what I recollect, > > it > > used to by direct block node names,
not symlinks. > > Maybe this could be of some help to you in debugging
this further. > > Apr 16 00:34:11 debian-sanboot systemd[1]: Started
Device-Mapper Multipath > Device Controller. > Apr 16 00:34:11
debian-sanboot systemd-udevd[949]: conflicting device node >
'/dev/mapper/sanroot' found, link to '/dev/dm-0' will not be created > >
> So I think my understanding was correct about Device Mapper's
behavior. Now I'm > not sure what 'systemd-udevd[949]' is reflecting
here as. I doubt if that is the > ID for the rules processing because
then I'd have seen the respective rule. > > I hope this is of some use
to you. My Debian time for today is running out :-) >

Hi Ritesh,

I am working with Mike on this problem.

We have seen this issue using multiple transports (FCoE, FC, and
iSCSI), although most of our testing has been using QLogic FC cards
with 3Par FC storage. I don't think this is a transport problem, but
is instead, some sort of udev/multipath interaction problem.

Here is a test I ran with multipath-tools_0.5.0-7 and
multipath-tools_0.5.0+git0.770e6d0d-1 on stretch using the same
hardware and configuration that Mike has been using:

Here is the storage:

1:0:0:1]    disk    3PARdata VV               3212  /dev/sdb
[1:0:0:2]    disk    3PARdata VV               3212  /dev/sdc
[1:0:0:254]  enclosu 3PARdata SES              3212  -       
[1:0:1:1]    disk    3PARdata VV               3212  /dev/sdd
[1:0:1:2]    disk    3PARdata VV               3212  /dev/sde
[1:0:1:254]  enclosu 3PARdata SES              3212  -       

And udev monitor output (udevadm monitor -u -k):

multipath-tools_0.5.0-7

# udevadm monitor -k -u
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[90540.185998] change   /devices/virtual/block/dm-0 (block)
KERNEL[90540.187857] change   /devices/virtual/block/dm-1 (block)
UDEV  [90540.265654] change   /devices/virtual/block/dm-1 (block)
UDEV  [90540.266138] change   /devices/virtual/block/dm-0 (block)

# ls -l /dev/mapper
total 0
crw------- 1 root root 10, 236 Apr 14 14:31 control
lrwxrwxrwx 1 root root       7 Apr 15 15:40 mpatha -> ../dm-0
lrwxrwxrwx 1 root root       7 Apr 15 15:40 mpathb -> ../dm-1

multipath-tools_0.5.0+git0.770e6d0d-1

# udevadm monitor -k -u
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[91354.622867] change   /devices/virtual/block/dm-0 (block)
KERNEL[91354.625218] change   /devices/virtual/block/dm-1 (block)
UDEV  [91354.699455] change   /devices/virtual/block/dm-0 (block)
UDEV  [91354.701663] change   /devices/virtual/block/dm-1 (block)

So the udev events look identical. We think that perhaps we have a
race and the new version of multipath is not waiting (or waiting long
enough) for udev to crate the device files and just goes and creates
the required files.

I am going to try and bisect the changes in the upstream multipath code
between
0.5.0 and 770e6d0d to see where this breaks.

This does not seem to that serious a problem on its own, since the
block device that is created is identical to what the symlink would
point to. But we are seeing other issue. But we are seeing other
issues like device files reappearing after the running multipath -f on
it, removing the LUN, and then running multipath -r which we think is
related.

-- 
Andrew Patterson
Hewlett-Packard Enterprise


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-lvm-maintainers/attachments/20160415/8a72adcf/attachment-0001.html>