Bug#821049: multipath-tools: reloading devmap results in block device being created incorrectly

[Ritesh Raj Sarraf]

Control: tag -1 +confirmed

Hello Mike,

All I can say is that I am able to see the behavior that you've mentioned (So
marking the bug as confirmed). THis is seen on my LIO iSCSI Setup running on 2
Guest VMs.

In doing that, I was surprised that those dev/mapper/ entries are now symlinks.
My storage knowledge has rusted for some time now, but from what I recollect, it
used to by direct block node names, not symlinks.


On Thu, 2016-04-14 at 17:47 -0600, Mike Bacco wrote:
> 
> Package: multipath-tools
> Version: 0.5.0+git1.656f8865-9
> Severity: normal
> 
> Dear Maintainer,
> 
> When executing 'multipath -r' the existing symlinks to dm-X devices 
> which are present in /dev/mapper are overwritten and replaced with block 
> devices as demonstrated below:
> 
> mike at dl380gen9-01:~$ sudo ls -l /dev/mapper
> total 0
> crw------- 1 root root 10, 236 Apr 14 14:31 control
> lrwxrwxrwx 1 root root 7 Apr 14 14:31 mpatha -> ../dm-0
> lrwxrwxrwx 1 root root 7 Apr 14 14:31 mpathb -> ../dm-1
> mike at dl380gen9-01:~$ sudo multipath -r
> reload: mpatha (360002ac0000000000000001d000028be) undef 3PARdata,VV
> size=10G features='1 queue_if_no_path' hwhandler='1 alua' wp=undef
> `-+- policy='queue-length 0' prio=50 status=undef
> > 
> > - 1:0:0:1 sdb 8:16 active ready running
> `- 1:0:1:1 sdd 8:48 active ready running
> reload: mpathb (360002ac0000000000000001e000028be) undef 3PARdata,VV
> size=20G features='1 queue_if_no_path' hwhandler='1 alua' wp=undef
> `-+- policy='queue-length 0' prio=50 status=undef
> > 
> > - 1:0:0:2 sdc 8:32 active ready running
> `- 1:0:1:2 sde 8:64 active ready running
> mike at dl380gen9-01:~$ sudo ls -l /dev/mapper
> total 0
> crw------- 1 root root 10, 236 Apr 14 14:31 control
> brw-rw---- 1 root disk 254, 0 Apr 14 14:40 mpatha
> brw-rw---- 1 root disk 254, 1 Apr 14 14:40 mpathb
> 


Bear with me. I'm interested to know if you have any specific reliance on those
dm-* device names? I ask because I believe this has been documented well enough
over the years, that those devices are not to be referenced direct, for any use.

But for the cosmetic part, I too am curious on who created those device links
and who deleted them. Because, `multipath -r` just generates a CHANGE event and
looking at the rules being processed, I don't see much that may be related to
those entry's creation/deletion.


> Additionally is has been observed that if the map is flushed with 
> multipath -f and then the backend storage removed a subsequent execution 
> of multipath -r results in the block device entry being recreated and 
> the map being loaded with failed paths
> 

I don't understand the part about "backend storage removed....". Did you mean
that after a flush of the map on the host, you deleted the LUN on the target ?


> This problem appears to have been introduced with 0.5.0+git0.770e6d0d-1. 
> Testing with 0.5.0-7 from snapshots results in the expected behavior of 
> the devmap reload resulting in symlink entries to dm-X devices not being 
> changed to block device types, and new entries being created as symlinks 
> to dm-X devices and not as block devices
> 

Hmmm.. So either somewhere in the past 2 years, the entries in /dev/mapper
changed to being symlinks. Sorry, I've been very rusted on the latest on DM
Multipath. :-)

With you help, I hope to improve it.

> When tracing the multipath process I have been able to observe the 
> creation of the block device entries as the device files are missing
> 
> --
> stat("/dev/mapper/mpatha", 0x7ffeae26abc0) = -1 ENOENT (No such file or 
> directory)
> umask(0)                                = 022
> mknod("/dev/mapper/mpatha", S_IFBLK|0660, makedev(254, 0)) = 0
> umask(022)                              = 0
> chown("/dev/mapper/mpatha", 0, 6)       = 0
> stat("/dev/mapper/mpathb", 0x7ffeae26abc0) = -1 ENOENT (No such file or 
> directory)
> umask(0)                                = 022
> mknod("/dev/mapper/mpathb", S_IFBLK|0660, makedev(254, 4)) = 0
> umask(022)                              = 0
> chown("/dev/mapper/mpathb", 0, 6)       = 0
> --
> 
> It has also been observed that if the maps are flushed and multipath is 
> re-run the resulting symlinks to dm-X devices are created in 
> /dev/mapper. Its only when mulitpath -r is executed that this behavior 
> is observed.
> 

Okay!! Could you please run `udevadm monitor` throughout the entire span on all
the command you mentioned. Let's see what all events are generated. That'll give
up pointers on who's involved in all the symlinking business. Because I'm more
(rusted) versed with the device node creation behavior.

> Please let me know if any additional details are needed or if there is 
> additional testing that can be done. We will continue working as well to 
> try and identify the specific commit which introduced this change in 
> behavior.
> 

Can you provide some information about the target? My guess is 3PAR is all iSCSI
? If so, what is the Host storage stack like ?

-- 
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: This is a digitally signed message part
URL: <http://lists.alioth.debian.org/pipermail/pkg-lvm-maintainers/attachments/20160416/1238632f/attachment.sig>