Bug#586182:

[Vincent.McIntyre at csiro.au]

>
>The issue on this particular system is that the binding file resides 
>on /var which isn't available early enough for
/etc/init.d/multipath-tools-boot.
>In that case the paths got assigned new names during each boot which
>causes the problem.
>A workaround is to use the multipath-tools-boot package so the path
>discovery happens in the initramfs with the bindings file available.

(a copy is placed in /var/lib/multipath/bindings of the initramfs.)

The workaround works. Whew.

This might also explain why using user-unfriendly names is regarded
as more robust.

The renaming process works correctly, but somehow not all of the 
/dev/mapper/mpathN block special files are created.

Here's an excerpt showing a successful set up for mpath1:
(grep -e rename -e bindings -e mpath[01234] /var/log/syslog)

Jul  7 10:18:58 install multipathd: Found matching wwid [2220e0001558b5168] in bindings file. Setting alias to mpath1
Jul  7 10:18:58 install multipathd: sdb: ownership set to mpath1
Jul  7 10:18:58 install multipathd: sdi: ownership set to mpath1
Jul  7 10:18:58 install multipathd: sdl: ownership set to mpath1
Jul  7 10:18:58 install multipathd: sdo: ownership set to mpath1
Jul  7 10:18:58 install multipathd: mpath1: pgfailback = -2 (controller setting)
Jul  7 10:18:58 install multipathd: mpath1: pgpolicy = multibus (controller setting)
Jul  7 10:18:58 install multipathd: mpath1: selector = round-robin 0 (controller setting)
Jul  7 10:18:58 install multipathd: mpath1: features = 1 queue_if_no_path (controller setting)
Jul  7 10:18:58 install multipathd: mpath1: hwhandler = 0 (controller setting)
Jul  7 10:18:58 install multipathd: mpath1: rr_weight = 1 (internal default)
Jul  7 10:18:58 install multipathd: mpath1: minio = 100 (controller setting)
Jul  7 10:18:58 install multipathd: mpath1: no_path_retry = 20 (controller setting)
Jul  7 10:18:58 install multipathd: 2220e0001558b5168: rename mpath3 to mpath1
Jul  7 10:18:58 install multipathd: mpath1: load table [0 27343744512 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 8:16 100 8:128 100 
8:176 100 8:224 100]
and a little later...
Jul  7 10:18:58 install multipathd: DEVLINKS=/dev/block/254:1 /dev/disk/by-id/dm-name-mpath1 /dev/disk/by-id/dm-uuid-mpath-2220e0001558b5168 /dev/disk/by-id/scsi-m

The same occurs for mpath0.

Something slightly different happens for mpath2:
Jul  7 10:18:58 install multipathd: Found matching wwid [222de000155468f10] in bindings file. Setting alias to mpath2
Jul  7 10:18:58 install multipathd: sdc: ownership set to mpath2
Jul  7 10:18:58 install multipathd: sdj: ownership set to mpath2
Jul  7 10:18:58 install multipathd: sdm: ownership set to mpath2
Jul  7 10:18:58 install multipathd: sdp: ownership set to mpath2
Jul  7 10:18:58 install multipathd: mpath2: pgfailback = -2 (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: pgpolicy = multibus (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: selector = round-robin 0 (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: features = 1 queue_if_no_path (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: hwhandler = 0 (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: rr_weight = 1 (internal default)
Jul  7 10:18:58 install multipathd: mpath2: minio = 100 (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: no_path_retry = 20 (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: set ACT_RELOAD (path group topology change)
Jul  7 10:18:58 install multipathd: mpath2: domap (0) failure for create/reload map

and there's no DEVLINKS output for mpath2.

The domap (0) failure recurs a four times. Each time multipathd goes
back to the "Found matchin wwid..." line and repeats all the steps shown.

The fourth attempt goes like this:
Jul  7 10:18:58 install multipathd: Found matching wwid [222de000155468f10] in bindings file. Setting alias to mpath2
Jul  7 10:18:58 install multipathd: sdc: ownership set to mpath2
Jul  7 10:18:58 install multipathd: sdj: ownership set to mpath2
Jul  7 10:18:58 install multipathd: sdm: ownership set to mpath2
Jul  7 10:18:58 install multipathd: sdp: ownership set to mpath2
Jul  7 10:18:58 install multipathd: mpath2: pgfailback = -2 (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: pgpolicy = multibus (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: selector = round-robin 0 (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: features = 1 queue_if_no_path (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: hwhandler = 0 (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: rr_weight = 1 (internal default)
Jul  7 10:18:58 install multipathd: mpath2: minio = 100 (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: no_path_retry = 20 (controller setting)
Jul  7 10:18:58 install multipathd: mpath2: set ACT_RELOAD (path group topology change)
Jul  7 10:18:58 install multipathd: mpath2: domap (0) failure for create/reload map
Jul  7 10:18:58 install multipathd: mpath4 devmap removed


Nonetheless, avoiding the need for renaming seems the right way to go.

For those following along at home, for debugging it was useful to:

  * edit /etc/default/multipath-tools to contain:
      # verbosity option for 'multipathd' - default=0
      VERBOSE=3

      if test "0" != "$VERBOSE" ; then
          DAEMON_OPTS="$DAEMON_OPTS -v $VERBOSE"
      fi
    This file is sourced by /etc/init.d/multipath-tools.

  * To see what multipath-tools-boot is doing,
    1. edit /usr/share/initramfs-tools/scripts/local-top/multipath
       (set VERBOSE=3)
    2. update-initramfs -u
    3. reboot, and capture the log from the serial console.

I think we can close this bug.
Thanks, Guido!