Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
Andrew Patterson
andrew.patterson at hpe.com
Tue Jun 14 23:30:25 UTC 2016
Package: multipath-tools
Version: 0.6.1-3
Multipath discovery fails to find some paths when a large number of
paths are used. I used the following to test this:
1. Present 64 FC LUNs to the OS with 4 paths each (256 paths total).
2. Run multipath -F to make sure we don't have any intermediate results.
3. Run the multipath command, e.g., multipath -v2. In the error case, you
will see "io_setup failed" for some devices.
4. Run multipath -l, checking each mulipath device to make sure
all paths are present (a random number will be missing in the error case).
You can run "multipath -l | grep -E '\|-|`- ' | grep active | wc -l)" to
count the active paths.
Example:
Testing on an amd64 system running sid with
multipath-tools/kpartx_0.6.1-3 from experimental. We hav 64 FC LUNs
with 4 paths each (total 256 paths used mutlipath). The kernel version
is 4.6.0-1-amd64.
With the following /etc/multipath.conf file:
defaults {
user_friendly_names "yes"
}
blacklist {
device {
vendor "HP"
product "LOGICAL VOLUME"
}
Here are the devices used (the boot device is blacklisted):
# lsscsi
[0:0:0:0] storage HP P244br 2.52 -
[0:1:0:0] disk HP LOGICAL VOLUME 2.52 /dev/sda
[1:0:0:0] disk 3PARdata VV 3212 /dev/sdj
[1:0:0:1] disk 3PARdata VV 3212 /dev/sdl
[1:0:0:2] disk 3PARdata VV 3212 /dev/sdn
[1:0:0:3] disk 3PARdata VV 3212 /dev/sdq
[1:0:0:4] disk 3PARdata VV 3212 /dev/sdr
[1:0:0:5] disk 3PARdata VV 3212 /dev/sdt
[1:0:0:6] disk 3PARdata VV 3212 /dev/sdv
[1:0:0:7] disk 3PARdata VV 3212 /dev/sdx
[1:0:0:8] disk 3PARdata VV 3212 /dev/sdah
[1:0:0:9] disk 3PARdata VV 3212 /dev/sdaj
[1:0:0:10] disk 3PARdata VV 3212 /dev/sdal
[1:0:0:11] disk 3PARdata VV 3212 /dev/sdan
[1:0:0:12] disk 3PARdata VV 3212 /dev/sdap
[1:0:0:13] disk 3PARdata VV 3212 /dev/sdar
[1:0:0:14] disk 3PARdata VV 3212 /dev/sdat
[1:0:0:15] disk 3PARdata VV 3212 /dev/sdav
[1:0:0:16] disk 3PARdata VV 3212 /dev/sday
[1:0:0:17] disk 3PARdata VV 3212 /dev/sdbs
[1:0:0:18] disk 3PARdata VV 3212 /dev/sdbu
[1:0:0:19] disk 3PARdata VV 3212 /dev/sdbw
[1:0:0:20] disk 3PARdata VV 3212 /dev/sdby
[1:0:0:21] disk 3PARdata VV 3212 /dev/sdca
[1:0:0:22] disk 3PARdata VV 3212 /dev/sdcc
[1:0:0:23] disk 3PARdata VV 3212 /dev/sdce
[1:0:0:24] disk 3PARdata VV 3212 /dev/sdcg
[1:0:0:25] disk 3PARdata VV 3212 /dev/sdci
[1:0:0:26] disk 3PARdata VV 3212 /dev/sdck
[1:0:0:27] disk 3PARdata VV 3212 /dev/sdcm
[1:0:0:28] disk 3PARdata VV 3212 /dev/sdco
[1:0:0:29] disk 3PARdata VV 3212 /dev/sdcq
[1:0:0:30] disk 3PARdata VV 3212 /dev/sdcs
[1:0:0:31] disk 3PARdata VV 3212 /dev/sdcu
[1:0:0:32] disk 3PARdata VV 3212 /dev/sdea
[1:0:0:33] disk 3PARdata VV 3212 /dev/sdeb
[1:0:0:34] disk 3PARdata VV 3212 /dev/sded
[1:0:0:35] disk 3PARdata VV 3212 /dev/sdef
[1:0:0:36] disk 3PARdata VV 3212 /dev/sdeh
[1:0:0:37] disk 3PARdata VV 3212 /dev/sdej
[1:0:0:38] disk 3PARdata VV 3212 /dev/sdel
[1:0:0:39] disk 3PARdata VV 3212 /dev/sden
[1:0:0:40] disk 3PARdata VV 3212 /dev/sdep
[1:0:0:41] disk 3PARdata VV 3212 /dev/sder
[1:0:0:42] disk 3PARdata VV 3212 /dev/sdet
[1:0:0:43] disk 3PARdata VV 3212 /dev/sdev
[1:0:0:44] disk 3PARdata VV 3212 /dev/sdex
[1:0:0:45] disk 3PARdata VV 3212 /dev/sdez
[1:0:0:46] disk 3PARdata VV 3212 /dev/sdfb
[1:0:0:47] disk 3PARdata VV 3212 /dev/sdfc
[1:0:0:48] disk 3PARdata VV 3212 /dev/sdff
[1:0:0:49] disk 3PARdata VV 3212 /dev/sdfh
[1:0:0:50] disk 3PARdata VV 3212 /dev/sdfj
[1:0:0:51] disk 3PARdata VV 3212 /dev/sdfl
[1:0:0:52] disk 3PARdata VV 3212 /dev/sdfn
[1:0:0:53] disk 3PARdata VV 3212 /dev/sdfp
[1:0:0:54] disk 3PARdata VV 3212 /dev/sdfr
[1:0:0:55] disk 3PARdata VV 3212 /dev/sdft
[1:0:0:56] disk 3PARdata VV 3212 /dev/sdfv
[1:0:0:57] disk 3PARdata VV 3212 /dev/sdfw
[1:0:0:58] disk 3PARdata VV 3212 /dev/sdfz
[1:0:0:59] disk 3PARdata VV 3212 /dev/sdgb
[1:0:0:60] disk 3PARdata VV 3212 /dev/sdgd
[1:0:0:61] disk 3PARdata VV 3212 /dev/sdgf
[1:0:0:62] disk 3PARdata VV 3212 /dev/sdgh
[1:0:0:63] disk 3PARdata VV 3212 /dev/sdgj
[1:0:0:254] enclosu 3PARdata SES 3212 -
[1:0:1:0] disk 3PARdata VV 3212 /dev/sdz
[1:0:1:1] disk 3PARdata VV 3212 /dev/sdaa
[1:0:1:2] disk 3PARdata VV 3212 /dev/sdab
[1:0:1:3] disk 3PARdata VV 3212 /dev/sdac
[1:0:1:4] disk 3PARdata VV 3212 /dev/sdad
[1:0:1:5] disk 3PARdata VV 3212 /dev/sdae
[1:0:1:6] disk 3PARdata VV 3212 /dev/sdaf
[1:0:1:7] disk 3PARdata VV 3212 /dev/sdag
[1:0:1:8] disk 3PARdata VV 3212 /dev/sdba
[1:0:1:9] disk 3PARdata VV 3212 /dev/sdbb
[1:0:1:10] disk 3PARdata VV 3212 /dev/sdbe
[1:0:1:11] disk 3PARdata VV 3212 /dev/sdbf
[1:0:1:12] disk 3PARdata VV 3212 /dev/sdbh
[1:0:1:13] disk 3PARdata VV 3212 /dev/sdbj
[1:0:1:14] disk 3PARdata VV 3212 /dev/sdbk
[1:0:1:15] disk 3PARdata VV 3212 /dev/sdbm
[1:0:1:16] disk 3PARdata VV 3212 /dev/sdbo
[1:0:1:17] disk 3PARdata VV 3212 /dev/sdcv
[1:0:1:18] disk 3PARdata VV 3212 /dev/sdcy
[1:0:1:19] disk 3PARdata VV 3212 /dev/sdcz
[1:0:1:20] disk 3PARdata VV 3212 /dev/sddc
[1:0:1:21] disk 3PARdata VV 3212 /dev/sdde
[1:0:1:22] disk 3PARdata VV 3212 /dev/sddg
[1:0:1:23] disk 3PARdata VV 3212 /dev/sddi
[1:0:1:24] disk 3PARdata VV 3212 /dev/sddk
[1:0:1:25] disk 3PARdata VV 3212 /dev/sddm
[1:0:1:26] disk 3PARdata VV 3212 /dev/sddo
[1:0:1:27] disk 3PARdata VV 3212 /dev/sddq
[1:0:1:28] disk 3PARdata VV 3212 /dev/sddt
[1:0:1:29] disk 3PARdata VV 3212 /dev/sddv
[1:0:1:30] disk 3PARdata VV 3212 /dev/sddx
[1:0:1:31] disk 3PARdata VV 3212 /dev/sddy
[1:0:1:32] disk 3PARdata VV 3212 /dev/sdgl
[1:0:1:33] disk 3PARdata VV 3212 /dev/sdgn
[1:0:1:34] disk 3PARdata VV 3212 /dev/sdgo
[1:0:1:35] disk 3PARdata VV 3212 /dev/sdgq
[1:0:1:36] disk 3PARdata VV 3212 /dev/sdgs
[1:0:1:37] disk 3PARdata VV 3212 /dev/sdgu
[1:0:1:38] disk 3PARdata VV 3212 /dev/sdgw
[1:0:1:39] disk 3PARdata VV 3212 /dev/sdgy
[1:0:1:40] disk 3PARdata VV 3212 /dev/sdha
[1:0:1:41] disk 3PARdata VV 3212 /dev/sdhd
[1:0:1:42] disk 3PARdata VV 3212 /dev/sdhg
[1:0:1:43] disk 3PARdata VV 3212 /dev/sdhi
[1:0:1:44] disk 3PARdata VV 3212 /dev/sdhk
[1:0:1:45] disk 3PARdata VV 3212 /dev/sdhm
[1:0:1:46] disk 3PARdata VV 3212 /dev/sdho
[1:0:1:47] disk 3PARdata VV 3212 /dev/sdhq
[1:0:1:48] disk 3PARdata VV 3212 /dev/sdhr
[1:0:1:49] disk 3PARdata VV 3212 /dev/sdht
[1:0:1:50] disk 3PARdata VV 3212 /dev/sdhv
[1:0:1:51] disk 3PARdata VV 3212 /dev/sdhx
[1:0:1:52] disk 3PARdata VV 3212 /dev/sdhz
[1:0:1:53] disk 3PARdata VV 3212 /dev/sdib
[1:0:1:54] disk 3PARdata VV 3212 /dev/sdid
[1:0:1:55] disk 3PARdata VV 3212 /dev/sdif
[1:0:1:56] disk 3PARdata VV 3212 /dev/sdih
[1:0:1:57] disk 3PARdata VV 3212 /dev/sdik
[1:0:1:58] disk 3PARdata VV 3212 /dev/sdil
[1:0:1:59] disk 3PARdata VV 3212 /dev/sdin
[1:0:1:60] disk 3PARdata VV 3212 /dev/sdip
[1:0:1:61] disk 3PARdata VV 3212 /dev/sdir
[1:0:1:62] disk 3PARdata VV 3212 /dev/sdit
[1:0:1:63] disk 3PARdata VV 3212 /dev/sdiw
[1:0:1:254] enclosu 3PARdata SES 3212 -
[2:0:0:0] disk 3PARdata VV 3212 /dev/sdb
[2:0:0:1] disk 3PARdata VV 3212 /dev/sdc
[2:0:0:2] disk 3PARdata VV 3212 /dev/sdd
[2:0:0:3] disk 3PARdata VV 3212 /dev/sde
[2:0:0:4] disk 3PARdata VV 3212 /dev/sdf
[2:0:0:5] disk 3PARdata VV 3212 /dev/sdg
[2:0:0:6] disk 3PARdata VV 3212 /dev/sdh
[2:0:0:7] disk 3PARdata VV 3212 /dev/sdi
[2:0:0:8] disk 3PARdata VV 3212 /dev/sdai
[2:0:0:9] disk 3PARdata VV 3212 /dev/sdak
[2:0:0:10] disk 3PARdata VV 3212 /dev/sdam
[2:0:0:11] disk 3PARdata VV 3212 /dev/sdao
[2:0:0:12] disk 3PARdata VV 3212 /dev/sdaq
[2:0:0:13] disk 3PARdata VV 3212 /dev/sdas
[2:0:0:14] disk 3PARdata VV 3212 /dev/sdau
[2:0:0:15] disk 3PARdata VV 3212 /dev/sdaw
[2:0:0:16] disk 3PARdata VV 3212 /dev/sdax
[2:0:0:17] disk 3PARdata VV 3212 /dev/sdbr
[2:0:0:18] disk 3PARdata VV 3212 /dev/sdbt
[2:0:0:19] disk 3PARdata VV 3212 /dev/sdbv
[2:0:0:20] disk 3PARdata VV 3212 /dev/sdbx
[2:0:0:21] disk 3PARdata VV 3212 /dev/sdbz
[2:0:0:22] disk 3PARdata VV 3212 /dev/sdcb
[2:0:0:23] disk 3PARdata VV 3212 /dev/sdcd
[2:0:0:24] disk 3PARdata VV 3212 /dev/sdcf
[2:0:0:25] disk 3PARdata VV 3212 /dev/sdch
[2:0:0:26] disk 3PARdata VV 3212 /dev/sdcj
[2:0:0:27] disk 3PARdata VV 3212 /dev/sdcl
[2:0:0:28] disk 3PARdata VV 3212 /dev/sdcn
[2:0:0:29] disk 3PARdata VV 3212 /dev/sdcp
[2:0:0:30] disk 3PARdata VV 3212 /dev/sdcr
[2:0:0:31] disk 3PARdata VV 3212 /dev/sdct
[2:0:0:32] disk 3PARdata VV 3212 /dev/sddz
[2:0:0:33] disk 3PARdata VV 3212 /dev/sdec
[2:0:0:34] disk 3PARdata VV 3212 /dev/sdee
[2:0:0:35] disk 3PARdata VV 3212 /dev/sdeg
[2:0:0:36] disk 3PARdata VV 3212 /dev/sdei
[2:0:0:37] disk 3PARdata VV 3212 /dev/sdek
[2:0:0:38] disk 3PARdata VV 3212 /dev/sdem
[2:0:0:39] disk 3PARdata VV 3212 /dev/sdeo
[2:0:0:40] disk 3PARdata VV 3212 /dev/sdeq
[2:0:0:41] disk 3PARdata VV 3212 /dev/sdes
[2:0:0:42] disk 3PARdata VV 3212 /dev/sdeu
[2:0:0:43] disk 3PARdata VV 3212 /dev/sdew
[2:0:0:44] disk 3PARdata VV 3212 /dev/sdey
[2:0:0:45] disk 3PARdata VV 3212 /dev/sdfa
[2:0:0:46] disk 3PARdata VV 3212 /dev/sdfd
[2:0:0:47] disk 3PARdata VV 3212 /dev/sdfe
[2:0:0:48] disk 3PARdata VV 3212 /dev/sdfg
[2:0:0:49] disk 3PARdata VV 3212 /dev/sdfi
[2:0:0:50] disk 3PARdata VV 3212 /dev/sdfk
[2:0:0:51] disk 3PARdata VV 3212 /dev/sdfm
[2:0:0:52] disk 3PARdata VV 3212 /dev/sdfo
[2:0:0:53] disk 3PARdata VV 3212 /dev/sdfq
[2:0:0:54] disk 3PARdata VV 3212 /dev/sdfs
[2:0:0:55] disk 3PARdata VV 3212 /dev/sdfu
[2:0:0:56] disk 3PARdata VV 3212 /dev/sdfx
[2:0:0:57] disk 3PARdata VV 3212 /dev/sdfy
[2:0:0:58] disk 3PARdata VV 3212 /dev/sdga
[2:0:0:59] disk 3PARdata VV 3212 /dev/sdgc
[2:0:0:60] disk 3PARdata VV 3212 /dev/sdge
[2:0:0:61] disk 3PARdata VV 3212 /dev/sdgg
[2:0:0:62] disk 3PARdata VV 3212 /dev/sdgi
[2:0:0:63] disk 3PARdata VV 3212 /dev/sdgk
[2:0:0:254] enclosu 3PARdata SES 3212 -
[2:0:1:0] disk 3PARdata VV 3212 /dev/sdk
[2:0:1:1] disk 3PARdata VV 3212 /dev/sdm
[2:0:1:2] disk 3PARdata VV 3212 /dev/sdo
[2:0:1:3] disk 3PARdata VV 3212 /dev/sdp
[2:0:1:4] disk 3PARdata VV 3212 /dev/sds
[2:0:1:5] disk 3PARdata VV 3212 /dev/sdu
[2:0:1:6] disk 3PARdata VV 3212 /dev/sdw
[2:0:1:7] disk 3PARdata VV 3212 /dev/sdy
[2:0:1:8] disk 3PARdata VV 3212 /dev/sdaz
[2:0:1:9] disk 3PARdata VV 3212 /dev/sdbc
[2:0:1:10] disk 3PARdata VV 3212 /dev/sdbd
[2:0:1:11] disk 3PARdata VV 3212 /dev/sdbg
[2:0:1:12] disk 3PARdata VV 3212 /dev/sdbi
[2:0:1:13] disk 3PARdata VV 3212 /dev/sdbl
[2:0:1:14] disk 3PARdata VV 3212 /dev/sdbn
[2:0:1:15] disk 3PARdata VV 3212 /dev/sdbp
[2:0:1:16] disk 3PARdata VV 3212 /dev/sdbq
[2:0:1:17] disk 3PARdata VV 3212 /dev/sdcw
[2:0:1:18] disk 3PARdata VV 3212 /dev/sdcx
[2:0:1:19] disk 3PARdata VV 3212 /dev/sdda
[2:0:1:20] disk 3PARdata VV 3212 /dev/sddb
[2:0:1:21] disk 3PARdata VV 3212 /dev/sddd
[2:0:1:22] disk 3PARdata VV 3212 /dev/sddf
[2:0:1:23] disk 3PARdata VV 3212 /dev/sddh
[2:0:1:24] disk 3PARdata VV 3212 /dev/sddj
[2:0:1:25] disk 3PARdata VV 3212 /dev/sddl
[2:0:1:26] disk 3PARdata VV 3212 /dev/sddn
[2:0:1:27] disk 3PARdata VV 3212 /dev/sddp
[2:0:1:28] disk 3PARdata VV 3212 /dev/sddr
[2:0:1:29] disk 3PARdata VV 3212 /dev/sdds
[2:0:1:30] disk 3PARdata VV 3212 /dev/sddu
[2:0:1:31] disk 3PARdata VV 3212 /dev/sddw
[2:0:1:32] disk 3PARdata VV 3212 /dev/sdgm
[2:0:1:33] disk 3PARdata VV 3212 /dev/sdgp
[2:0:1:34] disk 3PARdata VV 3212 /dev/sdgr
[2:0:1:35] disk 3PARdata VV 3212 /dev/sdgt
[2:0:1:36] disk 3PARdata VV 3212 /dev/sdgv
[2:0:1:37] disk 3PARdata VV 3212 /dev/sdgx
[2:0:1:38] disk 3PARdata VV 3212 /dev/sdgz
[2:0:1:39] disk 3PARdata VV 3212 /dev/sdhb
[2:0:1:40] disk 3PARdata VV 3212 /dev/sdhc
[2:0:1:41] disk 3PARdata VV 3212 /dev/sdhe
[2:0:1:42] disk 3PARdata VV 3212 /dev/sdhf
[2:0:1:43] disk 3PARdata VV 3212 /dev/sdhh
[2:0:1:44] disk 3PARdata VV 3212 /dev/sdhj
[2:0:1:45] disk 3PARdata VV 3212 /dev/sdhl
[2:0:1:46] disk 3PARdata VV 3212 /dev/sdhn
[2:0:1:47] disk 3PARdata VV 3212 /dev/sdhp
[2:0:1:48] disk 3PARdata VV 3212 /dev/sdhs
[2:0:1:49] disk 3PARdata VV 3212 /dev/sdhu
[2:0:1:50] disk 3PARdata VV 3212 /dev/sdhw
[2:0:1:51] disk 3PARdata VV 3212 /dev/sdhy
[2:0:1:52] disk 3PARdata VV 3212 /dev/sdia
[2:0:1:53] disk 3PARdata VV 3212 /dev/sdic
[2:0:1:54] disk 3PARdata VV 3212 /dev/sdie
[2:0:1:55] disk 3PARdata VV 3212 /dev/sdig
[2:0:1:56] disk 3PARdata VV 3212 /dev/sdii
[2:0:1:57] disk 3PARdata VV 3212 /dev/sdij
[2:0:1:58] disk 3PARdata VV 3212 /dev/sdim
[2:0:1:59] disk 3PARdata VV 3212 /dev/sdio
[2:0:1:60] disk 3PARdata VV 3212 /dev/sdiq
[2:0:1:61] disk 3PARdata VV 3212 /dev/sdis
[2:0:1:62] disk 3PARdata VV 3212 /dev/sdiu
[2:0:1:63] disk 3PARdata VV 3212 /dev/sdiv
[2:0:1:254] enclosu 3PARdata SES 3212 -
Here is the output of multipath -v3. I have only shown the output for
one succeeding and one failing case.
Jun 14 16:41:26 | sddn: udev property ID_WWN whitelisted
Jun 14 16:41:26 | sddn: mask = 0x3f
Jun 14 16:41:26 | sddn: dev_t = 71:80
Jun 14 16:41:26 | sddn: size = 2097152
Jun 14 16:41:26 | sddn: vendor = 3PARdata
Jun 14 16:41:26 | sddn: product = VV
Jun 14 16:41:26 | sddn: rev = 3212
Jun 14 16:41:26 | sddn: h:b:t:l = 2:0:1:26
Jun 14 16:41:26 | SCSI target 2:0:1 -> FC rport 2:0-4
Jun 14 16:41:26 | sddn: tgt_node_name = 0x2ff70002ac0028be
Jun 14 16:41:26 | sddn: path state = running
Jun 14 16:41:26 | sddn: 1011 cyl, 34 heads, 61 sectors/track, start at 0
Jun 14 16:41:26 | sddn: serial = 1410430
Jun 14 16:41:26 | sddn: get_state
Jun 14 16:41:26 | sddn: path_checker = directio (controller setting)
Jun 14 16:41:26 | sddn: checker timeout = 30 ms (internal default)
Jun 14 16:41:26 | directio: starting new request
Jun 14 16:41:26 | directio: io finished 4096/0
Jun 14 16:41:26 | sddn: state = up
Jun 14 16:41:26 | sddn: uid_attribute = ID_SERIAL (config file default)
Jun 14 16:41:26 | sddn: uid = 360002ac00000000000001997000028be (udev)
Jun 14 16:41:26 | sddn: detect_prio = no (config file default)
Jun 14 16:41:26 | sddn: prio = const controller setting)
Jun 14 16:41:26 | sddn: prio args = "" controller setting)
Jun 14 16:41:26 | sddn: const prio = 1
Jun 14 16:41:26 | sddp: udev property ID_WWN whitelisted
Jun 14 16:41:26 | sddp: mask = 0x3f
Jun 14 16:41:26 | sddp: dev_t = 71:112
Jun 14 16:41:26 | sddp: size = 2097152
Jun 14 16:41:26 | sddp: vendor = 3PARdata
Jun 14 16:41:26 | sddp: product = VV
Jun 14 16:41:26 | sddp: rev = 3212
Jun 14 16:41:26 | sddp: h:b:t:l = 2:0:1:27
Jun 14 16:41:26 | SCSI target 2:0:1 -> FC rport 2:0-4
Jun 14 16:41:26 | sddp: tgt_node_name = 0x2ff70002ac0028be
Jun 14 16:41:26 | sddp: path state = running
Jun 14 16:41:26 | sddp: 1011 cyl, 34 heads, 61 sectors/track, start at 0
Jun 14 16:41:26 | sddp: serial = 1410430
Jun 14 16:41:26 | sddp: get_state
Jun 14 16:41:26 | sddp: path_checker = directio (controller setting)
Jun 14 16:41:26 | sddp: checker timeout = 30 ms (internal default)
Jun 14 16:41:26 | io_setup failed
Jun 14 16:41:26 | sddp: checker init failed
There is no output in dmesg.
Changing /proc/sys/fs/aio-max-nr from the default 65536 to 1048576
seems to help. With this change, I can go up to 256 LUNs (1024 paths)
with no problems.
I have seen similar issues in earlier versions of multipath-tools.
--
Andrew Patterson
Hewlett-Packard Enterprise
More information about the pkg-lvm-maintainers
mailing list