Bug#740701: multipath-tools: mkfs fails "Add. Sense: Incompatible medium installed"

Bill MacAllister whm at stanford.edu
Tue Mar 4 07:49:26 UTC 2014


Package: multipath-tools
Version: 0.4.9+git0.9c3c5172-1
Severity: important

Dear Maintainer,

* What led up to the situation?

We use multipath connected devices on our OpenAFS servers.  We are in the 
process of attempting to upgrade the systems to wheezy.  Our normal 
procedure is to completely evacuate a server and rebuild it from scratch 
and reconfigure the storage.
     
* What exactly did you do (or not do) that was effective (or
  ineffective)?

On the systems we have tested:

  - mkfs on squeeze with a 2.6.32-5-amd64 kernel SUCCEEDS
  - mkfs on squeeze with a 3.2 backport kernel FAILS
  - mkfs on wheezy FAILS
  - mkfs on jessie FAILS

There is one exception to the above list.  We do have one wheezy system 
where we are able to mkfs file systems on multipath served devices.  These 
systems are built using FAI with the configuration be supplied by puppet.  
We have not been able to figure out what is unique about this system.

The current system we have available to test with has a qlogic HBA card.
We have also tested with Emulex cards with the same results.

When mkfs fails multipathd starts spewing errors on the console:

afssvr20:/etc/rsyslog.d# mkfs -t ext4 /dev/mapper/mpathb-part1
mke2fs 1.42.9 (4-Feb-2014)
[11712.731701] sd 3:0:0:20: [sdd] Unhandled sense code
[11712.800335] sd 3:0:0:20: [sdd]
[11712.838877] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[11712.905537] sd 3:0:0:20: [sdd]
[11712.944077] Sense Key : Medium Error [current]
[11712.998246] sd 3:0:0:20: [sdd]
[11713.036788] Add. Sense: Incompatible medium installed
[11713.097195] sd 3:0:0:20: [sdd] CDB:
[11713.139903] Unmap/Read sub-channel: 42 00 00 00 00 00 00 00 18 00
[11713.212987] end_request: I/O error, dev sdd, sector 128
[11713.276032] device-mapper: multipath: Failing path 8:48.
[11713.340113] sd 4:0:1:20: [sdc] Unhandled sense code
[11713.398455] sd 4:0:1:20: [sdc]
[11713.437003] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[11713.503669] sd 4:0:1:20: [sdc]
[11713.542218] Sense Key : Medium Error [current]
[11713.596408] sd 4:0:1:20: [sdc]
[11713.634948] Add. Sense: Incompatible medium installed
[11713.695357] sd 4:0:1:20: [sdc] CDB:
[11713.738077] Unmap/Read sub-channel: 42 00 00 00 00 00 00 00 18 00
[11713.811160] end_request: I/O error, dev sdc, sector 128
[11713.874213] device-mapper: multipath: Failing path 8:32.
[11713.938320] sd 3:0:1:20: [sde] Unhandled sense code
[11713.996662] sd 3:0:1:20: [sde]
[11714.035206] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[11714.101868] sd 3:0:1:20: [sde]
[11714.140413] Sense Key : Medium Error [current]
[11714.194590] sd 3:0:1:20: [sde]
[11714.233136] Add. Sense: Incompatible medium installed
[11714.293553] sd 3:0:1:20: [sde] CDB:
[11714.336268] Unmap/Read sub-channel: 42 00 00 00 00 00 00 00 18 00
[11714.409350] end_request: I/O error, dev sde, sector 128
[11714.472357] device-mapper: multipath: Failing path 8:64.
[11714.536430] sd 4:0:0:20: [sdb] Unhandled sense code
[11714.594768] sd 4:0:0:20: [sdb]
[11714.633306] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[11714.699970] sd 4:0:0:20: [sdb]
[11714.738506] Sense Key : Medium Error [current]
[11714.792681] sd 4:0:0:20: [sdb]
[11714.831221] Add. Sense: Incompatible medium installed
[11714.891625] sd 4:0:0:20: [sdb] CDB:
[11714.934333] Unmap/Read sub-channel: 42 00 00 00 00 00 00 00 18 00
[11715.007395] end_request: I/O error, dev sdb, sector 128
[11715.070374] device-mapper: multipath: Failing path 8:16.
[11715.134475] device-mapper: multipath: Failing path 8:48.
[11715.134690] sd 4:0:1:20: [sdc] Unhandled sense code
[11715.134691] sd 4:0:1:20: [sdc]
[11715.134692] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
.....

Here is the output from multipath -ll -v3

Mar 03 23:16:44 | loading /lib/multipath/libcheckdirectio.so checker
Mar 03 23:16:44 | loading /lib/multipath/libprioconst.so prioritizer
Mar 03 23:16:44 | sda: udev property ID_WWN whitelisted
Mar 03 23:16:44 | sda: not found in pathvec
Mar 03 23:16:44 | sda: mask = 0x25
Mar 03 23:16:44 | sda: dev_t = 8:0
Mar 03 23:16:44 | sda: size = 142082048
Mar 03 23:16:44 | sda: vendor = DELL    
Mar 03 23:16:44 | sda: product = PERC 6/i        
Mar 03 23:16:44 | sda: rev = 1.21
Mar 03 23:16:44 | sda: h:b:t:l = 0:2:0:0
Mar 03 23:16:44 | sda: path state = running

Mar 03 23:16:44 | sda: get_state
Mar 03 23:16:44 | sda: path checker = directio (internal default)
Mar 03 23:16:44 | sda: checker timeout = 90 ms (sysfs setting)
Mar 03 23:16:44 | directio: starting new request
Mar 03 23:16:44 | directio: io finished 4096/0
Mar 03 23:16:44 | sda: state = up
Mar 03 23:16:44 | sdc: udev property ID_WWN whitelisted
Mar 03 23:16:44 | sdc: not found in pathvec
Mar 03 23:16:44 | sdc: mask = 0x25
Mar 03 23:16:44 | sdc: dev_t = 8:32
Mar 03 23:16:44 | sdc: size = 11488198656
Mar 03 23:16:44 | sdc: vendor = NETAPP  
Mar 03 23:16:44 | sdc: product = LUN             
Mar 03 23:16:44 | sdc: rev = 811a
Mar 03 23:16:44 | sdc: h:b:t:l = 3:0:0:20
Mar 03 23:16:44 | SCSI target 3:0:0 -> FC rport 3:0-0
Mar 03 23:16:44 | sdc: tgt_node_name = 0x500a09808d44a0ea
Mar 03 23:16:44 | sdc: path state = running

Mar 03 23:16:44 | sdc: get_state
Mar 03 23:16:44 | loading /lib/multipath/libchecktur.so checker
Mar 03 23:16:44 | sdc: path checker = tur (controller setting)
Mar 03 23:16:44 | sdc: checker timeout = 30 ms (sysfs setting)
Mar 03 23:16:44 | sdc: state = up
Mar 03 23:16:44 | sde: udev property ID_WWN whitelisted
Mar 03 23:16:44 | sde: not found in pathvec
Mar 03 23:16:44 | sde: mask = 0x25
Mar 03 23:16:44 | sde: dev_t = 8:64
Mar 03 23:16:44 | sde: size = 11488198656
Mar 03 23:16:44 | sde: vendor = NETAPP  
Mar 03 23:16:44 | sde: product = LUN             
Mar 03 23:16:44 | sde: rev = 811a
Mar 03 23:16:44 | sde: h:b:t:l = 3:0:1:20
Mar 03 23:16:44 | SCSI target 3:0:1 -> FC rport 3:0-1
Mar 03 23:16:44 | sde: tgt_node_name = 0x500a09808d44a0ea
Mar 03 23:16:44 | sde: path state = running
Mar 03 23:16:44 | sde: get_state
Mar 03 23:16:44 | sde: path checker = tur (controller setting)
Mar 03 23:16:44 | sde: checker timeout = 30 ms (sysfs setting)
Mar 03 23:16:44 | sde: state = up
Mar 03 23:16:44 | sdb: udev property ID_WWN whitelisted
Mar 03 23:16:44 | sdb: not found in pathvec
Mar 03 23:16:44 | sdb: mask = 0x25
Mar 03 23:16:44 | sdb: dev_t = 8:16
Mar 03 23:16:44 | sdb: size = 11488198656
Mar 03 23:16:44 | sdb: vendor = NETAPP  
Mar 03 23:16:44 | sdb: product = LUN             
Mar 03 23:16:44 | sdb: rev = 811a
Mar 03 23:16:44 | sdb: h:b:t:l = 4:0:0:20
Mar 03 23:16:44 | SCSI target 4:0:0 -> FC rport 4:0-0
Mar 03 23:16:44 | sdb: tgt_node_name = 0x500a09808d44a0ea
Mar 03 23:16:44 | sdb: path state = running

Mar 03 23:16:44 | sdb: get_state
Mar 03 23:16:44 | sdb: path checker = tur (controller setting)
Mar 03 23:16:44 | sdb: checker timeout = 30 ms (sysfs setting)
Mar 03 23:16:44 | sdb: state = up
Mar 03 23:16:44 | sdd: udev property ID_WWN whitelisted
Mar 03 23:16:44 | sdd: not found in pathvec
Mar 03 23:16:44 | sdd: mask = 0x25
Mar 03 23:16:44 | sdd: dev_t = 8:48
Mar 03 23:16:44 | sdd: size = 11488198656
Mar 03 23:16:44 | sdd: vendor = NETAPP  
Mar 03 23:16:44 | sdd: product = LUN             
Mar 03 23:16:44 | sdd: rev = 811a
Mar 03 23:16:44 | sdd: h:b:t:l = 4:0:1:20
Mar 03 23:16:44 | SCSI target 4:0:1 -> FC rport 4:0-1
Mar 03 23:16:44 | sdd: tgt_node_name = 0x500a09808d44a0ea
Mar 03 23:16:44 | sdd: path state = running

Mar 03 23:16:44 | sdd: get_state
Mar 03 23:16:44 | sdd: path checker = tur (controller setting)
Mar 03 23:16:44 | sdd: checker timeout = 30 ms (sysfs setting)
Mar 03 23:16:44 | sdd: state = up
Mar 03 23:16:44 | sr0: blacklisted, udev property missing
Mar 03 23:16:44 | dm-0: blacklisted, udev property missing
Mar 03 23:16:44 | dm-1: blacklisted, udev property missing
Mar 03 23:16:44 | dm-2: blacklisted, udev property missing
===== paths list =====
uuid hcil     dev dev_t pri dm_st chk_st vend/prod/rev             dev_st 
     0:2:0:0  sda 8:0   -1  undef ready  DELL    ,PERC 6/i         running
     3:0:0:20 sdc 8:32  -1  undef ready  NETAPP  ,LUN              running
     3:0:1:20 sde 8:64  -1  undef ready  NETAPP  ,LUN              running
     4:0:0:20 sdb 8:16  -1  undef ready  NETAPP  ,LUN              running
Mar 03 23:16:44 | params = 4 queue_if_no_path pg_init_retries 50 retain_attached
_hw_handler 0 2 1 service-time 0 2 2 8:64 1 1 8:16 1 1 service-time 0 2 2 8:32 1
 1 8:48 1 1 
Mar 03 23:16:44 | status = 2 0 0 0 2 1 A 0 2 2 8:64 A 0 0 1 8:16 A 0 0 1 E 0 2 2
 8:32 A 0 0 1 8:48 A 0 0 1 
Mar 03 23:16:44 | mpathb: disassemble map [4 queue_if_no_path pg_init_retries 50
 retain_attached_hw_handler 0 2 1 service-time 0 2 2 8:64 1 1 8:16 1 1 service-t
ime 0 2 2 8:32 1 1 8:48 1 1 ]
Mar 03 23:16:44 | sde: mask = 0x8
Mar 03 23:16:44 | sde: path state = running

Mar 03 23:16:44 | sde: detect_prio = 2 (controller default)
Mar 03 23:16:44 | loading /lib/multipath/libprioalua.so prioritizer
Mar 03 23:16:44 | sde: prio = alua (detected setting)
Mar 03 23:16:44 | reported target port group is 0
Mar 03 23:16:44 | aas = 00 [active/optimized]
Mar 03 23:16:44 | sde: alua prio = 50
Mar 03 23:16:44 | sdb: mask = 0x8
Mar 03 23:16:44 | sdb: path state = running

Mar 03 23:16:44 | sdb: detect_prio = 2 (controller default)
Mar 03 23:16:44 | sdb: prio = alua (detected setting)
Mar 03 23:16:44 | reported target port group is 0
Mar 03 23:16:44 | aas = 00 [active/optimized]
Mar 03 23:16:44 | sdb: alua prio = 50
Mar 03 23:16:44 | sdc: mask = 0x8
Mar 03 23:16:44 | sdc: path state = running

Mar 03 23:16:44 | sdc: detect_prio = 2 (controller default)
Mar 03 23:16:44 | sdc: prio = alua (detected setting)
Mar 03 23:16:44 | reported target port group is 1
Mar 03 23:16:44 | aas = 01 [active/non-optimized]
Mar 03 23:16:44 | sdc: alua prio = 10
Mar 03 23:16:44 | sdd: mask = 0x8
Mar 03 23:16:44 | sdd: path state = running

Mar 03 23:16:44 | sdd: detect_prio = 2 (controller default)
Mar 03 23:16:44 | sdd: prio = alua (detected setting)
Mar 03 23:16:44 | reported target port group is 1
Mar 03 23:16:44 | aas = 01 [active/non-optimized]
Mar 03 23:16:44 | sdd: alua prio = 10
Mar 03 23:16:44 | mpathb: disassemble status [2 0 0 0 2 1 A 0 2 2 8:64 A 0 0 1 8
:16 A 0 0 1 E 0 2 2 8:32 A 0 0 1 8:48 A 0 0 1 ]
mpathb (360a9800064704842534a6c5063513577) dm-0 NETAPP  ,LUN             
size=5.3T features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_han
dle' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 3:0:1:20 sde 8:64 active ready running
| `- 4:0:0:20 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 3:0:0:20 sdc 8:32 active ready running
  `- 4:0:1:20 sdd 8:48 active ready running
Mar 03 23:16:44 | directio checker refcount 1
Mar 03 23:16:44 | tur checker refcount 4
Mar 03 23:16:44 | alua prioritizer refcount 4
Mar 03 23:16:44 | tur checker refcount 3
Mar 03 23:16:44 | alua prioritizer refcount 3
Mar 03 23:16:44 | tur checker refcount 2
Mar 03 23:16:44 | alua prioritizer refcount 2
Mar 03 23:16:44 | tur checker refcount 1
Mar 03 23:16:44 | alua prioritizer refcount 1
Mar 03 23:16:44 | unloading alua prioritizer
Mar 03 23:16:44 | unloading const prioritizer
Mar 03 23:16:44 | unloading tur checker
Mar 03 23:16:44 | unloading directio checker

Thanks for you help,

Bill

-- Package-specific info:
Contents of /etc/multipath.conf:
defaults {
   user_friendly_names yes
}
blacklist {
   devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
   devnode "^hd[a-z]"
   devnode "^cciss.*"
   wwid "36001e4f031414a000f8035050492d102"
}


-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 3.12-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/bash

Versions of packages multipath-tools depends on:
ii  initscripts         2.88dsf-51
ii  kpartx              0.4.9+git0.9c3c5172-1
ii  libaio1             0.3.109-4
ii  libc6               2.17-97
ii  libdevmapper1.02.1  2:1.02.83-2
ii  libgcc1             1:4.8.2-16
ii  libreadline6        6.2+dfsg-0.1
ii  libudev1            204-7
ii  lsb-base            4.1+Debian12
ii  udev                204-7

multipath-tools recommends no packages.

Versions of packages multipath-tools suggests:
pn  multipath-tools-boot  <none>

-- no debconf information



More information about the pkg-lvm-maintainers mailing list