Bug#740701: multipath-tools: mkfs fails "Add. Sense: Incompatible medium installed"

[Hans van Kranenburg]

Hi,

On 06/19/2014 06:49 PM, Veeraraghavan, Kugesh wrote:
> + Martin who can provide assistance going forward.
>
> -----Original Message-----
> From: Netapp-Linux-Community [mailto:netapp-linux-community-bounces at linux.netapp.com] On Behalf Of Sarraf, Ritesh
> Sent: Thursday, June 19, 2014 10:08 PM
> To: Hans van Kranenburg; 740701 at bugs.debian.org
> Cc: Sarraf, Ritesh; Bill MacAllister; netapp-linux-community at linux.netapp.com
> Subject: Re: [Netapp-Linux-Community] Bug#740701: multipath-tools: mkfs fails "Add. Sense: Incompatible medium installed"
>
> Hans,
>
> SCSI UNMAP functionality got complete very recently in the Linux kernel. From what I see so far, you seem to be running on 3.2 kernel. The Debian kernel team's policy for stable is to only backport important fixes, and some device driver refreshes. I highly doubt they'd have backported SCSI enhancements.
>
> If UNMAP is important to you as a feature, you may want to try evaluate a more recent kernel.
>
> I just checked on b.d.o, it currently has Linux 3.12. That should be a good start to verify against.
>
> https://packages.debian.org/search?keywords=linux-image&searchon=names&section=all&suite=wheezy-backports

As long as I have not succeeded to construct a proper isolated 
reproducible test case which fails consistently, there's not much reason 
to start changing or trying anything I guess.

So, right now I have built a copy of the production environment setup 
where this occured (see prev messages in debian bug report [1]), and I 
cannot trigger the same bug there yet, so something important must still 
be different. Same linux kernel, same ONTAP version, same layering of 
iscsi, multipath, dm_crypt and lvm...

In the meanwhile, I've been browsing around a bit in the kernel git 
history, also comparing latest (3.16 now) and stable linux-3.2.y.

When taking the error messages as starting point...

sd 12:0:0:0: [sdc] Unhandled sense code
sd 12:0:0:0: [sdc]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 12:0:0:0: [sdc]  Sense Key : Medium Error [current]
sd 12:0:0:0: [sdc]  Add. Sense: Incompatible medium installed
sd 12:0:0:0: [sdc] CDB: Unmap/Read sub-channel: 42 00 00 00 00 00 00 00 
18 00
end_request: I/O error, dev sdc, sector 4372576896

...I see that the only place in the linux kernel where the message 
"Unhandled sense code" occurs is in drivers/scsi/scsi_lib.c, around line 
900, where a case statement is executed on the sense key value.

SENSE KEYS are defined in drivers/include/scsi/scsi.h. One of them is 
MEDIUM_ERROR, 0x03, which seems to be the one we got here. MEDIUM_ERROR 
is not handled in the case statement, so the fall-through is executed:

897         default:
898             description = "Unhandled sense code";
899             action = ACTION_FAIL;
900             break;

This is the case in 3.2 and in 3.16 as well. There are changes related 
to UNMAP or WRITE_SAME in this area, but only in case of 
ILLEGAL_REQUEST. One of those changes (66a651a) is related to preventing 
complete path failures on failed operations. But, that codepath is not 
chosen in this case anyway, so not relevant.

I really want to understand what this message means and who generated 
it. If I assume it's the NetApp filer that returns this data when 
issuing an UNMAP command, I wonder why. Could any of you NetApp folks 
shed some light on when and how ontap 8.1.2 7-Mode might ever send this 
error back? This could help finding a test case which will trigger this 
error, and to find out why it does not occur in a different situation 
that seems to be identical.

The whole error still does not make a huge amount of sense to me. Why 
would my NetApp system return an error which I only would expect to see 
when using a CD-ROM drive or some USB removable device? It's not like 
all disks suddenly vanished from my FAS and disk shelves, like ejecting 
a CD drive. :-)

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=740701

-- 
Hans van Kranenburg - System / Network Engineer
T +31 (0)10 2760434 | hans.van.kranenburg at mendix.com | www.mendix.com