Bug#684708: mdadm: support external metadata arrays correctly

[Miquel van Smoorenburg]

On 08/13/2012 01:43 PM, Michael Tokarev wrote:
> On 13.08.2012 14:10, Miquel van Smoorenburg wrote:
>> Package: mdadm
>> Version: 3.2.5-1
>> Severity: serious
>> Tags: patch wheezy sid
>>
>> The initramfs hook supplied by mdadm doesn't install mdmon. Also, mdmon
>> is not included in the .udeb for the installer.
>>
>> This means that if you have an array with external metadata (ddf or,
>> more widely used, imsm - Intel Matrix Raid) that it will come up
>> readonly. This causes the installer to hang or the system not being
>> able to boot if root is on that array.
>
> I'm not sure this is the right course of actions -- speaking of the
> initramfs part, not about the d-i part.
>
> What's wrong with the array being started read-only?  Root filesystem
> has always been mounted readonly in initrd/initramfs.  There's no fsck
> tool included in initramfs.  Once we switch to real root, we may start
> mdmon and remount filesystem(s) read-write as appropriate, after
> fsck'ing them etc.
>
> Why the system is not being able to boot if root is on such an array?

Ah. You could ofcourse start mdmon very early in the bootprocess once 
you've switched to the real rootfilesystem, but that sounds fragile to 
me. mdadm automatically starts mdmon when needed, for the right array, 
at the right time. If you include mdmon in the initramfs, you do not 
need extra initscripts, you can debug stuff if you happen to get dropped 
in an initramfs prompt, you can boot with init=/bin/bash to repair 
things and easily mount the rootfs rw, etc.

>> - it adds a mdadm-waitidle script that runs just before reboot/halt. For all
>>    arrays that are still running, it sets safe_mode_delay to a low version,
>>    sets sync_action to idle, and waits for the array(s) to go idle.
>>    This is needed so that the array is clean, otherwise it will start
>>    to resync at the next boot.
>
> This is risky - we may never finish shutdown.  This is especially risky
> for things like raid - eg, stalled raid (resync) thread (we've seen
> these more than once) -- in such cases current code will shut down,
> but with this wait it wont anymore.  Especially useful for remote
> systems.  Such an approach should be tested with extra care, I'm
> not sure we have resources to do that for wheezy.  Generally it is a
> good idea I think.

Well no... /etc/init.d/mdadm-waitidle doesn't do anything if you don't 
have external metadata arrays, it runs for all arrays in parallel, and 
it has a timeout of a couple of seconds. It will NOT block shutdown.

The mdadm/mdmon documentation in fact says that you have to do this, and 
there is a special mdadm mode for it: mdadm --wait-clean --scan. You are 
supposed to run that after unmounting the rootfs. I have chosen a 
different implementation in /etc/init.d/mdadm-waitidle, because I think 
it is faster, and I did not want to patch mdadm.

>> - it adds 2 lines of code to mdmon.c so that it symlinks its pidfile
>>    into /run/sendsigs.omit.d - mdmon should not be killed at shutdown,
>>    we still need it after the rootfs has been unmounted.
>
> And I'm not sure this is needed, either: it can trivially be done in
> the initscript.

Hmm, not if you start/stop arrays (other than the rootfs ofcourse...) 
manually with mdadm, since mdadm takes care of starting/stopping mdmon 
for you. Because it does that, it should also take care of 
/run/sendsigs.omit.d, which is trivial (just a symlink).

>
>> I have added support for installation on Intel Matrix raid (imsm)
>> arrays using mdadm to d-i, and I'll be sending patches to the debian-boot
>> list soon. Please consider this patch for inclusion in wheezy.
>
> So far I think the only real change there is the inclusion of mdmon
> to the udeb, the other changes are a bit questionable for wheezy.

The changes only apply to systems with external metadata arrays, which 
are broken right now anyway if I'm not mistaken- the mdadm in initramfs 
starts all arrays, but right now nothing starts mdmon.. This is not a 
new feature, it's a bugfix.

 > (*) ofcourse it needs to be verified - the takeover after booting
 > into real system.  I can't do it, since I don't have any spare system
 > right now to install debian onto imsm array.  I can only verify that
 > the new takeover does not harm regular boot process without imsm
 > arrays.

One slight problem: mdadm only supports imsm on systems that actually 
have the Intel Matrix raid option rom installed. There are a lot of 
those systems, even laptops, but you do have to have one lying around.

 > This change - mdmon in initramfs - might be included to wheezy, it
 > isn't a dramatic change.  But I still don't think it is a good idea
 > in general -- using native md arrays is a much better way.

I agree, but the advantage of imsm raid is that the BIOS knows about it. 
So if a disk is broken, the system still boots. Also you can share the 
RAID setup with other OSes. But yes, we should really have BIOSes that 
understand linux md raid ..

Thank you,

Mike.