Bug#499643: Segmentation fault when attempting re-assembly of a failed RAID5 via "mdadm -v -A --run --force"

Sat Sep 20 22:49:49 UTC 2008

Package: mdadm
Version: 2.6.7-3.1
Severity: normal

The system is a clean 1-day old installation of Lenny.
Doing tests of RAID re-assembly, by failing devices and having mdadm rebuild
them. Had manually failed 2 devices on a 5-drive RAID5, causing it to stop.
All attempts at forced re-assembly ended with SegFault, e.g.

# mdadm -v -A --force --run /dev/md0 /dev/loop[0-4]
mdadm: looking for devices for /dev/md0
mdadm: /dev/loop0 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/loop1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/loop2 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/loop3 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/loop4 is identified as a member of /dev/md0, slot 1.
mdadm: forcing event count in /dev/loop4(1) from 74 upto 80
Segmentation fault

After many retries and failures, I downloaded older version of
mdadm v2.6.4 from neilb's site and compiled it and it worked fine
(no segfault):

# ./mdadm -v -A --run --force /dev/md0 /dev/loop0 /dev/loop2 /dev/loop3 /dev/loo
p4 /dev/loop1
mdadm: looking for devices for /dev/md0
mdadm: /dev/loop0 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/loop2 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/loop3 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/loop4 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/loop1 is identified as a member of /dev/md0, slot 5.
mdadm: forcing event count in /dev/loop4(1) from 74 upto 80
mdadm: clearing FAULTY flag for device 3 in /dev/md0 for /dev/loop4
mdadm: added /dev/loop4 to /dev/md0 as 1
mdadm: added /dev/loop2 to /dev/md0 as 2
mdadm: added /dev/loop3 to /dev/md0 as 3
mdadm: no uptodate device for slot 4 of /dev/md0
mdadm: added /dev/loop1 to /dev/md0 as 5
mdadm: added /dev/loop0 to /dev/md0 as 0
mdadm: /dev/md0 has been started with 4 drives (out of 5) and 1 spare.

Perhaps Lenny could consider the older, apparently more stable version
of mdadm?

-- Package-specific info:
--- mount output
/dev/hda1 on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/hda9 on /home type ext3 (rw)
/dev/hda8 on /tmp type ext3 (rw)
/dev/hda5 on /usr type ext3 (rw)
/dev/hda6 on /var type ext3 (rw)

--- mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
#DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This file was auto-generated on Sat, 20 Sep 2008 02:37:26 +0300
# by mkconf $Id$

--- /proc/mdstat:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
unused devices: <none>

--- /proc/partitions:
major minor  #blocks  name

   3     0   78177792 hda
   3     1     273073 hda1
   3     2          1 hda2
   3     5    4883728 hda5
   3     6    2931831 hda6
   3     7    2176776 hda7
   3     8     393561 hda8
   3     9   67513131 hda9
   7     0     262144 loop0
   7     1     262144 loop1
   7     2     262144 loop2
   7     3     262144 loop3
   7     4     262144 loop4
   7     5     262144 loop5

--- initrd.img-2.6.26-1-686:
29479 blocks
scripts/local-top/mdadm
sbin/mdadm
lib/modules/2.6.26-1-686/kernel/drivers/md/md-mod.ko
lib/modules/2.6.26-1-686/kernel/drivers/md/raid10.ko
lib/modules/2.6.26-1-686/kernel/drivers/md/multipath.ko
lib/modules/2.6.26-1-686/kernel/drivers/md/dm-mod.ko
lib/modules/2.6.26-1-686/kernel/drivers/md/raid456.ko
lib/modules/2.6.26-1-686/kernel/drivers/md/raid1.ko
lib/modules/2.6.26-1-686/kernel/drivers/md/dm-log.ko
lib/modules/2.6.26-1-686/kernel/drivers/md/dm-mirror.ko
lib/modules/2.6.26-1-686/kernel/drivers/md/linear.ko
lib/modules/2.6.26-1-686/kernel/drivers/md/raid0.ko
etc/mdadm
etc/mdadm/mdadm.conf

--- /proc/modules:
raid10 18560 0 - Live 0xee94a000
raid456 117264 0 - Live 0xee964000
async_xor 3520 1 raid456, Live 0xee91a000
async_memcpy 2304 1 raid456, Live 0xee918000
async_tx 6284 3 raid456,async_xor,async_memcpy, Live 0xee8cf000
xor 14696 2 raid456,async_xor, Live 0xee926000
raid1 18016 0 - Live 0xee920000
raid0 6368 0 - Live 0xee8ae000
multipath 6112 0 - Live 0xee83b000
linear 4544 0 - Live 0xee866000
md_mod 67036 6 raid10,raid456,raid1,raid0,multipath,linear, Live 0xee930000
dm_mirror 15104 0 - Live 0xee818000
dm_log 8452 1 dm_mirror, Live 0xee8a6000
dm_mod 46184 2 dm_mirror,dm_log, Live 0xee8c2000

--- /var/log/syslog:

--- volume detail:

--- /proc/cmdline
root=/dev/hda1 ro quiet 

--- grub:
kernel		/boot/vmlinuz-2.6.26-1-686 root=/dev/hda1 ro quiet 
kernel		/boot/vmlinuz-2.6.26-1-686 root=/dev/hda1 ro quiet single
kernel		/boot/vmlinuz-2.6.24-1-686 root=/dev/hda1 ro quiet 
kernel		/boot/vmlinuz-2.6.24-1-686 root=/dev/hda1 ro quiet single

-- System Information:
Debian Release: lenny/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: i386 (i686)

Kernel: Linux 2.6.26-1-686 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages mdadm depends on:
ii  debconf                       1.5.22     Debian configuration management sy
ii  libc6                         2.7-13     GNU C Library: Shared libraries
ii  lsb-base                      3.2-20     Linux Standard Base 3.2 init scrip
ii  makedev                       2.3.1-88   creates device files in /dev
ii  udev                          0.125-6    /dev/ and hotplug management daemo

Versions of packages mdadm recommends:
ii  module-init-tools             3.4-1      tools for managing Linux kernel mo
ii  postfix [mail-transport-agent 2.5.5-1.1  High-performance mail transport ag

mdadm suggests no packages.

-- debconf information:
* mdadm/autostart: true
  mdadm/mail_to: root
  mdadm/initrdstart_msg_errmd:
* mdadm/initrdstart: all
  mdadm/initrdstart_msg_errconf:
  mdadm/initrdstart_notinconf: false
  mdadm/initrdstart_msg_errexist:
  mdadm/initrdstart_msg_intro:
  mdadm/autocheck: true
  mdadm/initrdstart_msg_errblock:
  mdadm/start_daemon: true