Bug#621786: mdadm: invalid pointer or memory corruption on armel system

Arnaud arnaud.desmier at googlemail.com
Fri Apr 8 21:14:29 UTC 2011 

Package: mdadm
Version: 3.1.4.1-0
Severity: critical
Justification: breaks the whole system

One month ago I tried to install Debian Squeeze on my QNAP TS-409 nas (see configuration bellow). During the installation process I saw an error message like "invalid pointer" or "memory corruption", but the installer continue and ends as if nothing happened. After reboot I couldn't access anymore to the system. 

Then I plugged in a serial console to see what happened and the system couldn't start because it failed to find the root filesystem. In fact the mdadm tool crashed and randomly the error was one of the following after each reboot:
    - *** glibc detected *** /sbin/mdadm: munmap_chunk(): invalid pointer: 0x00089c00 ***
    - *** glibc detected *** /sbin/mdadm: malloc(): memory corruption: 0x0008b660 ***
    - double free

To boot my system, the only choice was to use the initramfs console, and use mdadm manually to activate my arrays. I also found my configuration file mdadm.conf empty after the installation process, maybe due to the error during the installation process.

I also tried to install a Lenny version, the Lenny installation process worked fine and I could get access to my system at the first boot. After some reboot it worked still fine so I tried a dist upgrade to Squeeze but after rebooting on Squeeze I get the same issue.

For each installation I made my test on new arrays rebuild from scratch, on new partitions.

Since one month I'm running on the git version (3 Mars 2011 snapshot) and I can boot without manual steps but I'm still getting some memory errors at startup or when I try to do the following:
/etc/init.d/mdadm-raid restart 
*** glibc detected *** /sbin/mdadm: munmap_chunk(): invalid pointer: 0x00084800 ***
*** glibc detected *** /sbin/mdadm: malloc(): memory corruption: 0x00084660 ***
Aborted
Generating udev events for MD arrays...done.


I've openned a thread on qnap forum and it seems that I'm not the only one to get this error
http://forum.qnap.com/viewtopic.php?f=147&t=41550&start=0



-- Package-specific info:
--- mdadm.conf
DEVICE partitions
CREATE owner=root group=disk mode=0660 auto=yes
HOMEHOST <system>
MAILADDR root
ARRAY /dev/md/debian:1 metadata=1.2 name=debian:1 UUID=fab8edec:a64fdc47:ead2b148:512a0ee9
ARRAY /dev/md/debian:0 metadata=1.2 name=debian:0 UUID=53dacd1d:be5456eb:ec80d301:3cc2fe01
ARRAY /dev/md2 metadata=1.2 name=debian:2 UUID=f42695a8:e8c1566b:a5cf49ce:9fde3762

--- /etc/default/mdadm
INITRDSTART='all'
AUTOSTART=true
AUTOCHECK=true
START_DAEMON=true
DAEMON_OPTIONS="--syslog"
VERBOSE=false

--- /proc/mdstat:
Personalities : [raid1] 
md0 : active raid1 sdb1[1] sda1[0]
      9763768 blocks super 1.2 [2/2] [UU]
      
md1 : active raid1 sdb2[1] sda2[0]
      499700 blocks super 1.2 [2/2] [UU]
      
md2 : active raid1 sda3[0] sdb3[1]
      966493272 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>

--- /proc/partitions:
major minor  #blocks  name

  31        0        512 mtdblock0
  31        1       2048 mtdblock1
  31        2       4096 mtdblock2
  31        3       1024 mtdblock3
  31        4        128 mtdblock4
  31        5        384 mtdblock5
   8        0  976762584 sda
   8        1    9764864 sda1
   8        2     499712 sda2
   8        3  966494432 sda3
   8       16  976762584 sdb
   8       17    9764864 sdb1
   8       18     499712 sdb2
   8       19  966494432 sdb3
   9        2  966493272 md2
   9        1     499700 md1
   9        0    9763768 md0

--- LVM physical volumes:
LVM does not seem to be used.
--- mount output
/dev/md0 on / type ext4 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/md2 on /data type ext4 (rw)
/data on /export/public type none (rw,bind)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)

--- initrd.img-2.6.32-5-orion5x:
11628 blocks
b62a6adcd76772d968d831e48feb5b70  ./etc/mdadm/mdadm.conf
744c01fafe631f39cb93fb950bbcaf70  ./lib/modules/2.6.32-5-orion5x/kernel/drivers/md/multipath.ko
6fc2182d097ff69f4271f9eb09c6132a  ./lib/modules/2.6.32-5-orion5x/kernel/drivers/md/raid6_pq.ko
5a32b7079fa214bcf946d2423da033b1  ./lib/modules/2.6.32-5-orion5x/kernel/drivers/md/md-mod.ko
ef2d3f70707f4f1d9078c8dd8c2cdf4e  ./lib/modules/2.6.32-5-orion5x/kernel/drivers/md/linear.ko
c932e3bc1c974e74058b46955c3bb276  ./lib/modules/2.6.32-5-orion5x/kernel/drivers/md/raid10.ko
c771cea128499bbc8f4fd15b86d7ef46  ./lib/modules/2.6.32-5-orion5x/kernel/drivers/md/raid1.ko
284eaa799a75bec45bd90daf54d567c1  ./lib/modules/2.6.32-5-orion5x/kernel/drivers/md/raid456.ko
32d0bb86800a1871319f9d39756d2857  ./lib/modules/2.6.32-5-orion5x/kernel/drivers/md/raid0.ko
6d36c9f17e93e076f9e15a816808bddb  ./sbin/mdadm
f4fbd9099399ab08ba9b9f6c71d77595  ./scripts/local-top/mdadm

--- initrd's /conf/conf.d/md:
MD_HOMEHOST='nashome'
MD_DEVPAIRS='/dev/md/debian:1:raid1 /dev/md/debian:0:raid1 /dev/md2:raid1'
MD_LEVELS='raid1 raid1 raid1'
MD_DEVS=all
MD_MODULES='raid1'

--- /proc/modules:
raid1 19757 3 - Live 0xbf0a6000
md_mod 80394 4 raid1, Live 0xbf088000

--- /var/log/syslog:

--- volume detail:
/dev/sda is not recognised by mdadm.
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 53dacd1d:be5456eb:ec80d301:3cc2fe01
           Name : debian:0
  Creation Time : Thu Mar  3 09:48:59 2011
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 19527680 (9.31 GiB 10.00 GB)
     Array Size : 19527536 (9.31 GiB 10.00 GB)
  Used Dev Size : 19527536 (9.31 GiB 10.00 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 6c29065f:1d4859ba:39babafd:7e1549e0

    Update Time : Fri Apr  8 22:42:29 2011
       Checksum : e4859f65 - correct
         Events : 1786


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
--
/dev/sda2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fab8edec:a64fdc47:ead2b148:512a0ee9
           Name : debian:1
  Creation Time : Thu Mar  3 11:15:16 2011
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 999400 (488.07 MiB 511.69 MB)
     Array Size : 999400 (488.07 MiB 511.69 MB)
    Data Offset : 24 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 1ca9bf9c:1a46824f:04671d24:dd760c93

    Update Time : Fri Apr  8 22:41:42 2011
       Checksum : e8ec54da - correct
         Events : 116


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
--
/dev/sda3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : f42695a8:e8c1566b:a5cf49ce:9fde3762
           Name : debian:2
  Creation Time : Thu Mar  3 13:52:03 2011
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1932986817 (921.72 GiB 989.69 GB)
     Array Size : 1932986544 (921.72 GiB 989.69 GB)
  Used Dev Size : 1932986544 (921.72 GiB 989.69 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b72b0ada:ef87ee04:987dcf55:c5cbe171

    Update Time : Fri Apr  8 22:14:11 2011
       Checksum : b15df46d - correct
         Events : 9728


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
--
/dev/sdb is not recognised by mdadm.
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 53dacd1d:be5456eb:ec80d301:3cc2fe01
           Name : debian:0
  Creation Time : Thu Mar  3 09:48:59 2011
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 19527680 (9.31 GiB 10.00 GB)
     Array Size : 19527536 (9.31 GiB 10.00 GB)
  Used Dev Size : 19527536 (9.31 GiB 10.00 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 659fccdc:ff29325d:792bc002:85551475

    Update Time : Fri Apr  8 22:42:29 2011
       Checksum : 9ef5a887 - correct
         Events : 1786


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing)
--
/dev/sdb2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fab8edec:a64fdc47:ead2b148:512a0ee9
           Name : debian:1
  Creation Time : Thu Mar  3 11:15:16 2011
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 999400 (488.07 MiB 511.69 MB)
     Array Size : 999400 (488.07 MiB 511.69 MB)
    Data Offset : 24 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 5caa5f7f:1040a213:56e5eaf8:bdd033f2

    Update Time : Fri Apr  8 22:41:42 2011
       Checksum : c3a12844 - correct
         Events : 116


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing)
--
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : f42695a8:e8c1566b:a5cf49ce:9fde3762
           Name : debian:2
  Creation Time : Thu Mar  3 13:52:03 2011
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1932986817 (921.72 GiB 989.69 GB)
     Array Size : 1932986544 (921.72 GiB 989.69 GB)
  Used Dev Size : 1932986544 (921.72 GiB 989.69 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 201ef31f:53a07223:f5d3d81b:62c758a2

    Update Time : Fri Apr  8 22:14:11 2011
       Checksum : c4b5135 - correct
         Events : 9728


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing)
--

--- /proc/cmdline
console=ttyS0,115200n8 root=/dev/ram rw initrd=0x800000,0x3fffff

--- udev:
ii  udev           164-3          /dev/ and hotplug management daemon
51a186a318e2d8f9119d9409f6375b99  /lib/udev/rules.d/64-md-raid.rules

--- /dev:
brw-rw---- 1 root disk 9, 0 Mar 31 19:38 /dev/md0
brw-rw---- 1 root disk 9, 1 Mar 31 19:38 /dev/md1
brw-rw---- 1 root disk 9, 2 Mar 31 19:38 /dev/md2

/dev/disk/by-id:
total 0
lrwxrwxrwx 1 root root  9 Mar 31 19:38 ata-WDC_WD10EADS-00L5B1_WD-WCAU45613335 -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 31 19:38 ata-WDC_WD10EADS-00L5B1_WD-WCAU45613335-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 31 19:38 ata-WDC_WD10EADS-00L5B1_WD-WCAU45613335-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Mar 31 19:38 ata-WDC_WD10EADS-00L5B1_WD-WCAU45613335-part3 -> ../../sda3
lrwxrwxrwx 1 root root  9 Mar 31 19:38 ata-WDC_WD10EADS-00L5B1_WD-WCAU45705889 -> ../../sdb
lrwxrwxrwx 1 root root 10 Mar 31 19:38 ata-WDC_WD10EADS-00L5B1_WD-WCAU45705889-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Mar 31 19:38 ata-WDC_WD10EADS-00L5B1_WD-WCAU45705889-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Mar 31 19:38 ata-WDC_WD10EADS-00L5B1_WD-WCAU45705889-part3 -> ../../sdb3
lrwxrwxrwx 1 root root  9 Mar 31 19:38 md-name-debian:0 -> ../../md0
lrwxrwxrwx 1 root root  9 Mar 31 19:38 md-name-debian:1 -> ../../md1
lrwxrwxrwx 1 root root  9 Mar 31 19:38 md-name-debian:2 -> ../../md2
lrwxrwxrwx 1 root root  9 Mar 31 19:38 md-uuid-53dacd1d:be5456eb:ec80d301:3cc2fe01 -> ../../md0
lrwxrwxrwx 1 root root  9 Mar 31 19:38 md-uuid-f42695a8:e8c1566b:a5cf49ce:9fde3762 -> ../../md2
lrwxrwxrwx 1 root root  9 Mar 31 19:38 md-uuid-fab8edec:a64fdc47:ead2b148:512a0ee9 -> ../../md1
lrwxrwxrwx 1 root root  9 Mar 31 19:38 scsi-SATA_WDC_WD10EADS-00_WD-WCAU45613335 -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 31 19:38 scsi-SATA_WDC_WD10EADS-00_WD-WCAU45613335-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 31 19:38 scsi-SATA_WDC_WD10EADS-00_WD-WCAU45613335-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Mar 31 19:38 scsi-SATA_WDC_WD10EADS-00_WD-WCAU45613335-part3 -> ../../sda3
lrwxrwxrwx 1 root root  9 Mar 31 19:38 scsi-SATA_WDC_WD10EADS-00_WD-WCAU45705889 -> ../../sdb
lrwxrwxrwx 1 root root 10 Mar 31 19:38 scsi-SATA_WDC_WD10EADS-00_WD-WCAU45705889-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Mar 31 19:38 scsi-SATA_WDC_WD10EADS-00_WD-WCAU45705889-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Mar 31 19:38 scsi-SATA_WDC_WD10EADS-00_WD-WCAU45705889-part3 -> ../../sdb3
lrwxrwxrwx 1 root root  9 Mar 31 19:38 wwn-0x50014ee257ac3600 -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 31 19:38 wwn-0x50014ee257ac3600-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 31 19:38 wwn-0x50014ee257ac3600-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Mar 31 19:38 wwn-0x50014ee257ac3600-part3 -> ../../sda3
lrwxrwxrwx 1 root root  9 Mar 31 19:38 wwn-0x50014ee2ad01e83a -> ../../sdb
lrwxrwxrwx 1 root root 10 Mar 31 19:38 wwn-0x50014ee2ad01e83a-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Mar 31 19:38 wwn-0x50014ee2ad01e83a-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Mar 31 19:38 wwn-0x50014ee2ad01e83a-part3 -> ../../sdb3

/dev/disk/by-label:
total 0
lrwxrwxrwx 1 root root 9 Mar 31 19:38 data -> ../../md2
lrwxrwxrwx 1 root root 9 Mar 31 19:38 system -> ../../md0

/dev/disk/by-path:
total 0
lrwxrwxrwx 1 root root  9 Mar 31 19:38 pci-0000:00:01.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 31 19:38 pci-0000:00:01.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 31 19:38 pci-0000:00:01.0-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Mar 31 19:38 pci-0000:00:01.0-scsi-0:0:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root  9 Mar 31 19:38 pci-0000:00:01.0-scsi-1:0:0:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Mar 31 19:38 pci-0000:00:01.0-scsi-1:0:0:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Mar 31 19:38 pci-0000:00:01.0-scsi-1:0:0:0-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Mar 31 19:38 pci-0000:00:01.0-scsi-1:0:0:0-part3 -> ../../sdb3

/dev/disk/by-uuid:
total 0
lrwxrwxrwx 1 root root 9 Mar 31 19:38 1cd71730-3f96-48af-bae9-4c3f6fa60b52 -> ../../md2
lrwxrwxrwx 1 root root 9 Mar 31 19:38 695192e7-148e-4d02-8588-4d2fc3f8c978 -> ../../md0
lrwxrwxrwx 1 root root 9 Mar 31 19:38 e38368bd-a842-4334-9e5c-4e28b70f8cb0 -> ../../md1

/dev/md:
total 0
lrwxrwxrwx 1 root root 6 Mar 31 19:38 debian:0 -> ../md0
lrwxrwxrwx 1 root root 6 Mar 31 19:38 debian:1 -> ../md1

Auto-generated on Fri, 08 Apr 2011 22:42:31 +0200
by mdadm bugscript 3.1.4.1-0

-- System Information:
Debian Release: 6.0.1
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: armel (armv5tel)

Kernel: Linux 2.6.32-5-orion5x
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages mdadm depends on:
ii  debconf                 1.5.36.1         Debian configuration management sy
ii  libc6                   2.11.2-10        Embedded GNU C Library: Shared lib
ii  libgcc1                 1:4.4.5-8        GCC support library
ii  lsb-base                3.2-23.2squeeze1 Linux Standard Base 3.2 init scrip
ii  udev                    164-3            /dev/ and hotplug management daemo

Versions of packages mdadm recommends:
ii  exim4-daemon-light [mail-tran 4.72-6     lightweight Exim MTA (v4) daemon
ii  module-init-tools             3.12-1     tools for managing Linux kernel mo

mdadm suggests no packages.

-- debconf information:
  mdadm/initrdstart_msg_errexist:
  mdadm/initrdstart_msg_intro:
  mdadm/autostart: true
* mdadm/autocheck: true
  mdadm/initrdstart_msg_errblock:
* mdadm/mail_to: root
  mdadm/initrdstart_msg_errmd:
* mdadm/initrdstart: all
  mdadm/initrdstart_msg_errconf:
  mdadm/initrdstart_notinconf: false
* mdadm/start_daemon: true

More information about the pkg-mdadm-devel mailing list