Bug#855871: non-metadata RAID arrays are limited to 27 component devices

Thu Feb 23 14:23:21 UTC 2017

It appears that the 27-component-device limit is specific to
non-metadata arrays ("mdadm --build"). More research:

When the RAID assembly fails --

# mdadm --build /dev/md/md-test --level=linear --raid-devices=28 /dev/loop{0..27}
mdadm: ADD_NEW_DISK failed for /dev/loop27: Device or resource busy

-- this shows up in the kernel log:

    kernel: [126679.205703] md: nonpersistent superblock ...
    kernel: [126679.205718] md: md125: array is limited to 27 devices
    kernel: [126679.205722] md: export_rdev(loop27)
    kernel: [126679.221644] md: md125 stopped.

The problem seems to be some kind of kernel limit. A Google search finds
this book:

https://books.google.ca/books?id=EbWbAgAAQBAJ&pg=PT75&lpg=PT75

quote: "MD_SB_DISKS indicates that each software array is limited to 27
        member disks."

This appears to be a constant relating to the MD-RAID v0.90 superblock
format:

http://lxr.free-electrons.com/ident?i=MD_SB_DISKS

some discussion:

    * The version-0.90 Superblock Format *

    Though it used to be the default format of raid superblock during
    array creation on most distributions until 2009, the older
    version-0.90 superblock format has several limitations that limit
    its applicability for use on large arrays or arrays with many
    component devices.

    The version-0.90 superblock limits the number of component devices
    within an array to 28, and limits each component device to a maximum
    size of 2TB on kernel version <3.1 and 4TB on kernel version >=3.1.

    ...

    * The version-1 Superblock Format *

    The newer and well-supported version-1 superblock format is
    more-expansion friendly than the previous format. It is the default
    as of v3.1.1. More specifically, --metadata=1.2 is used as of
    v3.1.2.

    The version-1 superblock is capable of supporting arrays with 384+
    component devices, and supports arrays with 64-bit sector lengths.

https://raid.wiki.kernel.org/index.php/RAID_superblock_formats

When the RAID assembly succeeds --

# mdadm --build /dev/md/md-test --level=linear --raid-devices=27 /dev/loop{0..26}
mdadm: array /dev/md/md-test built and started.
# 
# mdadm --detail /dev/md/md-test
/dev/md/md-test:
        Version : 
  Creation Time : Thu Feb 23 02:50:20 2017
     Raid Level : linear
     Array Size : 1769472 (1728.00 MiB 1811.94 MB)
   Raid Devices : 27
  Total Devices : 27

          State : clean 
 Active Devices : 27
Working Devices : 27
 Failed Devices : 0
  Spare Devices : 0

       Rounding : 64K

    Number   Major   Minor   RaidDevice State
       0       7        0        0      active sync   /dev/loop0
       1       7        1        1      active sync   /dev/loop1
       2       7        2        2      active sync   /dev/loop2
       3       7        3        3      active sync   /dev/loop3
       ...

-- notice that "Version:", the superblock type, is left unspecified.

This was a non-metadata array ("mdadm --build"). An internal-metadata
array ("mdadm --create") allows the use of more than 27 component block
devices, with a v1.2 superblock:

# mdadm --create /dev/md/md-test --level=linear --raid-devices=32 /dev/loop{0..31}
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/md-test started.
# 
# mdadm --detail /dev/md/md-test
/dev/md/md-test:
        Version : 1.2
  Creation Time : Thu Feb 23 03:12:21 2017
     Raid Level : linear
     Array Size : 2094592 (2045.50 MiB 2144.86 MB)
   Raid Devices : 32
  Total Devices : 32
    Persistence : Superblock is persistent

    Update Time : Thu Feb 23 03:12:21 2017
          State : clean 
 Active Devices : 32
Working Devices : 32
 Failed Devices : 0
  Spare Devices : 0

       Rounding : 0K

           Name : quadlie:md-test  (local to host quadlie)
           UUID : 79a9278b:125af417:91b0c7ab:75267ede
         Events : 0

    Number   Major   Minor   RaidDevice State
       0       7        0        0      active sync   /dev/loop0
       1       7        1        1      active sync   /dev/loop1
       2       7        2        2      active sync   /dev/loop2
       3       7        3        3      active sync   /dev/loop3
       ...

mdadm refuses to attempt creating the same array with a v0.90
superblock:

# mdadm --create /dev/md/md-test --level=linear --metadata=0 --raid-devices=32 /dev/loop{0..31}
mdadm: 0.90 metadata supports at most 27 devices per array

This error message is, however, a lot more informative than
"/dev/loop27: Device or resource busy", the initial one with a
non-metadata array.

mdadm won't allow the v1.2 superblock type with a non-metadata array:

# mdadm --build /dev/md/md-test --level=linear --metadata=1 --raid-devices=32 /dev/loop{0..31}
mdadm: :option --metadata not valid in build mode

-- so apparently, we're out of luck; it can't be done.

We may infer that when assembling a non-metadata array ("mdadm
--build"), the in-kernel superblock defaults to the v0.90 type, which
imposes the 27-component-device limit. Unfortunately, mdadm doesn't
allow you to override this default, and tell the kernel to use the v1.2
superblock. From the mdadm manpage:

    -e, --metadata=

    Declare the style of RAID metadata (superblock) to be used. The
    default is 1.2 for --create, and to guess for other operations.

Choosing the v0.90 superblock for "mdadm --build" is wrong. There
doesn't seem to be any reason why the in-kernel superblock in this case
shouldn't be the v1.2 type, which would eliminate this problem, and
other limitations of the v0.90 superblock.

This should be fixed. However, it seems to be an upstream (possibly
kernel), rather than specifically Debian, problem.

-- Ian Bruce