Bug#759063: mdadm RAID5 array intermittently stalls during a write operation

[Tim Boundy]

After some additional testing, my issue appears related to
http://thr3ads.net/ext3-users/2013/04/2300336-LONG-Delay-when-writing-to-ext4-LVM-after-boot

With clean restart of the array, followed by running dumpe2fs (presumably
to prefetch bitmaps), the initial write delay is resolved. However I'm
still confused as to why simply unmounting the filesystem, then flushing
caches (echo 3 > /proc/sys/vm/drop_caches as well as changing the md
stripe_cache_size) doesn't reproduce the issue. A full stop and reassemble
cycle via mdadm is required to reproduce the issue.

One other point of note is that after a clean restart of the array and then
running dumpe2fs, the only drive in the array that was read was /dev/sdd. I
find it strange that all the relevant filesystem metadata would sit
entirely on one drive in a multi drive array over a large range of LBAs.


On 24 August 2014 15:31, Tim Boundy <gigaplex at gmail.com> wrote:

> Thanks, I'll give that a try and report back later.
>
>
> On 24 August 2014 15:21, NeilBrown <neilb at suse.de> wrote:
>
>> On Sun, 24 Aug 2014 15:04:34 +1000 Tim Boundy <gigaplex at gmail.com> wrote:
>>
>> > I've tried unmounting the file system, dropping caches via
>> > /proc/sys/vm/drop_caches, then remounting the file system. I can't
>> > reproduce the stalling issue on demand this way, I have to stop then
>> > restart the array via mdadm to do so. Is there some caching of the mdadm
>> > array itself going on that wasn't dropped when I unmounted the
>> filesystem
>> > and requested a cache drop? If not, then I'm not entirely convinced
>> it's a
>> > file system issue, but I'll ask for their input too.
>> >
>>
>> There is some, but not much.
>>
>> If you write a small number to /sys/block/mdWATEVER/md/stripe_cache_size
>>
>> it will drop what little caching md does.  Then write the original number
>> back in (256 by default).
>>
>> Another thing to try is to use "blktrace" on both the md device and the
>> component device, then "blkparse" to look at the  result and see exactly
>> what blocks are being read, and maybe who is reading them.
>>
>> NeilBrown
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-mdadm-devel/attachments/20140825/3f74dbd6/attachment.html>