Bug#697585: lvm2: pvmove freezes the whole vg with running Xen VMs

Gael Le Mignot gael at pilotsystems.net
Mon Jan 7 09:24:12 UTC 2013


Package: lvm2
Version: 2.02.66-5
Severity: important

We use Xen and LVM2 with the following configuration : the DOM0 handles several 
LVM2 volume groups, and the virtual machines (DOMU) use the logical volumes for 
their paravirtualised disks.

We used pvmove to move data from one PV to another PV within a VG, while the
virtual machines where running.

pvmove froze at around 66% of completion, and afterwards, all access to the VG,
both from the virtual machines and the DOM0 froze. Any lvm command (pvs, lvs, 
...) would freeze too, and enter unkillable D state.

Here is a traceback we had on a frozen pvs :
Jan  4 12:12:24 buffy kernel: [584522.464110] INFO: task pvs:23197 blocked for more than 120 seconds.
Jan  4 12:12:24 buffy kernel: [584522.468010] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 12:12:24 buffy kernel: [584522.468010] pvs           D ffff88007dd5e350     0 23197  23196 0x00000000
Jan  4 12:12:24 buffy kernel: [584522.468010]  ffff88007d62a350 0000000000000286 ffff88007e230200 ffff88007cc3e2b0
Jan  4 12:12:24 buffy kernel: [584522.468010]  ffff88007ef0c208 0000000001d88023 000000000000f9e0 ffff88007ec53fd8
Jan  4 12:12:24 buffy kernel: [584522.468010]  0000000000015780 0000000000015780 ffff88007dd5e350 ffff88007dd5e648
Jan  4 12:12:24 buffy kernel: [584522.468010] Call Trace:
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff8106cbaa>] ? timekeeping_get_ns+0xe/0x2e
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff8130ce0a>] ? io_schedule+0x73/0xb7
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff811154a9>] ? __blockdev_direct_IO+0x910/0xa60
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff81113966>] ? blkdev_direct_IO+0x45/0x4a
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff81112c7b>] ? blkdev_get_blocks+0x0/0x8b
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff810b680b>] ? generic_file_aio_read+0xf6/0x536
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffffa01abb05>] ? dm_blk_open+0x10/0x5f [dm_mod]
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff81114021>] ? __blkdev_get+0x29c/0x342
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff811140ce>] ? blkdev_open+0x0/0x96
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff81114135>] ? blkdev_open+0x67/0x96
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff810f0101>] ? do_sync_read+0xce/0x113
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff810f3453>] ? cp_new_stat+0xe9/0xfc
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff81112f16>] ? block_ioctl+0x38/0x3c
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff810fbf62>] ? vfs_ioctl+0x21/0x6c
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff810fc4b0>] ? do_vfs_ioctl+0x48d/0x4cb
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff810f0b24>] ? vfs_read+0xa6/0xff
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff810f0c39>] ? sys_read+0x45/0x6e
Jan  4 12:12:24 buffy kernel: [584522.468010]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b

The virtual machines on other VG were still functionnal, only those using the
VG running pvmove were frozen (those using the source and the target PV were
both affected).

pvmove --abort froze too.

We had to do a physical reset on the whole server to recover from the situation.
No data was lost, but it created some significant downtime to our services.

The version of Xen we use is 4.0.1-5.5

-- System Information:
Debian Release: 6.0.6
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-xen-amd64 (SMP w/1 CPU core)
Locale: LANG=C, LC_CTYPE=fr_FR at euro (charmap=ISO-8859-15)
Shell: /bin/sh linked to /bin/dash

Versions of packages lvm2 depends on:
ii  dmsetup                 2:1.02.48-5      The Linux Kernel Device Mapper use
ii  libc6                   2.11.3-4         Embedded GNU C Library: Shared lib
ii  libdevmapper1.02.1      2:1.02.48-5      The Linux Kernel Device Mapper use
ii  libreadline5            5.2-7            GNU readline and history libraries
ii  libudev0                164-3            libudev shared library
ii  lsb-base                3.2-23.2squeeze1 Linux Standard Base 3.2 init scrip

lvm2 recommends no packages.

lvm2 suggests no packages.

-- no debconf information



More information about the pkg-lvm-maintainers mailing list