Bug#791869: lvm2: updating src:lvm2 from 2.02.111-2.2 to 2.02.122-1 breaks booting, mounting LVs other than / fails

Stefan Lippers-Hollmann s.l-h at gmx.de
Sun Jul 19 23:16:12 UTC 2015


On 2015-07-19, Bastian Blank wrote:
> On Thu, Jul 09, 2015 at 05:16:57AM +0200, Stefan Lippers-Hollmann wrote:
> > Upgrading src:lvm2 from 2.02.111-2.2 to 2.02.122-1 breaks booting due to
> > a new systemd unit dependency failures regarding lvmetad when mounting 
> > non-rootfs logical volumes. Jumping to the emergency shell and invoking
> > "vgchange -ay" and "mount -a" allows booting to finish.
> Please provide all information from the system regarding the storage.
> This includes:
> - /etc/fstab

already provided in the original submission:

# cat /etc/fstab
# /etc/fstab: filesystem table.
# filesystem                    mountpoint                      type    options                dump  pass
/dev/vg-redstone/debian64       /                               ext4    defaults,noatime,barrier=0                              1  1
LABEL=UEFI                      /boot/efi                       vfat    auto,user,exec,nodev,nosuid,noatime                     1  2

/dev/vg-redstone/swap           none                            swap    sw                     0  0

/dev/vg-redstone/var            /var                            ext4    auto,user,exec,dev,noatime,barrier=0                    1  2
/var/tmp                        /tmp                            none    bind                   0  0
/dev/vg-redstone/home           /home                           ext4    auto,user,exec,nodev,nosuid,noatime,barrier=0           1  2
/dev/vg-redstone/storage        /srv/storage                    ext4    auto,user,noexec,nodev,nosuid,noatime,barrier=0         1  2

LABEL=seagate                   /srv/seagate                    ext4    auto,user,noexec,nodev,nosuid,noatime                   1  2

> - /etc/lvm/lvm.conf

/etc/lvm/lvm.conf is unchanged from the package default of lvm2 
2.02.122-2, but I've attached it (gzipped) nevertheless.

$ md5sum -b /etc/lvm/lvm.conf 
de7411c6a935b065dd7dcde6208c364f */etc/lvm/lvm.conf

> - pvs, vgs, lvs

# pvs
  PV         VG          Fmt  Attr PSize   PFree  
  /dev/sdb2  vg-redstone lvm2 a--  800,00g 446,00g

# vgs
  VG          #PV #LV #SN Attr   VSize   VFree  
  vg-redstone   1   5   0 wz--n- 800,00g 446,00g

# lvs
  LV       VG          Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  debian64 vg-redstone -wi-ao----  10,00g                                                    
  home     vg-redstone -wi-ao----  30,00g                                                    
  storage  vg-redstone -wi-ao---- 300,00g                                                    
  swap     vg-redstone -wi-ao----   4,00g                                                    
  var      vg-redstone -wi-ao----  10,00g

> - lsblk

# lsblk
sda                         8:0    0   2,7T  0 disk 
├─sda1                      8:1    0   299M  0 part 
└─sda2                      8:2    0   2,7T  0 part /srv/seagate
sdb                         8:16   0 931,5G  0 disk 
├─sdb1                      8:17   0   300M  0 part /boot/efi
└─sdb2                      8:18   0   800G  0 part 
  ├─vg--redstone-debian64 254:0    0    10G  0 lvm  /
  ├─vg--redstone-var      254:1    0    10G  0 lvm  /var
  ├─vg--redstone-home     254:2    0    30G  0 lvm  /home
  ├─vg--redstone-swap     254:3    0     4G  0 lvm  [SWAP]
  └─vg--redstone-storage  254:4    0   300G  0 lvm  /srv/storage

> - systemctl status in broken state

Taken, and stored in a temporary file, from within the emergency shell.

$ zcat systemctl-status.log.gz
● redstone
    State: maintenance
     Jobs: 0 queued
   Failed: 1 units
    Since: Mo 2015-07-20 00:04:38 CEST; 2min 18s ago
   CGroup: /
           ├─1 /sbin/init
             │ └─200 /sbin/lvmetad -f
             │ ├─573 /bin/sh -c /sbin/sulogin; /bin/systemctl --job-mode=fail --no-block default
             │ ├─576 bash
             │ └─588 systemctl status
             │ └─199 /lib/systemd/systemd-journald
             │ └─369 /lib/systemd/systemd-networkd
               └─203 /lib/systemd/systemd-udevd

> > Kernel: Linux 4.1.0-1.slh.3-aptosid-amd64 (SMP w/2 CPU cores; PREEMPT)
> Ah, this is no Debian system at all.

It is a Debian system, I'm just usually running the kernel[1] I'm 
developing and working on - but even though I picked the "wrong 
one" when submitting the bug, the problem can be reproduced easily 
using Debian's linux-image-4.0.0-2-amd64:

All logs above have been gathered using:

$ dpkg -l | grep -e linux-image-amd64 -e linux-image-4.0.0-2-amd64 -e 2.02.122-2 -e 2:1.02.99-2
ii  dmeventd                                      2:1.02.99-2                         amd64        Linux Kernel Device Mapper event daemon
ii  dmsetup                                       2:1.02.99-2                         amd64        Linux Kernel Device Mapper userspace library
ii  libdevmapper-event1.02.1:amd64                2:1.02.99-2                         amd64        Linux Kernel Device Mapper event support library
ii  libdevmapper1.02.1:amd64                      2:1.02.99-2                         amd64        Linux Kernel Device Mapper userspace library
ii  liblvm2app2.2:amd64                           2.02.122-2                          amd64        LVM2 application library
ii  liblvm2cmd2.02:amd64                          2.02.122-2                          amd64        LVM2 command library
ii  linux-image-4.0.0-2-amd64                     4.0.8-1                             amd64        Linux 4.0 for 64-bit PCs
ii  linux-image-amd64                             4.0+65                              amd64        Linux for 64-bit PCs (meta-package)
ii  lvm2                                          2.02.122-2                          amd64        Linux Logical Volume Manager

$ cat /proc/version 
Linux version 4.0.0-2-amd64 (debian-kernel at lists.debian.org) (gcc version 4.9.3 (Debian 4.9.3-1) ) #1 SMP Debian 4.0.8-1 (2015-07-11)

$ cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-4.0.0-2-amd64 root=/dev/mapper/vg--redstone-debian64 ro quiet systemd.show_status=1

This particular system is running in UEFI mode, but I can reproduce it 
just as well on other (older) systems in BIOS mode.

Starting with lvm2 2.02.122-2, it suddenly boots fine ~40% of the time,
while it fails booting the during the other attempts (with the logs 
above). Interesting enough, systems using a SSD for the system 
mountpoints usually succeed booting most of the time, while systems 
with only spinning HDDs seem to be mouch more likely to be affected.

Using src:lvm2 2.02.122-1 it reliably failed all the time, while that
never happened with src:lvm2 <= 2.02.111-2.2.

I'm affected by this problem on more than 10, fairly different, systems.

After logging into the emergency shell, I can activate the volume group
manually using "vgchange -ay", mount the missing logical volumes with
"mount -a" and exit the emergency shell to continue the boot process.

	Stefan Lippers-Hollmann

[1]	Vcs-git: git://github.com/fullstory/linux-aptosid.git
	Vcs-Browser: https://github.com/fullstory/linux-aptosid
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lvm.conf.gz
Type: application/gzip
Size: 19667 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-lvm-maintainers/attachments/20150720/1acda7a6/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: Digitale Signatur von OpenPGP
URL: <http://lists.alioth.debian.org/pipermail/pkg-lvm-maintainers/attachments/20150720/1acda7a6/attachment-0001.sig>

More information about the pkg-lvm-maintainers mailing list