Bug#791869: lvm2: updating src:lvm2 from 2.02.111-2.2 to 2.02.122-1 breaks booting, mounting LVs other than / fails

[Ben Caradoc-Davies]

Booting succeeds for me with / and /home on separate LVs in a single 
crypto-luks PV (see lsblk below) with lvm2 2.02.122-2 amd64. However, 
after updating to the latest lvm2, "pvscan", "pvs", "vgs", and "lvs" all 
hang indefinitely until I manually run "pvscan --cache". They worked 
fine with 2.02.111-2.2, probably because lvmetad was not enabled.

- lvm.conf is unmodified
- lvm2-monitor.service, avahi-daemon.service, and avahi-daemon.socket 
are masked
- Fully dist-upgraded sid/amd64

Nothing helpful in the logs (journalctl or /var/log).

Before "pvscan --cache":

output of "strace pvs" while hung:
[...]
write(3, "request=\"pv_list\"\ntoken =\"filter"..., 36) = 36
write(3, "\n##\n", 4)                   = 4
read(3, "response = \"token_mismatch\"\nexpe", 32) = 32
read(3, "cted = \"update in progress\"\nrece"..., 1024) = 147
[...]

output of "strace vgs" while hung ("lvs" is similar):
[...]
write(3, "request=\"vg_list\"\ntoken =\"filter"..., 36) = 36
write(3, "\n##\n", 4)                   = 4
read(3, "response = \"token_mismatch\"\nexpe", 32) = 32
read(3, "cted = \"update in progress\"\nrece"..., 1024) = 147
[...]

Output of "strace -e read=3 pvs while hung:
[...]
write(3, "request=\"pv_list\"\ntoken =\"filter"..., 36) = 36
write(3, "\n##\n", 4)                   = 4
read(3, "response = \"token_mismatch\"\nexpe", 32) = 32
  | 00000  72 65 73 70 6f 6e 73 65  20 3d 20 22 74 6f 6b 65  response = 
"toke |
  | 00010  6e 5f 6d 69 73 6d 61 74  63 68 22 0a 65 78 70 65 
n_mismatch".expe |
read(3, "cted = \"update in progress\"\nrece"..., 1024) = 147
  | 00000  63 74 65 64 20 3d 20 22  75 70 64 61 74 65 20 69  cted = 
"update i |
  | 00010  6e 20 70 72 6f 67 72 65  73 73 22 0a 72 65 63 65  n 
progress".rece |
  | 00020  69 76 65 64 20 3d 20 22  66 69 6c 74 65 72 3a 30  ived = 
"filter:0 |
  | 00030  22 0a 72 65 61 73 6f 6e  20 3d 20 22 6c 76 6d 65  ".reason = 
"lvme |
  | 00040  74 61 64 20 63 61 63 68  65 20 69 73 20 69 6e 76  tad cache 
is inv |
  | 00050  61 6c 69 64 20 64 75 65  20 74 6f 20 61 20 67 6c  alid due to 
a gl |
  | 00060  6f 62 61 6c 5f 66 69 6c  74 65 72 20 63 68 61 6e  obal_filter 
chan |
  | 00070  67 65 20 6f 72 20 64 75  65 20 74 6f 20 61 20 72  ge or due 
to a r |
  | 00080  75 6e 6e 69 6e 67 20 72  65 73 63 61 6e 22 0a 0a  unning 
rescan".. |
  | 00090  23 23 0a                                          ##. 
       |
[...]

For your convenience, the message is:

response = "token_mismatch".expected = "update in progress".received = 
"filter:0".reason = "lvmetad cache is invalid due to a global_filter 
change or due to a running rescan"..##.

Then "man lvmetad" led me to "pvscan --cache".

After "pvscan --cache":

# pvs
   PV                     VG   Fmt  Attr PSize  PFree
   /dev/mapper/sda2_crypt vg   lvm2 a--  55.62g    0
# vgs
   VG   #PV #LV #SN Attr   VSize  VFree
   vg     1   2   0 wz--n- 55.62g    0
# lvs
   LV   VG   Attr       LSize  Pool Origin Data%  Meta%  Move Log 
Cpy%Sync Convert
   home vg   -wi-ao---- 36.99g 

   root vg   -wi-ao---- 18.62g

Other system information (before "pvscan --cache"):

# systemctl status lvm2-lvmetad
● lvm2-lvmetad.service - LVM2 metadata daemon
    Loaded: loaded (/lib/systemd/system/lvm2-lvmetad.service; disabled; 
vendor preset: enabled)
    Active: active (running) since Mon 2015-07-20 09:27:27 NZST; 38min ago
      Docs: man:lvmetad(8)
  Main PID: 508 (lvmetad)
    CGroup: /system.slice/lvm2-lvmetad.service
            └─508 /sbin/lvmetad -f
[...]

fstab excerpt:
/dev/mapper/vg-root / ext4 noatime,errors=remount-ro 0 1
/dev/sda1 /boot ext4 noatime,errors=remount-ro 0 2
/dev/mapper/vg-home /home ext4 noatime,errors=remount-ro 0 2

# lsblk
NAME           MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda              8:0    0 55.9G  0 disk
├─sda1           8:1    0  285M  0 part  /boot
└─sda2           8:2    0 55.6G  0 part
   └─sda2_crypt 254:0    0 55.6G  0 crypt
     ├─vg-root  254:1    0 18.6G  0 lvm   /
     └─vg-home  254:2    0   37G  0 lvm   /home

packages:
dmeventd 2:1.02.99-2 amd64
dmsetup 2:1.02.99-2 amd64
libdevmapper-event1.02.1:amd64 2:1.02.99-2 amd64
libdevmapper1.02.1:amd64 2:1.02.99-2 amd64
liblvm2app2.2:amd64 2.02.122-2 amd64
liblvm2cmd2.02:amd64 2.02.122-2 amd64
lvm2 2.02.122-2 amd64

kernel:
Linux ripley 4.0.0-2-amd64 #1 SMP Debian 4.0.8-1 (2015-07-11) x86_64 
GNU/Linux

Kind regards,

-- 
Ben Caradoc-Davies <ben at transient.nz>
Director
Transient Software Limited <http://transient.nz/>
New Zealand