Bug#787555: lvm2: Unmount of invalid (full) snapshot unmounts main device too

[Zdenek Kabelac]

On Tue, 02 Jun 2015 14:02:37 -0400 Matthew Gabeler-Lee <cheetah at fastcat.org> 
wrote:
> Package: lvm2
> Version: 2.02.111-2.2
> Severity: important
>
> Last night lvm2 tried to murder my server.
>
> OK, exaggerating a little bit, but it did something that seems not only
> wrong, but quite bad.
>
> The system was running a backup tool, using LVM snapshots to get coherent
> views of the filesystem.
>
> The snapshot for /var filled up.
>
> LVM decided to forcibly unmount the snapshot, which I guess is OK based on
> the information I've found on why this behavior was introduced, but it also
> forcibly unmounted the main device, resulting in the sudden and
> unceremonious loss of /var on the server, which had many unpleasant effects.
>
> Jun  2 01:32:32 cheetah kernel: [5957103.322426] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
> Jun  2 01:32:33 cheetah lvm[9693]: Unmounting invalid snapshot raid5-var--snap from /mnt/runbackup.8194.28259/var.
> Jun  2 01:32:33 cheetah lvm[9693]: Unmounting invalid snapshot raid5-var--snap from /var.
>
> It looks like something got quite confused as to what device was backing
> /var.


This doesn't look to me like lvm2 bug.
Unless  raid5-var/snap was not the volume mounted on /var ??

But lvm2 does umount based on major:minor pair - so wasn't /var incorrectly 
mounted from a snapshot instead of an /var origin device ?

The BZ then needs to show the table & mount state before and after.


Old snapshots are invalidated when they get full.

Such INVALID device cannot be used anymore - it's simply lost.

So whoever uses something there (.e.g. mounted fs) it is already doomed at 
this moment.

If users wants to continue to use origin & snapshots - I'd probably recommend 
to use thin-provisioning  - which is designed for such use case much better.

If user still want to use the old snapshot for this let him configure 
dmeventd monitoring of used space in snapshot - so there is some chance the 
system may resize COW space in case the used space there gets above configured 
threshold as the COW size is resized (when there is some free space in VG of 
course).

Regards

Zdenek