[Pkg-iscsi-maintainers] Bug#850211: open-iscsi: Boot with systemd hangs for iSCSI+LUKS+LVM volume

[Christian Seiler]

Control: severity -1 important
Control: clone -1 -2
Control: reassign -2 systemd 230-7
Control: affects -1 cryptsetup
Control: affects -2 open-iscsi cryptsetup
Control: retitle -1 open-iscsi: No support for disabling LUKS volumes on shutdown
Control: retitle -2 systemd: cryptsetup-generator does not support LUKS on network devices
Control: tags -1 + confirmed stretch sid
Control: tags -2 + upstream

First of all, sorry for the late reply, but I was ill for a while and
then I was quite busy with catching up with other stuff.

The immediate bug you're experiencing is a problem with systemd, so I
am cloning the bug report and reassigning it to the systemd package.
However, in reviewing your setup I've also noticed that there is a
separate problem for your kind of setup in open-iscsi.

I'm including the main part of your bug report for the systemd
maintainers' benefit.

> This is how I setup my volume:
> 
>     # The iSCSI part works fine.
>     iscsiadm … --login
> 
>     # Device shows up as /dev/sdb; I create a /dev/sdb1 partition using
>     # fdisk, of type 8e.
> 
>     # Create encrypted LUKS volume on /dev/sdb1, open and map as
>     # /dev/mapper/sdb1_crypt.
>     cryptsetup luksFormat /dev/sdb1
>     cryptsetup luksOpen /dev/sdb1 sdb1_crypt \
>         --key-file /root/blackbird-ullu
> 
>     # Set up LVM PV, VG, and LV mapped to /dev/mapper/blackbird-ullu,
>     # with an ext4 filesystem on top.
>     pvcreate /dev/mapper/sdb1_crypt
>     vgcreate blackbird /dev/mapper/sdb1_crypt
>     lvcreate -n ullu -l 100%VG blackbird
>     mkfs.ext4 /dev/mapper/blackbird-ullu
> 
>     mount /dev/mapper/blackbird-ullu /media/nas
> 
> I have an entry in /etc/crypttab like this:
> 
>     sdb1_crypt UUID=ae6b9263-d63c-4515-b7ce-51e5cc4caa9f /root/blackbird-ullu luks
> 
> And an entry in /etc/fstab like this (I've tried various variants here,
> see below):
> 
>     /dev/mapper/blackbird-ullu /media/nas ext4 defaults,nofail,_netdev 0 6
> 
> There are three devices involved:
> 
>     - /dev/disk/by-uuid/<x>: is the iSCSI target (/dev/sdb)
>     - /dev/mapper/sdb1_crypt: result of cryptsetup luksOpen /dev/sdb1 
>     - /dev/mapper/blackbird-ullu: LV built on top of sdb1_crypt
> 
> Now I suffer from the 90s wait on startup (before network-online), where
> systemd waits for the dev-mapper-blackbird\x2dullu.device to become
> available, along with dev-disk-by\x2duuid-<xxx>.device and
> dev-mapper-sdb1_crypt.device.

The problem here is that the cryptsetup-generator in systemd only
supports local block devices, and the dependencies it generates
assume that everything orders indirectly before the local
filesystems.

This causes systemd to wait for the block device underlying the
LUKS encryption _before_ open-iscsi is even started.

This can clearly also be seen by the ordering cycle that's
generated if you manually force the dependencies:

> open-iscsi.service: Found ordering cycle on open-iscsi.service/start
> open-iscsi.service: Found dependency on network-online.target/start
> open-iscsi.service: Found dependency on networking.service/start
> open-iscsi.service: Found dependency on local-fs.target/start
> open-iscsi.service: Found dependency on lvm2-activation.service/start
> open-iscsi.service: Found dependency on cryptsetup.target/start
> open-iscsi.service: Found dependency on systemd-cryptsetup at sdb1_crypt.service/start
> open-iscsi.service: Found dependency on dev-disk-by\x2duuid-ae6b9263\x2dd63c\x2d4515\x2db7ce\x2d51e5cc4caa9f.device/start
> open-iscsi.service: Found dependency on open-iscsi.service/start

Here you see that systemd-cryptsetup at sdb1_crypt.service depends
on open-iscsi.service (because of your manual dependency), but that
cryptsetup.target depends on systemd-cryptsetup at sdb1_crypt.service,
local-fs.target depends on cryptsetup.target and open-iscsi.service
(correctly) depends on local-fs.target.

Your trick in lowering the device timeout to 1s instead of the explicit
ordering makes this "work" because it causes systemd to immediately
fail systemd-cryptsetup at sdb1_crypt.service immediately at early boot
(before open-iscsi.service is started), and systemd will then start it
again once the device actually appears after open-iscsi logs in to the
target.

Ideally one would want to have an option like _netdev also for
/etc/crypttab, so that it's clear that this specific block device
will only appear later. Unfortunately that doesn't exist yet. [1] So
your workaround in explicitly lowering the device timeout (you can
just do that for
dev-disk-by\x2duuid-ae6b9263\x2dd63c\x2d4515\x2db7ce\x2d51e5cc4caa9f.device
btw., you don't need to set it for other devices) is the only thing
that will make this "work" currently though.





As for the open-iscsi bug I also noticed: on shutdown open-iscsi
tries to tear down all mounts on iSCSI, including layers such as LVM
or multipath, but it doesn't detect LUKS yet, so data may not be
written to disk properly once the iSCSI logout happens.

I plan on doing an upload of open-iscsi today (to get some other
stuff into Stretch), which will include a fix for the shutdown part
(I just need to test it first). However it won't be possible to fix
the shutdown issue in Jessie, because the code would be too invasive
(the shutdown code was reworked for Stretch and is much better now),
so you'll have to use the backport of open-iscsi for this (once that
hits jessie-backports, which will take a while, because of the time
it takes for packages to migrate to Stretch at the moment).


I'll also try to work on a patch for the systemd issue in the next
couple of days.



Regards,
Christian

[1] You can see that the string _netdev clearly appears in
http://sources.debian.net/src/systemd/232-14/src/fstab-generator/fstab-generator.c/
which parses /etc/fstab for systemd, but doesn't appear in
http://sources.debian.net/src/systemd/232-14/src/cryptsetup/cryptsetup-generator.c/
which parses /etc/crypttab for systemd.