[Pkg-iscsi-maintainers] multipath not automounting iscsi devices listed in fstab
Christian Seiler
christian at iwakd.de
Mon Jan 26 14:35:47 UTC 2015
Am 26.01.2015 um 08:47 schrieb Ritesh Raj Sarraf:
> On 01/25/2015 09:43 PM, Christian Seiler wrote:
>> The same fix that was implemented for open-iscsi in principle also
>> applies for multipath-tools, i.e. make sure that for systemd systems
>> the unit is ordered before remote-fs-pre.target. I don't use
>> multipath-tools myself, but I'll be able to prepare a patch that fixes
>> this on a minimal level tomorrow, you'll just have to test it
>> yourself.
>
> Thanks Christian.
>
> I'll wait for your patch.
So I did some testing with trivial multipath (only one device as
backend), and I came upon the following issues AFTER I fixed this in the
same way as the open-iscsi package. These issues don't seem to be
related to systemd, but a general problem of the multipath package
(although I didn't test it with sysvinit, so I don't know for sure):
1. open-iscsi init script (which is still called even by the new
systemd service file) does udevadm settle to make sure all device
nodes from logging in to iSCSI have been created, because immediately
after that, it wants to activate LVMs configured on iSCSI.
* On its own, that's not a problem, so if you have bare iSCSI with
or without LVM on top, that works fine.
* But, if you have multipath started and configured, there's
/lib/udev/rules.d/60-multipath-rules with the following entry:
# Coalesce multipath devices before multipathd is running
# (initramfs, early boot)
ACTION=="add|change", SUBSYSTEM=="block",
RUN+="/sbin/multipath -v0 /dev/$name"
The problem here is that multipath -v0 /dev/$name doesn't
complete because multipathd is not started. The problem is that
this rule is not only triggered for the devices first available
at boot, but also for the devices that appear due to iSCSI,
which in this case are even configured. Unfortunately, since
multipathd is not running, this is a new deadlock here.
udev now has a default timeout of 30s, so boot hangs for that
time and after that I get a bunch of log messages about
timeouts.[1]
After that, the system boots fine, udevadm settle completes,
open-iscsi init script continues, and then multipathd is
started, which properly activates the devices, which can then
be mounted.
I don't see anything systemd specific in here, and while I
haven't tried it, I would suspect that the same thing occurs
also with sysvinit.
2. Also, really curious, at shutdown I have the following situation:
multipath-tools does not seem to dismantle (or however that is
called properly) multipath volumes. So now, I have the following
situation:
- due to proper ordering with my fix for the 90s systemd issue,
remote filesystems get unmounted by systemd first, so nothing
is mounted anymore that's on multipath
- /etc/init.d/multipath-tools stop is called
- multipathd exits
- but apparently, /dev/mapper/mp{1,2} (that's how I called my
test devices) still exist
- /etc/init.d/open-iscsi stop is called, that logs out of the
iSCSI session
- later at shutdown, something (I don't know exactly what, since
shutdown is parallel) causes the kernel to try to access all
block devices in the system, making it notice that it can't
really access the multipath devices anymore (which still exit!),
so it complains about it. See [2] for log messages related to
this.
So basically you have two issues:
- 30s delay on boot because udevadm settle (in open-iscsi) waits for
multipath -v0 but that won't complete until multipathd is started,
which won't happen until the open-iscsi script is done (which waits
for udevadm settle) -> timeout
- note that if I comment out the udev rule in question, the
system boots immediately (total boot time only a couple of
seconds, including iSCSI + multipath setup), but obviously
that can't be a complete solution, because you DO want to
pick up multipath devices that were started in early boot
- this appears to be related to or the same as
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=580972
- on shutdown, multipath device mapper devices are not removed and then
something tries to access them in late shutdown phase, when iSCSI is
already gone, which produces weird log messages, which in the default
configuration of Jessie are shown on the screen for a short time
before rebooting (might irritate some people)
- since file systems umount cleanly and open-iscsi does a 'sync'
before logging out of all sessions, I think this is *probably*
only cosmetic
Therefore, my question would be: do you see the same to issues on
sysvinit? If so, I would then attach my patch to fix the
boot/shutdown ordering stuff of multipath-tools just on systemd and then
this bug may be closed, whereas the other stuff is something that I
probably can't really comment on too much because I don't use multipath.
Christian
[1] First:
iscsid[890]: Connection1:0 to [target:
iqn.2003-01.org.linux-iscsi.tkmlx74.x8664:sn.9284f4d8cb0e, portal:
192.168.15.100,3260] through [iface: default] is operational now
exactly 30s later:
systemd-udevd[145]: worker [157]
/devices/platform/host2/session1/target2:0:0/2:0:0:3/block/sdd timeout;
kill it
systemd-udevd[145]: seq 1357
'/devices/platform/host2/session1/target2:0:0/2:0:0:3/block/sdd' killed
systemd-udevd[145]: worker [159]
/devices/platform/host2/session1/target2:0:0/2:0:0:4/block/sde timeout;
kill it
systemd-udevd[145]: seq 1358
'/devices/platform/host2/session1/target2:0:0/2:0:0:4/block/sde' killed
systemd-udevd[145]: worker [157] terminated by signal 9 (Killed)
systemd-udevd[145]: worker [159] terminated by signal 9 (Killed)
(sdd and sde are configured in multipath via their wwid)
[2] First you have the stopping of multipath stuff:
[ 40.593235] systemd[1]: About to execute: /etc/init.d/multipath-tools
stop
[ ... some stuff ...]
[ 40.610542] systemd[1]: Child 3043 (multipath-tools) died
(code=exited, status=0/SUCCESS)
[ ... some stuff ...]
[ 40.647841] systemd[1]: Received SIGCHLD from PID 1081 (multipathd).
[ 40.647867] systemd[1]: Child 1081 (multipathd) died (code=exited,
status=0/SUCCESS)
(so basically, it tells me that the init script was successful)
And later on you've got:
[ 41.628858] device-mapper: multipath: Failing path 8:48.
[ 41.628865] end_request: I/O error, dev dm-5, sector 204672
[ 41.629431] end_request: I/O error, dev dm-5, sector 204784
[ 41.629897] end_request: I/O error, dev dm-5, sector 0
[ 41.630057] end_request: I/O error, dev dm-5, sector 8
[ 41.630466] end_request: I/O error, dev dm-5, sector 0
[ 41.631080] device-mapper: multipath: Failing path 8:64.
[ 41.631084] end_request: I/O error, dev dm-6, sector 204672
[ 41.631525] end_request: I/O error, dev dm-6, sector 204784
[ 41.631843] end_request: I/O error, dev dm-6, sector 0
[ 41.632029] end_request: I/O error, dev dm-6, sector 8
[ 41.632760] end_request: I/O error, dev dm-6, sector 0
[ 41.667776] device-mapper: multipath: Failing path 8:64.
[ 41.667790] Buffer I/O error on device dm-6, logical block 25584
[ 41.668517] device-mapper: multipath: Failing path 8:48.
[ 41.668522] Buffer I/O error on device dm-5, logical block 25584
[ 41.668707] Buffer I/O error on device dm-6, logical block 25584
[ 41.669120] Buffer I/O error on device dm-5, logical block 25584
[ 41.670072] Buffer I/O error on device dm-6, logical block 0
[ 41.670202] Buffer I/O error on device dm-6, logical block 1
[ 41.670322] Buffer I/O error on device dm-6, logical block 2
[ 41.670441] Buffer I/O error on device dm-6, logical block 3
[ 41.671130] Buffer I/O error on device dm-5, logical block 0
[ 41.671258] Buffer I/O error on device dm-5, logical block 1
[ 41.703846] device-mapper: multipath: Failing path 8:64.
[ 41.704643] device-mapper: multipath: Failing path 8:48.
[ 41.740753] device-mapper: multipath: Failing path 8:64.
[ 41.741484] device-mapper: multipath: Failing path 8:48.
[ 41.757937] systemd-udevd[3010]: '/sbin/kpartx -u -p -part /dev/dm-5'
[3427] terminated by signal 15 (Terminated)
The last one ist kind of weird, since that comes from a udev rule in
60-kpartx.rules that AFAICT should be run only if a device appears.
More information about the Pkg-iscsi-maintainers
mailing list