Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart
Alban Browaeys
prahal at yahoo.com
Fri Mar 31 04:05:47 UTC 2017
Package: multipath-tools
Version: 0.6.4-5.1
Severity: normal
Dear Maintainer,
multipathd does not respond to commands : list paths or list maps
returns "timed out".
The mutlipathd daemon when triggered by udev locks up in uev_add_path
per this locks is already helds by its caller uev_update_path.
Here in thread 5 uev_update_path and uev_add_path chain
(note that in thread 4 checkerloop also waits for this lock).
(gdb) t a a bt
Thread 6 (Thread 0x7f922663c700 (LWP 545)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f9228bd3602 in ?? () from /usr/lib/x86_64-linux-gnu/liburcu.so.4
#2 0x00007f92289ba424 in start_thread (arg=0x7f922663c700) at pthread_create.c:333
#3 0x00007f922825c9bf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
Thread 5 (Thread 0x7f9229734700 (LWP 543)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f92289bcb85 in __GI___pthread_mutex_lock (mutex=0x556fefc43080) at ../nptl/pthread_mutex_lock.c:80
#2 0x0000556fedcbe42d in lock (a=0x556fefc43080) at ../libmultipath/lock.h:12
#3 uev_add_path (vecs=0x556fefc43080, uev=<optimized out>, uev=<optimized out>) at main.c:627
#4 0x0000556fedcbe9c9 in uev_update_path (uev=0x7f9220001510, vecs=0x556fefc43080) at main.c:998
#5 0x0000556fedcbecdb in uev_trigger (uev=0x7f9220001510, trigger_data=0x556fefc43080) at main.c:1146
#6 0x00007f92292091b2 in service_uevq (tmpq=tmpq at entry=0x7f9229733b10) at uevent.c:89
#7 0x00007f9229209280 in uevent_dispatch (uev_trigger=<optimized out>, trigger_data=<optimized out>) at uevent.c:145
#8 0x0000556fedcbc2cc in uevqloop (ap=0x556fefc43080) at main.c:1177
#9 0x00007f92289ba424 in start_thread (arg=0x7f9229734700) at pthread_create.c:333
#10 0x00007f922825c9bf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
Thread 4 (Thread 0x7f9229745700 (LWP 542)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f92289bcb85 in __GI___pthread_mutex_lock (mutex=0x556fefc43080) at ../nptl/pthread_mutex_lock.c:80
#2 0x0000556fedcbfb45 in lock (a=0x556fefc43080) at ../libmultipath/lock.h:12
#3 checkerloop (ap=0x556fefc43080) at main.c:1827
#4 0x00007f92289ba424 in start_thread (arg=0x7f9229745700) at pthread_create.c:333
#5 0x00007f922825c9bf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
Thread 3 (Thread 0x7f9229810700 (LWP 541)):
#0 0x00007f9228253611 in __GI_ppoll (fds=0x7f92180021e0, nfds=nfds at entry=1, timeout=<optimized out>, timeout at entry=0x556fedecc020 <sleep_time>, sigmask=sigmask at entry=0x7f922980fa60) at ../sysdeps/unix/sysv/linux/ppoll.c:39
#1 0x0000556fedcc13ba in ppoll (__ss=0x7f922980fa60, __timeout=0x556fedecc020 <sleep_time>, __nfds=1, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:77
#2 uxsock_listen (uxsock_trigger=0x556fedcbb520 <uxsock_trigger>, trigger_data=0x556fefc43080) at uxlsnr.c:204
#3 0x0000556fedcbbd5a in uxlsnrloop (ap=0x556fefc43080) at main.c:1239
#4 0x00007f92289ba424 in start_thread (arg=0x7f9229810700) at pthread_create.c:333
#5 0x00007f922825c9bf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
Thread 2 (Thread 0x7f9229851700 (LWP 540)):
#0 0x00007f922825354d in poll () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f9229209f3a in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f9229850a88) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2 uevent_listen (udev=0x556fefbec040) at uevent.c:515
#3 0x0000556fedcbc235 in ueventloop (ap=0x556fefbec040) at main.c:1166
#4 0x00007f92289ba424 in start_thread (arg=0x7f9229851700) at pthread_create.c:333
#5 0x00007f922825c9bf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
Thread 1 (Thread 0x7f9229746f00 (LWP 537)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000556fedcc0aba in child (param=<optimized out>) at main.c:2407
#2 0x0000556fedcbb0df in main (argc=<optimized out>, argv=0x7fff81f9a0d8) at main.c:2664
Attached is a hash workaround that move uev_add_path call after the locks are released.
This also helps with shutdown/reboot which was waiting a long while for multipathd to terminate
before killing it under systemd.
Best regards
Alban
-- Package-specific info:
/etc/multipath.conf does not exist.
-- System Information:
Debian Release: 9.0
APT prefers unstable-debug
APT policy: (500, 'unstable-debug'), (500, 'unstable'), (500, 'testing'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.10.0-trunk-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages multipath-tools depends on:
ii init-system-helpers 1.47
ii kpartx 0.6.4-5.1
ii libaio1 0.3.110-3
ii libc6 2.24-9
ii libdevmapper1.02.1 2:1.02.137-2
ii librados2 10.2.5-6
ii libreadline7 7.0-2
ii libsystemd0 232-22
ii libudev1 232-22
ii liburcu4 0.9.3-3
ii lsb-base 9.20161125
ii sg3-utils-udev 1.42-2
ii udev 232-22
multipath-tools recommends no packages.
Versions of packages multipath-tools suggests:
ii multipath-tools-boot 0.6.4-5.1
-- no debconf information
-------------- next part --------------
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -994,9 +994,8 @@
pp->wwid_changed = 0;
}
- if (pp->initialized == INIT_REQUESTED_UDEV)
- retval = uev_add_path(uev, vecs);
- else if (mpp && ro >= 0) {
+ if ((pp->initialized != INIT_REQUESTED_UDEV)
+ && (mpp && ro >= 0)) {
condlog(2, "%s: update path write_protect to '%d' (uevent)", uev->kernel, ro);
if (mpp->wait_for_udev)
@@ -1010,6 +1009,9 @@
}
out:
lock_cleanup_pop(vecs->lock);
+ if (pp)
+ if (pp->initialized == INIT_REQUESTED_UDEV)
+ retval = uev_add_path(uev, vecs);
if (!pp)
condlog(0, "%s: spurious uevent, path not found", uev->kernel);
More information about the pkg-lvm-maintainers
mailing list