Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart

Alban Browaeys prahal at yahoo.com
Fri Mar 31 04:05:47 UTC 2017


Package: multipath-tools
Version: 0.6.4-5.1
Severity: normal

Dear Maintainer,
multipathd does not respond to commands : list paths or list maps
returns "timed out".

The mutlipathd daemon when triggered by udev locks up in uev_add_path
per this locks is already helds by its caller uev_update_path.

Here in thread 5 uev_update_path and uev_add_path chain
 (note that in thread 4 checkerloop also waits for this lock).

(gdb) t a a bt

Thread 6 (Thread 0x7f922663c700 (LWP 545)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9228bd3602 in ?? () from /usr/lib/x86_64-linux-gnu/liburcu.so.4
#2  0x00007f92289ba424 in start_thread (arg=0x7f922663c700) at pthread_create.c:333
#3  0x00007f922825c9bf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

Thread 5 (Thread 0x7f9229734700 (LWP 543)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f92289bcb85 in __GI___pthread_mutex_lock (mutex=0x556fefc43080) at ../nptl/pthread_mutex_lock.c:80
#2  0x0000556fedcbe42d in lock (a=0x556fefc43080) at ../libmultipath/lock.h:12
#3  uev_add_path (vecs=0x556fefc43080, uev=<optimized out>, uev=<optimized out>) at main.c:627
#4  0x0000556fedcbe9c9 in uev_update_path (uev=0x7f9220001510, vecs=0x556fefc43080) at main.c:998
#5  0x0000556fedcbecdb in uev_trigger (uev=0x7f9220001510, trigger_data=0x556fefc43080) at main.c:1146
#6  0x00007f92292091b2 in service_uevq (tmpq=tmpq at entry=0x7f9229733b10) at uevent.c:89
#7  0x00007f9229209280 in uevent_dispatch (uev_trigger=<optimized out>, trigger_data=<optimized out>) at uevent.c:145
#8  0x0000556fedcbc2cc in uevqloop (ap=0x556fefc43080) at main.c:1177
#9  0x00007f92289ba424 in start_thread (arg=0x7f9229734700) at pthread_create.c:333
#10 0x00007f922825c9bf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

Thread 4 (Thread 0x7f9229745700 (LWP 542)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f92289bcb85 in __GI___pthread_mutex_lock (mutex=0x556fefc43080) at ../nptl/pthread_mutex_lock.c:80
#2  0x0000556fedcbfb45 in lock (a=0x556fefc43080) at ../libmultipath/lock.h:12
#3  checkerloop (ap=0x556fefc43080) at main.c:1827
#4  0x00007f92289ba424 in start_thread (arg=0x7f9229745700) at pthread_create.c:333
#5  0x00007f922825c9bf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

Thread 3 (Thread 0x7f9229810700 (LWP 541)):
#0  0x00007f9228253611 in __GI_ppoll (fds=0x7f92180021e0, nfds=nfds at entry=1, timeout=<optimized out>, timeout at entry=0x556fedecc020 <sleep_time>, sigmask=sigmask at entry=0x7f922980fa60) at ../sysdeps/unix/sysv/linux/ppoll.c:39
#1  0x0000556fedcc13ba in ppoll (__ss=0x7f922980fa60, __timeout=0x556fedecc020 <sleep_time>, __nfds=1, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:77
#2  uxsock_listen (uxsock_trigger=0x556fedcbb520 <uxsock_trigger>, trigger_data=0x556fefc43080) at uxlsnr.c:204
#3  0x0000556fedcbbd5a in uxlsnrloop (ap=0x556fefc43080) at main.c:1239
#4  0x00007f92289ba424 in start_thread (arg=0x7f9229810700) at pthread_create.c:333
#5  0x00007f922825c9bf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

Thread 2 (Thread 0x7f9229851700 (LWP 540)):
#0  0x00007f922825354d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f9229209f3a in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f9229850a88) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  uevent_listen (udev=0x556fefbec040) at uevent.c:515
#3  0x0000556fedcbc235 in ueventloop (ap=0x556fefbec040) at main.c:1166
#4  0x00007f92289ba424 in start_thread (arg=0x7f9229851700) at pthread_create.c:333
#5  0x00007f922825c9bf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

Thread 1 (Thread 0x7f9229746f00 (LWP 537)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000556fedcc0aba in child (param=<optimized out>) at main.c:2407
#2  0x0000556fedcbb0df in main (argc=<optimized out>, argv=0x7fff81f9a0d8) at main.c:2664


Attached is a hash workaround that move uev_add_path call after the locks are released.

This also helps with shutdown/reboot which was waiting a long while for multipathd to terminate
before killing it under systemd.


Best regards
Alban







-- Package-specific info:
/etc/multipath.conf does not exist.


-- System Information:
Debian Release: 9.0
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'unstable'), (500, 'testing'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.10.0-trunk-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages multipath-tools depends on:
ii  init-system-helpers  1.47
ii  kpartx               0.6.4-5.1
ii  libaio1              0.3.110-3
ii  libc6                2.24-9
ii  libdevmapper1.02.1   2:1.02.137-2
ii  librados2            10.2.5-6
ii  libreadline7         7.0-2
ii  libsystemd0          232-22
ii  libudev1             232-22
ii  liburcu4             0.9.3-3
ii  lsb-base             9.20161125
ii  sg3-utils-udev       1.42-2
ii  udev                 232-22

multipath-tools recommends no packages.

Versions of packages multipath-tools suggests:
ii  multipath-tools-boot  0.6.4-5.1

-- no debconf information
-------------- next part --------------
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -994,9 +994,8 @@
 				pp->wwid_changed = 0;
 		}
 
-		if (pp->initialized == INIT_REQUESTED_UDEV)
-			retval = uev_add_path(uev, vecs);
-		else if (mpp && ro >= 0) {
+		if ((pp->initialized != INIT_REQUESTED_UDEV)
+		   && (mpp && ro >= 0)) {
 			condlog(2, "%s: update path write_protect to '%d' (uevent)", uev->kernel, ro);
 
 			if (mpp->wait_for_udev)
@@ -1010,6 +1009,9 @@
 	}
 out:
 	lock_cleanup_pop(vecs->lock);
+	if (pp)
+		if (pp->initialized == INIT_REQUESTED_UDEV)
+			retval = uev_add_path(uev, vecs);
 	if (!pp)
 		condlog(0, "%s: spurious uevent, path not found", uev->kernel);
 


More information about the pkg-lvm-maintainers mailing list