Bug#814625: open-iscsi: broken sanboot setup

Sun Feb 14 10:51:07 UTC 2016

On Sat, 2016-02-13 at 20:33 +0100, Christian Seiler wrote:
> 
> > Sorry about that. My setup was all broken and I was trying multiple
> > things.
> 
> I have a non-multipath test setup of open-iscsi that works well
> and I even recently upgraded to unstable before I closed the
> RC bug I had open to prevent open-iscsi from migrating to testing
> to verify that everything works as expected.
> 

Yes. I too upgraded and realized that the culprit wasn't iscsi.

> > But I don't recollect seeing any debconf prompt during the
> > downgrade.
> 
> It's in your log. And it's a critical prompt, so unless you
> explicitly use something that dramatically suppresses debconf
> prompts it should be shown.
> 

Not sure. Maybe I missed. In trying to find the bug, I may have not
paid attention.

> > The issue is weird in that the diff between multipath-tools -2 and
> > -4
> > is almost zero. But I can't see much on what would break my setup.
> > :-(
> 
> Are you sure it's multipath-tools?

That's what I think because with no other changes, if I upgrade -2 to
-4, and then update the intird, and reboot; multipath breaks.

Same above steps without the multipath upgrade, works fine.

> 
> Do you have an image I could try with some instructions on what
> LIO configuration you are using? Maybe I could help debug your
> issue...
> 

Thanks. I've attached the debdiff in between the 2 version. I doubt if
any of those changes introduced the bug. 

> Also: what exactly broke? Your bug doesn't actually specify
> what's wrong, beyond the fact that update-initramfs complains
> about iscsistart.
> 

Let me explain....

root at debian-sanboot:~# multipath -ll
sanroot (36001405ead943c8222140268e019ba49) dm-0 LIO-ORG,IBLOCK
size=8.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 3:0:0:0 sdb 8:16 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 5:0:0:0 sdd 8:48 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 4:0:0:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 2:0:0:0 sda 8:0  active ready running

So, on a working setup, this is my multipath report. For my root LUN
(sanroot), there are 4 paths available. Each path is an individual SCSI
device. These devices share the same VID/PID, by basis of which, the
overall setup coalesces them into a logical device mapper multipath
device named /dev/mapper/sanroot

root at debian-sanboot:~# mount | grep "/ "
/dev/mapper/sanroot on / type ext4 (rw,relatime,errors=remount-
ro,stripe=8191,data=ordered)

Now look at the package version.

root at debian-sanboot:~# apt-cache policy multipath-tools
multipath-tools:
  Installed: 0.5.0+git1.656f8865-2
  Candidate: 0.5.0+git1.656f8865-4
  Version table:
     0.5.0+git1.656f8865-4 500
        500 http://ftp.debian.org/debian testing/main amd64 Packages
        500 http://ftp.debian.org/debian unstable/main amd64 Packages
 *** 0.5.0+git1.656f8865-2 100
        100 /var/lib/dpkg/status

After the upgrade to the -4 version (debdiff attached), the setup looks
like below.

root at debian-sanboot:~# apt install multipath-tools multipath-tools-boot 
kpartx
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be upgraded:
  kpartx multipath-tools multipath-tools-boot
3 upgraded, 0 newly installed, 0 to remove and 119 not upgraded.
Need to get 246 kB of archives.
After this operation, 14.3 kB disk space will be freed.
Get:1 http://ftp.debian.org/debian testing/main amd64 kpartx amd64 0.5.
0+git1.656f8865-4 [31.8 kB]
Get:2 http://ftp.debian.org/debian testing/main amd64 multipath-tools a
md64 0.5.0+git1.656f8865-4 [197 kB]
Get:3 http://ftp.debian.org/debian testing/main amd64 multipath-tools-b
oot all 0.5.0+git1.656f8865-4 [17.0 kB]
Fetched 246 kB in 4s (51.5 kB/s)           
Preconfiguring packages ...
(Reading database ... 30314 files and directories currently installed.)
Preparing to unpack .../kpartx_0.5.0+git1.656f8865-4_amd64.deb ...
Unpacking kpartx (0.5.0+git1.656f8865-4) over (0.5.0+git1.656f8865-2)
...
Preparing to unpack .../multipath-tools_0.5.0+git1.656f8865-4_amd64.deb 
...
Warning: Stopping multipath-tools.service, but it can still be
activated by:
  multipathd.socket
Unpacking multipath-tools (0.5.0+git1.656f8865-4) over
(0.5.0+git1.656f8865-2) ...
Preparing to unpack .../multipath-tools-boot_0.5.0+git1.656f8865-
4_all.deb ...
Unpacking multipath-tools-boot (0.5.0+git1.656f8865-4) over
(0.5.0+git1.656f8865-2) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for libc-bin (2.21-7) ...
Processing triggers for systemd (228-6) ...
Setting up kpartx (0.5.0+git1.656f8865-4) ...
Setting up multipath-tools (0.5.0+git1.656f8865-4) ...
update-rc.d: warning: start and stop actions are no longer supported;
falling back to defaults
Setting up multipath-tools-boot (0.5.0+git1.656f8865-4) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for libc-bin (2.21-7) ...
Processing triggers for initramfs-tools (0.120) ...
update-initramfs: Generating /boot/initrd.img-4.3.0-1-amd64

And then, after the reboot, this is what I get.

root at debian-sanboot:~# mount | grep "/ "
/dev/sda on / type ext4 (rw,relatime,errors=remount-
ro,stripe=8191,data=ordered)

Also see the attached screenshot.

The thing is that all detection takes place in initrd, where it finds
and coalesces the paths into 1 logical multipath device.

The bug is somewhere in multipath changes in between -2 and -4.
Something I haven't been able to easily determine.

In initrd, the scripts are supposed the detect and create the logical
mutlipath device, and then as and when the real rootfs gets mounted,
the open-iscsi daemon reloads establishing further sessions, more
devices are detected, and added as additional paths to the same
multipath device.

But the fact that rolling -4 over -2 triggers the bug, has me grounded
and I'm trying to look as to where the bug really is.

> Btw. just now I noticed in your log that open-iscsi -13 was
> actually installed on your system and still the update-initramfs
> error came. Had you perhaps installed an older test package with
> -13 that didn't have the fix for the /sbin/iscsistart? It would
> be great to see the rest of your log to see what really happened.
> 

No. I think that (iscsistart symlink) bug later got fixed, when I
upgraded the setup

-- 
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
-------------- next part --------------
A non-text attachment was scrubbed...
Name: multipath-delta.debdiff
Type: text/x-patch
Size: 5435 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-lvm-maintainers/attachments/20160214/e78e3890/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot_debian-sanboot_2016-02-14_16:10:59.png
Type: image/png
Size: 14004 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-lvm-maintainers/attachments/20160214/e78e3890/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: This is a digitally signed message part
URL: <http://lists.alioth.debian.org/pipermail/pkg-lvm-maintainers/attachments/20160214/e78e3890/attachment-0001.sig>