[Pkg-ganeti-devel] Bug#700294: gnt-instance modify -t drbd should check if LV is still marked as online
Martin Zobel-Helas
zobel at debian.org
Mon Feb 11 09:04:15 UTC 2013
Package: ganeti2
Version: 2.5.2-1
Severity: normal
User: debian-admin at lists.debian.org
Usertags: needed-by-DSA-Team
Hi,
when trying to migrate from plain LV disk backend to DRBD using the
following the following command, the conversion fails if the LV is still
marked as online. Also the instance is unbootable afterwards.
root at clementi:~# gnt-job info 87671
Job ID: 87671
Status: error
Received: 2013-02-09 22:08:15.692172
Processing start: 2013-02-09 22:08:15.761730 (delta 0.069558s)
Processing end: 2013-02-09 22:08:19.548239 (delta 3.786509s)
Total processing time: 3.856067 seconds
Opcodes:
OP_INSTANCE_SET_PARAMS
Status: error
Processing start: 2013-02-09 22:08:15.761730
Execution start: 2013-02-09 22:08:15.830276
Processing end: 2013-02-09 22:08:19.548176
Input fields:
beparams: {}
comment: None
debug_level: 0
depends: None
disk_template: drbd
disks:
dry_run: False
force: False
force_variant: False
hvparams: {}
instance_name: kaufmann.debian.org
nics:
os_name: None
osparams: {}
priority: 0
remote_node: czerny.debian.org
wait_for_sync: True
Result:
OpExecError
[Can't create block device <DRBD8(hosts=clementi.debian.org/0-czerny.debian.org/0, port=11012, configured as 192.168.75.103:11012 192.168.75.109:11012, backend=<LogicalVolume(/dev/vg0/c3a00f49-53b2-43b6-95d5-d045e3db98b6.disk0_data, not visible, size=128m)>, metadev=<LogicalVolume(/dev/vg0/c3a00f49-53b2-43b6-95d5-d045e3db98b6.disk0_meta, not visible, size=128m)>, visible as /dev/disk/0, size=128m)> on node clementi.debian.org for instance kaufmann.debian.org: Can't assemble device after creation, unusual event: drbd0: can't attach local disk: /dev/drbd0: Failure: (114) Lower device is already claimed. This usually means it is mounted.
]
Execution log:
1:2013-02-09 22:08:16.301492:message Converting template to drbd
2:2013-02-09 22:08:16.533072:message Creating aditional volumes...
3:2013-02-09 22:08:18.695810:message Renaming original volumes...
4:2013-02-09 22:08:19.163656:message Initializing DRBD devices...
root at clementi:~# gnt-job info 87672
Job ID: 87672
Status: error
Received: 2013-02-09 22:09:40.513688
Processing start: 2013-02-09 22:09:40.579037 (delta 0.065349s)
Processing end: 2013-02-09 22:09:41.773141 (delta 1.194104s)
Total processing time: 1.259453 seconds
Opcodes:
OP_INSTANCE_STARTUP
Status: error
Processing start: 2013-02-09 22:09:40.579037
Execution start: 2013-02-09 22:09:40.646759
Processing end: 2013-02-09 22:09:41.773095
Input fields:
beparams: {}
comment: None
debug_level: 0
depends: None
dry_run: False
force: False
hvparams: {}
ignore_offline_nodes: False
instance_name: kaufmann.debian.org
no_remember: False
priority: 0
startup_paused: False
Result:
OpExecError
[Disk consistency error]
Execution log:
1:2013-02-09 22:09:41.099922:message - WARNING: Could not prepare block device disk/0 on node clementi.debian.org (is_primary=False, pass=1): Error while assembling disk: Can't activate lv /dev/vg0/9fc9ec56-ddf1-42ee-81a3-407478e34509.disk0: One or more specified logical volume(s) not found.\n
2:2013-02-09 22:09:41.184950:message - WARNING: Could not prepare block device disk/1 on node clementi.debian.org (is_primary=False, pass=1): Error while assembling disk: Can't activate lv /dev/vg0/16d72a18-b097-4bfb-aad3-42efea42073f.disk1: One or more specified logical volume(s) not found.\n
3:2013-02-09 22:09:41.259914:message - WARNING: Could not prepare block device disk/2 on node clementi.debian.org (is_primary=False, pass=1): Error while assembling disk: Can't activate lv /dev/vg0/6ae06036-07a1-465c-93ad-4bb0f21eeebd.disk2: One or more specified logical volume(s) not found.\n
4:2013-02-09 22:09:41.340190:message - WARNING: Could not prepare block device disk/0 on node clementi.debian.org (is_primary=True, pass=2): Error while assembling disk: Can't activate lv /dev/vg0/9fc9ec56-ddf1-42ee-81a3-407478e34509.disk0: One or more specified logical volume(s) not found.\n
5:2013-02-09 22:09:41.417764:message - WARNING: Could not prepare block device disk/1 on node clementi.debian.org (is_primary=True, pass=2): Error while assembling disk: Can't activate lv /dev/vg0/16d72a18-b097-4bfb-aad3-42efea42073f.disk1: One or more specified logical volume(s) not found.\n
6:2013-02-09 22:09:41.495581:message - WARNING: Could not prepare block device disk/2 on node clementi.debian.org (is_primary=True, pass=2): Error while assembling disk: Can't activate lv /dev/vg0/6ae06036-07a1-465c-93ad-4bb0f21eeebd.disk2: One or more specified logical volume(s) not found.\n
7:2013-02-09 22:09:41.708259:message Hint: If the message above refers to a secondary node, you can retry the operation using '--force'.
I renamed the LVs back to what ganeti expected them to be named, so the
instance gecomes bootable again.
I found out that the kernel knew about one LV having a partition:
zobel at clementi:~% sudo lvs | grep wi-a-
16d72a18-b097-4bfb-aad3-42efea42073f.disk1 vg0 -wi-a- 4.00g
6ae06036-07a1-465c-93ad-4bb0f21eeebd.disk2 vg0 -wi-a- 4.00g
zobel at clementi:/dev/mapper% ls vg0-9fc9ec56--ddf1--42ee--81a3--407478e34509.disk0p1
vg0-9fc9ec56--ddf1--42ee--81a3--407478e34509.disk0p1
from gnt-instance info:
...
Disk template: plain
Disks:
- disk/0: lvm, size 128M
access mode: rw
logical_id: vg0/9fc9ec56-ddf1-42ee-81a3-407478e34509.disk0
on primary: /dev/vg0/9fc9ec56-ddf1-42ee-81a3-407478e34509.disk0 (254:18)
- disk/1: lvm, size 4.0G
access mode: rw
logical_id: vg0/16d72a18-b097-4bfb-aad3-42efea42073f.disk1
on primary: /dev/vg0/16d72a18-b097-4bfb-aad3-42efea42073f.disk1 (254:20)
- disk/2: lvm, size 4.0G
access mode: rw
logical_id: vg0/6ae06036-07a1-465c-93ad-4bb0f21eeebd.disk2
on primary: /dev/vg0/6ae06036-07a1-465c-93ad-4bb0f21eeebd.disk2 (254:19)
zobel at clementi:/dev/mapper% sudo kpartx -d vg0-9fc9ec56--ddf1--42ee--81a3--407478e34509.disk0
zobel at clementi:/dev/mapper% sudo lvs | grep wi-a-
16d72a18-b097-4bfb-aad3-42efea42073f.disk1 vg0 -wi-a- 4.00g
6ae06036-07a1-465c-93ad-4bb0f21eeebd.disk2 vg0 -wi-a- 4.00g
9fc9ec56-ddf1-42ee-81a3-407478e34509.disk0 vg0 -wi-a- 128.00m
zobel at clementi:/dev/mapper% gnt-job info 88278
Job ID: 88278
Status: success
Received: 2013-02-10 21:12:42.642975
Processing start: 2013-02-10 21:12:42.700609 (delta 0.057634s)
Processing end: 2013-02-10 21:14:26.247909 (delta 103.547300s)
Total processing time: 103.604934 seconds
Opcodes:
OP_INSTANCE_SET_PARAMS
Status: success
Processing start: 2013-02-10 21:12:42.700609
Execution start: 2013-02-10 21:12:42.767784
Processing end: 2013-02-10 21:14:26.247886
Input fields:
beparams: {}
comment: None
debug_level: 0
depends: None
disk_template: drbd
disks:
dry_run: False
force: False
force_variant: False
hvparams: {}
instance_name: kaufmann.debian.org
nics:
os_name: None
osparams: {}
priority: 0
remote_node: czerny.debian.org
wait_for_sync: True
Result:
[disk_template, drbd]
Execution log:
1:2013-02-10 21:12:43.225613:message Converting template to drbd
2:2013-02-10 21:12:43.507703:message Creating aditional volumes...
3:2013-02-10 21:12:45.613282:message Renaming original volumes...
4:2013-02-10 21:12:46.269479:message Initializing DRBD devices...
5:2013-02-10 21:12:50.396838:message - INFO: Waiting for instance kaufmann.debian.org to sync disks.
6:2013-02-10 21:12:50.626711:message - INFO: - device disk/0: 24.30% done, 11s remaining (estimated)
7:2013-02-10 21:12:50.664605:message - INFO: - device disk/1: 3.60% done, 57s remaining (estimated)
8:2013-02-10 21:12:50.666871:message - INFO: - device disk/2: 0.60% done, 3m 57s remaining (estimated)
9:2013-02-10 21:13:50.857498:message - INFO: - device disk/1: 75.20% done, 19s remaining (estimated)
10:2013-02-10 21:13:50.859097:message - INFO: - device disk/2: 68.10% done, 35s remaining (estimated)
11:2013-02-10 21:14:26.031128:message - INFO: Instance kaufmann.debian.org's disks are in sync.
IMHO it should be easy to catch for "gnt-instance modify" if one of the
volumes is still marked as online and then should just stop before
renaming the LV and throw an error.
Best regards,
Martin
-- System Information:
Debian Release: 7.0
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 3.2.0-4-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages ganeti2 depends on:
ii bridge-utils 1.5-6
ii iproute 20120521-3
ii iputils-arping 3:20101006-1+b1
ii libjs-jquery 1.7.2+dfsg-1
ii lvm2 2.02.95-4
ii openssh-client 1:6.0p1-3
ii openssh-server 1:6.0p1-3
ii openssl 1.0.1c-4
ii python 2.7.3~rc2-1
ii python-openssl 0.13-2
ii python-paramiko 1.7.7.1-3.1
ii python-pycurl 7.19.0-5
ii python-pyinotify 0.9.3-1.1
ii python-pyparsing 1.5.6+dfsg1-2
ii python-simplejson 2.5.2-1
ii python-support 1.0.15
ii socat 1.7.1.3-1.4
Versions of packages ganeti2 recommends:
ii drbd8-utils 2:8.3.13-2
ii ganeti-instance-debootstrap 0.11-1
ii ndisc6 1.0.1-1+b1
ii qemu-kvm 1.1.2+dfsg-5
Versions of packages ganeti2 suggests:
pn drbd8-module-source <none>
-- no debconf information
--
Martin Zobel-Helas <zobel at debian.org> Debian System Administrator
Debian & GNU/Linux Developer Debian Listmaster
http://about.me/zobel Debian Webmaster
GPG Fingerprint: 6B18 5642 8E41 EC89 3D5D BDBB 53B1 AC6D B11B 627B
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/pkg-ganeti-devel/attachments/20130211/fcbd3d93/attachment.pgp>
More information about the Pkg-ganeti-devel
mailing list