[Pkg-iscsi-maintainers] Bug#699240: open-iscsi: Broadcom Offload Engine is unusable because of missing iscsiuio daemon
Rafal Ramocki
rafal.ramocki at gmail.com
Tue Jan 29 12:39:43 UTC 2013
Package: open-iscsi
Version: 2.0.873-3
Severity: normal
Hi,
I have iSCSI storage array, and dedicated to it broadcom interfaces with iSCSI Offload engine (BCM5709). After configuring open-iscsi to use offload engine (transport bnx2i) i have no connectivity.
When I'm forcing to log in to target by iscsiadm -m node --login it hangs. I figured out that it is sendig by NETLINK command to 'iscsid', and iscsid is trying to communicate with kernel without success.
Strace of iscsiadm shows:
socket(PF_FILE, SOCK_STREAM, 0) = 3
connect(3, {sa_family=AF_FILE, path=@"ISCSIADM_ABSTRACT_NAMESPACE"}, 110) = 0
write(...)
recvfrom(
(waiting and finnaly connection timed out)
While 'iscsid -d8 -f' shows:
iscsid: thread 01ad1128 wait some more
iscsid: thread 01ad1128 wait some more
iscsid: thread 01ad1128 wait some more
iscsid: thread 01ad1128 was scheduled at 1463:12, curtime 1475 q_forw 0x68b330 &pend_list 0x68b330
iscsid: thread 01ad1128 now in actor_list
iscsid: exec thread 01ad1128 callback
iscsid: iscsi_login_eh
iscsid: login failed ISCSI_CONN_STATE_XPT_WAIT/R_STAGE_NO_CHANGE
iscsid: in ktransport_ep_disconnect
iscsid: get ev context 0x1ad5020
iscsid: in ktransport_ep_connect
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: expecting event 29, got 103, handling...
iscsid: in nlpayload_read
iscsid: Received iferror -19: No such device.
iscsid: cannot make a connection to 10.2.90.69:3260 (-19,2)
iscsid: put ev context 0x1ad5020
iscsid: Requeue reopen attempt in 3 secs
iscsid: thread 01ad1128 delete: state 3
iscsid: thread 0x1ad1128 schedule: delay 12 state 3
iscsid: thread removed
iscsid: thread 01ad1128 wait some more
iscsid: thread 01ad1128 wait some more
iscsid: thread 01ad1128 wait some more
iscsid: thread 01ad1128 was scheduled at 1475:12, curtime 1487 q_forw 0x68b330 &pend_list 0x68b330
iscsid: thread 01ad1128 now in actor_list
iscsid: exec thread 01ad1128 callback
iscsid: iscsi_login_eh
iscsid: login failed ISCSI_CONN_STATE_XPT_WAIT/R_STAGE_NO_CHANGE
iscsid: Giving up on initial login attempt after 120 seconds.
iscsid: disconnect conn
iscsid: in ktransport_ep_disconnect
iscsid: Connection-1:0 to [target: iqn.1992-04.com.emc:cx.ckm00122000265.a7, portal: 10.2.90.69,3260] through [iface: bnx2i.00:10:18:e6:c3:55] is shutdown.
iscsid: mgmt_ipc_write_rsp: rsp to fd 5
iscsid: thread 01ad1128 delete: state 3
iscsid: thread 01ad1160 delete: state 0
iscsid: destroying session
In dmesg I have only:
[ 858.845517] cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.7 (July 20, 2011)
[ 858.845570] bnx2 0000:01:00.0: peth0: Added CNIC device
[ 858.846906] bnx2 0000:01:00.1: eth1: Added CNIC device
[ 858.846943] bnx2 0000:04:00.0: eth2: Added CNIC device
[ 858.848267] bnx2 0000:04:00.1: eth3: Added CNIC device
[ 858.851317] Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.0.3 (Jun 15, 2011)
[ 858.851359] iscsi: registered transport (bnx2i)
[ 858.851471] scsi6 : Broadcom Offload iSCSI Initiator
[ 858.851892] bnx2i [04:00.01]: ISCSI_INIT passed
[ 858.856678] scsi7 : Broadcom Offload iSCSI Initiator
[ 858.857106] bnx2i [04:00.00]: ISCSI_INIT passed
[ 858.864650] scsi8 : Broadcom Offload iSCSI Initiator
[ 858.864912] scsi9 : Broadcom Offload iSCSI Initiator
[ 858.865324] bnx2i [01:00.00]: ISCSI_INIT passed
But no information about connecting to target. I use following configuration:
# iscsiadm -m node -T iqn.1992-04.com.emc:cx.ckm00122000265.a7
# BEGIN RECORD 2.0-873
node.name = iqn.1992-04.com.emc:cx.ckm00122000265.a7
node.tpgt = -1
node.startup = automatic
node.leading_login = No
iface.hwaddress = 00:10:18:e6:c3:55
iface.ipaddress = 10.2.90.68
iface.iscsi_ifacename = bnx2i.00:10:18:e6:c3:55
iface.net_ifacename = <empty>
iface.transport_name = bnx2i
iface.initiatorname = <empty>
iface.bootproto = <empty>
iface.subnet_mask = <empty>
iface.gateway = <empty>
iface.ipv6_autocfg = <empty>
iface.linklocal_autocfg = <empty>
iface.router_autocfg = <empty>
iface.ipv6_linklocal = <empty>
iface.ipv6_router = <empty>
iface.state = <empty>
iface.vlan_id = 0
iface.vlan_priority = 0
iface.vlan_state = <empty>
iface.iface_num = 0
iface.mtu = 9000
iface.port = 0
node.discovery_address = <empty>
node.discovery_port = 0
node.discovery_type = static
node.session.initial_cmdsn = 0
node.session.initial_login_retry_max = 8
node.session.xmit_thread_priority = -20
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.nr_sessions = 1
node.session.auth.authmethod = None
node.session.auth.username = <empty>
node.session.auth.password = <empty>
node.session.auth.username_in = <empty>
node.session.auth.password_in = <empty>
node.session.timeo.replacement_timeout = 60
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.err_timeo.host_reset_timeout = 60
node.session.iscsi.FastAbort = Yes
node.session.iscsi.InitialR2T = Yes
node.session.iscsi.ImmediateData = No
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 2
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.conn[0].address = 10.2.90.69
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.DataDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
# END RECORD
# iscsiadm -m node -T iqn.1992-04.com.emc:cx.ckm00122000265.b7
# BEGIN RECORD 2.0-873
node.name = iqn.1992-04.com.emc:cx.ckm00122000265.b7
node.tpgt = -1
node.startup = automatic
node.leading_login = No
iface.hwaddress = 00:10:18:e6:c3:57
iface.ipaddress = 10.2.90.100
iface.iscsi_ifacename = bnx2i.00:10:18:e6:c3:57
iface.net_ifacename = <empty>
iface.transport_name = bnx2i
iface.initiatorname = <empty>
iface.bootproto = <empty>
iface.subnet_mask = <empty>
iface.gateway = <empty>
iface.ipv6_autocfg = <empty>
iface.linklocal_autocfg = <empty>
iface.router_autocfg = <empty>
iface.ipv6_linklocal = <empty>
iface.ipv6_router = <empty>
iface.state = <empty>
iface.vlan_id = 0
iface.vlan_priority = 0
iface.vlan_state = <empty>
iface.iface_num = 0
iface.mtu = 9000
iface.port = 0
node.discovery_address = <empty>
node.discovery_port = 0
node.discovery_type = static
node.session.initial_cmdsn = 0
node.session.initial_login_retry_max = 8
node.session.xmit_thread_priority = -20
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.nr_sessions = 1
node.session.auth.authmethod = None
node.session.auth.username = <empty>
node.session.auth.password = <empty>
node.session.auth.username_in = <empty>
node.session.auth.password_in = <empty>
node.session.timeo.replacement_timeout = 60
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.err_timeo.host_reset_timeout = 60
node.session.iscsi.FastAbort = Yes
node.session.iscsi.InitialR2T = Yes
node.session.iscsi.ImmediateData = No
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 2
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.conn[0].address = 10.2.90.101
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.DataDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
# END RECORD
After debugging and searching some information i believe that offload engine to operate needs:
- dedicated daemon available under GPL (file COPYING) called 'iscsiuio'
- interface between iscsid and iscsiuio (propobly patchers of open-iscsi are required)
I've been testing this on also on Centos, RHEL and SLES and in all of this distributions Offload engine is working gaining about 15% of performance. On thous distributions this problem is soled by including 'iscsiuio' and 'open-iscsi' into one 'initiator-tools' package, but I believe hat it dosn't have to be that way. They also use set of patches to make interface between this two components. There might be also a problem with DFSG since in iscsiuio there is release notes (file RELEASE.txt) leaved by Broadcom with copyrights but maybe 'iscsiuio' might be placed in non-free. I've tried to be precise, but if You need additional iformation please let me know.
-- System Information:
Debian Release: 7.0
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 3.2.0-4-amd64 (SMP w/24 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages open-iscsi depends on:
ii libc6 2.13-37
ii udev 175-7
open-iscsi recommends no packages.
open-iscsi suggests no packages.
-- Configuration Files:
/etc/iscsi/initiatorname.iscsi changed:
InitiatorName=iqn.1993-08.org.debian:01:e142dcfa5f5d
/etc/iscsi/iscsid.conf changed:
iscsid.startup = /usr/sbin/iscsid
node.startup = automatic
node.leading_login = No
node.session.timeo.replacement_timeout = 60
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.initial_login_retry_max = 8
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.xmit_thread_priority = -20
node.session.iscsi.InitialR2T = Yes
node.session.iscsi.ImmediateData = No
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
node.session.nr_sessions = 1
node.session.iscsi.FastAbort = Yes
-- no debconf information
More information about the Pkg-iscsi-maintainers
mailing list