[Pkg-iscsi-maintainers] Bug#687619: Bug#687619: iscsitarget restart fail if more than 32 session try reconnect

[Laszlo Fekete]

Hello!

On the iscsitarget-devel list got a suggestion:


The other possibility I have seen is the INCOMING_MAX problem:

-#define INCOMING_MAX           32
+
+/*
+ changed INCOMING_MAX from 32 to 256 since we have seen initiators unable
+ to login (connection RST) during the login phase
+
+ http://blog.wpkg.org/2007/09/09/solving-reliability-and-scalability-
problems-with-iscsi/
+*/
+#define INCOMING_MAX           255



Next week, will try this patch to see this cause the error.

regards, blackluck

On 2012. September 14. 15:31:50 Laszlo Fekete wrote:
> Hello,
> 
> On 2012. September 14. 17:40:48 you wrote:
> > On Friday 14 September 2012 03:36 PM, Laszlo Fekete wrote:
> > 
> > I have fresh debian squeeze with standard 2.6.32-5-amd64 kernel and
> 
> iscsitarget + iscsitarget-dkms packages and have some other squeeze with
> open- iscsi initiators.
> 
> > The problem is after a long testing, that if there is more than 32 active
> 
> sessions when try /etc/init.d/iscsitarget restart, it sometimes fails, so no
> more than 32 initiator can reconnect.
> 
> > What is the error you see on the initiator? Does it report that the
> 
> connection already exists?
> 
> In the logs reports iscsi connection error detected and try to recover.
> 
> > For more details:
> > I have 16 targets with summary 40 active sessions, try to restart the
> > iscsi
> 
> target and sometimes (not all the time, 20-60% of the tries for me) it fail.
> first 32 clients can reconnect without problem, but the others don't get
> answer from the target and don't working. There is no error on target or
> initiator side, just the initiators try to reconnect.
> 
> > There is two type of the initiators:
> > - using one session to access the target with only one ip, open-iscsi
> > config
> limits and timeout are the default
> 
> > - using 4 session to access the two target (some of the targets are
> 
> duplicated within 2 servers with drbd) with 2 session/ip per target. These
> initiators using multipath and minimal (1sec) timeouts.
> 
> > Both type contains sessions which are just connected, but the target isn't
> 
> mounted on that client which failed to reconnect.
> 
> > If the iscsi target restart fail it random which initiator stuck, I think
> > it
> only depend on who is the faster to be in the first 32 session.
> 
> > I am not sure here. The open-iscsi default replacement timeout is 120
> > secs.
> 
> Even then, when the target is back, it will poll it.
> 
> You're right, I meant for 5sec default settings this:
> node.conn[0].timeo.noop_out_interval = 5
> node.conn[0].timeo.noop_out_timeout = 5
> and with 1 sec also this settings on those connections where using
> multipath.
> > Tried to check, maybe this is a network connection, but if the restart
> > fail
> 
> and try to telnet to them sometimes it also don't answer (tcpdump show, that
> the target server got the request, but don't send any answer).
> 
> > Checked that maybe on the iscsi target stop stucked, but it seems to be
> 
> okay, the session closed, the modules unloaded, there isn't any error, tried
> to raise the sleep time before start to 10 sec from 1, but got the same
> error.
> > What 1 sec setting are you referring here?
> 
> Sleep time in init script which is in restart after the stop and before the
> start.
> 
> > So I think there is a limit somewere that the in a short time no more than
> 
> 32 initiator can connect, but don't find any of this.
> 
> > Checked the newer iscsi targets changelog and don't see any report that
> 
> describe this problem, so don't tried to upgrade wheezy.
> 
> > Debian GNU/Linux 6.0, kernel 2.6.32-5-amd64, libc 2.11.3-3
> > 
> > There are settings in ietd.conf where you can set the Max number of
> 
> connections/sessions. Have you explored them?
> 
> I tried to set these to a high number or set 0 to disable (which is the
> default for max sessions, as the manual said), but nothing changed.
> Also tried to change ImmediateData and InitialR2T settings and add
>         NOPInterval 1
>         NOPTimeout 1
> settings to all targets (every setting tried to set for some target and all
> target too for testing). These settings doesn't solve the problem
> But if I decrease the active session number to 32 or lower (stop some of the
> initiators) the restart working fine every time and after that if I can the
> stopped initiators start one by one, there is no problem, just when I try
> to restart the iscsi target if more than 32 active sessions.
> 
> Regards,  blackluck