Bug#358742: cyrus21-imapd process hangs on sent mailbox

[Frédéric Brière]

On Fri, Mar 24, 2006 at 06:26:53PM -0300, Henrique de Moraes Holschuh wrote:
> There is another way to help this.  The issue is definately a locking mishap
> in lib/lock_fcntl.c.  If you dig C, you can help me doing a fine combing
> over that thing and trying to see if the code fails to handle EINTR
> anywhere, or if there is a failure mode where a lock is failing to be
> released (on a error path, most probably).

Hi Henrique!  I hope I can help you on this one.

I just had the same thing happen to me today.  (I think it may not be
the first time, but if so it's been so long that I don't remember
anyway.)

My situation is pretty much like Patrick Scharrenberg detailed in the
message linked to by Matt Boes' first report; namely, I've got a stray
imapd process that's just sitting there, holding a lock and preventing
any further connection to that mailbox.

It would appear that under certain conditions, imapd will issue a
blocking read() while holding a lock, without any timeout.  That's bad.


Here's Sleeping Beauty:

  fbriere at goretex:~$ ps auxw | grep 29708
  cyrus    29708  0.0  0.2 22368 2772 ?        S    Feb28   0:00 imapd -s -U 30

Waiting for Prince Charming:

  fbriere at goretex:~$ sudo gdb /usr/lib/cyrus/bin/imapd 29708
  [...]
  (gdb) bt       
  #0  0x403058de in read () from /lib/tls/libc.so.6
  #1  0x40197c0e in BIO_sock_non_fatal_error () from /usr/lib/i686/cmov/libcrypto.so.0.9.7

While clutching to the, erm, Golden Scepter or what-have-you:

  fbriere at goretex:~$ sudo lsof /var/spool/cyrus/mail/.../Sent/cyrus.header
  COMMAND   PID  USER   FD   TYPE DEVICE SIZE   NODE NAME
  imapd   29708 cyrus  mem-W  REG  253,2  178 229398 /var/spool/cyrus/mail/.../Sent/cyrus.header
  imapd   29708 cyrus   12uW  REG  253,2  178 229398 /var/spool/cyrus/mail/.../Sent/cyrus.header


What happened today was that we had a shitty connection between client
and server; the Windows client eventually gave up on the TCP connection
(disappeared from netstat), but it still shows up as ESTABLISHED on the
Linux server.  And since imapd doesn't use SO_KEEPALIVE (not even in
2.2, according to grep), that connection and that process will linger
there forever.

(Not that SO_KEEPALIVE is the appropriate solution, but it would be a
nice complement.)


I should point out that I've got four other forlorn imap processes also
waiting in vain for some love letter from their long-departed
sweetheart.  Yet, none of these is holding a lock, so there does seem to
be a specific condition that triggers this event.


-- 
Vista.  (cue laughter)