[Pkg-varnish-devel] Bug#789658: varnish: varnishlog/varnishncsa killed by logrotate under systemd

[Paul Wise]

Package: varnish
Version: 4.0.2-1
Severity: important
X-Debbugs-CC: debian-admin at lists.debian.org
User: debian-admin at lists.debian.org
Usertags: needed-by-DSA-Team

Since lw07.d.o and sibelius.d.o (snapshot.d.o frontends) were upgraded
to jessie, we have been getting the below errors from the logrotate
cron job on a daily basis. The issue seems to be something to do with
the VSM file being deleted but varnish holding the file open, something
writes a new one and then varnishlog thinks varnishd has abandoned the
file, varnishlog exits and reloads fail after that.

/etc/cron.daily/logrotate:
Job for varnishlog.service failed. See 'systemctl status varnishlog.service' and 'journalctl -xn' for details.
invoke-rc.d: initscript varnishlog, action "reload" failed.
error: error running non-shared postrotate script for /var/log/varnish/varnish.log of '/var/log/varnish/varnish.log '
Job for varnishncsa.service failed. See 'systemctl status varnishncsa.service' and 'journalctl -xn' for details.
invoke-rc.d: initscript varnishncsa, action "reload" failed.
error: error running non-shared postrotate script for /var/log/varnish/varnishncsa.log of '/var/log/varnish/varnishncsa.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

pabs at lw07:~$ sudo systemctl status varnishlog
● varnishlog.service - Varnish HTTP accelerator log daemon
   Loaded: loaded (/lib/systemd/system/varnishlog.service; enabled)
   Active: failed (Result: exit-code) since Sun 2015-06-07 11:14:57 UTC; 2 weeks 1 days ago
 Main PID: 1642 (code=exited, status=1/FAILURE)

Jun 14 06:25:11 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 15 06:25:07 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 16 06:25:07 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 17 06:25:07 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 18 06:25:06 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 19 06:25:06 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 20 06:25:06 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 21 06:25:06 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 22 06:25:09 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 23 06:25:06 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.

pabs at lw07:~$ sudo journalctl -u varnish -u varnishlog -u varnishncsa
-- Logs begin at Sun 2015-06-07 11:14:37 UTC, end at Tue 2015-06-23 07:10:44 UTC. --
...
Jun 07 11:14:57 lw07 varnishlog[1642]: Can't open VSM file (Abandoned VSM file (Varnish not running?) /var/lib/varnish/lw07/_.vsm
Jun 07 11:14:57 lw07 varnishlog[1642]: )
Jun 07 11:14:57 lw07 systemd[1]: varnishlog.service: main process exited, code=exited, status=1/FAILURE
Jun 07 11:14:57 lw07 varnishncsa[1643]: Can't open VSM file (Abandoned VSM file (Varnish not running?) /var/lib/varnish/lw07/_.vsm
Jun 07 11:14:57 lw07 varnishncsa[1643]: )
Jun 07 11:14:57 lw07 systemd[1]: Unit varnishlog.service entered failed state.
Jun 07 11:14:57 lw07 systemd[1]: varnishncsa.service: main process exited, code=exited, status=1/FAILURE
Jun 07 11:14:57 lw07 systemd[1]: Unit varnishncsa.service entered failed state.
Jun 08 06:25:06 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 08 06:25:07 lw07 systemd[1]: Unit varnishncsa.service cannot be reloaded because it is inactive.
Jun 09 06:25:07 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 09 06:25:07 lw07 systemd[1]: Unit varnishncsa.service cannot be reloaded because it is inactive.
Jun 10 06:25:06 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 10 06:25:06 lw07 systemd[1]: Unit varnishncsa.service cannot be reloaded because it is inactive.
Jun 11 06:25:08 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 11 06:25:08 lw07 systemd[1]: Unit varnishncsa.service cannot be reloaded because it is inactive.
Jun 12 06:25:06 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 12 06:25:06 lw07 systemd[1]: Unit varnishncsa.service cannot be reloaded because it is inactive.
Jun 13 06:25:05 lw07 systemd[1]: Unit varnishlog.service cannot be reloaded because it is inactive.
Jun 13 06:25:05 lw07 systemd[1]: Unit varnishncsa.service cannot be reloaded because it is inactive.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://lists.alioth.debian.org/pipermail/pkg-varnish-devel/attachments/20150623/bb31da74/attachment.sig>