[Logcheck-devel] Bug#636810: logcheck: Should concatenate pattern files before execution

[Witold Baryluk]

Package: logcheck
Version: 1.3.13
Severity: normal

Hi!

I have logcheck running and running, for very long time now.
This is mostly because I have very big logfiles currently
due bug in one of deamons. It take so long that
it take few hours to process (and logcheck is started every 2 hours,
no enough to process this). I often need to killall -9 -u logcheck,
as it starts to be eating lots of CPU and making my laptop really hot. :/

So, I looked why actually it take so long for logcheck
and ways of improving.

One of things I immediately see, was that in top I see things
like this:


.... egrep --text -v -f /tmp/logcheck.SVwK21/ignore/dovecot
/tmp/logcheck.SVwK21/checked


In this particular moment it runs (already for 10 minutes) negative matching
using grep -E, with preprocessed dovecot pattern files
(probably removing empty lines and lines beging with # - comments - I guess).
On a file /tmp/logcheck.SVwK21/checked, and this files have about 310k lines
and 61MB.

Immediately I asked myself, why logcheck doesn't actually create one pattern
file
from all files in ignore.d, and pass it to egrep ? (Similar for positive
matches).

So, finding no excuse I filling this report.

Merging all pattern files (both for positive and negative match passes), will
greatly
improve performance by doing only single pass over log file. It will also
in principle allow egrep to optimize regular expressions better (because
there is lots of commonality beetwen patterns), make it even
slightly faster overally.

Please implement it, it is very simple change. Do not have particular
benchmark numbers, but if you want I can perform some.

Thanks.




-- System Information:
Debian Release: wheezy/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 3.0.0-t43-prod-03124-g81d6743-dirty
Locale: LANG=pl_PL.utf8, LC_CTYPE=pl_PL.utf8 (charmap=UTF-8) (ignored: LC_ALL set to pl_PL.utf8)
Shell: /bin/sh linked to /bin/dash

Versions of packages logcheck depends on:
ii  adduser                       3.113      add and remove users and groups
ii  cron                          3.0pl1-118 process scheduling daemon
ii  exim4-daemon-light [mail-tran 4.76-2     lightweight Exim MTA (v4) daemon
ii  lockfile-progs                0.1.15     Programs for locking and unlocking
ii  logtail                       1.3.13     Print log file lines that have not
ii  mime-construct                1.11       construct/send MIME messages from 
ii  rsyslog [system-log-daemon]   5.8.3-1    reliable system and kernel logging

Versions of packages logcheck recommends:
ii  logcheck-database             1.3.13     database of system log rules for t

Versions of packages logcheck suggests:
ii  syslog-summary                1.14-2     summarize the contents of a syslog

-- Configuration Files:
/etc/cron.d/logcheck changed [not included]
/etc/logcheck/logcheck.conf [Errno 13] Brak dostępu: u'/etc/logcheck/logcheck.conf'
/etc/logcheck/logcheck.logfiles [Errno 13] Brak dostępu: u'/etc/logcheck/logcheck.logfiles'

-- no debconf information