[Logcheck-devel] Bug#445215: logcheck: egrep is soooo slow

[Frédéric Brière]

Package: logcheck
Version: 1.2.62
Severity: wishlist

Yesterday, while running logcheck against all my syslogs for the week, I
started bemoaning how long the whole thing was taking (over 9 minutes
for 4 megs).  I wondered if maybe one bad regex was stalling the whole
thing, but the debug output showed that all rulefiles were taking up
time proportional to their size.  (Besides, egrep being based on a DFA,
it doesn't care much about how a regex is written.)

Out of curiosity, and realizing that an egrep regex should, AFAIK, work
just the same in Perl, I whipped up a one-liner to test out one
rulefile.  egrep took over 20 seconds to match ignore.d.server/spamd
against my logs; perl took less than 2 to produce the same results.

So, I wrote up the attached script as a quick hack to try out perl as a
substitute for egrep.  This brought the run time down to less than a
minute and a half.  As Mr. Brian Norris said: "I'm convinced".  :)


Now, I'm not advocating immediate action, as such a switch should
certainly not be taken lightly, especially given the security role of
logcheck.  Nevertheless, I think it's something worth mulling over,
given the speed difference.  What do you think?


-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: i386 (i686)

Kernel: Linux 2.6.21-2-k7 (SMP w/1 CPU core)
Locale: LANG=en_CA.UTF-8, LC_CTYPE=en_CA.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages logcheck depends on:
ii  adduser          3.105                   add and remove users and groups
ii  cron             3.0pl1-100              management of regular background p
ii  lockfile-progs   0.1.11                  Programs for locking and unlocking
ii  logtail          1.2.62                  Print log file lines that have not
ii  mailx            1:8.1.2-0.20070424cvs-1 A simple mail user agent
ii  postfix [mail-tr 2.4.5-4                 High-performance mail transport ag
ii  sysklogd [system 1.5-1                   System Logging Daemon

Versions of packages logcheck recommends:
ii  logcheck-database             1.2.62     database of system log rules for t

-- no debconf information
-------------- next part --------------
A non-text attachment was scrubbed...
Name: minigrep.pl
Type: application/x-perl
Size: 1279 bytes
Desc: not available
Url : http://lists.alioth.debian.org/pipermail/logcheck-devel/attachments/20071004/2eaa443e/attachment.bin