[Pkg-awstats-devel] RFC - cron-related stuff

Thu Apr 16 12:51:20 UTC 2009

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, Apr 16, 2009 at 02:18:13PM +0400, Sergey B Kirpichev wrote:
>On Mon, Apr 13, 2009 at 12:25:54PM +0200, Jonas Smedegaard wrote:
>> This is your approach (please correct me if I am wrong:
>> 
>> logrotate[root] < httpd > logs[root:awstats]
>> awstats[awstats] < logs[:awstats] > stats[awstats:www-data]
>> awstats[www-data] < stats[:www-data] > httpd
>> 
>> It won't work out of the box: It requires changes to a logrotate 
>> script not owned by our package, so cannot even optionally be 
>> automated by us.
>
>Yes.  But it's easy to customize by local admin (one line changes in 
>logrotate.d/* configs).

We should strive towards working-by-default.

>But, after our discussion, the second line is:
>awstats[awstats] < logs[:awstats] > stats[awstats:awstats]
>No web access.

Here's a summary of how I understand your proposal (added leading 
interface and paranthesis around routines breaking the chain from 
working out-of-the-box):

cron:   logrotate[root] < httpd > logs[root:adm]
admin: (edit logrotate to output logs to root:awstats)
cron:   awstats[awstats] < logs[:awstats] > stats[awstats:awstats]
admin: (edit cron to output stats to awstats:www-data)
cgi:    awstats[www-data] < stats[:www-data] > httpd

In other words, you propose to have neither stats nor CGI usable without 
admin customization.  Or did I miss something?

(please explicitly write all lines of your proposed workflow, similar to 
above, instead of only describing your changes to my description - to 
improve our chance of understanding each other)

>> When working, raw logfiles are only usable by awstats - other packages 
>> (e.g. alternative log analyzers, perhaps even security surveillance) 
>> cannot concurrently access them without even more custom changes.
>
>Yet simple...  Just add them to awstats group (no security issues).

Right.

Simple for an admin to adjust, but unlikely that other packages would 
want to support special-casing for awstats.

Here's an improved draft of my proposal:

cron:   logrotate[root] < httpd > logs[root:adm]
cron:   cp[awstats-adm] < logs[:adm] > logs[awstats-adm:awstats]
cron:   awstats < logs[:awstats] > stats[awstats:awstats]
cron:   cp[awstats-adm] < stats[:awstats] > stats[awstats-adm:adm]
admin: (edit /etc/default/awstats to cp > stats[awstats-adm:www-data]
web:    awstats[www-data] < stats[:www-data] > httpd

Provides stats out-of-the-box (but readable only my admins).

Can later be enhanced:

  * admin task can be implemented as debconf question in postinst
  * cp can be replaced with smarter wrapper (gzip, ip resolving, etc.)
  * tiny piping tool can be provided, by default useless but optionally
    (through debconf) suid awstatsadm, for safer-than-direct realtime
    access to logfiles

>> This is my approach, improved to not use root access but another user 
>> member of both adm and awstats groups ([:awstats] means read-only by 
>> members of awstats group, writable only by same user):
>> 
>> logrotate[root] < httpd > logs[root:adm]
>> awstats-adm[awstats-adm] < logs[:adm] > logs[awstats-adm:awstats]
>> awstats < logs[:awstats] > stats[awstats:awstats]
>> awstats[www-data] < stats[:www-data] > httpd
>> 
>> It won't work out of the box: Stats are generated but accessible to 
>> noone by default.  Changes needed are possible to automate, however, 
>> as they involve only groups memberships and/or our own config files.
>> 
>> When working, raw logfiles are untouched, so usable by other packages 
>> too.
>
>Personally, I'm dislike this copying stuff (as more error prone).

I agree that it requires more disk space.  And that it restricts ability 
to use fancy features of AWStats.  But I cannot see how it is more 
risky. But how is "cp" more error-prone than allow AWStats access to the 
file (and everything else with same access rights!)?

Please elaborate.

>> Yes, we can throw all sorts of files in as examples.  Examples are 
>> unreliable, it is something that we do not support.
>> 
>> With my proposal we can extend our *supported* routines to support 
>> several approaches to virtual hosting.  And we can extend our 
>> *supported* routines to automatically enable fully working, 
>> web-accessible stats (yes, I _do_ want that to be possible, I just 
>> don't want it to be the default).
>
>> With my approach we need the location, possibly in multiple scripts, 
>> so it makes sense instead to save the location in 
>> /etc/default/awstats.
>
>If user want's piped logs?  Or gzip'ed?  Now it all can be handled in 
>just one place (/etc/awstats/awstats.example.com.conf).

We should provide a working default setup, not (necessarily) an 
infrastructure supporting all features of AWStats: Our users are always 
welcome to do things differently than what we provide by default.

If the user feels too constrained by the safe default workflow we offer, 
then the user can choose to replace parts of our workflow with less 
secure but more flexible parts - e.g. use awstats to resolve, uncompress 
and read input data.

>> Yet another thing the local admin needs to do manually which we 
>> cannot later automate :-(
>
>If we have a working setup for parsing stuff (as package is installed), 
>then we would enable crontab entry.

[Please quote only the parts that you respond to.  Above does not seem 
to be a response to the specific quote, but a more general comment]

Do you have ideas on how we get can stats analyzing out-of-the-box?  
Sorry if I missed such idea pointed out earlier.  As I noted further up, 
it seems to me that your proposal cannot (safely) provide this without 
local editing.

>Right now (and before), default awstats setup doesn't work from
>begining.
>
>So, my suggestion is to merge first static pages stuff (and use 
>separate awstats:awstats user/group for /var/lib/awstats).  I will do 
>this at weekend or on the next week.

I agree that switching default config to produce static pages by default 
is first step, no matter what.

>And then we can try to tune crontab entries (include some wrapper to 
>access logs in adm group).

Agreed - but without releasing in between.

Either we should roll back the current changes to cron script before our 
next packaging release, or we postpone package release until we have 
something that works by default (i.e. produces analysis, even if not 
accessible by www-data).

Currently _all_ awstats users customize the install.  Which means _all_ 
users will need to re-customize each time we change even a comment in 
the packaged conffiles.  So we really should avoid such changes unless 
we provide something that there is a chance that _some_ users can use 
as-is.

>> It is better to provide master config below /etc/default than 
>> requiring the admin to edit multiple configfiles in different 
>> formats.
>> 
>> Also, files below /etc/default must use a specific format that we can 
>> later automate and provide a debconf interface for.
>> 
>> Would be _very_ nice if we some day could make it possible to preseed 
>> awstats for use out-of-the-box at various security levels.
>
>Then, anyway, we _must_ automate also /etc/awstats/awstats._site_.conf 
>stuff (virtual host names etc).  Doesn't look nice.
>
>M.b. we can instead use just some sed/awk/grep stuff in 
>/etc/cron.d/awstats to receive LogFile info?

If by default a cron job written by us deliver the logs to awstats, then 
default awstats.conf should simply point to that path we use, and we can 
add a comment recommending to look at /etc/default/awstats before 
changing it (in /etc/default/awstats the user can either change the 
source path to point to some custom location, or comment it out to 
disable the cron job).

Yes, later we can consider doing fancier stuff like parsing all 
awstats.*.conf files (and all files that they include!!!), but I suggest 
we avoid that as it gets too ugly too quick: In addition to supporting 
includes, AWStats also made no promise to keep its logfile format!

  - Jonas

- -- 
* Jonas Smedegaard - idealist og Internet-arkitekt
* Tlf.: +45 40843136  Website: http://dr.jones.dk/

  [x] quote me freely  [ ] ask before reusing  [ ] keep private
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAknnKcgACgkQn7DbMsAkQLj7BwCgmLZ0d/zT3o8VrvEcffIPka6T
yXwAoI8SUTHqyIeRRsk5uATqc6h5WeZc
=08Ic
-----END PGP SIGNATURE-----