[BlueOnyx:25870] Re: Real Time access analysis (and better stats)

[Michael Stauber]

Hi Juerg,

 > My unterstanding of split_logs is now better than before. I'll
 > check if array for STDIN also works in logrotate-environment.
 > If so, I would change the script for me: I don't like any scripts
 > I can't run manually... :) I'll give you feedback tomorrow if
 > script change also worked in logrotate.

This "split_log" script? We inherited it from the BlueQuartz sources and 
it may already have been there during the Cobalt days. It's in a coding 
format and principle I don't like very much either, but short of a 
complete rewrite there isn't much that can be done about it.

> When you start split_logs in the command line,
> system("$GoAccess_cmd $go_access_params");
> clears the STDIN. If it's run in logrotate I think STDIN keeps alive, 
> else the file wouldn't be created. But maybe it would be good to backup 
> STDIN in script start (near variable declaration) with
> 
> my @stdin;
> while (<STDIN>) {
>          push(@stdin, $_);
> }
> 
> and then loop trough the array instead of STDIN
> while (my $line = <STDIN>) {
> while (my $line = shift(@stdin)) {

That may be one way to do it. As you can see in /etc/logrotate.d/apache 
it calls split_logs this way:

https://devel.blueonyx.it/trac/browser/BlueOnyx/5211R/ui/base-sitestats.mod/src/sitestats-scripts/apache.logrotate

/usr/local/sbin/split_logs web < /tmp/.logrotate_apache_access 2> 
/dev/null || true

So we're telling split_logs that this is a 'web' log and we're piping 
/tmp/.logrotate_apache_access through it. This may be a frigging large 
file, so that's why it's not shoved into a variable, but directly 
processed by slit_logs.

Once *that* is done and we have the Server's web.log, we simply call 
GoAccess to do it's deeds, because at that point we don't have to worry 
about STDIN anymore, as there will be nothing else delivered by STDIN 
anymore.

> I have to test it if it's also work with logrotate.

I guess it'll work. Until you eventually try to shove several gigabytes 
of logfiles through it to parse. :o)

> And then I saw, you create a /etc/logrotate.d/siteX script for each 
> vsite which clears the web.log, so this is ok, that there's a 0 Byte 
> file (it's ok, the complete server logs in /var/logs/https are rotated, 
> so we don't need it).

That's fine. By the time web.log is rotated and set to 0 all relevant 
data has been extracted by it already. But for testing purpose you can 
comment out or temporarily remove the /etc/logrotate.d/siteX files.

-- 
With best regards

Michael Stauber