[BlueOnyx:25872] Re: Real Time access analysis (and better stats)

[Juerg Sommer]

Hi Michael,


> That may be one way to do it. As you can see in 
> /etc/logrotate.d/apache it calls split_logs this way:
>
> https://devel.blueonyx.it/trac/browser/BlueOnyx/5211R/ui/base-sitestats.mod/src/sitestats-scripts/apache.logrotate 
>
>
> /usr/local/sbin/split_logs web < /tmp/.logrotate_apache_access 2> 
> /dev/null || true
>
> So we're telling split_logs that this is a 'web' log and we're piping 
> /tmp/.logrotate_apache_access through it. This may be a frigging large 
> file, so that's why it's not shoved into a variable, but directly 
> processed by slit_logs.
>
> Once *that* is done and we have the Server's web.log, we simply call 
> GoAccess to do it's deeds, because at that point we don't have to 
> worry about STDIN anymore, as there will be nothing else delivered by 
> STDIN anymore.
>
>> I have to test it if it's also work with logrotate.
>
> I guess it'll work. Until you eventually try to shove several 
> gigabytes of logfiles through it to parse. :o)

I agree, large logfiles are a problem with variables...

The problem is line ~170 (I modified the script)
system("$GoAccess_cmd $go_access_params");

This is called BEFORE <STDIN> loop in ~270:
while (my $line = <STDIN>) {

GoAccess grabs the STDIN run on command line (both way "cat xy | script" 
or "script < xy")

SAMPLE:

------------
#!/usr/bin/perl
#system("/usr/bin/xargs /usr/bin/echo YY ");
my @stdin = <STDIN>;
#system("/usr/bin/xargs /usr/bin/echo YY ");
while ($line = shift(@stdin)) {
      print "XX:".$line;
}
------------

If you activate only the first system(), output is

[root at blueonyx9 admin]# echo test |./stdin.pl
YY test
(system grabs stdin, loop is empty)

If you activate only the second system(), output is
[root at blueonyx9 admin]# echo test |./stdin.pl
YY
XX:test
(@stdin grabs the input and system is empty)

So you have to call GoAccess AFTER parsing STDIN or backup STDIN in a 
variable. It works when called in logrotate, but I don't know how, 
GoAccess or perl must be handle different, when they are not executed on 
an interactive shell. I don't like any script which reacts different 
when they are called manually or by server daemon. And somewhere in the 
future, the STDIN is maybe also grabbed when executed on logrotate...

Regards,
Juerg