[BlueOnyx:25867] Re: Real Time access analysis (and better stats)

[Juerg Sommer]

Hi Michael

I debugged the split_logs script, not quite easy.....

>> > Yeah, it worked now. Was a special environment: I tested it with a vm
>> > and only log-files from today. Running split_logs manually created a
>> > folder for 2022/12/27 (yesterday, normally wanted), but in this folder
>> > were only records for today (which would be filtered, I think).
>>
>> Yes, this is intentional. When logrotate runs we create a copy of the 
>> log and work through it. The results go in a folder with yesterdays 
>> date, as "todays" data is still in the live log.
>>
>> If logrotate doesn't run at midnight we might end up with some data 
>> from yesterday and today in it, but each dated folder after a while 
>> will have consistently data that was aggregated until the logrotate 
>> kicked in. So I don't consider this an issue.
>
> I checked the logs today after automatic logrotate. The files are 
> created (and I don't know why), but it's strange. I have to less 
> traffic on the test-site to debug it (only manual 5 hits), but
> web.log is a null byte file (also on other older installations)
> there's a valid json file in /home/sites/{sitename}/var/logs/2022/12/28
> in GUI I can only choose 2022/12/29, which shows no results (correct, 
> no json). 2022/12/28 is not clickable.
>
> Should web.log have data or would it removed after statistic is 
> processed? I have to debug the process, when I migrated a site with 
> more traffic....

When you start split_logs in the command line,
system("$GoAccess_cmd $go_access_params");
clears the STDIN. If it's run in logrotate I think STDIN keeps alive, 
else the file wouldn't be created. But maybe it would be good to backup 
STDIN in script start (near variable declaration) with

my @stdin;
while (<STDIN>) {
         push(@stdin, $_);
}

and then loop trough the array instead of STDIN
while (my $line = <STDIN>) {
while (my $line = shift(@stdin)) {

I have to test it if it's also work with logrotate.

And then I saw, you create a /etc/logrotate.d/siteX script for each 
vsite which clears the web.log, so this is ok, that there's a 0 Byte 
file (it's ok, the complete server logs in /var/logs/https are rotated, 
so we don't need it).

My unterstanding of split_logs is now better than before. I'll check if 
array for STDIN also works in logrotate-environment. If so, I would 
change the script for me: I don't like any scripts I can't run 
manually... :) I'll give you feedback tomorrow if script change also 
worked in logrotate.

Regards,
Juerg