[BlueOnyx:25854] Re: Real Time access analysis (and better stats)

[Michael Stauber]

Hi all,

> Have you ever heard of https://goaccess.io/?

Soo ... I've been busy. :p

The base-sitestats on BlueOnyx 5211R was a bit of a train-wreck and the 
one on BlueOnyx 5210R isn't much better.

I just tackled the base-sitestats on BlueOnyx 5211R and published YUM 
updates for it and will also backport the relevant changes to BlueOnyx 
5210R.

FTP Usage Statistics:
======================

These stopped working a while ago and the effort to fix them vs. the 
actual benefit of *having* them working? Not worth the hassles. So I 
removed them from the GUI.


Net Usage Statistics:
======================

These used to use "iptables" accounting rules to record traffic 
statistics and as both 5210R and 5211R use Firewalld these "iptables" 
rules simply don't cut it for us anymore. So I removed this from the GUI 
as well.


Web Usage Statistics:
======================

Now this is something we really want. Both for the server as a whole as 
well as on a Vsite level.

The old statistics for this worked this way: The daily logrotate ran 
split_logs and split the central Apache access_log up into chunks. Each 
Vsite then got a var/log/web.log logfile with the accesses that 
pertained to this Vsite since the last logrotate (usually the night before).

Then "Analog" ran over that and condensed the statistics into cache 
files that got stored in dated directories.

Like var/log/2022/12/26/web.cache - for example.

The GUI then parsed these cache files and was supposed to display the 
web usage statistics stored therein. Long story short: "Analog" stopped 
working for us at one point or another and the resulting files were garbage.

Enter GoAccess: It can not only display real time web usage statistics. 
It can also parse logfiles and dump out either a static HTML page with 
the presentation of the gathered statistics. Or it can dump out a JSON 
file with just the data. Which is ideal for us.

So I modified split_logs to run goaccess over the individual server and 
vsite web.log files and dump out JSON files (web.json) into the date 
directories.

And an entirely new "Usage Information" / "Web" page (both on the server 
and Vsite level) shows the GoAccess statistics.

As "admin" or similarly privileged user you get to see the live stats 
and can use a date picker to see older (archived and therefore static) 
statistics.

On a Vsite level "Usage Information" / "Web" will only show static 
GoAccess statistics, as they're all served from the Vsite's statistical 
archive under var/log/<Date>

The separate menu entry "Server Management" / "Usage Information" / 
"GoAccess" has been removed, as the display of the live data has been 
merged into "Server Management" / "Usage Information" / "Web".

ATTENTION:
----------

In 5211R the live stats will apparently only work if AdmServ has a valid 
SSL cert. If the cert is self signed or expired, then the GoAccess HTML 
page fails to maintain a permanent connection with the GoAccess web socket.

All in all? This should serve us nicely. Many thanks to Dirk Estenfeld 
for the recommendation.


Monitorix:
==========

This is new for us. To replace the removed FTP and Net statistics on the 
server level (not Vsite!) I added the menu entry "Monitorix" to "Usage 
Information". In there you'll find the most detailed statistics about 
anything that makes sense to be displayed as graphics.

 From System Load averages, Kernel and I/O stuff, Network traffic stats 
to mail related graphs and much more. Monitorix is custom tailored for 
RHEL and clones and still under active development, so it'll do. Even if 
its look is a bit dated.

By default it shows the last 24h of activity, but if you click on the 
bent arrow icon in the upper right corner of the Monitorix page and open 
it up in a separate window you'll get the see a selector that allows you 
to specify different date ranges as well as types of statistics to show.


Please note that both GoAccess and Monitorix only will show data that 
was aggregated since the services came aboard via a YUM update (or on a 
fresh BlueOnyx install).

"Active Monitor" has also been extended with something called "Usage 
Information Services", which will monitor the two new daemons related to 
the statistics gathering and presentation: "monitorix" and "goaccess".

So if these services fail, then "Active Monitor" will restart them 
automatically.


GDPR/DSGVO conformity (for the EU folks):
==========================================

It's the same as before: The real time statistics aren't anonymized, but 
anything that is archived will have the IP address partially anonymized 
and the stored data will be expired at the chosen expiry date that you 
have configured under "System Settings" / "Data Retention".

Further plans:
===============

As said: I'll backport this to BlueOnyx 5210R as well. Gimme a few days 
to sort this out, as the changes were extensive and backporting from the 
new GUI to the old one feels like potty training a baby-monkey. :p

-- 
With best regards

Michael Stauber