[BlueOnyx:20543] Re: GUI Admserv Segfault (5209R) ** sort of solved **

[Michael Stauber]

Hi all,

Just to give some feedback to the list on the problem with AdmServ on
5209R that Michael Aronoff reported.

I spent two days digging around on the box and failed to identify the
exact source of the problem. Which is pretty impressive.

Symptoms:
=========

Upon login to the GUI (both on port 444 and 81) the GUI would throw a
fit and the browser would show connection loss. /var/log/adm_error
reported several instances of this:

[core:notice] [pid 9902] AH00052: child pid 9906 exit signal
Segmentation fault (11)

Like said: The problem happend with SSL and without. But not on every
PHP page that AdmServ served. A simple phpinfo() or "hello world" page?
Worked fine. The "404 Page Not Found" that is routed through CodeIgniter
as well? Worked fine.

In essence: Any GUI page that required a login with username and
password (or username and valid SessionID) would throw a segmentation fault.

The box was fairly standard with no unusual software or modifications
present. A few PKGs from the shop were present and that was that. Verify
of the RPM database? Nothing stuck out as being modified beyond what one
would expect with the software that was present.

The only unusual part was that near the event of the "segmentation
fault" or during a "systemctl restart admserv" we would sporadically see
a Systemd Policy Toolkit violation in /var/log/messages.

After throwing the entire book at the box for two days I still was not
one step closer to finding cause or fix.

Fix (rather a work around):
============================

Then I did a "yum reinstall" of all 1470 RPMs that were available via
YUM (which excluded any RPMs that came with Shop PKGs). Which is pretty
drastic and was a last ditch effort before recommending an OS restore.

That solved the problem. Another "rpm -Va" and a compare between that
and the one before the "yum reinstall" turned up nothing. Which leads to
the only sensible conclusion that something related to Systemd broke and
whatever directory, file, symlik or permission broke was is not directly
"owned" by (or attributed to) an RPM. Such as a directory, symlink or
file that's created (and perhaps chowned) during a RPM install without
noting in the RPM database that a certain RPM owns it.

If someone runs into a similar problem of AdmServ crashing on a 5209R
with "Segmentation fault (11)" reported in /var/log/admserv/adm_error,
then please let me know so that I can further investigate this problem.

-- 
With best regards

Michael Stauber