[BlueOnyx:06641] Re: Server processes slowly grow, admin GUI down, cced hangs on shutdown

[Darrell D. Mobley]

> -----Original Message-----
> From: blueonyx-bounces at blueonyx.it [mailto:blueonyx-bounces at blueonyx.it]
> On Behalf Of Jeff Jones
> Sent: Tuesday, March 08, 2011 9:43 AM
> To: BlueOnyx General Mailing List
> Subject: [BlueOnyx:06638] Server processes slowly grow, admin GUI down,
> cced hangs on shutdown
> 
> Hi all,
> 
> A while back I reported an issue of a BX server that mysteriously hung in
> the night - all websites down, basically ran out of RAM:
> 
> http://www.blueonyx.it/pipermail/blueonyx/2010-September/005484.html
> 
> The only thing I could see beforehand that was fishy, was that the number
> of processes running on the box suddenly started to increase, by 1 or 2 -
> every 15 mins - until the server RAM went.

It's gremlins, I tell ya.  I have a BX server running on an old powerhog, an
HP DL380 G3, which runs fine until a new kernel comes in through YUM and I
edit grub.conf to use it.  When the reserver reboots under the new kernel
(which is now default=0), sometime around 4AM that night my server will
simply stop responding to anything, except pings. No SSH, HTTP, FTP, nothing
(useful) can access it.

I have the HP health scripts with the watchdog timer running, but since the
server isn't locked up, it just keeps sitting there until I wake up, hit the
IP-based power switch and force a hard reboot.  Not good for the SCSI
drives, but there is nothing else I can do (short of buying different
equipment) to determine why it does this, since it is remote to me.