[BlueOnyx:06638] Server processes slowly grow, admin GUI down, cced hangs on shutdown

[Jeff Jones]

Hi all,

A while back I reported an issue of a BX server that mysteriously hung in the night - all websites down, basically ran out of RAM:

http://www.blueonyx.it/pipermail/blueonyx/2010-September/005484.html

The only thing I could see beforehand that was fishy, was that the number of processes running on the box suddenly started to increase, by 1 or 2 - every 15 mins - until the server RAM went.

So what I did was stick a monitor on the number of processes, and if it went over 'X' in the future, to send me an alert.

Well - I just got my first high process alert!

The first thing I noticed was that the admserv was not working - I was unable to get into GUI admin. I restarted admsrv, but no dice.
All the websites on the server were fine - this is the same as last time, the sites only fail when the number of processes gets so high that the server runs out of ram.
I tried to restart apache, but that hung.
On a different SSH session, I then attempted a reboot - but the reboot hung when trying to kill cced. 

So there was something majorly wrong with the server.

After the hard reset - it rebooted - and fine again.

It's disappointing that although I can now detect this issue very early on - I still need to perform a hard reset again to 'fix' it. I guess the bonus now is that I get early warning, so I don't get woken up at 3am!!!

So I was wondering - are there any other ideas as to why this might be happening? 

Any suggestions for logs etc to look at?

Cheers,

Jeff

Jeff Jones
jeffrhysjones at mac.com