[BlueOnyx:07073] Re: cced gone wild

[Jeff Jones]

Hi all,

Just had another occurrence of this mysterious cced hang problem (as documented in my early mail here) this morning - this time solved it without a hardware reset.

First check running processes for httpd and cced

ps ax | grep http | awk '{print $1}' 
ps ax | grep cced | awk '{print $1}' 

What's interesting with this is that there are only a few cced processes running, although a large number of httpd.

So first I just try to kill all the http ones in one go:

ps ax | grep http | awk '{print $1}' | xargs kill

That kills all the httpd. However I am still unable to stop cced.

I now have to go:

kill -9 (cced pid)

That kills cced dead.

Now it's just a case of:

Start ceed.init
Restart admin server
Start httpd

Done - server back and processes now dropped from 150 back to 100. 

So now I think this might be a case of httpd hanging cced - rather than the other way round?

Perhaps a 'bad app' ?

No idea why it appears to occur so randomly thought..... the last event was 1 month ago....

Cheers,

Jeff




On 16 Mar 2011, at 22:40, Jeff Jones wrote:

> Hi Rashid,
> 
> I think this may be a similar issue to mine.
> 
> For me - the following occurs:
> 
> Steady, linear rise in new processes created on the server, 2 every 5 minutes
> Admin GUI stops working
> Restarting AdmSrv doesn't get GUI working
> Websites continue to work until RAM on server is exhausted
> Unable to restart HTTPD
> Unable to restart cced
> Unable to restart server
> 
> The only way to recover is a physical power cycle. 
> 
> I only have one BX server that has this problem, and it occurs about once every 6 months. 
> 
> I have set up an SNMP monitoring alert on system processes that alerts me when the number of processes rises above normal - in this way I can manually intervene and restart the box at night in a 'controlled' manner. 
> 
> But I would really like some tips on how to investigate this problem further in order to find the culprit.....
> 
> Cheers,
> 
> Jeff
> 
> On 16 Mar 2011, at 22:18, MuntadaNet Webmaster wrote:
> 
>> On one of my servers, cced keeps spawning more and more 
>> processes.  Can someone give me an idea of what to look for to 
>> prevent this?  It is causing a Denial of Service eventually in which 
>> the only recovery is cycling the power.
>> 
>> Regards,
>> 
>> Rashid
>> 
>> _______________________________________________
>> Blueonyx mailing list
>> Blueonyx at blueonyx.it
>> http://www.blueonyx.it/mailman/listinfo/blueonyx
> 
> _______________________________________________
> Blueonyx mailing list
> Blueonyx at blueonyx.it
> http://www.blueonyx.it/mailman/listinfo/blueonyx