[BlueOnyx:06724] Re: [bluequartz] Re: cced gone wild

[Jeffrey Pellin]

Hi Chris - you're right I did ask!

BTW a solution that means both we and our clients are completely unaware of
any issues or service problems sounds pretty damn good to me and far from
imperfect.

I'm sorry Rashid - but maybe the techniques that Chris used to keep our
server up might help.

Regards

Jeffrey


On Thu, 17 Mar 2011 08:11:38 -0500, Chris Gebhardt - VIRTBIZ Internet
<cobaltfacts at virtbiz.com> wrote:
> Jeffrey Pellin wrote:
>> This might not be the same thing but we had a similar problem with lots
>> of
>> processes bringing down one of our  BO v-servers. 
>> 
>> Our Aventurine box is with Chris over at Virtbiz and it appears that one
>> of
>> his excellent team nailed the problem, as we haven't had an issue for
>> some
>> time. I never thought to ask Chris what it was.
>> 
>> If it's the same thing maybe he'll post.
> 
> Hi Jeffrey,
> Ah yes, this would be our old Albatros, right?  :)  I just had a quick 
> look back at the history on this one and it looks like it wasn't exactly 
> the same issue.  On your box it was Apache instead of cced.init.  What 
> would happen is that Apache would get hammered and then get very out of 
> sorts to the point that it begins to block network traffic for the 
> entire host (not just the VPS).  Restarting Apache on the affected VPS 
> seems to cure the symptom.
> 
> We tightened up some web security and that seemed to bring the number 
> incidents down significantly but not entirely.  We also loaded in a 
> quick script to watch for network to fail and if it does, issue a 
> restart for Apache.  It looks like that restart is happening about once 
> every 1.5 days or so.   An imperfect solution, to be sure, but the 
> system continues to run.
> 
> As a failsafe, we've also got the physical system monitored and if 
> network goes AWOL on it for more than 5 minutes the box gets 
> power-cycled by the PDU.  The 5 minute window leaves time for things 
> like intentional reboots.
> 
> I don't think the above will help Rashid with his issue, but since you 
> asked... I told.  ;)