[BlueOnyx:06723] Re: [bluequartz] Re: cced gone wild

[Chris Gebhardt - VIRTBIZ Internet]

Jeffrey Pellin wrote:
> This might not be the same thing but we had a similar problem with lots of
> processes bringing down one of our  BO v-servers. 
> 
> Our Aventurine box is with Chris over at Virtbiz and it appears that one of
> his excellent team nailed the problem, as we haven't had an issue for some
> time. I never thought to ask Chris what it was.
> 
> If it's the same thing maybe he'll post.

Hi Jeffrey,
Ah yes, this would be our old Albatros, right?  :)  I just had a quick 
look back at the history on this one and it looks like it wasn't exactly 
the same issue.  On your box it was Apache instead of cced.init.  What 
would happen is that Apache would get hammered and then get very out of 
sorts to the point that it begins to block network traffic for the 
entire host (not just the VPS).  Restarting Apache on the affected VPS 
seems to cure the symptom.

We tightened up some web security and that seemed to bring the number 
incidents down significantly but not entirely.  We also loaded in a 
quick script to watch for network to fail and if it does, issue a 
restart for Apache.  It looks like that restart is happening about once 
every 1.5 days or so.   An imperfect solution, to be sure, but the 
system continues to run.

As a failsafe, we've also got the physical system monitored and if 
network goes AWOL on it for more than 5 minutes the box gets 
power-cycled by the PDU.  The 5 minute window leaves time for things 
like intentional reboots.

I don't think the above will help Rashid with his issue, but since you 
asked... I told.  ;)

-- 
Chris Gebhardt
VIRTBIZ Internet Services
Access, Web Hosting, Colocation, Dedicated
www.virtbiz.com | toll-free (866) 4 VIRTBIZ