[BlueOnyx:02316] Re: Protection Against Root Toolkit Attack

[Michael Stauber]

Hi Phil,

> Does anyone think that a Root Toolkit Detection system would be beneficial
> on a BO server?

I think there is no good answer to that question that suits everybody.

But in my opinion the typical implementations of Chkrootkit, Portsentry and 
similar tools very often a waste of time and CPU cycles. It's like fighting 
todays threats with the tools of 2001. Or in other words: It's inadequate.

At least in an automated fashion. Nothing speaks against running rootkit 
detectors manually from time to time. But especially people with multiple 
servers will get tired of the daily emails and frequent false alarms or the 
information overload that these tend to generate. Up to the point where you 
simply overlook the one email that should have gotten your attention, because 
it reported a true and sucessful attack.

> I am looking at Rkdet, chkrootkit, Tripwire, psionic and similar software
> one of which should run ok with a little configuration.

I don't know Rkdet, but over the years I had a lot of (mixed) experience with 
Tripwire. I'm running it on my border router, so it sees all traffic that hits 
the pipe facing the outside world. Finding the right mix of Tripwire rules and 
tresholds that suits your specific environment is almost an art. I'm down to 
like 10-15 false alarms a day now - after years of fine tuning. There is a set 
of rules that I could drop and would get rid of most false alarms, but that 
would also prevent the detection of some stuff that I'd like to keep on top 
of. 

I certainly would NOT build Tripwire into the standard BlueOnyx distribution, 
because it would be a support nightmare. Everyone would come with exceptions 
to the rules that need to be built in to cater his specific needs. Which in 
turn would make the GUI for it overly complex. I also don't want to fall into 
the same pits that Cobalt did when they built their security stuff into the 
RaQ550 GUI. Some here may remember it. It was pretty much useless. Partially 
due to horrible design and implementation flaws (it scanned outbound traffic 
only due to a mixup in the code!), partially because due to the "appliance 
idea" they made the GUI for it too simplicistic to really allow you to adjust 
it to your specific needs. Imperfect tools can create a false sense of 
security and then do more harm then good.

> I have been told (although I can not verify just how true that it is) that
> a good packet sniffer could possibly build a set of usernames and passwords
> for a linux system.

Well, "sniffing" has always been an issue and always will be. But in order to 
"sniff" someone needs to have "an ear on the track". Like he must have the 
ability to tie into your network traffic. Like having root access to one of 
the boxes in your network segment. If that's the case, then the child has 
already fallen into the well. Or the attacker must sit somewhere between you 
and the person(s) that frequently login to the server through unencrypted 
protocols (POP3, IMAP, SMTP, FTP, etc.).

In that case he can more or less easily sniff packets and can gather the 
usernames and passwords. A good start there is the emphasis on using SSL 
whenever possible. All network services support it - more or less.

Intelligent network design can also prevent some if it. Like don't put all 
eggs in one basket and have all client boxes in the same network segment 
(switched or not).

I've also seen that large ISPs run custom kernels which prevent you from using 
sniffers as they hacked the kernel in a way that it can no longer switch the 
network cards into promiscuous mode through the usual means.

Or there is OpenVZ, Virtuozzo or Aventurin{e}, where the virtualization layer 
prevents anyone (including "root") in a VPS to see any network traffic but the 
one that's relevant to him.

None of that provides an all around protection, but it can help.

> I already run fail2ban and use the inbuilt Pam intrusion protection but
> guess that would not afford protection against a root toolkit attack.

Then that's already a good start. You have to keep in mind that rootkits don't 
infect a system out of the blue. It has been a while since a daemon was found 
that had a network exploitable vulnerability that - if exploited - instantly 
granted you "root" access. The typical exploits you see these days involve 
multiple steps. During the first step the attacker gains unprivileged shell 
access. Through a vulnerable web application (PHP or Perl script) for example. 
Or by brute force password guessing attempts. To some degree BlueOnyx already 
protects against these due to the PHP security settings, which limit the 
damage that an attacker can do or which prevent remote code inclusion. Or 
there is the new brute force login attempt blocker, which blocks IPs after 
repeated failed login attempts.

Still, if someone gets past that layer and gets unprivileged local access, 
then that's bad. But he still doesn't have root access yet. So he needs to 
find another locally exploitable vulnerability to get to that point. If you 
don't allow shell access to your boxes, then the attacker is furtherly 
hindered. He'll have to upload exploit code through FTP and has to hope that 
the browser allows him to run the commands he needs for the task. The default 
PHP security settings for sites will certainly p*ss him off quite a bit, but 
it's an obstacle that can be overcome with some efforts. So in oder to load 
the rootkit, the attacker needs to find that vulnerability that allows him to 
gain root access first. The rootkit is then used to hide the intrusion and to 
mask the changes made to the system and to cloak the hackers presence in the 
system.

Personally I haven't seen a thoroughly hacked (i.e.: "root" access was 
established) BlueOnyx yet, but I've heard of two cases. However, when I look 
at past experience with hacked boxes (I used to be frequently asked to do 
post-mortem forensics on hacked boxes or give an estimate if it was 
salvageable), then I can say this:

Over the last few years it appears that both BlueQuartz (and BlueOnyx) have 
been subject to the usual (but declining) number of unauthorized and 
unprivileged non-root accesses. Typically for the purpose of sending SPAM or 
phishing emails. The drop in numbers there can be explained with the fact that 
most of this SPAM is nowadays sent through bot-nets of infected home user PCs, 
which are a much softer and easier target. Also the implementation of SMTP-
Auth a few years ago made it easy for most admins to make sure that they 
unintentionally turned their boxes into open relays. Of course that's a 
different topic as exploiting that isn't a hack in the traditional sense.

But the number of root-compromises or hostile takeovers of servers has dropped 
considerably. At least in our small and concealed world of BlueQuartz and 
BlueOnyx. On the Cobalts it was MUCH worse during their days of glory. Part of 
this can be attributed to the OS we use now, which is more modern and better 
maintained (although CentOS really slacked a lot with CentOS4!). Part of it 
may be a tribute to the limited "spread" that BlueQuartz and BlueOnyx servers 
have when you compare the numbers with the enourmous spread that Cobalt boxes 
had in their days of old, or with todays market share of other target groups 
such as Plesk, Ensim or Cpanel. But it may also be a tribute to dropping 
prices for root servers.

A few years ago you had to pay like $250-500 US a month for a well connected 
data center hosted server - when this used to be a lot of money. And typically 
such contracts had a duration of one year.  

Nowadays you can get a small VPS with root access for the price of a pizza and 
you can cancel that contract pretty much at any time or to the end of the 
month at the latest. Plus the average home user now has access to bandwiths 
which - a few years ago - were only available in data centers or through 
expensive dedicated lines.

Don't get me wrong: I don't want to say that there is no threat, nothing to 
see and move along please <grin>. Not at all. Some well designed extra 
security, an investment in network security and proper network design and 
being prepared and vigilant is always good and recommended. Especially if you 
host websites which - for political or practical reasons - can be considered 
"high value targets". 

But the threats have changed and a lot of recommendations you find on the net 
in regards to security may no longer be really adequate. They should still be 
taken into some sort of consideration, but don't use them as main line of 
defense. In a recent (private) security discussion someone praised his 
implementation of Portsentry and I though to myself: "Dude, you just 
disqualified yourself." :o)

-- 
With best regards

Michael Stauber