[BlueOnyx:12261] Re: Kernel 0-day vulnerability + SSHd Spam Exploit (libkeyutils.so.1.9)

[Michael Stauber]

Hi Dan and all,

After a few hours of reading up on various sources I think I finally
managed to wrap my head around this. This will be a long post with a lot
of details. However, I'll summarize the important conclusions from the
end of this message in the very beginning, too:

The bottom line is this:

1.) Current Linux kernel on RHEL 5 and 6 clones are vulnerable to a
privilege escalation performed by a local user (or process). Updated
kernels should hit the official vendor repositories soon. ETA unknown,
though.

2.) SSHd Spam Exploit (libkeyutils.so.1.9): Aside from that there is
currently an ongoing attack which started sometime in December 2012 and
which seems to primarily target boxes with control panels. This attack
modifies OpenSSH's behavior via a manually supplemented
libkeyutils.so.1.9 library, which allows the attacker to login via SSH
and to perform commands on the trojaned servers. It appears that this
trojan also sends your "root" password to a certain collector box and
that hacked servers are mostly used to send SPAM. The entry vector how
this infection is/was carried out is yet unclear. It affects various
OS's (RHEL6 clones) of different versions and various boxes with or
without control panels.

3.) In the absence of more solid information a lot of band-aids are
currently being suggested. Like firewalling SSH and only allowing access
from certain IP address ranges, or switching entirely to key based SSH
logins. Which are good and sensible precautions to begin with. However,
it doesn't fix the underlying problems, of which at least one is yet
unknown.

After posting the conclusions first, here is how they were obtained:

a.) The topic of this thread and the one in WebHostingTalk (WHT) is a
bit misleading. Yes, there is a recently discovered kernel vulnerability:

CVE-2013-0871
https://bugzilla.redhat.com/show_bug.cgi?id=911937
https://access.redhat.com/security/cve/CVE-2013-0871

Race condition in the ptrace functionality in the Linux kernel before
3.7.5 allows local users to gain privileges via a PTRACE_SETREGS ptrace
system call in a crafted application, as demonstrated by ptrace_death.

This bug affects all kernels shipped with Red Hat Enterprise Linux 5, 6,
Fedora Core and clones such as CentOS, Scientific Linux or Cloud Linux.

The scope of the vulnerability is local. Means: An unprivileged user
that already has local access to the box can gain root access through a
complicated ptrace system call. Multi CPU boxes and timing issues
probably make this a hell of a lot more complicated to exploit.

b.) The WHT forums are a mess and always have been. The discussion about
this topic (see http://www.webhostingtalk.com/showpost.php?p=8562898) is
well beyond +50 pages now. There are some useful contributions to it,
but the majority of the posts are useless repetitions or nitpickings
about irrelevant details or wrong approaches.

- If a server is hacked and the attacker gained "root" access, there is
only one remedy: OS restore, fully patch, don't trust any data on the
old box and start fresh.

- A lot of the discussion on WHT is about how to "clean" a compromised
box again, suggestion deletion of a library and fixing a symlink. Which
is applying a band aid to an owned box which will get rooted again via
the same approach vector.

- What the attack mentioned in WHT does is this:

http://www.webhostingtalk.com/showpost.php?p=8564042&postcount=329

Boxes with control panels that have been affected: cPanel, DirectAdmin,
and Plesk, but I personally know of at least one BlueOnyx box that was
most likely affected as well. I don't have access to that box anymore to
verify this.

The attacker can login via SSH to the box in a fashion that it will
temporarily disable logging via syslog, __syslog_chk,
audit_log_user_message, and audit_log_acct_message. The write() hook
will also prevent logging via stderr. However, if the log level
verbosity is raised to the maximum, then it'll show these logins as well.

The common denominator is that most of the compromised boxes have been
used for sending SPAM. When this happens, lots of SSH logins (which
won't show up in the logs) are made and via remote command execution via
SSH the emails are sent. While this attack is ongoing (i.e.: While the
SSH connections happen) the process list and "lsof" might shed some
light on this.

However: In order to get to this point the attacker would already need
"root" permission to substitute a library (/lib64/libkeyutils.so.1.9 or
/lib/libkeyutils.so.1.9) that messes with SSH. Just the presence of this
library doesn't spell doom, as there is a legitimate RPM that - in rare
cases - might be installed and which brings a "good" version of that
file aboard.

A good reading is this blog page by John Williams, who sourced the crux
of the info from WHT and supplemented it with his own findings:

http://blog.solidshellsecurity.com/2013/02/18/0day-linuxcentos-sshd-spam-exploit-libkeyutils-so-1-9/

c.) Entry vector for this attack:

As of now it is not 100% clear how the attacker(s) got access first. Did
they first get unprivileged local access and then used the Kernel
vulnerability CVE-2013-0871 to gain root access? Or is there another
locally exploitable vulnerability that allows privilege escalation?

At this time there is a lot of speculation going on about what the entry
vector could be.

This could be PHP related (which wouldn't surprise anyone). It could be
a vulnerable daemon or (can't be ruled out entirely) even a problem with
OpenSSH itself. While this is darn unlikely, it can't be ruled out
entirely yet.

There is even speculation about compromised credentials via the admin's
PC. Which doesn't sound too far fetched if you figure in all the *very*
serious Adobe and Java related vulnerabilities of the recent weeks and
months.

The bottom line is this:
========================

1.) Current Linux kernel on RHEL 5 and 6 clones are vulnerable to a
privilege escalation performed by a local user (or process). Updated
kernels should hit the official vendor repositories soon. ETA unknown,
though.

2.) SSHd Spam Exploit (libkeyutils.so.1.9): Aside from that there is
currently an ongoing attack which started sometime in December 2012 and
which seems to primarily target boxes with control panels. This attack
modifies OpenSSH's behavior via a manually supplemented
libkeyutils.so.1.9 library, which allows the attacker to login via SSH
and to perform commands on the trojaned servers. It appears that this
trojan also sends your "root" password to a certain collector box and
that hacked servers are mostly used to send SPAM. The entry vector how
this infection is/was carried out is yet unclear. It affects various
OS's (RHEL6 clones) of different versions and various boxes with or
without control panels.

3.) In the absence of more solid information a lot of band-aids are
currently being suggested. Like firewalling SSH and only allowing access
from certain IP address ranges, or switching entirely to key based SSH
logins. Which are good and sensible precautions to begin with. However,
it doesn't fix the underlying problems, of which at least one is yet
unknown.

---

Lastly: I'll watch WHT and other sources for more information about this
and will post updates here if there is a major breakthrough.

If you suspect that your BlueOnyx server is sending SPAMs and has either
/lib64/libkeyutils.so.1.9 or /lib/libkeyutils.so.1.9 present, then
please contact me offlist (!) and allow me to take a look at the box.

-- 
With best regards

Michael Stauber