[BlueOnyx:14475] Re: updatedb strange behavior

[Michael Stauber]

Hi Dirk,

>> 2.) Verify the integrity of your PS command via "rpm -V procps". 
>> It should come back blank without flagging any of the files that this RPM contains.
> 
> rpm -V procps
> S.5....T.    /bin/ps

Ok, so /bin/ps has been replaced. Probably with a trojaned version that
hides the suspicious processes.

>> 4.) Use "lsof" to see which files, pipes, devices and/or sockets are 
>> held open by the process with the PID you noted:
> 
> lsof -nl|grep 19806
> updatedb  19806        0  cwd       DIR              253,0      4096          2 /
> updatedb  19806        0  rtd       DIR              253,0      4096          2 /
> updatedb  19806        0  txt       REG              253,0    592464      67098 /root/gpu/updatedb (deleted)
> updatedb  19806        0  mem       REG              253,0    156928     303158 /lib64/ld-2.12.so
> updatedb  19806        0  mem       REG              253,0   1926800     303663 /lib64/libc-2.12.so
> updatedb  19806        0  mem       REG              253,0    145896     305799 /lib64/libpthread-2.12.so
> updatedb  19806        0  mem       REG              253,0     47064     306403 /lib64/librt-2.12.so
> updatedb  19806        0    0r      CHR                1,3       0t0       3662 /dev/null
> updatedb  19806        0    1w      CHR                1,3       0t0       3662 /dev/null
> updatedb  19806        0    2w      CHR                1,3       0t0       3662 /dev/null
> updatedb  19806        0    3u     IPv4          273202025       0t0        TCP <myip>:57006->184.106.96.142:domain (ESTABLISHED)

Dead give away:

/root/gpu/updatedb (deleted)

There is a process running, but the program which launched that process
has been deleted? That's *really* fishy. Programs that behave nice don't
do this.

Also, I don't recognize the directory /root/gpu/ as legitimate. Might be
worth to check out what else is in that directory.

> TCP <myip>:57006->184.106.96.142:domain (ESTABLISHED)

Why is that deleted program (which is still running) holding open a TCP
connection to port 53 on the IP 184.106.96.142?

I'd suggest to fire up tcpdump to examine the network traffic related to
that fishy IP:

tcpdump -i eth0 -n|grep 184.106.96.142

Replace eth0 with the network interface that faces to the internet.
Might be venet0 in an OpenVZ VPS.

>> 5.) If lsof says it's the same /usr/bin/updatedb, we verify its integrity as well:
> 
> rpm -q --whatprovides /usr/bin/updatedb
> mlocate-0.22.2-4.el6.x86_64

Looks good.

>> 6.) You also might want to check the directory /proc/<PID>/
> 
> ls -la /proc/13816
> lrwxrwxrwx   1 root root 0 Feb  7 07:50 exe -> /root/gpu/updatedb (deleted)

Same as above. It tells us where the program responsible for that
process was located. And that it's no longer there.

So we have two conclusions here:

1.) System binaries have been replaced, and the attacker had stuff
launched out of /root/gpu/ - for both he needed "root" access.

2.) The box is holding up a network connection to 184.106.96.142, which
is a "Rackspace Hosting" IP. They're in San Antonio, Texas.

I'd call that one up as a total loss which needs reinstallation. That's
of course the official "party line", because once a box has been rooted
you never know what kind of easter eggs you might overlook even after a
very thorough cleaning.

Sorry for the bad news, Dirk.

-- 
With best regards

Michael Stauber