[BlueOnyx:14476] Re: updatedb strange behavior

[Dirk Estenfeld]

Michael,

we figured it out.

It seems like it was an attack over DNS (Port 53) to get an overflow and gain root access.
Afterward ps was replaced and an application for generating bitcoins (I am not the technician here so my explanation will be not very detailed) was started.
We have not reinstalled procps and coreutils and are now blocking port 53 in the firerwall.
Additional we have permanently blocked ip 184.106.96.142 in the firewalls.

We will have an eye on that.

Regards,
Dirk

-----------------------------------------------
Black Point Arts Internet Solutions GmbH - Hanauer Landstrasse 423a - 60314 Frankfurt


-----Ursprüngliche Nachricht-----
Von: blueonyx-bounces at mail.blueonyx.it [mailto:blueonyx-bounces at mail.blueonyx.it] Im Auftrag von Michael Stauber
Gesendet: Freitag, 7. Februar 2014 16:02
An: BlueOnyx General Mailing List
Betreff: [BlueOnyx:14475] Re: updatedb strange behavior

Hi Dirk,

>> 2.) Verify the integrity of your PS command via "rpm -V procps". 
>> It should come back blank without flagging any of the files that this RPM contains.
> 
> rpm -V procps
> S.5....T.    /bin/ps

Ok, so /bin/ps has been replaced. Probably with a trojaned version that hides the suspicious processes.

>> 4.) Use "lsof" to see which files, pipes, devices and/or sockets are 
>> held open by the process with the PID you noted:
> 
> lsof -nl|grep 19806
> updatedb  19806        0  cwd       DIR              253,0      4096          2 /
> updatedb  19806        0  rtd       DIR              253,0      4096          2 /
> updatedb  19806        0  txt       REG              253,0    592464      67098 /root/gpu/updatedb (deleted)
> updatedb  19806        0  mem       REG              253,0    156928     303158 /lib64/ld-2.12.so
> updatedb  19806        0  mem       REG              253,0   1926800     303663 /lib64/libc-2.12.so
> updatedb  19806        0  mem       REG              253,0    145896     305799 /lib64/libpthread-2.12.so
> updatedb  19806        0  mem       REG              253,0     47064     306403 /lib64/librt-2.12.so
> updatedb  19806        0    0r      CHR                1,3       0t0       3662 /dev/null
> updatedb  19806        0    1w      CHR                1,3       0t0       3662 /dev/null
> updatedb  19806        0    2w      CHR                1,3       0t0       3662 /dev/null
> updatedb  19806        0    3u     IPv4          273202025       0t0        TCP <myip>:57006->184.106.96.142:domain (ESTABLISHED)

Dead give away:

/root/gpu/updatedb (deleted)

There is a process running, but the program which launched that process has been deleted? That's *really* fishy. Programs that behave nice don't do this.

Also, I don't recognize the directory /root/gpu/ as legitimate. Might be worth to check out what else is in that directory.

> TCP <myip>:57006->184.106.96.142:domain (ESTABLISHED)

Why is that deleted program (which is still running) holding open a TCP connection to port 53 on the IP 184.106.96.142?

I'd suggest to fire up tcpdump to examine the network traffic related to that fishy IP:

tcpdump -i eth0 -n|grep 184.106.96.142

Replace eth0 with the network interface that faces to the internet.
Might be venet0 in an OpenVZ VPS.

>> 5.) If lsof says it's the same /usr/bin/updatedb, we verify its integrity as well:
> 
> rpm -q --whatprovides /usr/bin/updatedb
> mlocate-0.22.2-4.el6.x86_64

Looks good.

>> 6.) You also might want to check the directory /proc/<PID>/
> 
> ls -la /proc/13816
> lrwxrwxrwx   1 root root 0 Feb  7 07:50 exe -> /root/gpu/updatedb (deleted)

Same as above. It tells us where the program responsible for that process was located. And that it's no longer there.

So we have two conclusions here:

1.) System binaries have been replaced, and the attacker had stuff launched out of /root/gpu/ - for both he needed "root" access.

2.) The box is holding up a network connection to 184.106.96.142, which is a "Rackspace Hosting" IP. They're in San Antonio, Texas.

I'd call that one up as a total loss which needs reinstallation. That's of course the official "party line", because once a box has been rooted you never know what kind of easter eggs you might overlook even after a very thorough cleaning.

Sorry for the bad news, Dirk.

--
With best regards

Michael Stauber
_______________________________________________
Blueonyx mailing list
Blueonyx at mail.blueonyx.it
http://mail.blueonyx.it/mailman/listinfo/blueonyx