[BlueOnyx:03716] Re: Possible breach of server - assistance appreciated

[Stephanie Sullivan]

Just off the top of my head the following comes to mind:

Do you have an iptables firewall setup? If not you might want to try
blocking outgoing connections to unpriviledged ports from unpriviledged
ports. I'm not sure what BX (aka BO) has built-in that does these kinds of
connections, if anything.

Also, by default deny outgoing connections to priviledged ports except those
you would need to access - like UDP on port 53 for DNS, for example or ports
20/21 for ftp and maybe port 22 for ssh outgoing. If you log the blocked
connections you'll be able to establish a time related pattern and catch the
culprit with a netstat -avpn command.

I don't know how many sites you have on this server but looking for bogus
files - I've seen instances of vulnerabilities in CMS and shopping carts
(zen for instance) that would drop various php files into folders. Not nice!
I've mostly seen .php files with some random or weird seeming names.

Consider installing and running the rootkit checker from
http://www.chkrootkit.org/. 

To be really sure you might consider rolling the sites off with cmuExport,
reinstalling the BX iso, then rolling the sites back with cmuImport. You
might also save a copy of your /etc directory just in case. When you
re-import be sure the check for errant or vulnerable php, perl or python
apps. Most likely culprits are probably CMS, blogs, or shopping carts of
older versions. Joomla has lots of reports on security focus. Drual has far
less. Wordpress has regular updates and who knows what other web apps might
have old versions or vulnerabilities.

I use php_safe_mode on and use safe_mode_gid on  too to mitigate some of the
safe_mode headaches. Keeps a number of issues more under control than
running without. With careful safe_mode configuring wordpress and zen cart
(for example) run just fine as does drupal. I'd give details for each here
but I do it rarely enough that I just figure it out again each time. Not all
that hard.

So, to summarize: restrict outgoing connections to mitigate the effect of
your infection on other systems while you troubleshoot, check for errant web
files, check for old versions of CMS, cart or blog software, run a root kit
checker, if that fails, rebuild a new server and import your sites.

Considerinig you are in a university, can I assume you have students or
facilty with accounts? If so, don't discount "misguided" student or even
faculty member with a legit account... think "outside the box"... hackers
do.

Oh yes, you might want to change all your server admin passwords (something
strong please!) and consider (if you have a small user population) forcing
everyone to get a new strong password. Toward that end you might want to run
a password strength checker on your /etc/shadow file to look for really
crummy passwords. For exmaple: http://www.openwall.com/john/

Hope this counts as a point in the right direction. Hard to be more specific
with just some port scan output.

	Thanks,
		-Stephanie



> -----Original Message-----
> From: blueonyx-bounces at blueonyx.it [mailto:blueonyx-
> bounces at blueonyx.it] On Behalf Of Richard Morgan
> Sent: Sunday, February 28, 2010 6:31 PM
> To: BlueOnyx General Mailing List
> Subject: [BlueOnyx:03715] Possible breach of server - assistance
> appreciated
> 
> Hi
> 
> I hope someone can point me in the right direction because this is a
> little
> stressful (to say the least).
> 
> We've been sent a message, via the abuse team in the data centre we
> use,
> that indicates port scans or something malicious is originating from
> our
> server.  It's actually the third time now.
> 
> The message is below, but my problem is I don't have enough
> familarity with
> Linux security to dig in and know what I'm looking for.  Therefore
> any
> pointers in the right direction would be appreciated.
> 
> Thanks, Richard
> 
> The message received was as follows:
> --------------------------------------
> 
> This is an automated message from Columbia University IT Security.
> You are
> receiving it because you are listed as the abuse contact for the
> machine
> referred to below.  This machine attempted to gain unauthorized
> access to one or more machines at Columbia University.
> 
> Details are provided below.  Please take all necessary
> steps to mitigate such attacks.
> 
> If you have received this message in error, or if this incident
> reported is inappropriate, please contact security at columbia.edu
> so that we can update our procedures.  Please include the entire
> body of this message.
> 
> Thank You.
> 
> Columbia University IT Security
> security at columbia.edu
> 
> Name:     s1.XXXXXX.net
> Address:  217.112.XX.XX
> 
> Incident type: 6000/tcp
> First attempt: 27-feb-2010 05:50:00 GMT-0500
> Last attempt: 27-feb-2010 05:50:00 GMT-0500
> Total attempts: 131061
> 
> 27-Feb-2010 05:51:46 GMT-0500 217.112.XX.XX:62518 ->
> 156.111.227.14:6000
> 6 48
> 27-Feb-2010 05:51:46 GMT-0500 217.112.XX.XX:62518 ->
> 156.111.227.40:6000
> 6 48
> 27-Feb-2010 05:51:46 GMT-0500 217.112.XX.XX:62518 ->
> 156.111.227.45:6000
> 6 48
> 27-Feb-2010 05:51:46 GMT-0500 217.112.XX.XX:62518 ->
> 156.111.227.46:6000
> 6 48
> 27-Feb-2010 05:51:46 GMT-0500 217.112.XX.XX:62518 ->
> 156.111.227.48:6000
> 6 48
> 27-Feb-2010 05:51:46 GMT-0500 217.112.XX.XX:62518 ->
> 156.111.227.81:6000
> 6 48
> 27-Feb-2010 05:51:46 GMT-0500 217.112.XX.XX:62518 ->
> 156.111.227.67:6000
> 6 48
> 
> There was lots more, all of a simalar nature.
> 
> _______________________________________________
> Blueonyx mailing list
> Blueonyx at blueonyx.it
> http://www.blueonyx.it/mailman/listinfo/blueonyx