[BlueOnyx:09397] Re: /var/spool is at 3GB causing /var to fil
Chuck Tetlow
chuck at tetlow.net
Sun Jan 15 02:28:20 -05 2012
Jimmy,
You've either still got another script generating that e-mail, or someone is using a local account to access a webmail package and send it that way.
You might look in the /var/log/maillog file and search for that string you noticed "netease.com". If you find it - look in that same log entry for the "ctladdr" (who is the sender). Hopefully, it will be a username and domain that is on your server. That would be the exploited account. Either suspend that account or change its password. Or you might look in /var/log/httpd/access_log to see what IP the offending traffic is coming from and block them out in the firewall.
You also might look in the /var/log/httpd access and error files for more information. If its a script being run through Apache - it should leave a record of that in the access_log file.
A lot of times, a webmail package like OpenWebMail will have its own logfile tracking who is accessing it and what they're doing. On my server, the Nuonce OpenWebMail package logs everything done by every user in /var/log/openwebmail.log, and it also logs the Apache actions in /var/log/httpd/access_log file. Check the /var/log/openwebmail.log for a lot of sending actions (grep "send message" /var/log/openwebmail.log | less". Especially check for a lot of sending action by a single account or during the night. Again, that will be the offending account and can be suspended or password changed to block it.
Good luck Jimmy. I've had to track down and clean up a lot of these in the past - especially in the BQ days. Give a shout direct if you need more help. I'm not a programmer like Michael or Chris - but I've administered a lot of Linux boxes over the years and have some experience tracking down nasty crap like this.
Chuck
---------- Original Message -----------
From: Jimmy Gross <jimmy at constantino.net>
To: "BlueOnyx General Mailing List" <blueonyx at mail.blueonyx.it>
Sent: Sun, 15 Jan 2012 00:38:29 -0600
Subject: [BlueOnyx:09396] Re: /var/spool is at 3GB causing /var to fil
> I found a script that was being used to send out spam and removed it.
>
> I have someone else spammin through my server but cannot find out how.
>
> This is from my mailog:
> maillog
>
> Jan 11 03:24:18 hosting2 sendmail[720]: q0B97l0p000689: to=<jiyu0312 at 163.com>, ctladdr=<apache at hosting2.cgeinternet.com> (48/48), delay=01:16:30, xdelay=01:16:30, mailer=esmtp, pri=121546, relay=163mx00.mxmail.netease.com. [220.181.12.76], dsn=4.0.0, stat=Deferred: Connection timed out with 163mx00.mxmail.netease.com.
>
> There are several of these.
>
> When I look at the Running Processes I see several entries like this one:
> 12496 root 22:39 sendmail: ./q0F5KiUI030059 163mx01.mxmail.netease.com.: user open
>
> When I view File and Connections for that PID it pulls up:
> Current dir Directory 12288 262152 /var/spool/mqueue
> Root dir Directory 4096 2 /
> Program code Regular file 806460 699239 /usr/sbin/sendmail.sendmail
> Shared library Regular file 1011760 1343532 /lib/libdb-4.3.so
> Shared library Regular file 31344 1343573 /lib/libwrap.so.0.7.6
> Shared library Regular file 14012 696053 /usr/lib/libhesiod.so.0.0.0
> Shared library Regular file 99060 702926 /usr/lib/libsasl2.so.2.0.22
> Shared library Regular file 53792 698079 /usr/lib/liblber-2.3.so.0.2.31
> Shared library Regular file 1297124 1343685 /lib/libcrypto.so.0.9.8e
> Shared library Regular file 240584 698075 /usr/lib/libldap-2.3.so.0.2.31
> Shared library Regular file 1693812 1343646 /lib/libc-2.5.so
> Shared library Regular file 7812 1343674 /lib/libcom_err.so.2.1
> Shared library Regular file 20668 1343656 /lib/libdl-2.5.so
> Shared library Regular file 190712 696891 /usr/lib/libgssapi_krb5.so.2.2
> Shared library Regular file 50848 1345260 /lib/libnss_files-2.5.so
> Shared library Regular file 613716 694940 /usr/lib/libkrb5.so.3.3
> Shared library Regular file 157336 692654 /usr/lib/libk5crypto.so.3.1
> Shared library Regular file 93508 1343672 /lib/libselinux.so.1
> Shared library Regular file 245376 1343671 /lib/libsepol.so.1
> Shared library Regular file 75120 1343676 /lib/libz.so.1.2.3
> Shared library Regular file 129900 1343507 /lib/ld-2.5.so
> Shared library Regular file 80636 1343670 /lib/libresolv-2.5.so
> Shared library Regular file 14752 758091 /usr/lib/sasl2/liblogin.so.2.0.22
> Shared library Regular file 137908 1343657 /lib/libpthread-2.5.so
> Shared library Regular file 905200 758118 /usr/lib/sasl2/libsasldb.so.2.0.22
> Shared library Regular file 21948 1344343 /lib/libnss_dns-2.5.so
> Shared library Regular file 33968 692650 /usr/lib/libkrb5support.so.0.1
> Shared library Regular file 16832 758084 /usr/lib/sasl2/libcrammd5.so.2.0.22
> Shared library Regular file 14848 758095 /usr/lib/sasl2/libplain.so.2.0.22
> Shared library Regular file 293428 1343492 /lib/libssl.so.0.9.8e
> Shared library Regular file 7880 1343667 /lib/libkeyutils-1.2.so
> Shared library Regular file 109740 1343717 /lib/libnsl-2.5.so
> Shared library Regular file 47172 758088 /usr/lib/sasl2/libdigestmd5.so.2.0.22
> Shared library Regular file 45432 1343707 /lib/libcrypt-2.5.so
> Shared library Regular file 14372 757564 /usr/lib/sasl2/libanonymous.so.2.0.22
> 0r Character special
> 1600 /dev/null
> 1w Character special
> 1600 /dev/null
> 2w Character special
> 1600 /dev/null
> 4uw Regular file 1174 262185 /var/spool/mqueue/qfq0F5KiUI030059
> 5r Regular file 53033 262167 /var/spool/mqueue/dfq0F5KiUI030059
> 6r Regular file 172032 66405 /etc/mail/virtusertable.db
> 7r Regular file 172032 66405 /etc/mail/virtusertable.db
> 8r Regular file 12288 66408 /etc/mail/mailertable.db
> 9r Regular file 12288 66408 /etc/mail/mailertable.db
>
>
> Open Network Connections
> IPV4 TCP 10u 65.39.71.4:44891 -> 220.181.12.63:smtp SYN_SENT
>
> copy of message header from mqueue:
> Return-Path <g>
> Received from hosting2.cgeinternet.com (localhost [127.0.0.1])by hosting2.cgeinternet.com (8.13.8/8.13.8) with ESMTP id q0F5IaOh006469(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)for <cmq at cnuninet.com>; Sat, 14 Jan 2012 22:18:37 -0700
> Full-Name Apache
> Received (from apache at localhost)by hosting2.cgeinternet.com (8.13.8/8.13.8/Submit) id q0F5ITBF004465;Sat, 14 Jan 2012 22:18:29 -0700
> Date Sat, 14 Jan 2012 22:18:29 -0700
> Message-Id <201201150518.q0F5ITBF004465 at hosting2.cgeinternet.com>
> To cmq at cnuninet.com
> Subject Your order has been completed
> From "American Airlines" <account-no572 at aa.com>
> X-Mailer aerobacterkatowicefairport
> Reply-To "American Airlines" <account-no572 at aa.com>
> Mime-Version 1.0
> Content-Type multipart/mixed;boundary="----------13266047094F1261A50346C"
> X-Virus-Scanned clamav-milter 0.97.3 at hosting2.cgeinternet.com
> X-Virus-Status Clean
>
> Can someone please point me in the right direction?
>
> Thank you.
>
> jimmy
> -----Original Message-----
> From: blueonyx-bounces at mail.blueonyx.it [mailto:blueonyx-bounces at mail.blueonyx.it]On Behalf Of Chuck Tetlow
> Sent: Sunday, January 08, 2012 3:01 PM
> To: BlueOnyx General Mailing List
> Subject: [BlueOnyx:09335] Re: /var/spool is at 3GB causing /var to fil
>
> > Jimmy Gross wrote:
> > >
> > > /var/spool is at 3GB which is making the directory get full which is causing
> > > mail to stop.
> > >
> > > How do I clear this?
> >
> > What file(s) inside /var/spool are growing? Find that out, and you'll
> > know what to clear.
> >
> > If your /var/spool/mqueue directory is growing with a bunch of files,
> > you'll want to find out why. Maybe you've been mailbombed, hacked,
> > spammed or have some sort of delivery problem. In /var/spool, that
> > would be my first guess to look for.
> >
> > > I have rebooted the server twice but no change.
> >
> > Right. Rebooting a server isn't going to remove files. Besides, I try
> > and reboot as infrequently as possible, using reboots only for when
> > there is no other option, or to load a new kernel.
> >
> > --
> > Chris Gebhardt
>
> In my experience - its almost always /var/spool/mqueue. I've had to clean at least a dozen servers that have had the same thing happen. A single user account on the box gets hacked because of a guessable password - and then some scumbag starts relaying millions of SPAM through your server.
>
> And I'm not exaggerating! One of my BQ customers had a couple of accounts guessed because they had passwords like "password" or a password same as the username. In the four/five days it took them to realize there was a problem and call me - the logs recorded just over 2 million SPAMS dropped into that machine for relay. In fact, the reason they knew there was a problem - the big companies like YahooMail and Gmail were blacklisting their server!!
>
> To find what in /var/spool is growing so much - go to the command line and run "du -hs /var/spool/*". That will give you a list of the files/directories in /var/spool along with their size (in KB, MB, and GB). Look to see what is ridiculously large and that directory is your problem.
>
> If its /var/spool/mqueue - its probably someone using your server to relay SPAM through a hacked account. Its actually a easy fix. But you risk loosing a valid e-mail or two. To fix it - just shutdown your e-mail server with "service sendmail stop". E-mail waiting to be sent out to their destination server is stored in /var/spool/mqueue. So just go into that directory and run "rm -f /var/spool/mqueue/*". That will get rid of all the build-up in /var/spool/mqueue. Then restart the mail server with "service sendmail start".
>
> (Oh - once or twice I've had to do a similar cleanup, and there were so many messages that the server couldn't delete them with a simple "rm -f *". I had to do it in chunks, like "rm -f dfq0*", "rm -f dfq1*", "rm -f dfq2*", and so on)
>
> BUT! Don't forget to find the root cause and fix it. The best bet is to carefully go through /var/log/maillog and see who is sending/relaying a lot of messages. Other places to look for clues is /var/log/messages and /var/log/secure. Of course, if you've got a webmail package installed - you may have to go through /var/log/httpd/access to see who is sending those messages. But where ever you find the clues - either shutdown the account being used or change its password. That will lock out the scum using your server.
>
> And this is a very good answer to that discussion on the list last week - about disabling strong password enforcement! If you let users choose weak passwords, they will. Those weak passwords will be guessed and exploited - the the point that it can deny services to your valid users, take down your server, and possibly get your server blacklisted as a SPAM relay. Then you'll spend hours cleaning up the mess, tracking down the exploited account, getting your server un-blacklisted, and apologizing to your customers about the service interruption (IF you can keep them). Its just not worth it - leave the system set to REQUIRE strong passwords!!
>
> Chuck
>
>
------- End of Original Message -------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.blueonyx.it/pipermail/blueonyx/attachments/20120115/8b9e7e38/attachment.html>
More information about the Blueonyx
mailing list