[BlueOnyx:09396] Re: /var/spool is at 3GB causing /var to fil
Jimmy Gross
jimmy at constantino.net
Sun Jan 15 01:38:29 -05 2012
I found a script that was being used to send out spam and removed it.
I have someone else spammin through my server but cannot find out how.
This is from my mailog:
maillog
Jan 11 03:24:18 hosting2 sendmail[720]: q0B97l0p000689:
to=<jiyu0312 at 163.com>, ctladdr=<apache at hosting2.cgeinternet.com> (48/48),
delay=01:16:30, xdelay=01:16:30, mailer=esmtp, pri=121546,
relay=163mx00.mxmail.netease.com. [220.181.12.76], dsn=4.0.0, stat=Deferred:
Connection timed out with 163mx00.mxmail.netease.com.
There are several of these.
When I look at the Running Processes I see several entries like this one:
12496 root 22:39 sendmail: ./q0F5KiUI030059 163mx01.mxmail.netease.com.:
user open
When I view File and Connections for that PID it pulls up:
Current dir Directory 12288 262152 /var/spool/mqueue
Root dir Directory 4096 2 /
Program code Regular file 806460 699239 /usr/sbin/sendmail.sendmail
Shared library Regular file 1011760 1343532 /lib/libdb-4.3.so
Shared library Regular file 31344 1343573 /lib/libwrap.so.0.7.6
Shared library Regular file 14012 696053 /usr/lib/libhesiod.so.0.0.0
Shared library Regular file 99060 702926 /usr/lib/libsasl2.so.2.0.22
Shared library Regular file 53792 698079 /usr/lib/liblber-2.3.so.0.2.31
Shared library Regular file 1297124 1343685 /lib/libcrypto.so.0.9.8e
Shared library Regular file 240584 698075 /usr/lib/libldap-2.3.so.0.2.31
Shared library Regular file 1693812 1343646 /lib/libc-2.5.so
Shared library Regular file 7812 1343674 /lib/libcom_err.so.2.1
Shared library Regular file 20668 1343656 /lib/libdl-2.5.so
Shared library Regular file 190712 696891 /usr/lib/libgssapi_krb5.so.2.2
Shared library Regular file 50848 1345260 /lib/libnss_files-2.5.so
Shared library Regular file 613716 694940 /usr/lib/libkrb5.so.3.3
Shared library Regular file 157336 692654 /usr/lib/libk5crypto.so.3.1
Shared library Regular file 93508 1343672 /lib/libselinux.so.1
Shared library Regular file 245376 1343671 /lib/libsepol.so.1
Shared library Regular file 75120 1343676 /lib/libz.so.1.2.3
Shared library Regular file 129900 1343507 /lib/ld-2.5.so
Shared library Regular file 80636 1343670 /lib/libresolv-2.5.so
Shared library Regular file 14752 758091 /usr/lib/sasl2/liblogin.so.2.0.22
Shared library Regular file 137908 1343657 /lib/libpthread-2.5.so
Shared library Regular file 905200 758118 /usr/lib/sasl2/libsasldb.so.2.0.22
Shared library Regular file 21948 1344343 /lib/libnss_dns-2.5.so
Shared library Regular file 33968 692650 /usr/lib/libkrb5support.so.0.1
Shared library Regular file 16832 758084 /usr/lib/sasl2/libcrammd5.so.2.0.22
Shared library Regular file 14848 758095 /usr/lib/sasl2/libplain.so.2.0.22
Shared library Regular file 293428 1343492 /lib/libssl.so.0.9.8e
Shared library Regular file 7880 1343667 /lib/libkeyutils-1.2.so
Shared library Regular file 109740 1343717 /lib/libnsl-2.5.so
Shared library Regular file 47172 758088
/usr/lib/sasl2/libdigestmd5.so.2.0.22
Shared library Regular file 45432 1343707 /lib/libcrypt-2.5.so
Shared library Regular file 14372 757564
/usr/lib/sasl2/libanonymous.so.2.0.22
0r Character special
1600 /dev/null
1w Character special
1600 /dev/null
2w Character special
1600 /dev/null
4uw Regular file 1174 262185 /var/spool/mqueue/qfq0F5KiUI030059
5r Regular file 53033 262167 /var/spool/mqueue/dfq0F5KiUI030059
6r Regular file 172032 66405 /etc/mail/virtusertable.db
7r Regular file 172032 66405 /etc/mail/virtusertable.db
8r Regular file 12288 66408 /etc/mail/mailertable.db
9r Regular file 12288 66408 /etc/mail/mailertable.db
Open Network Connections
IPV4 TCP 10u 65.39.71.4:44891 -> 220.181.12.63:smtp SYN_SENT
copy of message header from mqueue:
Return-Path <g>
Received from hosting2.cgeinternet.com (localhost [127.0.0.1])by
hosting2.cgeinternet.com (8.13.8/8.13.8) with ESMTP id
q0F5IaOh006469(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256
verify=NO)for <cmq at cnuninet.com>; Sat, 14 Jan 2012 22:18:37 -0700
Full-Name Apache
Received (from apache at localhost)by hosting2.cgeinternet.com
(8.13.8/8.13.8/Submit) id q0F5ITBF004465;Sat, 14 Jan 2012 22:18:29 -0700
Date Sat, 14 Jan 2012 22:18:29 -0700
Message-Id <201201150518.q0F5ITBF004465 at hosting2.cgeinternet.com>
To cmq at cnuninet.com
Subject Your order has been completed
>From "American Airlines" <account-no572 at aa.com>
X-Mailer aerobacterkatowicefairport
Reply-To "American Airlines" <account-no572 at aa.com>
Mime-Version 1.0
Content-Type multipart/mixed;boundary="----------13266047094F1261A50346C"
X-Virus-Scanned clamav-milter 0.97.3 at hosting2.cgeinternet.com
X-Virus-Status Clean
Can someone please point me in the right direction?
Thank you.
jimmy
-----Original Message-----
From: blueonyx-bounces at mail.blueonyx.it
[mailto:blueonyx-bounces at mail.blueonyx.it]On Behalf Of Chuck Tetlow
Sent: Sunday, January 08, 2012 3:01 PM
To: BlueOnyx General Mailing List
Subject: [BlueOnyx:09335] Re: /var/spool is at 3GB causing /var to fil
> Jimmy Gross wrote:
> >
> > /var/spool is at 3GB which is making the directory get full which is
causing
> > mail to stop.
> >
> > How do I clear this?
>
> What file(s) inside /var/spool are growing? Find that out, and you'll
> know what to clear.
>
> If your /var/spool/mqueue directory is growing with a bunch of files,
> you'll want to find out why. Maybe you've been mailbombed, hacked,
> spammed or have some sort of delivery problem. In /var/spool, that
> would be my first guess to look for.
>
> > I have rebooted the server twice but no change.
>
> Right. Rebooting a server isn't going to remove files. Besides, I try
> and reboot as infrequently as possible, using reboots only for when
> there is no other option, or to load a new kernel.
>
> --
> Chris Gebhardt
In my experience - its almost always /var/spool/mqueue. I've had to clean
at least a dozen servers that have had the same thing happen. A single user
account on the box gets hacked because of a guessable password - and then
some scumbag starts relaying millions of SPAM through your server.
And I'm not exaggerating! One of my BQ customers had a couple of accounts
guessed because they had passwords like "password" or a password same as the
username. In the four/five days it took them to realize there was a problem
and call me - the logs recorded just over 2 million SPAMS dropped into that
machine for relay. In fact, the reason they knew there was a problem - the
big companies like YahooMail and Gmail were blacklisting their server!!
To find what in /var/spool is growing so much - go to the command line and
run "du -hs /var/spool/*". That will give you a list of the
files/directories in /var/spool along with their size (in KB, MB, and GB).
Look to see what is ridiculously large and that directory is your problem.
If its /var/spool/mqueue - its probably someone using your server to relay
SPAM through a hacked account. Its actually a easy fix. But you risk
loosing a valid e-mail or two. To fix it - just shutdown your e-mail server
with "service sendmail stop". E-mail waiting to be sent out to their
destination server is stored in /var/spool/mqueue. So just go into that
directory and run "rm -f /var/spool/mqueue/*". That will get rid of all the
build-up in /var/spool/mqueue. Then restart the mail server with "service
sendmail start".
(Oh - once or twice I've had to do a similar cleanup, and there were so
many messages that the server couldn't delete them with a simple "rm -f *".
I had to do it in chunks, like "rm -f dfq0*", "rm -f dfq1*", "rm -f dfq2*",
and so on)
BUT! Don't forget to find the root cause and fix it. The best bet is to
carefully go through /var/log/maillog and see who is sending/relaying a lot
of messages. Other places to look for clues is /var/log/messages and
/var/log/secure. Of course, if you've got a webmail package installed - you
may have to go through /var/log/httpd/access to see who is sending those
messages. But where ever you find the clues - either shutdown the account
being used or change its password. That will lock out the scum using your
server.
And this is a very good answer to that discussion on the list last week -
about disabling strong password enforcement! If you let users choose weak
passwords, they will. Those weak passwords will be guessed and exploited -
the the point that it can deny services to your valid users, take down your
server, and possibly get your server blacklisted as a SPAM relay. Then
you'll spend hours cleaning up the mess, tracking down the exploited
account, getting your server un-blacklisted, and apologizing to your
customers about the service interruption (IF you can keep them). Its just
not worth it - leave the system set to REQUIRE strong passwords!!
Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.blueonyx.it/pipermail/blueonyx/attachments/20120115/a6bc0b0a/attachment.html>
More information about the Blueonyx
mailing list