<HTML>
<HEAD>
<META content="text/html; charset=iso-8859-1" http-equiv=Content-Type>
<META content="OPENWEBMAIL" name=GENERATOR>
</HEAD>
<BODY bgColor=#ffffff>
Jimmy,
<br />
<br />You've either still got another script generating that e-mail, or someone is using a local account to access a webmail package and send it that way.
<br />
<br />You might look in the /var/log/maillog file and search for that string you noticed "<font face="Geneva, Arial, Helvetica"><span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">netease.com</font></span></font>". If you find it - look in that same log entry for the "ctladdr" (who is the sender). Hopefully, it will be a username and domain that is on your server. That would be the exploited account. Either suspend that account or change its password. Or you might look in /var/log/httpd/access_log to see what IP the offending traffic is coming from and block them out in the firewall.
<br />
<br />You also might look in the /var/log/httpd access and error files for more information. If its a script being run through Apache - it should leave a record of that in the access_log file.
<br />
<br />A lot of times, a webmail package like OpenWebMail will have its own logfile tracking who is accessing it and what they're doing. On my server, the Nuonce OpenWebMail package logs everything done by every user in /var/log/openwebmail.log, and it also logs the Apache actions in /var/log/httpd/access_log file. Check the /var/log/openwebmail.log for a lot of sending actions (grep "send message" /var/log/openwebmail.log | less". Especially check for a lot of sending action by a single account or during the night. Again, that will be the offending account and can be suspended or password changed to block it.
<br />
<br />Good luck Jimmy. I've had to track down and clean up a lot of these in the past - especially in the BQ days. Give a shout direct if you need more help. I'm not a programmer like Michael or Chris - but I've administered a lot of Linux boxes over the years and have some experience tracking down nasty crap like this.
<br />
<br />
<br />
<br />Chuck
<br />
<br />
<br />
<br /><font size="2"><b>---------- Original Message
-----------</b>
<br />
From: Jimmy Gross <jimmy@constantino.net>
<br />
To: "BlueOnyx General Mailing List" <blueonyx@mail.blueonyx.it>
<br />
Sent: Sun, 15 Jan 2012 00:38:29 -0600
<br />
Subject: [BlueOnyx:09396] Re: /var/spool is at 3GB causing /var to fil
<br />
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">I
found a script that was being used to send out spam and removed
it.</font></span>
<br />> <span class="137083306-15012012"></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">I
have
someone else spammin through my server but cannot find out
how.</font></span>
<br />> <span class="137083306-15012012"></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">This
is from my
mailog:</font></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">maillog</font></span>
<br />>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">Jan 11
03:24:18 hosting2 sendmail[720]: q0B97l0p000689: to=<<a href="mailto:jiyu0312@163.com">jiyu0312@163.com</a>>, ctladdr=<<a href="mailto:apache@hosting2.cgeinternet.com">apache@hosting2.cgeinternet.com</a>>
(48/48), delay=01:16:30, xdelay=01:16:30, mailer=esmtp, pri=121546,
relay=163mx00.mxmail.netease.com. [220.181.12.76], dsn=4.0.0, stat=Deferred:
Connection timed out with
163mx00.mxmail.netease.com.</font></span>
<br />> <span class="137083306-15012012"></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">There
are several of
these.</font></span>
<br />> <span class="137083306-15012012"></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">When I
look at the Running Processes I see several entries like this
one:</font></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">12496
root 22:39 sendmail: ./q0F5KiUI030059 163mx01.mxmail.netease.com.: user
open</font></span>
<br />> <span class="137083306-15012012"></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">When I
view File and Connections for that PID it pulls
up:</font></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">Current dir Directory 12288 262152 /var/spool/mqueue
<br />> Root
dir
Directory 4096 2 /
<br />> Program code Regular file 806460 699239
/usr/sbin/sendmail.sendmail
<br />> Shared library Regular file 1011760 1343532
/lib/libdb-4.3.so
<br />> Shared library Regular file 31344 1343573
/lib/libwrap.so.0.7.6
<br />> Shared library Regular file 14012 696053
/usr/lib/libhesiod.so.0.0.0
<br />> Shared library Regular file 99060 702926
/usr/lib/libsasl2.so.2.0.22
<br />> Shared library Regular file 53792 698079
/usr/lib/liblber-2.3.so.0.2.31
<br />> Shared library Regular file 1297124
1343685
/lib/libcrypto.so.0.9.8e
<br />> Shared library Regular file 240584 698075
/usr/lib/libldap-2.3.so.0.2.31
<br />> Shared library Regular file 1693812
1343646
/lib/libc-2.5.so
<br />> Shared library Regular file 7812 1343674
/lib/libcom_err.so.2.1
<br />> Shared library Regular file 20668 1343656
/lib/libdl-2.5.so
<br />> Shared library Regular file 190712 696891
/usr/lib/libgssapi_krb5.so.2.2
<br />> Shared library Regular file 50848
1345260
/lib/libnss_files-2.5.so
<br />> Shared library Regular file 613716 694940
/usr/lib/libkrb5.so.3.3
<br />> Shared library Regular file 157336 692654
/usr/lib/libk5crypto.so.3.1
<br />> Shared library Regular file 93508 1343672
/lib/libselinux.so.1
<br />> Shared library Regular file 245376 1343671
/lib/libsepol.so.1
<br />> Shared library Regular file 75120 1343676
/lib/libz.so.1.2.3
<br />> Shared library Regular file 129900 1343507
/lib/ld-2.5.so
<br />> Shared library Regular file 80636 1343670 /lib/libresolv-2.5.so
<br />> Shared
library Regular file 14752 758091 /usr/lib/sasl2/liblogin.so.2.0.22
<br />>
Shared
library Regular file 137908 1343657 /lib/libpthread-2.5.so
<br />> Shared
library
Regular file 905200 758118 /usr/lib/sasl2/libsasldb.so.2.0.22
<br />> Shared
library
Regular file 21948 1344343 /lib/libnss_dns-2.5.so
<br />> Shared library
Regular
file 33968 692650 /usr/lib/libkrb5support.so.0.1
<br />> Shared library Regular
file
16832 758084 /usr/lib/sasl2/libcrammd5.so.2.0.22
<br />> Shared library Regular
file
14848 758095 /usr/lib/sasl2/libplain.so.2.0.22
<br />> Shared library Regular
file
293428 1343492 /lib/libssl.so.0.9.8e
<br />> Shared library Regular file 7880
1343667 /lib/libkeyutils-1.2.so
<br />> Shared library Regular file 109740
1343717
/lib/libnsl-2.5.so
<br />> Shared library Regular file 47172 758088
/usr/lib/sasl2/libdigestmd5.so.2.0.22
<br />> Shared library Regular file 45432
1343707 /lib/libcrypt-2.5.so
<br />> Shared library Regular file 14372 757564
/usr/lib/sasl2/libanonymous.so.2.0.22
<br />> 0r Character special
<br />>
1600
/dev/null
<br />> 1w Character special
<br />> 1600 /dev/null
<br />>
2w Character
special
<br />> 1600 /dev/null
<br />> 4uw Regular file 1174 262185
/var/spool/mqueue/qfq0F5KiUI030059
<br />> 5r Regular file 53033 262167
/var/spool/mqueue/dfq0F5KiUI030059
<br />> 6r Regular file 172032 66405
/etc/mail/virtusertable.db
<br />> 7r Regular file 172032 66405
/etc/mail/virtusertable.db
<br />> 8r Regular file 12288 66408
/etc/mail/mailertable.db
<br />> 9r Regular file 12288 66408
/etc/mail/mailertable.db
</font></span>
<br />> <span class="137083306-15012012"></span>
<br />> <span class="137083306-15012012"></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">Open
Network Connections
<br />> IPV4 TCP 10u 65.39.71.4:44891 ->
220.181.12.63:smtp
SYN_SENT
</font></span>
<br />> <span class="137083306-15012012"></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">copy
of message header from mqueue:
<br />> Return-Path <g>
<br />> Received
from
hosting2.cgeinternet.com (localhost [127.0.0.1])by hosting2.cgeinternet.com
(8.13.8/8.13.8) with ESMTP id q0F5IaOh006469(version=TLSv1/SSLv3
cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)for <<a href="mailto:cmq@cnuninet.com">cmq@cnuninet.com</a>>; Sat, 14 Jan 2012
22:18:37 -0700
<br />> Full-Name Apache
<br />> Received (from <a href="mailto:apache@localhost)by">apache@localhost)by</a>
hosting2.cgeinternet.com (8.13.8/8.13.8/Submit) id q0F5ITBF004465;Sat, 14 Jan
2012 22:18:29 -0700
<br />> Date Sat, 14 Jan 2012 22:18:29 -0700
<br />>
Message-Id
<<a href="mailto:201201150518.q0F5ITBF004465@hosting2.cgeinternet.com">201201150518.q0F5ITBF004465@hosting2.cgeinternet.com</a>>
<br />> To <a href="mailto:cmq@cnuninet.com">cmq@cnuninet.com</a>
<br />>
Subject Your
order has been completed
<br />> From "American Airlines" <<a href="mailto:account-no572@aa.com">account-no572@aa.com</a>>
<br />>
X-Mailer
aerobacterkatowicefairport
<br />> Reply-To "American Airlines" <<a href="mailto:account-no572@aa.com">account-no572@aa.com</a>>
<br />>
Mime-Version
1.0
<br />> Content-Type
multipart/mixed;boundary="----------13266047094F1261A50346C"
<br />>
X-Virus-Scanned
clamav-milter 0.97.3 at hosting2.cgeinternet.com
<br />> X-Virus-Status Clean
</font></span>
<br />> <span class="137083306-15012012"></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">Can
someone please point me in the right
direction?</font></span>
<br />> <span class="137083306-15012012"></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">Thank
you.</font></span>
<br />> <span class="137083306-15012012"></span>
<br />> <span class="137083306-15012012"><font face="Arial" color="#0000ff" size="2">jimmy</font></span>
<blockquote>
<br />> <font face="Tahoma" size="2">-----Original Message-----
<br />> <b>From:</b>
blueonyx-bounces@mail.blueonyx.it
[mailto:blueonyx-bounces@mail.blueonyx.it]<b>On Behalf Of </b>Chuck
Tetlow
<br />> <b>Sent:</b> Sunday, January 08, 2012 3:01 PM
<br />>
<b>To:</b> BlueOnyx
General Mailing List
<br />> <b>Subject:</b> [BlueOnyx:09335] Re: /var/spool
is at
3GB causing /var to fil
<br />>
<br />> </font><font size="2">> Jimmy Gross
wrote:
<br />> > >
<br />> > > /var/spool is at 3GB which is
making the
directory get full which is causing
<br />> > > mail to stop.
<br />>
> >
<br />> > > How do I clear this?
<br />> >
<br />> > What
file(s) inside
/var/spool are growing? Find that out, and you'll
<br />> > know
what to
clear.
<br />> >
<br />> > If your /var/spool/mqueue directory is
growing with a
bunch of files,
<br />> > you'll want to find out why. Maybe you've
been
mailbombed, hacked,
<br />> > spammed or have some sort of delivery
problem.
In /var/spool, that
<br />> > would be my first guess to look for.
<br />> >
<br />> > > I have rebooted the server twice but no
change.
<br />> >
<br />> > Right. Rebooting a server isn't going to
remove files.
Besides, I try
<br />> > and reboot as infrequently as possible,
using
reboots only for when
<br />> > there is no other option, or to load a new
kernel.
<br />> >
<br />> > --
<br />> > Chris Gebhardt
<br />>
<br />> </font>In
my experience - its almost always /var/spool/mqueue. I've had to clean
at least a dozen servers that have had the same thing happen. A single
user account on the box gets hacked because of a guessable password - and then
some scumbag starts relaying millions of SPAM through your server.
<br />>
<br />> And I'm not exaggerating! One of my BQ customers had a
couple of
accounts guessed because they had passwords like "password" or a password same
as the username. In the four/five days it took them to realize there was
a problem and call me - the logs recorded just over 2 million SPAMS dropped
into that machine for relay. In fact, the reason they knew there was a
problem - the big companies like YahooMail and Gmail were blacklisting their
server!!
<br />>
<br />> To find what in /var/spool is growing so much - go
to the
command line and run "du -hs /var/spool/*". That will give you a list of
the files/directories in /var/spool along with their size (in KB, MB, and
GB). Look to see what is ridiculously large and that directory is your
problem.
<br />>
<br />> If its /var/spool/mqueue - its probably someone
using your
server to relay SPAM through a hacked account. Its actually a easy
fix. But you risk loosing a valid e-mail or two. To fix it - just
shutdown your e-mail server with "service sendmail stop". E-mail waiting
to be sent out to their destination server is stored in
/var/spool/mqueue. So just go into that directory and run "rm -f
/var/spool/mqueue/*". That will get rid of all the build-up in
/var/spool/mqueue. Then restart the mail server with "service sendmail
start".
<br />>
<br />> (Oh - once or twice I've had to do a similar
cleanup, and
there were so many messages that the server couldn't delete them with a simple
"rm -f *". I had to do it in chunks, like "rm -f dfq0*", "rm -f dfq1*",
"rm -f dfq2*", and so on)
<br />>
<br />> BUT! Don't forget to find
the root
cause and fix it. The best bet is to carefully go through
/var/log/maillog and see who is sending/relaying a lot of messages.
Other places to look for clues is /var/log/messages and /var/log/secure.
Of course, if you've got a webmail package installed - you may have to go
through /var/log/httpd/access to see who is sending those messages. But
where ever you find the clues - either shutdown the account being used or
change its password. That will lock out the scum using your server.
<br />>
<br />> And this is a very good answer to that discussion on the
list last
week - about disabling strong password enforcement! If you let users
choose weak passwords, they will. Those weak passwords will be guessed
and exploited - the the point that it can deny services to your valid users,
take down your server, and possibly get your server blacklisted as a SPAM
relay. Then you'll spend hours cleaning up the mess, tracking down the
exploited account, getting your server un-blacklisted, and apologizing to your
customers about the service interruption (IF you can keep them). Its
just not worth it - leave the system set to REQUIRE strong passwords!!
<br />>
<br />> Chuck
<br />>
<br />>
</blockquote>
<br /><b>------- End of Original Message
-------</b>
<br />
</font>
</BODY>
</HTML>