[BlueOnyx:09335] Re: /var/spool is at 3GB causing /var to fil

[Chuck Tetlow]

> Jimmy Gross wrote: 
> > 
> > /var/spool is at 3GB which is making the directory get full which is causing 
> > mail to stop. 
> > 
> > How do I clear this? 
> 
> What file(s) inside /var/spool are growing?  Find that out, and you'll 
> know what to clear. 
> 
> If your /var/spool/mqueue directory is growing with a bunch of files, 
> you'll want to find out why.  Maybe you've been mailbombed, hacked, 
> spammed or have some sort of delivery problem.  In /var/spool, that 
> would be my first guess to look for. 
> 
> > I have rebooted the server twice but no change. 
> 
> Right.  Rebooting a server isn't going to remove files.  Besides, I try 
> and reboot as infrequently as possible, using reboots only for when 
> there is no other option, or to load a new kernel. 
> 
> -- 
> Chris Gebhardt

In my experience - its almost always /var/spool/mqueue.  I've had to clean at least a dozen servers that have had the same thing happen.  A single user accounton the box gets hacked because of a guessable password - and then some scumbag starts relaying millions of SPAM through your server. 

And I'm not exaggerating!  One of my BQ customers had a couple of accounts guessed because they had passwords like "password" or a password same as the username.  In the four/five days it took them to realize there was a problem and call me - the logs recorded just over 2 million SPAMS dropped into that machine for relay.  In fact, the reason they knew there was a problem - the big companies like YahooMail and Gmail were blacklisting their server!!

To find what in /var/spool is growing so much - go to the command line and run "du -hs /var/spool/*".  That will give you a list of the files/directories in /var/spool along with their size (in KB, MB, and GB).  Look to see what is ridiculously large and that directory is your problem.

If its /var/spool/mqueue - its probably someone using your server to relay SPAM through a hacked account.  Its actually a easy fix.  But you risk loosing a valid e-mail or two.  To fix it - just shutdown your e-mail server with "service sendmail stop".  E-mail waiting to be sent out to their destination server is stored in /var/spool/mqueue.  So just go into that directory and run "rm -f /var/spool/mqueue/*".  That will get rid of all the build-up in /var/spool/mqueue.  Then restart the mail server with "service sendmail start".

(Oh - once or twice I've had to do a similar cleanup, and there were so many messages that the server couldn't delete them with a simple "rm -f *".  I had to do it in chunks, like "rm -f dfq0*", "rm -f dfq1*", "rm -f dfq2*", and so on)

BUT!  Don't forget to find the root cause and fix it.  The best bet is to carefully go through /var/log/maillog and see who is sending/relayinga lot of messages.  Other places to look for clues is /var/log/messagesand /var/log/secure.  Of course, if you've got a webmail package installed - you may have to go through /var/log/httpd/access to see who is sending those messages.  But where ever you find the clues - either shutdown the account being used or change its password.  That will lock out the scum using your server.

And this is a very good answer to that discussion on the list last week - about disabling strong password enforcement!  If you let users choose weak passwords, they will.  Those weak passwords will be guessed and exploited - the the point that it can deny services to your valid users, take down your server, and possibly get your server blacklisted as a SPAM relay.  Then you'll spend hours cleaning up the mess, tracking down the exploited account, getting your server un-blacklisted, and apologizing to your customers about the service interruption (IF you can keep them).  Its just not worth it - leave the system set to REQUIRE strong passwords!!

Chuck

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.blueonyx.it/pipermail/blueonyx/attachments/20120108/54ce770f/attachment.html>