[BlueOnyx:02629] Re: PHP 404 Not Found error in daily report

[Jeff Folk]

On Oct 14, 2009, at 10:45 AM, Gerald Waugh wrote:

> On Wednesday 14 October 2009 10:06, Jeff Folk wrote:
>> And if you install dfix, it will block them after a couple attempts.
>> That way you won't end up with 250K of 404 hits like I did a couple
>> months ago...
>
> So you don't want web search engines crawling your sites
> How do your sites rank in the search engines?

What legitimate search engine hits a site with 250K non-existent  
pages? The bots hitting me announced themselves as "Googlebot" but did  
not honor a robots.txt exclude of the pages (mostly submit.php direct  
calls), so I let dfix take care of them.

Jeff