[BlueOnyx:10178] Re: Yup - there's an app for that!

[Michael Stauber]

Hi Stephanie,

> We ended up using the sitexx.include file to 
> 
> SetEnvIfNoCase User-Agent ^SiteSucker.* goaway
> <Location /home/.sites/xx/sitexx/web>
> Order allow,deny
> Allow from all
> Deny from env=goaway
> </Location>
> 
> So, want to impede a CMS based web site?

Oh, that is really nice. I like that, Stephanie. :-)

Just the other day I could have used it. I noticed that the server load of 
devel.blueonyx.it was constantly hovering at around 10. I checked an both 
Google and Baidu were spidering every page that the installed Trac web 
frontend to the SVN had to offer. And it's a lot of pages as we by now have 
roughly 1000 SVN commits.

The best part: Neither Google nor Baidu were honoring the robots.txt which 
denied indexing of the /trac/ directory.

I ended up firewalling 66.249.71.0/24 and 180.76.6.0/24 to end the madness.

-- 
With best regards

Michael Stauber