[BlueOnyx:10181] Re: Yup - there's an app for that!

[Stephanie Sullivan]

Yeah - I pasted a non-working version of the code by mistake, but Michael's
solution is better for multiple bad actors and I've changed to that version.
Sorry for my earlier bad cut/paste.

Thank you Michael!

Maybe instead of forbidden we should 301 them to their localhost or just
give them 404's? :-)

	Thanks,
		-Stephanie

> -----Original Message-----
> From: Michael Stauber [mailto:mstauber at blueonyx.it]
> Sent: Wednesday, April 18, 2012 6:47 PM
> To: BlueOnyx General Mailing List
> Subject: [BlueOnyx:10180] Re: Yup - there's an app for that!
> 
> Hi all,
> 
> > # Get rid of bad bots:
> > RewriteEngine on
> > RewriteCond %{HTTP_USER_AGENT} .*google.* [OR]
> > RewriteCond %{HTTP_USER_AGENT} .*yahoo.* [OR]
> > RewriteCond %{HTTP_USER_AGENT} .*bot.* [OR]
> > RewriteCond %{HTTP_USER_AGENT} .*spider.* [OR]
> > RewriteCond %{HTTP_USER_AGENT} "^Black.Hole"
> > RewriteRule .* - [F]
> 
> Speaking of "bad robots": The Chinese Baidu search engine is the
> worst of the
> worst.
> 
> It ignores robotx.txt - which is already bad enough.
> 
> But once I had started to block it with iptables (address range
> 180.76.0.0/16)
> and the above code, it came back.
> 
> >From different IP address ranges:
> 
> #### Baidu Spider:
> 180.76.0.0/16
> 
> # ShenZhen Sunrise Technology Co.,Ltd.
> 202.46.32.0/10
> 
> # SADF - CN
> 123.125.71.0/24
> 
> # CHINANET-IDC-BJ
> 220.181.0.0/16
> 
> # BAIDUJP
> 119.63.192.0/21
> 
> # CTIHK - City Telecom (H.K.) Ltd
> 183.178.0.0/16
> 
> # Victor Villar - Montevideo (errrr ... ok)
> 200.40.50.0/24
> 
> The return happened fairly quickly. In a matter of hours it came back
> from all
> over the place, even in disguise by simply giving the user agent
> string
> "Mozilla" instead of anything related to Baidu.
> 
> There was even a crawl from Russia which is most likely related to
> Baidu and I
> blocked it for good measure as well.
> 
> I wonder which part of "No" they didn't understand.
> 
> --
> With best regards
> 
> Michael Stauber
> 
> _______________________________________________
> Blueonyx mailing list
> Blueonyx at mail.blueonyx.it
> http://mail.blueonyx.it/mailman/listinfo/blueonyx