CatchBot lies
Comments(1)From catchbot.com:
Will my servers or site be affected by CatchBot?
It is unlikely that your servers or site will be affected by CatchBot. As Catchbot typically only crawls a few pages of any website slowly, CatchBot has no material negative impact on the load on your servers or the availability of your website.
…and by slowly they mean only 2 requests per second, even if the target URL is a 404 every time.
Thanks for the info. I have just started using the .htaccess file to block some robots like twiceler,yandex,and exbot. I tried using the robots.txt but it would not stop them. Even though the .htacces does block them they hammer the site constantly. Anyway, any time i see something new i check it out to see if i should block it. Catchbot seems to be okay.
mc