This is all in the readme.

These bots were cluttering up my error logging database table for 404 and 500 errors on a server holding about 250 websites.

Like you, I took small steps initially in case any of the bots turned out to be legit. I have found no trace of any negative consequences.

I am due to write a new post about all the different useless user-agents that I am blocking without using an IP address match as well. The amount of crap out there is astounding.

I found a site “Project Honey Pot” that tracks these things and confirms them by seeing if they spam or not, it might be of use (it’s possible I found the link on this blog somewhere).
http://www.projecthoneypot.org/harvester_useragents.php

At this point I’m not getting hit that hard by these “bots”, but if it becomes abusive, to the point it starts taking a lot of CPU then I will have to do something about it and appreciate the info.

Bandwidth for the few K that they take isn’t a problem (yet). What I am way more worried about is losing some possible indexing that could send traffic to my site.

All they seem to do so far is load the first level pages and go no deeper, they load no images.

Yes, they should properly identify themselves, but maybe they don’t want to because people might use that ID to “game” their system.

If I ever do start to block, I’m probably going to start with the IP list from that “honey pot” place since those are confirmed and see how that goes.

Plus, if these guys who do this get a clue, they can just change the agent string randomly to any number of known browser types. After that the only way you could tell is by their behavior, which may not be the best indication, or by the honey pot method.

I think we agree that the extra load on our web servers caused by java bots is useless, and blocking these robots is so easy that it simply makes sense.

I am about to update the post with some simplified rewrite rules that I have been using for a couple weeks.

Thanks again for your reply. Yes that’s what i want to accomplish, to show users only my domain. However, the home/default.html path is not a ‘physical’ file in my server, it is based on MCMS. I think your approach will only work if there is an existing physical file of home/default.html. I saw the generated logs and it is clearly looking for the physical file. Is my goal still possible with my current setup? Kindly advice. Thanks.

RewriteCond %{SERVER_NAME} ([^\.] )\.mysite\.com$ [I]

This first line means check all requests for this domain. My server has a few hundred sites that are all configured to use the IIRF ISAPI filter (right click website profile properties isapi filters tab). This condition identifies the site I want to modify from the bunch.

RewriteRule ^/?$ /home/default.htm

This rule says match requests on the root for empty string or a slash…

^ beginning of string
/? optional one slash
$ end of string

..and rewrite that location to /home/default.htm. The result is that any time the root level of the domain is requested, mysite.com or mysite.com/, return the contents of /home/default.htm for that request. The user sees just mysite.com/ in their browser’s address bar, but they are looking at /home/default.htm.

I believe this is what you are trying to accomplish based on your comments.

Thanks for your reply, I tried your approach but still resulted to error page. Actually I don’t want to redirect the page, I just want to mask it, so if a user browsed to my domain (http://www.mysite.com) he/she will not be redirected to (http://www.mysite.com/home/default.html)

My goal is really not to show the /home/default.html path (more like to strip it in the address/URL bar). Is this possible in IIRF? Thanks again.