You can definitely use IIRF to block ranges with exclusions. Here’s an example I wrote for the Wget bot:

#wget
RewriteCond %{HTTP_USER_AGENT} Wget.*
#allow ip range 162.69.226.0 to 162.69.226.24
RewriteCond %{REMOTE_ADDR} ^(?!162\.69\.226\.([0-9]|1[0-9]|2[0-4]))([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})(.*)$
#allow 64.170.133.110
RewriteCond %{REMOTE_ADDR} ^(?!64\.170\.133\.110)([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})(.*)$
RewriteRule ^/(.*)$ /$1 [F]

These rules will block all Wget bots except when the IP address matches the exceptions.

At first it is painful to go through the logfiles everyday but it gets very easy once you have blocked so many of the bots. The pattern is easy to find sorting with Excel. I did write a log analyzer because the free ones were not specific enough for my kind of traffic. But I rather use excel and sort on columns to see the patterns. Because the patterns may change and my script isn’t smart enough to see it.

What I see mostly from my logfiles are:

- same IP address hitting at minute intervals to the same URL.
- several different IP addresses in the range of xxx.xxx.xx.* or xxx.xxx.*.* hitting the server at staggered intervals so in the log file, it looks like a human is hitting the URL. But when looking at the range of ip address also sorted by time, you can tell that its a robot doing the equivelant of a per minute hit to the same URL.

Honeypotting your pages so the log file reflects how the person is hitting the page is important. That way it supports your guess that what you are seeing is more than likely a robot.

I also wanted to caution that all robots are not java. There are a lot of languages out there that’s why I filter by behaviour.

My site is relatively new and so is my use of IIRF. I would like to let one IP address of a range of addresses use my site. Like 127.0.10.11 when I have blocked the range 127.0.10.*. The reason is that I block a range of IP addresses when I see that the robot is using more than three IP addresses. Does anyone know how to do that. Thank you.

This is all in the readme.

These bots were cluttering up my error logging database table for 404 and 500 errors on a server holding about 250 websites.

Like you, I took small steps initially in case any of the bots turned out to be legit. I have found no trace of any negative consequences.

I am due to write a new post about all the different useless user-agents that I am blocking without using an IP address match as well. The amount of crap out there is astounding.

I found a site “Project Honey Pot” that tracks these things and confirms them by seeing if they spam or not, it might be of use (it’s possible I found the link on this blog somewhere).
http://www.projecthoneypot.org/harvester_useragents.php

At this point I’m not getting hit that hard by these “bots”, but if it becomes abusive, to the point it starts taking a lot of CPU then I will have to do something about it and appreciate the info.

Bandwidth for the few K that they take isn’t a problem (yet). What I am way more worried about is losing some possible indexing that could send traffic to my site.

All they seem to do so far is load the first level pages and go no deeper, they load no images.

Yes, they should properly identify themselves, but maybe they don’t want to because people might use that ID to “game” their system.

If I ever do start to block, I’m probably going to start with the IP list from that “honey pot” place since those are confirmed and see how that goes.

Plus, if these guys who do this get a clue, they can just change the agent string randomly to any number of known browser types. After that the only way you could tell is by their behavior, which may not be the best indication, or by the honey pot method.

I think we agree that the extra load on our web servers caused by java bots is useless, and blocking these robots is so easy that it simply makes sense.

I am about to update the post with some simplified rewrite rules that I have been using for a couple weeks.

Thanks again for your reply. Yes that’s what i want to accomplish, to show users only my domain. However, the home/default.html path is not a ‘physical’ file in my server, it is based on MCMS. I think your approach will only work if there is an existing physical file of home/default.html. I saw the generated logs and it is clearly looking for the physical file. Is my goal still possible with my current setup? Kindly advice. Thanks.