How Do You Deal With Too Much Bot Traffic

Recently I noticed the load on one of my servers way beyound what I would expect it to be. I run multi processor servers and even during a backup the load is only around 1.5.

But lately I noticed peak loads that high under normal web traffic.

I know 1.5 is low on an multi processor server, but I am hoping to add much more to those machines and with sustained load that high it leaves no room for expansion. The servers are not cheap, so adding another server to the cluster can only be done if I make money from the last one I added.

I checked the traffic levels and they were very high. After further review I had some bots hitting sites at over 1200 pages a minute. Multiply that by a few hundred bots and clearly I could have a load issue. The potential is there to bring any server to its knees when delivering those volumes.

I created programing to watch connections and block the abusive bots. While logging I became aware of over 600 bots crawling my servers. Many bots from, Japan, China, Germany and so on and on, useless to my customers even if they are legit search indexes.

Another problem I see is that the bots are running from many ip addresses and hitting the same sites from multiple ips at the same time. Why would the need to do that?

Among other things I decided to validate googlebot, msn and yahoo with dns lookups so I could determine that they were actually their bots and not imposters. In 24 hours I found valid bots from the big three hitting one server from 1100 different ips.

Now we are looking at thousands of vaild bots and thousands more email harvesters and content theives.

As a host, the number of sites I can host on a server is greatly reduced by the bot traffic. My customers do not want to hear that their website was being crawled at 3,000 pages a minute and that is why they could not access it. Of course they will blame it on me.

I was able to filter the bots at a firewall level and drop connections based on reverse dns lookups and site crawl rates and my server sits around 0.05 most of the time even with hundreds of pages a minute being accessed.

I am wondering how the rest of you hosts deal with this problem. Do you leave it up to your hosting customers? Or do you have some type of filter to get rid of the bots.

When you have a few sites it is not really a problem, but as you grow it grows exponetially out of control.