Aren Web robots controversial?
Robots are a fact of life on the Internet. If you’ve used Lycos, Alta Vista, InfoSeek or any other search engine, you’ve tapped into databases which were generated using robots. Robots are software tools. They are not inherently good or bad. However, poorly written robots can do serious damage to a Web site. Since a robot is an automated program, it can submit HTTP requests to a server at a very, very fast rate. This can overload the server to the point where it can’t respond to requests from real human beings using real Web browsers. The solution to this is for the robot to take its time processing your Web site. Our robots pause between each access to your site. That means it will take time to process your entire site, but the alternative could be serious for you and anyone else sharing your server.
