Aren Robots Dangerous?
In the early years of the World Wide Web (circa 1992) ill-mannered robots were a problem. Web crawlers would blunder through a site without regard for the privacy of the data or the effect on server load that their visit would have The Robot Exclusion Standard was developed to allow the web developer to control/limit the access to their site. Our Server Check Pro robot abides by this standard. If you have excluded robots from your site, our robot will not be able to monitor your site unless you add the following entry into the robots.txt file for your server. User-agent: NetMechanic Disallow: In addition, our robot will only retrieve files designated by you as part of your account setup. Our robot will also visit your site no more than once every 15 minutes per monitoring station. So the load impact of our robots on your server is minimal.
In the early years of the World Wide Web (circa 1992) ill-mannered robots were a problem. Web crawlers would blunder through a site without regard for the privacy of the data or the effect on server load that their visit would have The Robot Exclusion Standard was developed to allow the web developer to control/limit the access to their site. Our Server Check Pro robot abides by this standard. If you have excluded robots from your site, our robot will not be able to monitor your site unless you add the following entry into the robots.txt file for your server.
