What are Robots, Spiders, WebAnts, and Worms?
If you own a website you need to understand how data collecting companies find and explore your website. Once you understand the basics you can easily master tools to see who is visiting your site and how to stop malicious robots from hacking into your site. It is also important to understand what robots are so that you can create meta data and optimize your website to keep robots coming back and re-indexing your site. Spiders, ants, and worms, oh my! What are they and what do they do? Robots is a catch-all, or generic term for programs and automated scripts that “crawl” through the web (the Internet) and collect data from websites and anything else on the Internet that they can find. • Spiders: Spiders are the same thing as robots and the terms can be used interchangeably. • Worms: Worms are robots but are distinguished because worms are replicating programs (unlike other robots). • Web crawlers: Web crawlers (or, webcrawlers) are also the same as robots. Note: WebCrawler is a specifi
