How does robots.txt differ from the ROBOTS META tag?
robots.txt is used to tell robots which parts of a site should not be visited, whereas the ROBOTS META tag tells a robot how to treat a specific page. from The Web Robots Pages: The ROBOTS META tag allows HTML authors to indicate to visiting robots if a document may be indexed, or used to harvest more links. No server administrator action is required. Note that currently only a few robots implement this. In this simple example: a robot should neither index this document, nor analyse it for links.
