How do I prevent the search engine from indexing Web pages?
To prevent the Penn State Search Engine from indexing your Web site or your entire Web server, your Web server administrator should use the Robots Exclusion Protocol. The User-Agent string used by the Penn State search engine is PennStateSpider. To prevent the Penn State search engine from indexing or following links on an individual Web page you can use the following Robots Meta tag between the
and tags of an HTML page. If you wish to prevent an entire document from being indexed by the search engine, then insert the tag within the tags and omit the tag. NOTE: It may take up to seven days for the search engine to recognize the update to your document(s) and that it needs to remove all or part of the document(s) from the index. By default, when a server is indexed (for example, www.someserver.psu.edu) all documents, provided that all files/directories are referenced from th