What areas of university web sites exclude search engine robots and why?
Universities which exclude search engines robots The ostensible reason for using REP is, of course, to exclude search engines from accessing the university’s web pages. The degree to which the engines were excluded from the sites varied widely. Among the universities which excluded robots, the most circumscribed restrictions which were limited to sites under development. According to the webmaster at the university where this limited restriction was applied, student pages were not excluded from search engines, however, the student pages frequently included links to incomplete sites within the university. It was to these incomplete sites alone that access was blocked. More extreme cases were found in universities which blocked access to any pages other than official university pages. This included both student web pages and internal department pages which contained faculty members’ personal pages. Two main reasons emerged for this type of exclusion: conservation of computer resources an
