How does the Fast Spider Used By Lycos Work?
April 2004 The spider goes to a web site and follows hyperlinks throughout the site. The spider parses (reads in) certain pages and downloads them into a file or database as it moves through a web site. The spider is set to only parse a set number of pages on the same site. When the spider has reached the limit of URLs determined by the user’s subscription service, it stops spidering your web site. The file of spidered URLs are then built into a search catalog. These URL’s are displayed as part of search results when a relevant match is made. Spidering starts at the root URL (ex: www.lycos.com is the root URL vs. www.lycos.com/search) and discovers the total number of URL’s in your Web Site, up to a maximum of 500 URL’s. Then, the spider collects and submits text from the parsed page into the catalog used to display search results. The spider will gather text from the number of web pages that the user included in the subscription, accessing pages by breadth and then depth. For example,
