Why is the Ask Jeeves crawler downloading the same page on my site multiple times?
Generally, the Ask Jeeves crawler should only download one copy of each file from your site during a given crawl. There are two exceptions: • A URL may contain commands that “redirect” the crawler to a different URL. This may be done with the HTML command: < META HTTP-EQUIV="REFRESH" CONTENT="0; URL=http://www.your page address here.html" > or with the HTTP status codes 301 or 302. In this case the crawler downloads the second page in place of the first one. If many URLs redirect to the same page, then this second page may be downloaded many times before the crawler realises that all these pages are duplicates. • An HTML page may be a “frameset.” Such a page is formed from several component pages, called “frames.” If many frameset pages contain the same frame page as components, then the component page may be downloaded many times before the crawler realises that all these components are the same. Q: Why is the Ask Jeeves crawler trying to download incorrect links from my server? Or fr
