How does the appliance determine whether a URL has changed since the previous crawl?
Unless the URL matches the patterns in a “Force Recrawl” field we’ve set up, the appliance performs a HEAD request to determine the Last Modified date of the document. If the host claims that the document has not changed in the last 20 days and our indexed version is less than 20 days old, the appliance will assume that we have the most recent version of the document. Once the document is downloaded, the appliance uses a content checksum to determine whether the document has changed.
