How do I request “304 Not Modified” answers from HTTP servers?
To send “Last Modified: xx” headers and get “304 Not Modified” answers from HTTP servers, add following line to the gatherer’s configuration file: HTTP-If-Modified-Since: Yes If the document hasn’t changed since last gathering, the gatherer will use the data from its database, instead of retrieving it again. This will save bandwidth and speed up gathering significantly.