Important Notice: Our web hosting provider recently started charging us for additional visits, which was unexpected. In response, we're seeking donations. Depending on the situation, we may explore different monetization options for our Community and Expert Contributors. It's crucial to provide more returns for their expertise and offer more Expert Validated Answers or AI Validated Answers. Learn more about our hosting issue here.

Why does Harvest gather different URLs between gatherings?

April 26, 2017different gather Gatherings harvest URLs

0

Posted

Why does Harvest gather different URLs between gatherings?

1 Answer

0

Posted

When HTTP-If-Modified-Since is enabled, the candidate selection scheme of the http enumerators will change for successful database lookups. For unchanged URLs, the enumerators will behave more like depth first gatherer. The result of the gatherings should be the same if you are gathering all URLs of a site, but if you gather only parts of a site by using URL=n with n < number of URLs of a site you will get different subset of the system you gather.