Important Notice: Our web hosting provider recently started charging us for additional visits, which was unexpected. In response, we're seeking donations. Depending on the situation, we may explore different monetization options for our Community and Expert Contributors. It's crucial to provide more returns for their expertise and offer more Expert Validated Answers or AI Validated Answers. Learn more about our hosting issue here.

Why does the gatherer dump core?

April 26, 2017core dump gatherer

0

10 Posted

Why does the gatherer dump core?

1 Answer

0

10 Posted

Core dumps can occur for a variety of reasons. However – a number of people have reported core dumps occuring in the gatherer before any objects are fetched. This is most often caused by the site having a malformed robots.txt file. If you’re attempting to fetch a page http://www.blob.org/blobby/blah.html then Harvest will first check the robots.txt file at http://www.blob.org/robots.txt The correct format of this file is documented at http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html Under some circumstances Harvest may dump core if this file is corrupt.