Important Notice: Our web hosting provider recently started charging us for additional visits, which was unexpected. In response, we're seeking donations. Depending on the situation, we may explore different monetization options for our Community and Expert Contributors. It's crucial to provide more returns for their expertise and offer more Expert Validated Answers or AI Validated Answers. Learn more about our hosting issue here.

Does Harvest support the Robot Exclusion Protocol?

April 26, 2017exclusion harvest Protocol Robot support

0

Posted

Does Harvest support the Robot Exclusion Protocol?

2 Answers

0

Posted

Yes. Both robots.txt files and META robots tags are supported. The correct format for robots.txt files is documented at http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html, Harvest may have problems gathering from sites which have incorrectly formed robots.txt files. The format for META robots tags, which give users control over indexing on a page by page basis, is available from http://info.webcrawler.com/mak/projects/robots/meta-user.

0

0 Posted

Yes. Both robots.txt files and META robots tags are supported. The correct format for robots.txt files is documented at http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html, Harvest may have problems gathering from sites which have incorrectly formed robots.txt files.