Important Notice: Our web hosting provider recently started charging us for additional visits, which was unexpected. In response, we're seeking donations. Depending on the situation, we may explore different monetization options for our Community and Expert Contributors. It's crucial to provide more returns for their expertise and offer more Expert Validated Answers or AI Validated Answers. Learn more about our hosting issue here.

Can I insert the crawl download directly into a MYSQL database instead of into an ARC file on disk while crawling?

April 26, 2017ARC Crawl database directly disk download file insert MySQL

0

Posted

Can I insert the crawl download directly into a MYSQL database instead of into an ARC file on disk while crawling?

1 Answer

0

Posted

Yes. See RE: [archive-crawler] Inserting information to MYSQL during crawl for pointers on how but also see the rest of this thread for why you might rather do database insertion post-crawl rather than during.