Important Notice: Our web hosting provider recently started charging us for additional visits, which was unexpected. In response, we're seeking donations. Depending on the situation, we may explore different monetization options for our Community and Expert Contributors. It's crucial to provide more returns for their expertise and offer more Expert Validated Answers or AI Validated Answers. Learn more about our hosting issue here.

Im having a problem getting the online spider to work, what could be wrong?

April 26, 2017online problem Spider wrong

0

Posted

Im having a problem getting the online spider to work, what could be wrong?

1 Answer

0

Posted

There are a few things that can go wrong with the online spider app. Sometimes it gets tied up if the servers are being slammed with link requests, or it may get stuck on the robots.txt file and seem to go nowhere. If this is the case, in the spider app is a link for settings, open that link and you can adjust all the settings for the spider. For example you can select for the spider to ignore the robots.txt file, you can adjust the speed and style of contextual parsing etc. If you continue to have problems with the online spider, you can always download the personal desktop version of the application and run it directly from your desktop.