Important Notice: Our web hosting provider recently started charging us for additional visits, which was unexpected. In response, we're seeking donations. Depending on the situation, we may explore different monetization options for our Community and Expert Contributors. It's crucial to provide more returns for their expertise and offer more Expert Validated Answers or AI Validated Answers. Learn more about our hosting issue here.

Is there an optimal algorithm for web crawling and indexing?

April 26, 2017ALGORITHM crawling indexing optimal web

0

Posted

Is there an optimal algorithm for web crawling and indexing?

1 Answer

0

Posted

Answer Hello Nancy… Thanks for the email. Sorry that I have not responded earlier but I just received it today (Sat)…… There are a number of resources that I would recommend to you. There are no optimal algorithmic scripts as all are based upon your specific needs and requirements. Some initial resources to help you on your way to knowledge discovery are: http://www.bottechnology.com/AIIP/ These are bookmarks that I recommended from my keynote speech last month in New Orleans for the Association of Independent Information Professionals (AIIP) I would also recommend the following URLs as well: http://www.asp101.com/articles/chris/spider/default.asp http://www.sunworld.com/sunworldonline/swol-11-2000/swol-1103-regex.html http://www.planet-source-code.com/xq/ASP/txtCodeId.6216/lngWId.4/qx/vb/scripts/S… http://www.thiink.com/eSpider/default.htm http://www.xav.com/scripts/xavatoria/search.txt http://www.acme.com/resources/classes/Acme/Spider.java http://www.w3.org/Robot/ http://www.