Whats different about SmartInfoSearchs crawler?
Like all search engine spiders, our crawler can only see the text on a webpage. But unlike the others, our crawler reads a page intelligently – that is, it tries to work out what the text actually means in context. Context partly depends on the layout of a page, so our crawler uses structured text to approximate how the page would appear in an ordinary browser. Other crawlers only see a “flat” plain text version of the page, so our crawler has a much better chance of understanding what the page is really about – which means better access to the information you want. • Why is the layout different from the original page? There are many different ways of writing and structuring a webpage, and what our crawler sees won’t always be the same as what you see. It’s a bit like using a different browser. As our crawler develops it will be able to follow ever more layout methods. • Where have all the links gone? Links are not currently shown on cached pages. • Where have all the images gone? Sinc
