How big are the results?
The Crawled URLs files average about 25-30MB per million URLs. The Analyzed URLs files can be much larger because you control the size of the analysis results. For example, the method mentioned above to extract all the page contents will produce huge results files that you will need to access. For example, a 100m page crawl where each result was 10KB will return 1TB (one terabyte) of results that you will need to download. This will take over 18 days to download if you have a fairly typical 5Mbps broadband connection and will still take 1 day if you have a 100Mbps connection. So please consider doing as much processing inside 80legs as possible to reduce the size of your results!
