Is “web scraping” legal?
You should always check the Terms of Use for a given website you wish to use to determine if they allow automated access to their site (for example, Google does not, except via their API), and remember that Anthracite is specifically licensed for use only with materials for which you have the proper rights, so you should really ask this question of your attorney. There is also a lot of public domain material out there (eg, government laws & publications) and your “fair use” rights may give you wide latitude with how you use certain materials. There is also a movement to help clarify license usage terms for copyrighted material, with explicit clearances granted for certain types of uses, check out the Creative Commons.
When it comes to web scraping public information, then there definitely is no legal issue behind it. There is nothing illegal about grabbing the exchange rates from remote sites or scraping thousands and even millions of documents, movie files, and PDFs from other sites. Some websites, however, limit web scraping by mentioning it within their terms of use. But to this day, the legality of web scraping remains ambiguous. Danish Maritime and Commercial Court (Copenhagen) has found that web scraping is not in conflict with the database directive of the European Union. Within the United States, many cases of web scraping have been dismissed. However, in 2008, an Irish airline filed a suit against a website that was web scraping its ticket availability information to sell tickets. Courts are yet to release a verdict in this case.
