Webmaster Claims Spider Entered Contract In Suit

The Web and artificial intelligence have brought about some surreal, science fiction like questions. The most recent mind-bending concept is whether or not robots can enter into contracts – that is, is a Web crawler implicitly entering a contract posted on a website announcing copyright conditions?

A little while back, we explored the idea that RSS, as an automatic distribution agent, could robots.txt file) to prevent Internet Archive's robot from scanning, copying, and storing the pages on Shell's website. She discovered that the Wayback Machine had reproduced the contents of her website about 87 times in five years, "and displayed her entire website to the public daily during that period." 

Shell sued the company for conversion, civil theft, breach of contract, and violations of the Racketeering Influence and Corrupt Organizations Act (RICO) and the Colorado Organized Crime Control Act (COCCA).

As might be guessed, most of these claims were dismissed. But the breach of contract claim is still under consideration, awaiting more information. The question that will be decided, ultimately, is whether a web crawler that is not blocked by a website can legally be bound by a contract posted on the site.

The outcome of that question could also have important impact on Web-crawling and Internet copyright law itself. Search engines like Google have leaned on Fair Use principles when scouring the Web (and off-line libraries) for information. The Google Book Search project defenders have claimed that publishers can "
Of a similar vein, then, is Web crawling. Webmasters must opt-out of indexing via the robots.txt file, preventing the spider from crawling the site. If the courts continue to back search engines' and other Internet companies' right to copy and index at will, then the whole copyright system, by default, it would seem, goes opt-out, too.

Copyright law blogger John Ottaviani, on Internet Archive v. Shell case and its implications in greater detail.