A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (as Wikipedia says) Googlebot, YahooBot, and MSNbot an ...