Web Crawlers & Bots
Topic Thread:      Business Center   »   Internet Business & eCommerce   »   Internet Marketing   »   Search Engines   »   Web Crawlers   »   Web Crawlers & Bots

Recommended:   Robots.txt File,    Website Promotion,    Advantages of Link Exchange


Web Crawlers & Bots


Web crawlers, also referred to as web robots, bots or search engine spiders, are computer programs designed to interrogate websites, collect information about web page content, documents and hyperlinks discovered during this process, and return the information for inclusion in search engine databases. Each search engine uses its own set of web crawlers, and at any given moment may have numerous crawlers active. In the case of distributed web crawlers such as Grub and Boitho, there may be hundreds or thousands of web crawlers active on the internet at any given time.

Once a web crawler or bot has gathered information about a web page, the information must be collated and indexed. Only when indexing has been completed will (updated) contents of a web page be available for listing in the search results of a search engine query. Because of the size and complexity of the worldwide web, a search engine may require many months to completely crawl the entire web.


Follow links to the right to learn more about web crawlers, web robots and the role they play in updating search engine results. At the left margin, Related Links address topics of interest pertaining to internet business and ecommerce. View the Internet Business & eCommerce SiteMap for a complete list of internet business, web business, website promotion and ecommerce topics.


Receive updates to this and other pages on Twitter!


Ten Spider™ and tenspider™ are trademarks of Ten Spider Enterprises, LLC, and are protected by United States and international trademark laws.
Valid XHTML 1.0!