List of User-Agents
(Spiders, Robots,
Crawler, Browser)
Andreas Staeding, Psychedelix.com
Psychedelix.com provides a comprehensive database of
user agent strings identifying browsers, search
engine spiders and crawlers, web directories,
download managers, link checkers, proxy servers,
web filtering tools, harvesters, spambots and
badbots.
|
|
|
|
|
Web crawlers, also referred to as web robots, bots
or search engine spiders, are computer programs designed to
interrogate websites, collect information about web page content,
documents and hyperlinks discovered during this process, and return the
information for inclusion in search engine databases. Each search
engine uses its own set of web crawlers, and at any given moment may have
numerous crawlers active. In the case of distributed web crawlers such as
Grub
and Boitho, there may be hundreds or thousands of web crawlers
active on the internet at any given time.
Once a web crawler or bot has gathered information about a web page, the
information must be collated and indexed. Only when indexing
has been completed will (updated) contents of a web page be available for
listing in the search results of a search engine query. Because of the
size and complexity of the worldwide web, a search engine may require many
months to completely crawl the entire web.
Follow links to the right to learn more about web crawlers, web robots and the
role they play in updating search engine results.
At the left margin, Related Links address topics of interest
pertaining to internet business and ecommerce. View the
Internet Business & eCommerce SiteMap
for a complete list of internet business, web business, website promotion and ecommerce topics.
|
|
Receive updates to this and other pages on
Twitter!
|
