Search engine spider agent names
Find below a list of the search engine spider user agent names. These names can help you identify which search engines have sent out a crawler to look at your site.
(Also helpful if you would like to use agent name delivery for your pages)
Excite
ArchitextSpider
Altavista
Scooter/2.0 G.R.A.B. V1.1.0
vscooter/2.0 G.R.A.B. V1.1.0
Scooter/1.0
Scooter/2.0 G.R.A.B V1.0
Infoseek
Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)
Mozilla/4.5 [en] (Win95; I)
Mozilla/4.5 [en] (Win95; I)
InfoSeek Sidewinder/0.9
InfoSeek Sidewinder/0.9
InfoSeek Sidewinder/0.9
Mozilla/3.01 (Win95; I)
Mozilla/3.01 (Win95; I)
Ultraseek
Ultraseek
Mozilla/4.5 (Macintosh; I; PPC)
Mozilla/3.01 (Win95; I)
InfoSeek Sidewinder/0.9
InfoSeek Sidewinder/0.9
Ultraseek
Mozilla/4.04 [en] (Win95; I)
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)
Mozilla/4.5 [en] (Win95; I)
InfoSeek Sidewinder/0.9
Lycos
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)
Lycos_Spider_(T-Rex)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_(T-Rex) ; Lycos_Spider_(T-Rex) )
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_Beta2(T-Rex) ; Lycos_Spider_Beta2(T-Rex) )
Google
Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/)
Googlebot/2.0 beta (googlebot(at)googlebot.com)
Googlebot/2.1 (+http://googlebot.com/bot.html)
Googlebot/2.0 (+http://googlebot.com/bot.html)
Googlebot/1.0 (googlebot(at)googlebot.com)
Googlebot/2.1w (+http://googlebot.com/bot.html)
Googlebot-w/2.1 (+http://googlebot.com/bot.html)
Googlebot/1.0
Northern Light
Gulliver/1.3
Gulliver/1.2
Fast/AlltheWeb
FAST-WebCrawler/2.1.pre.2000-04-14.1 (ashen@looksmart.net)
FAST-WebCrawler/2.1-pre2 (ashen@looksmart.net)
FAST-WebCrawler/2.1.prealpha.2000-04-07.1 (ashen@looksmart.net)
fastlwspider/1.0
FAST-WebCrawler/2.1.pre.2000-04-18.1 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.0.9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-SoccerCrawler/2.2-pre-cvs (oyvinda@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.0.10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre1 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre2 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre3 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre12 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre13 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre14 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
For more information about the spiders then have a look at this page. It has more comprehensive information on this site about the spider IP addresses and host names and even sometimes the network block that a search engine uses.