List of crawlers / bots needed

ASP Free Forums Sponsor:

April 20th, 2008, 08:12 PM

Dr_Rock

Wizard Down Under

Join Date: Dec 2006

Location: Australia

Posts: 249

Time spent in forums: 3 Days 3 h 52 m 35 sec
Reputation Power: 211

List of crawlers / bots needed

Hi folks, I am in search of a list of user-agent or IP ranges used by crawlers so that I can exclude them from a hit counter, can anyone point me in the right direction?

Cheers!

__________________
For my first trick watch me turn a zero into a one...

April 21st, 2008, 01:12 AM

Dr_Rock

Wizard Down Under

Join Date: Dec 2006

Location: Australia

Posts: 249

Time spent in forums: 3 Days 3 h 52 m 35 sec
Reputation Power: 211

Got it!

Ok, I found one, seems to be short a couple though...

Use this array to check against the http_user_agent, should be good for asp or php...

Code:

Array("Teoma", "alexa", "froogle", "inktomi", "looksmart", "URL_Spider_SQL", "Firefly", "NationalDirectory", "Ask Jeeves", "TECNOSEEK", "InfoSeek", "WebFindBot", "girafabot", "crawler", "www.galaxy.com", "Googlebot", "Scooter", "Slurp", "appie", "FAST", "WebBug", "Spade", "ZyBorg", "rabaz", "msnbot")

Feel free to extend the list!

Last edited by Dr_Rock : April 21st, 2008 at 01:45 AM. Reason: forgot msnbot

April 22nd, 2008, 06:49 AM

Shadow Wizard

Moderator From Beyond

Join Date: Sep 2004

Location: Israel

Posts: 26,892

Folding Points: 338184 Folding Title: Super Ultimate Folder - Level 1

Time spent in forums: 3 Months 1 Week 5 Days 7 h 56 m 32 sec
Reputation Power: 1537

nice find.. thanks for sharing!

found one bot on this forum that is not in your list:
Gigabot

__________________

Visit The Windows Scripting Guru

Last edited by Shadow Wizard : April 22nd, 2008 at 06:59 AM.

April 27th, 2008, 07:39 PM

Dr_Rock

Wizard Down Under

Join Date: Dec 2006

Location: Australia

Posts: 249

Time spent in forums: 3 Days 3 h 52 m 35 sec
Reputation Power: 211

Thanks SW, heres my function:

Code:

Private Function IsBot()
    Dim arrBotList, intCount, blnIsBot
    arrBotList = Array("Teoma", "alexa", "froogle", "inktomi", "looksmart", "URL_Spider_SQL", "Firefly", "NationalDirectory", "Ask Jeeves", "TECNOSEEK", "InfoSeek", "WebFindBot", "girafabot", "crawler", "www.galaxy.com", "Googlebot", "Scooter", "Slurp", "appie", "FAST", "WebBug", "Spade", "ZyBorg", "rabaz", "msnbot", "Gigabot")
    blnIsBot = False
    for intCount = 0 to Ubound(arrBotList)

        If (InStr(Request.ServerVariables("HTTP_USER_AGENT"), arrBotList(intCount))) Then
	    blnIsBot = True
	    Exit For
	End If
    Next
    IsBot = blnIsBot
End Function

April 28th, 2008, 04:28 AM

Shadow Wizard

Moderator From Beyond

Join Date: Sep 2004

Location: Israel

Posts: 26,892

Time spent in forums: 3 Months 1 Week 5 Days 7 h 56 m 32 sec
Reputation Power: 1537

cheers, this can be slightly improved though:

Code:

Private Function IsBot()
	Dim arrBotList, intCount, blnIsBot
	Dim strUserAgent
	arrBotList = Array("Teoma", "alexa", "froogle", "inktomi", "looksmart", "URL_Spider_SQL", "Firefly", "NationalDirectory", "Ask Jeeves", "TECNOSEEK", "InfoSeek", "WebFindBot", "girafabot", "crawler", "www.galaxy.com", "Googlebot", "Scooter", "Slurp", "appie", "FAST", "WebBug", "Spade", "ZyBorg", "rabaz", "msnbot", "Gigabot")
	blnIsBot = False
	strUserAgent = Request.ServerVariables("HTTP_USER_AGENT")
	for intCount = 0 to Ubound(arrBotList)
		If (InStr(strUserAgent, arrBotList(intCount))) Then
			blnIsBot = True
			Exit For
		End If
	Next
	IsBot = blnIsBot
End Function

searching the Request.ServerVariables collection will take some CPU cycles,
having the same search in loop will waste more than you need..

Comments on this post

Dr_Rock agrees: didnt event think of that, cheers!

Thread Tools	Search this Thread
Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread: