Tag Archives: crawlers

Developer

Common crawlers or bots

Leave a comment

Archive.org (Wayback Machine) user-agent:

  • archive.org_bot

Bing user-agents:

  • Bingbot
  • MSNBot
  • MSNBot-Media
  • AdIdxBot
  • BingPreview

Google user-agents:

  • Googlebot
  • Googlebot-News
  • Googlebot-Image
  • Googlebot-Video
  • Googlebot-Mobile
  • Mediapartners-Google
  • Adsbot-Google

References:

Archive.org – https://archive.org/details/archive.org_bot

Bing – http://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0

Google – from the Google Search Console robots.txt Tester20150528-googlebot

Listing of all Google bots20150528-all-google-bots