Preview Mode Links will not work in preview mode

Search Off the Record takes you behind the scenes of Google Search and its inner workings! In each episode, the folks from the Search Relations team will give you background info on the decision-making behind launches, feature prioritization in Search Console, and the projects Google Search teams are working on. They will share fun stories from the many conferences they attend as well as from their day-to-day working life at Google. They will also dive into the currently trending conversations in the SEO community at large. Have a listen!

  1. All Episodes
  2. Google crawlers behind the scenes

Google crawlers behind the scenes

Mar 12, 2026

Developers often talk about Googlebot as if it were a single program you could just run as “googlebot.exe”, but that is not how Google’s crawling actually works. In this episode of Search Off the Record, Martin and Gary from the Search Relations team unpack how Google’s crawling infrastructure is really built and operated.​
They cover why “Googlebot” is a misnomer and how it relates to a central crawling software-as-a-service used by many Google products​, how crawl behavior is controlled centrally to avoid overwhelming sites (throttling, handling 503s, and “don’t break the internet” safeguards)​ and more!
If you build for the web, work on SEO, or just want a more accurate mental model of how Google crawls pages, this behind‑the‑scenes discussion is for you.

Resources:
​Crawlers → https://developer.google.com/crawling 

Episode transcript → https://goo.gle/sotr107-transcript 

Listen to more Search Off the Record → https://goo.gle/sotr-yt  

Subscribe to Google Search Channel → https://goo.gle/SearchCentral 

Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.

 #SOTRpodcast #SEO #GoogleSearch

Speakers: Martin Splitt, Gary Illyes