Preview Mode Links will not work in preview mode

Search Off the Record takes you behind the scenes of Google Search and its inner workings! In each episode, the folks from the Search Relations team will give you background info on the decision-making behind launches, feature prioritization in Search Console, and the projects Google Search teams are working on. They will share fun stories from the many conferences they attend as well as from their day-to-day working life at Google. They will also dive into the currently trending conversations in the SEO community at large. Have a listen!

Mar 14, 2024

In this episode of Search Off the Record, Gary Illyes and Lizzi Sassman take a deep dive into crawling the web: what is a web crawler, and how does it really work? Listen along as the Search team is joined by an expert web developer in the SEO community, Dave Smart, for an in-depth and technical discussion of all things crawling, and maybe dispel some myths along the way. 



Episode transcript → 

Managing your crawl budget →

Dave Smart on LinkedIn →

Tame the Bots →

Search Central Help Forum →

Indexing API docs →


Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.




Speaker: Gary Illyes, Lizzi Sassman

Products Mentioned: Search Off The Record