SEO in Orbit: Unlocking the secrets of indexing

The first ever webinar series sending SEO into space!


Unlocking the secrets of indexing

Aired on: June 12th, 2019

Kevin Indig shares his thoughts on getting pages indexed, how the pages indexed for a site influence site-wide rankings, and what pages shouldn’t be indexed. What is the right approach towards this intermediary step between getting pages discovered and getting them to appear on SERPs?


Kevin Indig
Kevin Indig has helped startups acquire +100M users over the last 10 years. He is VP SEO & CONTENT @ G2, a mentor for Growth @ GermanAccelerator, and ran SEO @ Atlassian and Dailymotion previously. His specialty is user acquisition, brand building, and user retention. Companies Kevin worked with include eBay, Eventbrite, Bosch, Samsung, Pinterest, Columbia, UBS, and many others. You can also find him at


Hosted by: Rebecca Berbel
Rebecca Berbel is the Content Manager at OnCrawl. Fascinated by NLP and machine models of language in particular, and by systems and how they work in general, Rebecca is never at a loss for technical SEO subjects to get excited about. She believes in evangelizing tech and using data to understand website performance on search engines.

Webinar Summary

Presenting Kevin Indig and Rebecca Berbel – [01:10]
Definitions – [03:13]
– Crawling – [03:44]
– Google’s 3-step process – [04:14]
– Indexing – [04:50]
– Log files – [05:18]

Ways to check indexing – [07:09]
– Not recommended: “site:” queries – [07:20]
– Recommended: Search Console URL Inspection – [07:30]
– Recommended: XML sitemaps and the Coverage Report – [07:40]

Importance in distinguishing between crawl-render-index – [08:33]
Different Googlebots – [08:54]
Changing priorities in Google’s process – [09:48]
Crawl: ways Google finds URLs to index – [10:22]
– How Google prioritizes URLs (Crawl budget) – [10:50]
– Popularity: backlinks and PageRank – [11:15]
– Status codes – [11:23]
– Site hygiene – [11:35]
– Page speed and server response time – [11:45]

Rendering: Caffeine update – [12:07]
Indexing: Different clusters for content types – [12:22]
Ranking: Separate algorithms – [12:48]

Improving indexing speed – [13:40]
Improving number of pages indexed – [14:04]
Variation in crawl rates according to page profile – [14:55]
Relation between crawl rate, indexing speed, and rank – [15:32]
Measuring crawl rate – [18:00]
– Focusing on before/after – [18:26]
– Focusing on difference between page types – [18:52]
– Common observations in crawl behavior – [18:42]

Distribution of crawl budget based on business model – [20:19]
Addressing crawl waste – [22:32]
Indexing issues due to similar pages – [25:38]
– Canonical errors – [26:09]
– Site migrations – [26:38]
– HTTP to HTTPS – [27:40]
– Finding source of error when Google ignores the declared canonical – [27:52]

Types of pages that contribute to index bloat – [28:55]
How indexing affects a site as a whole – [31:25]
XML sitemaps for quick indexing – [33:00]
Growing prominence of technical SEO – [33:12]
Q&A – [36:14]
– Bing and indexing 10,000 URLs/day – [36:24]
– When Google ignores robots.txt – [38:28]

Get onboard

Subscribe to SEO in Orbit to be notified as soon as a new webinar is out, and receive a compilation of the best tips and the aftermovie.

About OnCrawl

OnCrawl is an award-winning technical SEO platform that helps you make smarter SEO decisions. OnCrawl combines your content, log files and search data at scale so that you can open Google’s blackbox and build a SEO strategy with confidence. Backed by a SEO crawler, a log analyzer and third-party integrations, OnCrawl currently works with over 800 clients in 66 countries including e-commerce websites, online publishers and travel websites. OnCrawl produces actionable dashboards and reports to support your entire search engine optimization process and helps you improve your rankings, traffic and revenues.