Crawl budget is the maximum number of pages a search engine crawls on a website in a given amount of time. It is basically the attention that Google will give to your website. If your crawl budget is insufficient, parts of your site will be very slow to be discovered and ranked by Google and by other search engines. This means missed opportunities to drive qualified traffic and conversions. But unfortunately there’s no way to increase your crawl budget, though major changes on a website can spur Google to “rediscover” large portions of the site. For most websites, crawl budget is managed by Google and never needs to be a concern. But the larger your website is, the more important the crawl budget becomes.
When your crawl budget is “spent” on pages that should not be searchable, it can prevent priority pages from being indexed and ranked. And though you can’t get “more” crawl budget, you can direct bots to priority pages in order to better spend the budget you have. With Oncrawl, you can combine data from crawls and server logs to detect parts of your site that waste crawl budget, find the pages that need more attention, and confirm changes in Googlebot behavior after you make improvements.