Google Rewrote Its Crawl Budget Documentation. Here's What Actually Changed.

Published July 23, 2026 by Aldo Giammusso

Google quietly rewrote a large part of how it explains crawling. Buried in the edits are three things it had never put in writing before, including one that changes how you should think about image and product crawlers.

Google has significantly updated its crawl budget documentation. The change was spotted by Shauvik Kumar and reported by Barry Schwartz, and it is more than a tidy-up. Several sentences in the new version state things Google had never actually put in writing.

The documentation also moved. It now lives under a new Google Crawling Infrastructure section at developers.google.com/crawling/docs/crawl-budget, which is a signal in itself that Google is treating crawling as its own topic rather than a subsection of Search.

Here is every meaningful change, what it means in practice, and an honest answer to whether it affects your business at all.

The Changes at a Glance

We pulled the exact wording from Google's live documentation and lined the old text up against the new text.

1. Every Site Starts With the Same Conservative Crawl Budget

This is the headline, and the exact wording matters:

"Every site starts with the same default, conservative crawl capacity limit. If there is demand to crawl more and the site remains healthy, Google's systems will automatically adjust this limit over time."

Note the word same. Google is not saying new sites get a small budget and established sites get a large one from the start. It is saying everyone begins at the same conservative baseline, and the limit rises from there if two conditions are met: there is demand to crawl more, and the site stays healthy.

That is a useful thing to be able to tell a client launching a new site. You do not earn a bigger crawl allowance by being an established brand. You earn it by giving Google a reason to want more pages and by serving them reliably when it asks.

2. Crawl Capacity Is Shared Across All of Google's Crawlers

This is the change most people missed, and it is the one with real operational consequences:

"While each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers. This means that high demand from one crawler can reduce the capacity available for others."

Google does not run one crawler. It runs many, including Googlebot for pages, Googlebot-Image for images, AdsBot for ad landing pages, and a Google Shopping crawler for merchant feeds. The new documentation confirms these all draw from the same pool of capacity for your site.

The practical version: if you dump ten thousand uncompressed product images onto your site, the effort Googlebot-Image spends fetching them is capacity that is no longer available for crawling your actual pages. The same applies if you run dynamic ad targets, since Google notes AdsBot has higher demand when a site is running them.

This makes image weight and media bloat a crawling issue, not just a page speed issue. It is another argument for compressing images properly, which is something we cover in our guide to image SEO and website design.

3. Crawl Demand Is Now Defined by Four Named Factors

The old text was vague about what drives demand. The new text is specific:

"For Googlebot, demand varies based on a site's size, update frequency, page quality, and relevance, compared to other sites."

Two words in there deserve attention. The first is quality. Google is stating plainly that page quality influences how often it bothers to come back. Thin, duplicated, or low-value pages do not just fail to rank, they reduce Google's appetite for crawling your site at all.

The second is compared to other sites. Crawl demand is relative. You are not measured against a fixed standard, you are measured against everyone else competing for the same finite crawling resources.

Google also clarified that each crawler has its own demand, driven by factors specific to that crawler. AdsBot wants more when you run dynamic ad targets, and Google Shopping wants more when you have products in merchant feeds.

4. Crawl Health Now Names Latency and Time to First Byte

The old documentation said that if a site "responds quickly for a while, the limit goes up." The new version is far more precise:

"If the site responds consistently and its response times (including latency and Time to First Byte) remain stable or improve, the limit goes up, meaning more connections can be used to crawl. If the site slows down (latency increases or response times become longer), or responds with server errors or rate-limiting signals (such as HTTP 429), the limit goes down and Google crawls less."

Three specifics are new here. Google now names Time to First Byte, it names HTTP 429 rate-limiting responses, and it emphasises consistency rather than raw speed.

That last point is easy to miss. Google is not only asking whether your site is fast, it is asking whether it is predictable. A site that usually responds in 200ms but regularly spikes to 4 seconds under load can be treated worse than a site that steadily returns everything in 600ms. Cheap shared hosting tends to fail exactly this test, which is one reason hosting quality affects SEO.

5. New Guidance: Support HTTP 304 Responses

This addition got almost no attention and is the most directly actionable item in the whole update. Google added two new best practices:

• "Improve loading speed: Optimize your server response times and resources to make pages load faster."
• "Use HTTP caching: Support 304 (Not Modified) HTTP status codes. If a page hasn't changed since Google last crawled it, returning a 304 code tells Google to reuse the cached version, saving your server bandwidth and resources."

A 304 response means "nothing has changed since you last asked." Instead of sending the entire page again, your server sends a tiny response telling Google to use what it already has.

This is a genuine efficiency win. Every 304 is crawl capacity that gets spent on a page that actually changed instead of one that did not. If you run a site with thousands of largely static pages, correct handling of Last-Modified and ETag headers is now something Google explicitly asks for.

6. Google Softened How It Talks About Its Own Limits

A small change with a revealing tone shift. The old line was almost casual: "Google has lots of machines, but not infinite machines."

The new line is corporate and pointed:

"While Google's resources are extensive, they are finite, and we must prioritize resource allocation across the web."

Read that as what it is. Google is telling site owners that crawling is a cost it is actively managing, and that resources get allocated to sites that justify them. In an era of AI crawlers consuming enormous bandwidth across the web, Google is setting expectations.

One More Clarification: Subdomains Get Separate Budgets

The updated documentation is explicit that Google's crawling infrastructure defines a site as a unique hostname. It gives the example that https://www.example.com/ and https://code.example.com/ are treated as separate sites and have separate crawl budgets.

If you run a blog, a shop, or a documentation site on separate subdomains, each one has its own crawl budget rather than drawing from a shared pool. Google also repeats an important reminder: not every page that gets crawled will be indexed. After crawling, each page still has to be evaluated and assessed for suitability.

The Original Diff

For anyone who wants to see the raw change, here is the side-by-side comparison of the old and new documentation, with the edits highlighted. The clean summary above is easier to read, but this is the source.

Does Any of This Affect Your Business?

Here is the honest answer, which is not what most SEO coverage will tell you: for the majority of small business websites, crawl budget is not your problem and never was.

Google's own guidance is explicit that this material is aimed at very large sites. The thresholds it gives are sites with more than a million unique pages whose content changes weekly, or more than ten thousand unique pages whose content changes daily. A Buffalo contractor with forty pages does not have a crawl budget problem. If those pages are not in Google, the cause is almost always something else, such as thin content, no internal links, or a page Google has judged not worth indexing.

So who should genuinely care?

• Ecommerce sites with large catalogs, faceted navigation, and filter URLs that multiply into thousands of near-duplicate pages.
• Sites with heavy media libraries, because of the shared capacity change above.
• Publishers and directories with large archives and frequent updates.
• Anyone on slow or unreliable hosting, since crawl health is now defined in terms of consistency.

That said, the underlying advice is worth following at any size, because it is the same work that makes a site fast and clean for actual human beings.

What to Actually Do

Based on the updated documentation, the practical checklist is:

• Make your server respond consistently. Stable response times now matter as much as fast ones. Watch Time to First Byte.
• Support HTTP 304 responses. Configure Last-Modified and ETag headers correctly so unchanged pages do not get re-sent.
• Compress your images. With capacity shared across crawlers, bloated media now competes with your own pages.
• Kill duplicate and thin pages. Google names page quality as a demand factor, so low-value pages actively cost you.
• Block what does not need crawling with robots.txt, such as infinite filter combinations and sorted duplicates. Google notes that blocking with robots.txt works better than noindex here, because a noindex page still has to be crawled to be seen.
• Keep sitemaps current and use the lastmod tag so Google can tell what actually changed.
• Check the Page Indexing report in Search Console. Google also renamed this from the old index coverage report in the updated text.

The Takeaway

Google did not change how crawling works this week. It changed how much it is willing to tell you about it, and the new details are genuinely useful: a shared capacity pool across crawlers, a level starting line for every site, quality named as a crawl demand factor, and an explicit request to support caching.

The theme running through all of it is that crawling is a finite resource Google allocates deliberately. Sites that are fast, consistent, well-structured, and worth revisiting get more of it. That has always been true. Google has just now written it down.

Sources

• Google, Crawl Budget Management documentation
• Barry Schwartz, Search Engine Roundtable
• Change originally spotted by Shauvik Kumar; additional analysis by Glenn Gabe

Frequently Asked Questions

What is crawl budget?

Google defines a site's crawl budget as the set of URLs that Google can and wants to crawl. It comes from two things working together: the crawl capacity limit, which is how much crawling your server can handle without being overloaded, and crawl demand, which is how much Google actually wants to crawl your content.

Does every website start with the same crawl budget?

Yes, according to the updated documentation. Google now states that every site starts with the same default, conservative crawl capacity limit. If there is demand to crawl more and the site stays healthy, Google's systems adjust that limit upward automatically over time.

Do Google's different crawlers share crawl capacity?

Yes. This is one of the most useful additions to the documentation. Google now says the crawl capacity limit is shared across all crawlers, so high demand from one crawler can reduce the capacity available for others. If a crawler like Googlebot-Image is working hard on your site, that can leave less capacity for regular page crawling.

Do subdomains have their own crawl budget?

Yes. Google's crawling infrastructure defines a site as a unique hostname. The documentation gives the example that https://www.example.com/ and https://code.example.com/ are treated as separate sites and have separate crawl budgets.

Does crawl budget matter for a small business website?

For most small business sites, no. Google's own guidance says this material is aimed at very large sites, such as those with over a million unique pages with content that changes weekly, or over ten thousand pages with content that changes daily. A typical local business site with a few dozen pages will normally be crawled fine without any crawl budget work.

What is an HTTP 304 response and why does Google recommend it?

A 304 means Not Modified. If a page has not changed since Google last crawled it, your server can return a 304 instead of sending the whole page again. Google's updated documentation recommends supporting these responses because it tells Google to reuse the cached version, which saves your server bandwidth and resources and makes crawling more efficient.