Google: Moving to a CDN Can Overload Your Server or Block Googlebot. Here's How to Switch Safely
Google's crawl team published advice on Nov. 11 for sites moving behind a CDN: let verified crawlers through, warm the cache first, and use 503 or 429, never 403, if the server needs a break. We wrote two small scripts for the first two steps and ran them on this site.
On Nov. 11, 2026, Google published guidance for a Googlebot CDN migration: a site moving behind a CDN can overload its own server with cache misses or block Googlebot with bot protection.
Google recommends allowing verified crawlers through CDN and firewall bot rules, checked against its published IP ranges, rather than challenging every automated visitor.
Google says to pre-warm the CDN cache by requesting your busiest and most crawled URLs at a controlled pace before sending all traffic through it.
If the origin server needs relief, Google says to return HTTP 429 or 503 for a few hours to one or two days at most, and never a 403 or a challenge page to crawlers.
Google's common crawler IP list had 317 ranges on Nov. 11, 2026, published as specific blocks, so a rule that trusts a whole 66.249.x.x range lets in addresses Google does not list.
Google published advice on Nov. 11, 2026 for anyone planning a Googlebot CDN migration: putting a site behind a content delivery network such as Cloudflare, Fastly or Akamai. Two things go wrong, Google says. The new CDN's empty cache sends a flood of requests to your own server, and blanket bot protection can stop Googlebot at the door, which can drop pages out of Search.
The post, by Cherry Prommawin and Gary Illyes of Google Search, comes with the final episode of the first season of Google's video series on crawling. Below is what it says, what to change before a switch, and two short scripts we wrote and ran for the steps that are easy to get wrong.
What did Google say about CDNs and crawling?
Google's crawlers pace themselves by how your site responds. When responses slow down, they back off; when responses are fast and steady, they speed up. A CDN usually makes responses faster, so crawling can ramp up right after the switch.
At the same moment, the CDN starts with a cold cache. The first request for every URL is a miss that goes through to your own server (the "origin"). People mostly visit popular pages, which warm up quickly. Crawlers, Google points out, ask for a much wider range of URLs, so they hit far more misses. The result can be more load on your server right after a change that was meant to reduce it.
The second risk is protection. Many CDNs and web application firewalls offer bot rules that challenge every automated visitor with a CAPTCHA or a block page. Googlebot can't solve a challenge. Google says a crawler that keeps getting blocked may back off, and pages may drop from Search.
What Google recommends
Three steps, all from the post:
Let verified crawlers through. If you block bots, allow verified crawlers in your CDN and firewall settings, checked against Google's published IP ranges. To keep out bots you don't want, use robots.txt or targeted rules for those bots instead of blocking all automated clients.
Pre-warm the cache. Before routing all traffic through the CDN, request your highest-traffic pages, pages you expect to get busy soon, and URLs that are crawled often, at a controlled pace.
Throttle the right way. If the origin needs a break during warm-up, return HTTP 429 or 503 for a short window: a few hours, or one or two days at most. Don't send a 403 or a challenge page to crawlers.
Google's crawl rate documentation explains why the limit matters. A significant number of 500, 503 or 429 responses lowers the crawl rate for the whole hostname, and it rises again once the errors stop. But if Googlebot sees those codes on the same URL over multiple days, the URL may be dropped from the index. For Google Ads, the same page warns that campaigns may be paused and ads may stop serving.
How do you verify a Googlebot IP?
Google gives two ways. The manual one: run a reverse DNS lookup on the IP from your logs, check that the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that name and confirm it returns the same IP. The automatic one: match the IP against Google's published JSON lists, one for each kind of crawler (common crawlers like Googlebot, special-case crawlers like AdsBot, and user-triggered fetchers).
Many CDNs offer a verified bots setting that does this for you. If yours doesn't, or you want to check what your firewall logged, this Node.js script (Node 18 or later, no packages) checks any IP against all five of Google's lists:
// is-google.mjs: node is-google.mjs <ip> [<ip> ...]const LISTS = { 'common crawler (Googlebot)': 'common-crawlers.json', 'special-case crawler (e.g. AdsBot)': 'special-crawlers.json', 'user-triggered fetcher': 'user-triggered-fetchers.json', 'user-triggered fetcher (google.com)': 'user-triggered-fetchers-google.json', 'user-triggered agent': 'user-triggered-agents.json',};const BASE = 'https://developers.google.com/static/crawling/ipranges/';function toBigInt(ip) { if (ip.includes(':')) { const [head, tail] = ip.split('::'); const h = head ? head.split(':') : []; const t = tail ? tail.split(':') : []; const parts = tail === undefined ? h : [...h, ...Array(8 - h.length - t.length).fill('0'), ...t]; return { v: 6, n: parts.reduce((a, p) => (a << 16n) + BigInt(parseInt(p || '0', 16)), 0n) }; } return { v: 4, n: ip.split('.').reduce((a, p) => (a << 8n) + BigInt(+p), 0n) };}function inRange(ip, cidr) { const [net, bits] = cidr.split('/'); const a = toBigInt(ip), b = toBigInt(net); if (a.v !== b.v) return false; const shift = (a.v === 4 ? 32n : 128n) - BigInt(bits); return a.n >> shift === b.n >> shift;}const ranges = [];for (const [label, file] of Object.entries(LISTS)) { const data = await (await fetch(BASE + file)).json(); for (const p of data.prefixes) ranges.push([label, p.ipv4Prefix || p.ipv6Prefix]);}for (const ip of process.argv.slice(2)) { const hit = ranges.find(([, cidr]) => inRange(ip, cidr)); console.log(`${ip.padEnd(24)} ${hit ? `Google: ${hit[0]} (${hit[1]})` : 'not in any Google crawler range'}`);}
We ran it on Nov. 11 against the lists Google generated that morning (the common crawler file is stamped 15:46 UTC). The common crawler list had 317 ranges (170 IPv4, 147 IPv6); special-case crawlers had 272, and the user-triggered fetcher lists 1,088 and 504. The results for a few test addresses:
66.249.66.1 Google: common crawler (Googlebot) (66.249.66.0/27)
66.249.79.40 Google: common crawler (Googlebot) (66.249.79.32/27)
66.249.79.100 not in any Google crawler range
108.177.2.5 Google: special-case crawler (e.g. AdsBot) (108.177.2.0/27)
2001:4860:4801:10::1 Google: common crawler (Googlebot) (2001:4860:4801:10::/64)
1.1.1.1 not in any Google crawler range
The third line is the lesson. Google lists its crawler addresses as small blocks, most of them /27s of 32 addresses each. On Nov. 11 the 66.249.79.x blocks it listed skipped 66.249.79.96 to .127. A firewall rule that trusts all of 66.249.0.0/16 because "that's Googlebot" lets in addresses Google doesn't list. Allow the published ranges, and refresh them; Google regenerates the files.
How do you warm a CDN cache?
Request the pages before the crowd does. This script reads a sitemap (it follows a sitemap index one level down), requests each URL at a fixed rate, and counts the status codes and the CDN's cache header (cf-cache-status on Cloudflare, x-cache on many others):
// prewarm.mjs: node prewarm.mjs <sitemap-url> [requests-per-second] [max-urls]const [sitemap, rps = '1', max = '500'] = process.argv.slice(2);const locs = (xml) => [...xml.matchAll(/<loc>\s*([^<\s]+)\s*<\/loc>/g)].map((m) => m[1]);const xml = await (await fetch(sitemap)).text();let urls = locs(xml);if (/<sitemapindex/.test(xml)) { const child = []; for (const u of urls) child.push(...locs(await (await fetch(u)).text())); urls = child;}urls = urls.slice(0, +max);const gap = 1000 / +rps, tally = {};for (const url of urls) { const t0 = performance.now(); const res = await fetch(url, { headers: { 'user-agent': 'cache-prewarm/1.0' } }); await res.arrayBuffer(); const ms = Math.round(performance.now() - t0); const cache = res.headers.get('cf-cache-status') || res.headers.get('x-cache') || '-'; const key = `${res.status} ${cache}`; tally[key] = (tally[key] || 0) + 1; if (res.status !== 200) console.log(`${res.status} ${cache} ${ms}ms ${url}`); await new Promise((r) => setTimeout(r, Math.max(0, gap - ms)));}console.log(`\n${urls.length} URLs requested at ${rps}/s`);for (const [k, n] of Object.entries(tally)) console.log(` ${k.padEnd(20)} ${n}`);
Three things to know before you run it:
Keep the rate low. The point is to spread the misses out. One or two requests a second won't trouble most servers; raise it only if your server is comfortable.
Run it against the CDN, not the origin. Point DNS (or a test hostname) at the CDN first, warm it, then send everyone through.
A CDN caches per location. Your requests warm the edge locations nearest to you. Visitors and crawlers elsewhere can still miss at first. It's still worth doing: Google names your busiest and most crawled URLs as the place to start.
We ran it on this site on Nov. 11 at two requests a second: 58 URLs in about 30 seconds, all 58 returned 200, and every one reported cf-cache-status: DYNAMIC. That is Cloudflare's label for a response its cache didn't store, and on this site it is expected. The Search Post is a static site on Cloudflare Pages, and its pages are sent with max-age=0, must-revalidate, so there is no server of ours behind the CDN to overload. On a site with a WordPress or other application server behind the CDN, you'd want to see MISS turn into HIT on a second pass. If it stays DYNAMIC or BYPASS, your HTML isn't being cached at all, and the origin is still answering every crawl.
What should you check this week?
Our view, whether you are planning a CDN move or made one in the past year:
Look at your bot settings. If a bot-blocking or challenge rule is on, confirm verified search crawlers are exempt. Your CDN's security or firewall event log shows what was challenged; filter it for Googlebot and Bingbot user agents, then check those IPs against the lists above.
Check Search Console's Crawl stats report (Settings, then Crawl stats; Domain and root-level properties only) for a change in host status, a jump in "Server error (5XX)" responses, or a fall in requests around your switch date. Google's post doesn't mention the report, but it is where crawl trouble after a switch shows up.
Never use 403 to slow crawlers. If the server is struggling, 503 or 429 for hours, not days. A 403 tells Google the page is forbidden.
Warm the cache before the switch, at a pace your server can take.
Kiran runs Kihan Marketing, a web design, SEO and ads agency, and manages dozens of websites across real estate, home services and healthcare. He writes about SEO, AI search and paid ads news, and what it means for the people who run websites.