Meta has paused the updated business terms it planned to switch on Nov. 20, 2026. Its Business Help Center notice says the updates to its Commercial Terms, Product Catalog Terms, Self-Serve Ad Terms and Ad Creative Generative AI Terms "will not take effect on November 20, 2026," and that the existing terms stay in effect. Nieman Lab reported the pause on Nov. 5.
That is good news for publishers who objected to the new terms, but it changes less than it sounds for advertisers. The Self-Serve Ad Terms in force today already give Meta the right to crawl the websites of anyone who places an ad. If you run Meta ads, the useful question is not what the paused terms said. It is what Meta's crawlers can read on your site right now.
The updates would have added crawling language to Meta's Commercial Terms, which apply to businesses using Meta's tools, not only to advertisers. Critics, publishers among them, read that as letting Meta crawl the sites of any business with a Meta account. A Meta spokesperson told Reuters the company is pausing the changes "to get this right." Search Engine Journal, which compared the documents, reports that the Commercial Terms in force today (dated March 6, 2024) have no crawling paragraph, and that the paused version would have added one.
Meta has not given a new date. Its notice says the existing terms remain in effect.
What the ad terms in force already allow
We read Meta's current Self-Serve Ad Terms, which list an effective date of Aug. 22, 2025. One paragraph says that by placing an order, the advertiser grants Meta "the right to access, index, cache, analyze or crawl the content" available through the advertiser's "domains, webpages or URLs." The stated purposes are developing and improving "advertising and other business-related products," related uses, "research and training," and maintaining the integrity of Meta's products.
That paragraph does not mention robots.txt or any way to opt out. It is a contract, and the pause leaves it exactly as it was.
Meta's crawler documentation lists five user agents. Each has a different job, so blocking one does not block the others:
- facebookexternalhit: fetches a page when someone shares a link on Meta's apps, for the title, description and thumbnail. Meta says it "might bypass robots.txt" for security or integrity checks.
- Meta-WebIndexer: crawls to improve Meta AI search results. Meta says allowing it helps Meta cite and link to your content in Meta AI's answers.
- Meta-ExternalAds: crawls for "improving advertising and other business-related products and services." Meta shows how to block it with a robots.txt
Disallow rule.
- Meta-ExternalAgent: crawls for "training foundation AI models" or for improving products by indexing content directly. Also blockable in robots.txt.
- Meta-ExternalFetcher: fetches a single link when a user asks for it. Meta says it "may bypass robots.txt rules."
Meta also says robots.txt changes can take up to 24 hours to apply, because its crawlers may cache the file that long.
On Nov. 10, 2026, we fetched the robots.txt files of 56 large US sites: 20 news publishers, 15 retailers, 11 travel, health, jobs, real estate and local review sites, and 10 SEO and website platform sites. Fifty returned a file we could read; six did not respond to our script. For each of Meta's five crawlers, we recorded whether the file names it and whether the rules let it fetch the home page, using Python's standard robots.txt parser.
What we found:
- Meta-ExternalAgent (AI training) is named by 20 of the 50 sites and blocked from the home page on 18, including The New York Times, The Washington Post, The Guardian, the BBC, Reuters, Amazon, eBay, Zillow and Tripadvisor.
- Meta-ExternalFetcher is blocked on 12 and Meta-WebIndexer on 8.
- Meta-ExternalAds is named by just one site, Reuters, which gives it the same path rules as most crawlers and leaves the home page open. It is blocked from the home page on only two sites, The Wall Street Journal and Yelp, and neither names it: both block every crawler they don't list.
- 16 of the 50 sites block Meta's AI training crawler but let its ads crawler in.
Big sites are not typical advertisers, and a robots.txt file is not the whole story: a site can also block crawlers at its firewall or CDN, which our test can't see. But the pattern is clear. Sites that took the time to shut out Meta's AI crawlers mostly have not thought about the ads crawler, whose stated purposes overlap.
What we don't know
- Whether robots.txt overrides the contract. Meta's documentation shows advertisers how to block Meta-ExternalAds, and the ad terms grant crawling rights without mentioning robots.txt. Meta has not said which one wins for a site that advertises and blocks the crawler.
- What blocking does to your ads. Meta does not say whether blocking Meta-ExternalAds affects ad review, delivery or automated creative for an advertiser. We have not tested it, and you should not assume there is no cost.
- What the next version of the terms will say. Meta says it will share an update later.
What to do this week
Our view: the pause is not a reason to stop paying attention. If you advertise on Meta, the crawling grant you care about is already in force.
- Read your robots.txt for all five Meta user agents. Most sites name none of them. Know which of Meta's crawlers you allow today, not only Meta-ExternalAgent.
- Keep link previews working. Leave facebookexternalhit allowed on public pages, or shared links lose their title and image.
- Keep gated and private sections out of reach. Member areas, client portals, staging copies and paid content should not be crawlable by anyone. Block them for Meta-ExternalAds and Meta-ExternalAgent by name, and better, put them behind a login.
- Decide on Meta-ExternalAgent on its own merits. Blocking AI training crawlers is a reasonable choice for many sites. Blocking only that one while leaving the ads crawler open covers less than it seems, because the ad terms list "research and training" among their purposes.
- Don't block Meta-ExternalAds on pages you advertise until Meta explains the effect. Those are the pages its ad systems need to read, and the terms already let Meta crawl them.
If you don't advertise on Meta, the Self-Serve Ad Terms don't apply to you. The paused Commercial Terms would have reached businesses that only use Meta's tools, so if your business has a Meta account, read the next version when Meta publishes it.