Most small businesses should not block AI crawlers. A local physio, builder or accountant wants to be found, and in late 2026 a growing share of “who should I call” questions get answered by ChatGPT, Gemini, Copilot or Perplexity. If those tools can’t read your site, they’ll describe you from whatever else they can find, or recommend someone else.
There is a sensible middle path, though, and a smaller group of businesses for whom blocking makes real sense. The key is understanding that “AI crawler” covers two very different jobs.
Training crawlers and search crawlers are not the same thing
AI companies send out different bots for different purposes.
Training crawlers collect web content that may be used to train future AI models. Blocking them says “don’t learn from my content.” It doesn’t remove you from AI answers today, because those answers increasingly come from live search, not from what the model memorised.
Search and retrieval crawlers fetch pages so an AI tool can cite them in an answer, often with a link. Blocking these is what actually makes you invisible in AI search.
User-triggered fetchers visit a page because a person asked the AI tool to look at it. These behave more like a browser than a crawler, and some companies say robots.txt may not apply to them.
The main AI crawlers, as of late 2026
Bot names change and new ones appear, so check each company’s documentation before you write rules. These are the ones that matter most right now.
| Company | User agent | What it does |
|---|---|---|
| OpenAI | GPTBot | Collects content that may be used for model training |
| OpenAI | OAI-SearchBot | Finds pages to show in ChatGPT search answers |
| OpenAI | ChatGPT-User | Visits a page when a user asks; robots.txt “may not apply” |
| Anthropic | ClaudeBot | Collects content that may be used for model training |
| Anthropic | Claude-SearchBot | Indexes pages to improve Claude’s search results |
| Anthropic | Claude-User | Fetches pages when a user’s question needs it |
| Google-Extended | A robots.txt token controlling Gemini training and grounding (not a separate crawler) | |
| Perplexity | PerplexityBot | Indexes pages for Perplexity’s answers |
| Common Crawl | CCBot | Builds a public web archive many AI models are trained on |
| Apple | Applebot-Extended | Token controlling whether Apple uses content for AI training |
| Meta | Meta-ExternalAgent | Collects content for Meta’s AI models |
The sources: OpenAI’s crawler documentation, Anthropic’s page on its crawlers and Google’s list of common crawlers.
The Google problem
Google is the awkward one. Its documentation says Google-Extended “does not impact a site’s inclusion in Google Search” and isn’t a ranking signal. Blocking it keeps your content out of Gemini model training and Gemini’s grounding, but it doesn’t keep you out of AI Overviews or AI Mode, which are built from the normal Googlebot index.
Robots.txt isn’t the tool for AI Overviews. Since late August 2026, Search Console has had a site-level switch under Settings, then Search generative AI, that keeps your pages out of AI Overviews, AI Mode and generative AI in Discover while leaving your normal rankings alone (Google’s help page on the control). Tags like nosnippet are the blunter option, because they also strip the snippet from your normal search listing. For a small business that relies on Google for enquiries, opting out either way is almost never worth it: you give up the citations and the clicks they bring.
How robots.txt works
Robots.txt is a plain text file at yoursite.com.au/robots.txt. It lists user agents and the paths they’re allowed or disallowed from fetching. Legitimate crawlers from the big companies check it and follow it. Scrapers and bad actors ignore it, so it’s a polite request, not a lock.
On a WordPress site, you can usually edit it through your SEO plugin (in Yoast SEO it’s under Yoast SEO > Tools > File editor) or directly on the server. If you don’t know who has access to your hosting, sort that out first. Our guide to who should own your domain, hosting and logins covers why.
Three settings to choose from
Option 1: Allow everything (our default for most clients)
If your robots.txt doesn’t mention AI bots, they’re allowed. Most WordPress sites already look something like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com.au/sitemap_index.xml
This is right for tradies, health practices, professional services, hospitality and most retailers. Your content is marketing. You want it read, quoted and linked.
Option 2: Block training, allow search
This suits businesses that are happy to be cited in AI answers but don’t want their writing or photos used to train models.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot are untouched, so you stay eligible for AI search answers. Be realistic about what this achieves: it applies to future crawling, and content that was collected before you added the rule isn’t recalled.
Option 3: Block AI entirely
Add the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) to the list above. This makes sense if your content is the product: a recipe developer, a photographer selling prints, a paid course creator, a niche publisher whose income depends on people visiting the page rather than reading a summary.
Even then, think it through. If an AI tool can’t read your site, it can still talk about you based on reviews, social posts and other sites. You lose the ability to shape what it says.
Check you’re not blocking AI by accident
This one catches people out. We regularly find sites blocking crawlers they never meant to:
- Cloudflare and other CDNs. Cloudflare began offering to block AI crawlers by default for new sites in 2025. If your site sits behind Cloudflare, check the bot settings in the dashboard.
- Security plugins. Some WordPress firewall plugins block unfamiliar user agents, and new AI bots look unfamiliar.
- Leftover development settings. A site launched with “Discourage search engines from indexing this site” still ticked (Settings, then Reading in WordPress) blocks everyone, AI included.
- Copy-pasted rules. A robots.txt borrowed from a news publisher’s website may block every AI bot going.
To check, open yoursite.com.au/robots.txt in a browser and read it. If you see Disallow: / under any user agent you care about, that’s a block.
Our recommendation
For most businesses we work with, Option 1 is the right answer and Option 2 is a reasonable choice if training use bothers you. Option 3 is for publishers.
Whatever you choose, it’s only half the picture. Being readable doesn’t mean being recommended: your content still has to answer real questions clearly. Our guide to llms.txt covers another file you’ll hear about, and why it matters far less than robots.txt.
Not sure whether a security plugin or CDN rule is blocking a crawler you want? Checking robots.txt, CDN and plugin settings together is a quick job for our WordPress support team, and it’s one of the first things we look at in an AI search audit.