Should you block AI crawlers in robots.txt?
Blocking AI crawlers can keep your business out of AI answers. Learn which bots to allow, which to block, and how to check your robots.txt file.
robots-txt, ai-search, local-seo
Short answer: it can. But it depends on which bots you block. Some AI crawlers collect content to train models. Others fetch pages so an AI tool can show your business in an answer. Blocking the second kind can make you harder to find.
This post helps you decide, without needing to be technical.
What a robots.txt file does for your Twin Cities business
Your robots.txt file is a small text file at the root of your website. It asks automated visitors (called crawlers or bots) which pages they may fetch. Reputable bots follow it. It is a request, not a lock.
Most small business sites never touch this file. But some website platforms, security plugins, and hosts now add AI bot blocks on their own. So you may be blocking AI bots and not know it.
You can look at yours in a minute. Type your web address into a browser, then add /robots.txt at the end. For example, yourbusiness.com/robots.txt.
Training bots and search bots are not the same thing
This is the part that matters most. AI companies often run more than one bot, and each has a different job.
OpenAI (ChatGPT)
OpenAI explains this in its own crawler documentation. It lists OAI-SearchBot and GPTBot as separate robots.txt tags, so site owners can manage them on their own. For OAI-SearchBot, which is for search, OpenAI says that to help your site appear in search results, it recommends allowing that bot in your robots.txt file.
So if you block OAI-SearchBot, you are asking ChatGPT's search feature to skip your pages. That is a visibility choice, not just a privacy one.
Google (Gemini and AI features)
Google works differently. Its common crawlers page describes Google-Extended as a standalone product token. Publishers can use it to manage whether content Google crawls may be used to train future Gemini models, and for grounding.
Google-Extended is a robots.txt token, not a separate bot with its own user agent string. Google also says its common crawlers obey robots.txt rules when they crawl automatically.
One thing to be careful about: blocking Google-Extended is not the same as blocking Googlebot. Googlebot is what builds the search index. If you block Googlebot, you can disappear from regular Google results. Never do that by accident.
Other AI tools
Other companies, like Anthropic and Perplexity, publish their own bot names and rules. They change over time. Check each company's current help page before you copy a block list from a forum or a blog.
When blocking AI crawlers can hurt you
Think about how a customer finds a plumber, a dentist, or a bakery in St. Paul now. Some still search Google. Some ask an AI assistant. If an AI search bot cannot read your site, it has less to work with when it builds an answer.
Blocking can hurt when:
- You block a search-focused bot, such as OAI-SearchBot, and want to appear in ChatGPT search.
- You use a blanket rule that blocks every bot except a short list.
- A plugin added a block you never reviewed.
- Your host or firewall quietly blocks unknown bots, even though robots.txt allows them.
I can't tell you that allowing a bot will earn you a mention. No one can promise that. Allowing access only removes one barrier. Your content still has to be clear, useful, and specific to your area.
When blocking can make sense
Blocking is a fair choice in some cases. You might want to block training bots if:
- You publish original writing, photos, or guides that you sell or want to protect.
- You have a members-only area or private pricing pages.
- You simply don't want your content used to train AI models.
Most local service businesses want to be found. Their pages are public already: services, hours, service areas, contact details. For those sites, a block on training bots is a values decision. A block on search bots is a trade-off you should make on purpose.
A simple way to decide
Ask yourself two questions.
- Do I want AI tools to be able to mention my business? If yes, allow the search-focused bots and Googlebot.
- Am I comfortable with my public pages being used for model training? If not, block the training tokens, such as GPTBot and Google-Extended, and leave the search bots alone.
That split lets you opt out of training while staying readable to search. It is not perfect. Companies change what their bots do, and policies shift. Recheck every few months.
What the file might look like
Here is a simple example that blocks two training controls but does not touch search bots:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Anything not named in the file stays allowed. Check the current bot names in each company's documentation before you use this. Names and purposes can change.
How to check your own file
- Open yourbusiness.com/robots.txt in a browser.
- Look for lines that say Disallow: / under a named bot. That means the whole site is blocked for that bot.
- Look for a line that says User-agent: * followed by Disallow: /. That blocks every compliant bot, including Googlebot. Fix it right away unless your site is meant to be hidden.
- Check for rules from plugins or your host. Some security tools add them without a clear note.
- Ask your web designer or host whether a firewall also blocks bots. This is separate from robots.txt.
If you are on WordPress, Squarespace, Wix, or Shopify, the file may be edited in a settings page rather than as a raw file. Look in your SEO or crawler settings.
Common mistakes to avoid
- Copying a block list from the internet. Many lists block search bots along with training bots.
- Blocking everything "to be safe." Safe from what? You may be hiding from the customers you want.
- Treating robots.txt as security. Bots that ignore the rules can still fetch public pages. Private information should sit behind a login.
- Never checking again. Your site may have been rebuilt, and the file may have changed with it.
What matters just as much as the file
Even with every door open, AI tools need something worth quoting. Clear service pages, your address and service area, consistent business details, and honest answers to common questions all help. A robots.txt file only decides whether bots can read your site. It does not make them pick you.
If you want a second pair of eyes, Ellment Creative, a web design and search visibility studio in St. Louis Park, offers a $99 one time Website and AI-readiness audit that includes a look at which bots can reach your site.
Questions owners ask about this
Will blocking GPTBot remove me from Google?
No. GPTBot is an OpenAI tag. Google search relies on Googlebot, which is separate.
Will blocking Google-Extended hurt my Google rankings?
Google describes it as a control for Gemini training and grounding use, separate from the crawler that builds the search index. Still, test any change and confirm Googlebot is not blocked.
Do I need to do anything if I never touched the file?
Just look at it once. Plugins and hosts sometimes add rules without telling you.
Can I change my mind later?
Yes. Edit the file and remove the block. Bots will pick up the change the next time they read it, which may take a while.
Drafted with AI by the Ellment Creative studio and published without individual human review.
Tell us what you need
We reply within one business day, and you get a scoped, fixed-price quote in writing before any work begins. We also check, free, whether ChatGPT and Claude recommend your business. Founder-led, from St. Louis Park, MN.