AI Crawler Controls for Search, Training and User Fetches

Direct answer: control AI crawler access by documenting a separate decision for search discovery, model training and user-triggered retrieval. OpenAI uses different tokens for these purposes; Bing and Copilot share Bing’s crawl and index foundation; Google Search AI features use Googlebot, while Google-Extended covers separate Gemini uses. Publish only the rules the content owner approves, repeat required wildcard restrictions in specific groups, and verify genuine crawler traffic. No rule guarantees indexing, citation, ranking or traffic.

This guide owns the technical permission and verification decision. For page structure, sourcing and authorship, use PAKWM’s AI search visibility content guide.

Map each crawler token to its documented purpose

Token or channel	Documented purpose	Policy question
OAI-SearchBot	Automatic crawling for ChatGPT search results	Should public pages be eligible for ChatGPT search discovery?
GPTBot	Content that may be used to train OpenAI foundation models	Does the owner permit this separate training use?
ChatGPT-User	User-triggered fetches from ChatGPT and Custom GPTs	Are private resources protected by authentication rather than robots.txt?
bingbot	Bing’s standard crawler; Bing and Copilot share Bing’s crawl and index foundation	Should the URL be discoverable in Bing and eligible for its grounded experiences?
Googlebot	Google Search crawling, including AI Overviews and AI Mode	Should the page be available to Google Search?
Google-Extended	A control for certain Gemini training and grounding uses of Google-crawled content	Does the owner permit those uses outside Google Search?

Target the documented product token, not a full versioned user-agent string. Provider versions and infrastructure can change.

Approve a policy by page class before editing robots.txt

Group URLs by their real access requirement: public and discoverable pages; public pages intentionally excluded from search; licensed or restricted material; and private accounts, customer data, drafts or internal tools. Record the owner, desired search visibility, training choice, reason, approval date and review trigger for each class. Rights, privacy or contract questions need the appropriate legal or content owner.

Robots.txt is not access control. Protect private content with authentication and authorisation. A crawler rule is a request to compliant crawlers, not a barrier against a browser, a malicious bot or a leaked URL.

Separate OpenAI search discovery from training

OpenAI’s current crawler documentation treats OAI-SearchBot and GPTBot independently. Allowing OAI-SearchBot can make content eligible for ChatGPT search while disallowing GPTBot indicates that the content should not be used to train OpenAI foundation models. If both are allowed, OpenAI may reuse one crawl for both purposes.

If a reviewed wildcard group already permits public search crawling, the smallest training opt-out may be:

User-agent: GPTBot
Disallow: /

This example does not create an OAI-SearchBot group, so that bot follows the applicable existing policy. Do not paste it until the complete file, restricted paths and approved outcome have been checked. OpenAI says its search systems may take about 24 hours to adjust after a robots change. A blocked page may still appear as a navigational link when OpenAI learns the URL elsewhere, so robots.txt is not a guaranteed URL-removal control.

Treat ChatGPT-User as a user-triggered fetch

ChatGPT-User represents a request initiated by a person using ChatGPT or a Custom GPT. OpenAI says it is not an automatic search crawler, its requests may not be affected by robots.txt, and it does not determine ChatGPT search eligibility. That makes authentication and authorisation essential for private resources. Do not depend on a robots rule to protect a login, document or account endpoint.

Keep Googlebot and Google-Extended decisions separate

Google’s AI features documentation says AI Overviews and AI Mode use ordinary Google Search technical requirements. There is no separate AI Search crawler: a page must be indexed and eligible for a snippet, and normal Googlebot, robots and preview controls apply. Eligibility still does not guarantee crawling, indexing, inclusion, a supporting link or ranking.

Google-Extended is different. It is a standalone robots token, not a separate HTTP user-agent. It controls whether content Google crawls may be used to improve future Gemini models and for grounding in Gemini Apps and Vertex AI. Google states that it does not affect inclusion or ranking in Google Search. Blocking Googlebot to make a training choice would also block ordinary Search crawling, so use the control that matches the approved purpose.

Apply Bing controls with their documented effects

Bing’s Webmaster Guidelines explain that Bing Search and Copilot experiences use the same core crawl and index foundation; Microsoft does not document a separate Copilot crawler token. Robots.txt controls crawling, not indexing. If Bingbot can process a page, NOINDEX excludes it from Bing Search, Copilot experiences and grounding results.

Bing documents more specific presentation effects: NOARCHIVE prevents content use in Copilot answers and grounding, NOCACHE limits Copilot to the URL, title and snippet, and NOSNIPPET or DATA-NOSNIPPET can reduce preview or citation depth. Apply them only when that outcome is intended. None guarantees ranking, grounding, citations or traffic.

Do not expect a specific group to inherit wildcard rules

Under the Robots Exclusion Protocol, the User-agent: * group is the fallback when no specific matching group exists. If a file adds a named group, repeat any sensitive-path restrictions that bot still needs. Before release, save the existing file and test important URLs, resources and directories against the complete merged policy.

For the separate choice among redirects, canonicals, crawlable noindex directives for supported search engines, removal responses and access controls, use PAKWM’s canonical, redirect and noindex guide.

Verify identity before trusting crawler traffic

A user-agent string is easy to spoof. For OpenAI, compare requests with its current published ranges for OAI-SearchBot, GPTBot and ChatGPT-User. For Bing, use the official Verify Bingbot process or its published ranges. For Google, use forward-confirmed reverse DNS or official IP lists. Refresh cached ranges; do not create a firewall allowlist from an unverified third-party list.

Deploy and monitor a minimal change

1. Capture the live robots.txt response and every existing group.
2. Approve search, training and user-fetch decisions by page class.
3. Merge the smallest rule change and repeat required wildcard restrictions.
4. Test valuable URLs, blocked paths and rendering resources.
5. Publish, then fetch /robots.txt without login and confirm the intended response.
6. Monitor verified crawler requests, response codes and unexpected blocked resources.
7. Review after provider documentation changes and at the organisation’s documented interval.

Measure observable outcomes rather than claiming “AI rankings.” Bing’s AI Performance report describes citations and grounding activity, not ranking or importance. Track verified crawler activity, landing-page referrals when identifiable and qualified outcomes. Record the date, system and query when reviewing citations because answer interfaces change. Use PAKWM’s SEO measurement guide for the broader reporting framework.

Common implementation mistakes

• using GPTBot as if it controls ChatGPT search visibility;
• blocking Google-Extended and expecting Google AI Overviews or AI Mode to disappear;
• blocking Googlebot when the intended choice concerns Google-Extended;
• assuming ChatGPT-User requests are automatic crawler traffic;
• expecting a named group to inherit every wildcard restriction;
• blocking a URL and expecting an index directive on that page to be read;
• treating robots.txt as security or a legal rights-management system;
• trusting a claimed bot name without network verification.

Primary sources

• OpenAI crawler documentation
• Bing Webmaster Guidelines
• Bing crawler list
• Bing AI Performance documentation
• Google Search AI features documentation
• Google common crawlers and Google-Extended
• RFC 9309: Robots Exclusion Protocol

For implementation support, see PAKWM’s SEO service. Need a crawler policy built around your real content classes? Request a quote from PAKWM with your search goals, training choices and restricted paths.