3 of 4
1 of 4state no such exception
Assistant makers whose docs say user-triggered fetches may ignore robots.txt
Does robots.txt stop an AI assistant from reading my page?
For automatic crawling, yes. For a fetch a user triggered, three of the four major assistant makers publish that robots.txt may not apply.
3 of 4
Assistant makers whose docs say user-triggered fetches may ignore robots.txt
October 2026 · Global
1 of 4
state no such exception
For comparison
This is the gap most robots.txt advice misses. Blocking a training crawler is well documented and works. But when a person asks an assistant about your hotel and it goes to look, OpenAI, Google and Perplexity all publish that the rule may be ignored, because the fetch was requested by a human rather than scheduled by a machine. Anthropic's documentation states no such exception.
Source
- Reference period
- Geography
- Global
- Evidence type
- Analysis
- Sample
- OpenAI, Google and Perplexity each publish that user-initiated fetches may ignore robots.txt; Anthropic's documentation states no equivalent exception
- For comparison
- state no such exception: 1 of 4
- Last checked
Method
Read from each publisher's own crawler documentation in October 2026. OpenAI: "Because these actions are initiated by a user, robots.txt rules may not apply." Google: "these fetchers generally ignore robots.txt rules." Perplexity: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Anthropic states its bots honour robots.txt and names no exception. Silence is not a commitment, so read the fourth as undocumented rather than as a guarantee.
Related questions
Related indicators
42%
29%sometimes fact-checkConsumers who always fact-check AI outputs
25%
Travellers who received outdated or inaccurate AI travel information
70%
Tourism businesses already using AI
94%
60%properties with under 10 staffCybersecurity readiness gap by property size
52%
Executives reporting AI agents in production
6
