robots.txt
The file that tells crawlers which parts of a site they may fetch.
robots.txt now governs AI visibility as well as search indexing, and the two need separate decisions. A common and costly error in travel is a blanket disallow inherited from a template, which quietly removes a property from the retrieval layer AI answers are built on. Strict parsers end a rule group at the first blank line, so formatting mistakes can disable rules silently.
Related terms
Related indicators
Every figure carries its publisher, sample and field period.