Website owners increasingly want to express preferences for AI-related crawlers. Many crawler operators publish user-agent names that can be addressed in robots.txt.
Example directives
User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: /
These rules communicate a crawl preference to compliant bots. They do not create access control, and crawler behavior can vary.
Decide by purpose
Before blocking a crawler, consider whether discovery, search visibility, citations or content licensing matter to your goals. Maintain a short documented list and review operator documentation periodically because crawler names and policies may change.
Use technical protection for private data
Confidential material requires authentication, server access controls or removal from the public web. Robots.txt is public and advisory.