A robots.txt file is one of the smallest files on a website, but a single incorrect rule can stop important pages from being crawled. This guide explains what the file does, where it belongs and how to use it safely.
How robots.txt works
Compliant crawlers request the file at the root of a domain before exploring URLs. Rules are grouped by user-agent. An asterisk applies to all crawlers, while names such as Googlebot target a specific crawler.
User-agent: * Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Crawling is not indexing
Blocking a URL in robots.txt does not guarantee removal from search results. A search engine may discover the URL through external links. Use noindex when a crawlable page should not appear in search results.
Safe implementation checklist
- Place the file at the root of the domain.
- Use exact paths and correct capitalization.
- Do not block CSS or JavaScript required for rendering.
- Add an absolute sitemap URL.
- Test changes before and after publishing.
A focused generator can reduce syntax mistakes, but the website owner still needs to review whether each rule matches the site structure.