WordPress can serve a virtual robots.txt file automatically, but many site owners need custom rules for administrative paths and a sitemap. The safest setup is usually short and easy to audit.
A practical WordPress example
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/wp-sitemap.xml
The allow rule keeps a commonly used WordPress endpoint accessible. Avoid copying large lists of disallow rules without understanding them.
Check your sitemap
Modern WordPress versions commonly expose a sitemap at /wp-sitemap.xml. SEO plugins may use /sitemap_index.xml instead. Open the URL before placing it in robots.txt.
Common mistakes
- Blocking the entire site with Disallow: /.
- Using a staging rule after launching publicly.
- Blocking uploads when images should appear in search.
- Adding a sitemap URL that returns an error.
After updating the file, open yourdomain.com/robots.txt in a private browser window and confirm the public version is correct.