Robots.txt mistakes often come from copying generic files or forgetting temporary launch settings. Here are ten checks to make before publishing.
- Accidentally using Disallow: / on a live site.
- Uploading the file outside the domain root.
- Misspelling User-agent or Disallow.
- Blocking CSS and JavaScript needed for rendering.
- Expecting robots.txt to secure confidential content.
- Using relative sitemap URLs instead of absolute URLs.
- Leaving an old staging-domain sitemap.
- Blocking image folders when image search matters.
- Creating conflicting rules without testing them.
- Never reviewing the file after a redesign.
Build a review habit
Open the file directly, check server status, verify the sitemap and test important URLs. Keep a dated copy of changes so accidental blocks can be reversed quickly.