38 / SEO
Robots.txt Tester & Generator
Build or paste a robots.txt file, validate crawler groups and sitemaps, then test the exact Allow or Disallow rule for any bot and path.
Crawler rule workbench
Write the file. Test the exact path.
Build or paste a robots.txt file, catch unsupported directives, and see which longest matching rule applies to a crawler and URL. Everything stays in this tab.
Quick builder
Start with one clean group.
robots.txt
Edit rules
Ready. 0 uploads.
Path test
Would this crawler get in?
Blocked by disallow: /private/.
Matched group: *Line review
Issues and receipts
Field note
What this robots.txt tester checks—and what the file cannot do
Robots.txt manages crawling, not guaranteed indexing. A disallowed URL can still appear without a snippet if other pages link to it. Use a visible noindex rule or authentication when a page must stay out of search.
The tester selects the most specific matching user-agent group, finds matching Allow and Disallow paths, then uses the longest rule. Allow wins an equal-length tie. Wildcard * and end anchor $ are supported. Query strings are included when you test a full URL.
Google supports User-agent, Allow, Disallow, and Sitemap fields. It ignores Crawl-delay, so Siftlet warns instead of pretending that directive controls Googlebot. Put the final file at the top level of the exact host and protocol it should govern.