robots.txt tester — will this URL be crawled?
Paste a robots.txt, enter a path and a crawler, and see whether it is allowed or blocked, and by which rule. It runs entirely in your browser (0 uploads, works offline).
How to use it
- Paste your robots.txt.
- Enter the path and the user-agent to test.
- Read whether it is allowed and which rule matched.
FAQ
How do I test a robots.txt rule for free?
Paste the file, enter a path like /admin/secret and a crawler name, and this page applies the longest-match rules the way search engines do, telling you whether the path is allowed and which rule decided it — on your device with no upload.
How does allow-versus-disallow work?
The most specific (longest) matching rule wins, and on an equal-length tie an Allow beats a Disallow. The tool follows that behaviour so results match real crawlers.
Is my robots.txt uploaded anywhere?
No. Testing runs entirely in your browser and nothing is transmitted or stored, which you can verify by turning off your Wi-Fi.
Does it fetch my live robots.txt?
No — it tests exactly the text you paste, so you can try changes before you deploy them.
Limits
Applies standard user-agent, Allow, Disallow and wildcard rules; crawl-delay and non-standard extensions are not evaluated.
Related
The sitemap validator checks your sitemap; the hreflang generator builds language tags.
Bookmark this page (Ctrl+D, or ⌘D on Mac) or install the app — it works offline the next time you need it.