Home
/ Free tools / Agent-friendly tools
Free tools / Agent-friendly tools
Robots.txt Rule Tester
Paste robots.txt rules and test whether a user-agent would be allowed to crawl a particular URL path using a simplified rule matcher.
How it works
A simple tool for a specific job.
Use the result as a starting point; review important decisions independently.
01 / STEP
Paste policy
Use a robots.txt file or draft.
02 / STEP
Choose crawler and path
Check a named agent with a path.
03 / STEP
Inspect effective rule
Understand which Allow or Disallow was selected.
Frequently asked questions
Good to know.
Will this fetch my live robots.txt file?
No. Browser cross-origin restrictions mean you must paste the text.
Does robots.txt secure private content?
No. Sensitive resources need server-side access control, not crawler instructions.
Keep exploring