Tool 02 · Technical
Robots.txt Tester
Paste a robots.txt file and test whether a URL is allowed for a given crawler. Matching follows Google's documented rules: the most specific rule wins, and Allow wins a tie.
Results
What this checks, and what it doesn't
A crawler obeys the most specific group that matches its user agent, falling back to the * group. Within that group, the longest matching path wins, * matches any sequence and $ anchors the end of the URL. If an Allow and a Disallow rule are equally long, Allow wins.
Work with me
Need help applying this?
Reading is the easy part. If you want this implemented on your site, tell me what you're working on.
Work With Me Follow the SEO RoadmapEmail — [email protected]
Based in — India · working remotely
Response — usually within 2 working days