Robots.txt Tester
Enter a URL to instantly test whether Googlebot and other bots can crawl that page based on robots.txt.
What is the Robots.txt Tester?
Robots.txt is a file in your site's root directory that tells search engine bots which pages they may crawl. This tool evaluates the URL you enter against the site's robots.txt rules and shows whether a bot like Googlebot can crawl that page and which rule applies.
How to use it
- 1Enter the full URL you want to test (e.g. https://yoursite.com/private-page).
- 2Pick a bot (default: Googlebot) and press 'Check'.
- 3See the result: whether the page is crawlable, which rule matched and the sitemaps.
Frequently asked questions
My page shows 'Blocked' — what should I do?
If you want that page to appear in Google, remove or narrow the relevant Disallow rule in robots.txt. Blocked pages are usually not crawled.
If robots.txt blocks a page, will it never show in Google?
Robots.txt blocks crawling; however, a URL linked from other sites can still be indexed without its content being crawled. To fully exclude a page, use a 'noindex' meta tag instead of robots.txt.
If Allow and Disallow conflict, which wins?
In Google, the longest (most specific) matching rule wins; if lengths are equal, Allow takes precedence. This tool applies the same logic.
What if I have no robots.txt?
Without a robots.txt, bots crawl all pages by default. That is not a problem; robots.txt is not required for most small sites.