Seo Doktoru
Robots.txt Tester

Robots.txt Tester

Enter a URL to instantly test whether Googlebot and other bots can crawl that page based on robots.txt.

What is the Robots.txt Tester?

Robots.txt is a file in your site's root directory that tells search engine bots which pages they may crawl. This tool evaluates the URL you enter against the site's robots.txt rules and shows whether a bot like Googlebot can crawl that page and which rule applies.

Robots.txt Tester

How to use it

  1. 1Enter the full URL you want to test (e.g. https://yoursite.com/private-page).
  2. 2Pick a bot (default: Googlebot) and press 'Check'.
  3. 3See the result: whether the page is crawlable, which rule matched and the sitemaps.

How to read the result

Allowed
Crawlable — Googlebot can reach this URL. That does not guarantee indexing, only access.
Blocked
Not crawlable — robots.txt is blocking it. For a live page this is usually not what you want.
Blocked but indexed
Confusing — robots.txt does not prevent indexing. External links can still put the page in the index; use noindex to hide it.
Blocking CSS and JS
Harmful — Googlebot cannot render the page the way you see it and rankings can drop. Never block these files.
Per-bot rules
Check them — A path open to Googlebot may be closed to Bingbot or AI crawlers.

When to use it

Pre-Launch Staging Verification

Confirm that the global 'Disallow: /' directive applied during staging was properly removed before pushing your site live to production.

Parameter Filtering Audits

Test whether complex disallow patterns targeting e-commerce sorting and filtering query strings actually catch dynamic URLs without breaking canonical paths.

Landing Page Isolation

Verify that private PPC funnels or internal checkout paths are strictly hidden from crawl budgets before launching advertising campaigns.

Common mistakes

Adding Noindex Rules

Google officially ignores 'Noindex' lines in robots.txt; doing this stops crawling but does not de-index URLs, requiring on-page meta tags instead.

Overusing Broad Wildcards

Careless use of '/*' or '?' patterns in disallow lines can unintentionally block crucial category pages or vital site-wide assets from crawlers.

Subfolder File Placement

Placing robots.txt in subdirectories like /en/robots.txt makes it invalid; web crawlers look exclusively for the file at the site's top-level root.

Frequently asked questions

My page shows 'Blocked' — what should I do?

If you want that page to appear in Google, remove or narrow the relevant Disallow rule in robots.txt. Blocked pages are usually not crawled.

If robots.txt blocks a page, will it never show in Google?

Robots.txt blocks crawling; however, a URL linked from other sites can still be indexed without its content being crawled. To fully exclude a page, use a 'noindex' meta tag instead of robots.txt.

If Allow and Disallow conflict, which wins?

In Google, the longest (most specific) matching rule wins; if lengths are equal, Allow takes precedence. This tool applies the same logic.

What if I have no robots.txt?

Without a robots.txt, bots crawl all pages by default. That is not a problem; robots.txt is not required for most small sites.

How long does it take for Google to detect robots.txt changes?

Googlebot typically fetches and updates cached robots.txt files within 24 hours. For critical emergency fixes, you can request an instant refresh through Google Search Console's Crawl tools. Otherwise, standard algorithmic visits will naturally discover the modified directives on their next cycle.

Does blocking a directory automatically block all subpages?

Yes, robots.txt directives operate on a prefix-matching basis and naturally cover every subfolder path. For example, 'Disallow: /admin/' locks /admin/settings, /admin/users, and everything nested underneath. To leave a specific subfolder accessible, you must add an explicit 'Allow' exception.