Seo Doktoru
Robots.txt & Sitemap Generator

Robots.txt & Sitemap Generator

Create and download robots.txt rules and sitemap.xml in a few clicks.

robots.txt

User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /cart
Disallow: /checkout

Sitemap: https://ornek.com/sitemap.xml

What is the Robots.txt & Sitemap Generator?

This tool builds, in a few clicks, the robots.txt file that tells search engines which parts of your site to crawl, and the sitemap.xml that lists all your important pages. Correct configuration helps you use your crawl budget efficiently.

Robots.txt & Sitemap Generator

How to use it

  1. 1Enter the allow/disallow paths and the sitemap address for robots.txt.
  2. 2Add your page addresses (one URL per line) for the sitemap.
  3. 3Download the robots.txt and sitemap.xml files and upload them to your site's root.

How to read the result

Allow: /
Default — The whole site is crawlable. This is the right starting point for most sites.
Disallow: /account/
Private areas — Block paths that have no business in search results — login, cart, account pages.
Disallow is not noindex
Common mistake — robots.txt blocks crawling, not indexing. To truly hide a page use a noindex tag.
Sitemap line
Do not forget — Add your sitemap URL at the end of the file; it helps search engines find content faster.
AI crawlers
A deliberate choice — Blocking GPTBot or ClaudeBot means you will not appear in AI-generated answers.

When to use it

Custom Stack Deployments

Ideal for custom or headless sites lacking automated SEO plugins, ensuring search crawlers discover clean root directives and primary routes immediately.

E-Commerce Crawl Optimization

Blocks facet filters, internal search, and checkout loops from burning crawl budget while funneling Googlebot directly to high-margin canonical products.

Post-Staging Environment Migration

Removes hazardous 'Disallow: /' development rules during site launch, preventing catastrophic drops in organic impressions and complete de-indexation.

Common mistakes

Blocking CSS and JS

Restricting asset folders prevents search engines from rendering pages correctly. Always keep design and script paths crawlable to pass mobile friendliness checks.

Submitting Non-200 URLs

Never place 404 errors, 301 redirects, or noindex pages into your sitemap. The XML document should strictly feature indexable, canonical 200 OK endpoints.

Missing Trailing Slashes

Writing 'Disallow: /blog' unintentionally blocks paths like '/blog-post'. Always add a trailing slash like 'Disallow: /blog/' to restrict only the intended directory.

Frequently asked questions

What is robots.txt for?

Robots.txt tells search engine bots which parts of your site they may crawl; blocking unnecessary or private pages protects your crawl budget.

Why do I need a sitemap?

A sitemap presents all your important pages to search engines in one file, helping new or weakly linked pages get discovered faster.

Where do I upload the files?

Both files belong in your site root: robots.txt → site.com/robots.txt, sitemap → site.com/sitemap.xml. Also submit the sitemap in Search Console.

Is there a limit to how many URLs a sitemap can hold?

A single sitemap.xml file cannot exceed 50,000 URLs or 50 MB uncompressed. If your website exceeds either threshold, split your links across multiple sitemaps and reference them inside a parent sitemap index file.

How quickly does Google recognize robots.txt changes?

Updates take effect as soon as Googlebot recrawls your robots.txt file, which typically happens within a few hours up to 24 hours. You can verify and accelerate updates using the Robots Testing Tool in Google Search Console.