Overview
Introduction
It's easy to deploy a Cloudflare Pages project with the exact same robots.txt on both its production domain and its pages.dev staging subdomain, which quietly invites search engines to index a duplicate copy of your site.
This tool generates the right robots.txt for whichever environment you're deploying: open for production, closed for staging.
What Is Cloudflare Pages robots.txt Generator?
A robots.txt generator with two modes: production (allow all crawlers, list any private paths to exclude, and add a Sitemap line pointing at your real domain) and staging (a blanket Disallow: / for every crawler).
It's meant to be deployed as two different robots.txt files, one committed for your production build, one for your pages.dev preview/staging build, if your build process can tell the two apart.
How Cloudflare Pages robots.txt Generator Works
In production mode, the generator writes a `User-agent: *` block with a Disallow line for each private path you list (or a bare `Disallow:` meaning nothing is excluded), followed by a Sitemap line built from the production domain you enter.
In staging mode, it ignores all other inputs and outputs the standard two-line block that tells every well-behaved crawler not to touch any path on that subdomain.
When To Use Cloudflare Pages robots.txt Generator
Use it once per Cloudflare Pages project to get both the production and staging robots.txt right from the start, rather than discovering months later that Google indexed your pages.dev URL.
It's also useful whenever you add a new private section to your site and want it excluded from crawling going forward.
Often used alongside Cloudflare Staging Domain Blocker, pages.dev Duplicate Content Checker and AI Crawler Policy Generator.
Features
Advantages
- Makes the production/staging distinction explicit as a single toggle, instead of hoping someone remembers to write two different files.
- Automatically builds the Sitemap line from your production domain, one less manual detail to get wrong.
- Supports any number of private paths without needing to hand-format each Disallow line.
Limitations
- robots.txt is advisory only, disallowing a path doesn't remove it from search results if it's already indexed or linked to from elsewhere, use noindex meta tags or the X-Robots-Tag header for that.
- Doesn't generate the sitemap.xml file itself, it only references its expected URL.
Examples
Best Practices & Notes
Best Practices
- Make sure your build pipeline actually serves different robots.txt content on staging versus production, a single static file committed to your repo will apply to both unless you branch on environment.
- Combine a staging Disallow: / with an X-Robots-Tag: noindex header (see the Staging Domain Blocker tool) for stronger protection than robots.txt alone provides.
- Keep the Sitemap line pointed at your actual production domain, not the pages.dev URL, even if you generate this file from a staging build.
Developer Notes
Production mode always emits a `User-agent: *` block; each private path is validated to start with "/" and rendered as its own `Disallow:` line, with a trailing `Sitemap:` line built by stripping any protocol/trailing slash from the domain input and appending `/sitemap.xml`. Staging mode short-circuits all other options and returns the fixed disallow-all text.
Cloudflare Pages robots.txt Generator Use Cases
- Setting up correct, differentiated robots.txt files for a project's production and staging environments
- Excluding a newly added admin or internal section from search crawling
- Auditing an existing robots.txt by comparing it against the tool's recommended production/staging templates
Common Mistakes
- Shipping the same robots.txt to both the production domain and the pages.dev staging subdomain, letting search engines index duplicate content.
- Assuming Disallow: / on staging is sufficient on its own, without also blocking indexing at the header level for crawlers that ignore robots.txt.
Tips
- Run the Staging Domain Blocker tool next, it bundles this same staging robots.txt together with the matching _headers and _redirects entries in one step.