Robots.txt Generator

A robots.txt file is a plain text file at the root of a website (https://example.com/robots.txt) that gives instructions to web crawlers about which pages or directories they may or may not access. It follows the Robots Exclusion Protocol. Well-behaved crawlers like Googlebot and Bingbot respect these rules, but robots.txt is not a security measure — it is publicly visible and malicious bots can ignore it. For truly sensitive pages, use authentication or firewall rules instead.

Disallow in robots.txt prevents crawlers from fetching a URL. Noindex is a meta tag or HTTP header that lets crawlers access the page but instructs them not to include it in search results. If you Disallow a URL, crawlers cannot read any noindex tag on it — which can paradoxically prevent de-indexing of already-discovered pages. Use noindex to remove pages from search results; use Disallow to save crawl budget on genuinely unnecessary pages like admin areas and search result pages.

Yes — use the User-agent directive to target specific crawlers. User-agent: * applies rules to all bots. User-agent: Googlebot applies rules only to Google's crawler. Rules for a specific User-agent take precedence over * rules for that bot. You can use User-agent: GPTBot with Disallow: / to block OpenAI's training crawler, or User-agent: CCBot to block Common Crawl. Each bot should be in a separate User-agent/Disallow block.

No — never disallow CSS and JavaScript files. Google needs to crawl and render these resources to understand your page layout and content. Blocking them with robots.txt can cause Googlebot to misrender your pages, leading to poor indexing quality. Older SEO advice recommended blocking these file types to save crawl budget, but this is now counterproductive. Only disallow directories that are truly irrelevant to search (admin panels, internal APIs, duplicate parameter URLs).