Robots.txt: What It Is, How It Works, and How to Use It for SEO

Robots.txt is a small file, but it plays a big role in SEO. If it’s set up incorrectly, search engines may miss important pages or waste crawl budget on useless URLs. Many ranking and indexing issues start with a bad robots.txt file.

In this guide, I’ll explain what robots.txt is, how search engines use it, common mistakes to avoid, and how to configure it properly so your site stays crawlable and optimized.

What Is Robots.txt?

Robots.txt is a text file placed in the root of your website. It tells search engine bots which pages or sections they can crawl and which ones they should avoid.

Search engines check this file before crawling your site. If something is blocked here, Google usually won’t crawl it, even if the page is important

Where Is the Robots.txt File Located?

The robots.txt file always lives in the root domain.

Example:
https://example.com/robots.txt

Each subdomain needs its own robots.txt file.

How Robots.txt Works

Robots.txt works using simple rules called directives. These rules are read by bots like Googlebot, Bingbot, and others.

It controls crawling, not indexing. That means:

• A page blocked in robots.txt can still appear in search results if linked elsewhere
• But Google won’t crawl the content

Basic Robots.txt Directives Explained

User-agent

Defines which search engine bot the rule applies to.

Example:

User-agent: Googlebot

To apply rules to all bots:

User-agent: *

Disallow

Tells bots which pages or folders not to crawl.

Example:

Disallow: /admin/

This blocks all URLs under /admin/.

Allow

Overrides a disallow rule and lets bots crawl specific pages.

Example:

Allow: /blog/post-1/

Sitemap

Helps search engines find your sitemap faster.

Example:

Sitemap: https://example.com/sitemap.xml

Example of a Simple Robots.txt File

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml

This setup blocks WordPress admin pages but allows necessary files for functionality.

What Robots.txt Is Commonly Used For

Blocking admin or login pages
Blocking staging or test environments
Preventing crawl waste on filters and parameters
Managing crawl budget for large websites

Common Robots.txt Mistakes That Hurt SEO

Blocking Important Pages

Accidentally blocking key pages like product pages, blog posts, or category pages can kill rankings.

Always double-check before publishing changes.

Blocking CSS or JavaScript Files

If Google can’t access CSS or JS files, it may not render your page correctly.

This can hurt Core Web Vitals and mobile usability.

Using Robots.txt to Handle Indexing

Robots.txt does not remove pages from Google’s index.

To remove indexed pages, use:

• noindex meta tag
• Google Search Console removal tool

Blocking the Entire Website by Mistake

This happens more often than you think.

Example of a dangerous rule:

User-agent: *
Disallow: /

This blocks everything.

Robots.txt vs Noindex: What’s the Difference?

Robots.txt controls crawling.
Noindex controls indexing.

If you want a page not to appear in search results, use noindex, not robots.txt.

How to Test Your Robots.txt File

Use Google Search Console → Robots.txt Tester to:

• See blocked URLs
• Test specific pages
• Validate rules before publishing

Always test after making changes.

Best Practices for Robots.txt

Keep it simple
Only block what’s necessary
Never block important pages
Always include your sitemap
Test before and after updates

When You Should Update Robots.txt

After a site migration
When launching new sections
When fixing crawl budget issues
When staging sites go live

Conclusion

Robots.txt is a powerful SEO file when used correctly. A few wrong lines can block traffic, rankings, and growth without you even noticing.

If your site has crawling or indexing issues, reviewing robots.txt should always be one of the first steps in a technical SEO audit.