Does robots.txt hide a page from search results?

Not reliably. Disallowing a page in robots.txt prevents crawling, but the page can still appear in search results if other sites link to it. Use a noindex meta tag on the page itself for reliable exclusion.

Is robots.txt a security measure?

No. It is a voluntary instruction that well-behaved crawlers respect, but it does not prevent access — anyone can view a robots.txt file and any crawler can choose to ignore it.

What does “User-agent: *” mean?

The asterisk applies the following rules to all crawlers. A specific bot name in that field instead targets only that particular crawler.

Robots.txt Generator

Build robots.txt rules visually, apply presets, and download the file for your site.

Quick Presets

Rules

User-agentAllow / DisallowPath

Sitemap URL (optional)

Generated robots.txt

# robots.txt

Note: robots.txt is advisory only — well-behaved bots follow it, but malicious crawlers may ignore it. It is not a security mechanism.

How to Use

Click a Quick Preset to load a common configuration, or build rules from scratch.
Each rule row has a User-agent (which bot), an Allow/Disallow directive, and a Path.
Use * as the User-agent to target all crawlers.
Paths must start with /. Use / alone to match everything.
Add your Sitemap URL to help search engines discover your content.
Click Copy or Download to use the generated file. Place it at the root of your domain: https://yourdomain.com/robots.txt

About this robots.txt Generator

robots.txt is a plain text file placed at a website's root that tells search engine crawlers which parts of the site they're allowed to access. It's a voluntary standard — well-behaved crawlers (Google, Bing) respect it, but it is not a security or access control mechanism.

Basic syntax

User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xml

"User-agent: *" applies the rules to all crawlers; a specific bot name targets only that crawler. "Disallow" blocks a path from being crawled; "Allow" explicitly permits a path, useful for carving out an exception inside a disallowed folder.

robots.txt does not hide content

Disallowing a page in robots.txt prevents crawling, but the page can still appear in search results (typically without a description) if other sites link to it. To reliably keep a page out of search results, use a noindex meta tag on the page itself, not just a robots.txt disallow rule.

Why reference a sitemap here

Adding a Sitemap line to robots.txt is one of the standard ways to point crawlers to your full sitemap.xml, helping them discover all the indexable pages on the site efficiently, especially for larger sites.