robots.txt
Generate robots file
Rule 1
User-agent: * Allow: /
Frequently Asked Questions
What is robots.txt?
Robots.txt is a file that tells search engine crawlers which pages or sections of your site should not be crawled. It is placed in the root directory of your website.
Does robots.txt block pages from appearing in search results?
No, robots.txt only prevents crawling, not indexing. Pages can still appear in search results if linked from other sites. Use noindex meta tag to prevent indexing.
What is the difference between Disallow and Noindex?
Disallow in robots.txt prevents crawling but pages may still be indexed. Noindex meta tag prevents indexing but allows crawling. Use both for complete blocking.