Home › SEO Tools › Technical SEO
Robots.txt Generator
robots.txt is the first file crawlers read on your site — small mistakes can block important pages or waste crawl budget on junk. Set your defaults, list the paths to block, and generate a clean, valid file.
Use the tool
How to use the Robots.txt Generator
- Add your sitemap URL and any paths you want blocked.
- Adjust any optional settings to match your page, business or campaign.
- Click the button to generate your result instantly — nothing is uploaded; the tool runs in your browser.
- Copy the result, or download it as a file where that option appears, and paste it into your site, document or campaign.
Frequently asked questions
Where does robots.txt live?
At the root of your domain: yoursite.com/robots.txt. It must be a plain text file with exactly that name.
Does Disallow hide a page from Google?
It stops crawling, but a blocked page can still appear in results if other sites link to it. Use noindex to remove a page from results.
Should I block my admin area?
Blocking obvious admin and checkout paths is common and sensible, but never block CSS/JS files Google needs to render your pages.