robots.txt Generator

robots.txt Generator Online

The robots.txt file tells search crawlers (Googlebot, Yandexbot, Bingbot, etc.) which pages and sections of a website to crawl or skip. Use this tool to create a correct robots.txt without syntax errors.

Key directives

• User-agent — the name of the bot the rules apply to. * matches all bots;
• Disallow — prevents crawling of a path;
• Allow — explicitly permits crawling (overrides Disallow);
• Crawl-delay — delay in seconds between requests (not supported by Google);
• Sitemap — URL of the sitemap for faster indexing.

Frequently Asked Questions

• How do I verify that my robots.txt is working?

  After generating and uploading the file, open https://yoursite.com/robots.txt in a browser. A 404 or blank page means the file is missing. Use Google Search Console (the robots.txt tester) or Yandex.Webmaster to validate it. Enter the paths you blocked and confirm that their status shows as "Disallowed".

• Can one robots.txt file contain rules for different bots?

  Yes, that is standard practice. Each User-agent gets its own block of directives. For example, you can block all bots from /temp/, yet allow Googlebot to crawl a specific file inside it. Our generator lets you add multiple User-agent blocks in sequence so the rule hierarchy stays clear.

• What should I do if my site is not being indexed after setting up robots.txt?

  First check the file for a Disallow: / directive that blocks the entire site. Make sure there are no typos in paths. Then clear the cache in your webmaster tools panel and request a recrawl. If the problem persists, temporarily remove robots.txt or regenerate it via 101ip.ru, keeping only the Sitemap directive and allow rules.