robots.txt Builder

Ask Vibe Bot

Questions about robots.txt rules? Ask away.

Build your robots.txt

Create user-agent groups, allow/disallow rules, and add sitemap/crawl-delay. Copy or download the result.

User-agents
Rules
Extras
Tip: Use /$ to only allow the homepage while blocking the rest. 


      

        
Best practices:

        
    
          
  • Blocking a path doesn’t remove it from Google. Use noindex meta or password if needed.
    • 
          
  • Wildcard * and end anchor $ are widely supported by major crawlers.
    • 
          
  • Add all your sitemap URLs at the end of the file.
    •