#tell compliant bots to crawl slower
User-agent: * 
Crawl-Delay: 5