How to prevent Googlebot from overwhelming site?

Question

I\'m running a site with a lot of content, but little traffic, on a middle-of-the-road dedicated server.

Occasionally, Googlebot will stampede us, resulting in Apache

Answer 1

You can set how your site is crawled using google's webmaster tools. Specifically take a look at this page: Changing Google's crawl rate

You can also restrict the pages that the google bot searches using a robots.txt file. There is a setting available for crawl-delay, but it appears that it is not honored by google.