Does anyone have a solution for getting the hits from Googlebot? It's creating a ton (literally 1476) session files on my server (causing them to be very unhappy) all with zero byte count. I tried blocking the IP and they come in on another one. They are hitting twice a minute . . . they keep trying my old calendar install. When they get the 404 Silverstripe is showing the "page missing" message and I suspect Google is going "ok, and we'll try back again since you didn't send a 404 message". Any ideas?
Nope . . . still hammering it
ok, you can tweak the robots file to tell various agents to ignore various parts of the site.
Maybe if you don't care for SEO just disallow all?
User-agent: * Disallow: /