Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

General Questions

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Googlebot issues


Reply

4 Posts   520 Views

Avatar
HeartlandTech

31 March 2012 at 1:30pm Community Member, 17 Posts

Does anyone have a solution for getting the hits from Googlebot? It's creating a ton (literally 1476) session files on my server (causing them to be very unhappy) all with zero byte count. I tried blocking the IP and they come in on another one. They are hitting twice a minute . . . they keep trying my old calendar install. When they get the 404 Silverstripe is showing the "page missing" message and I suspect Google is going "ok, and we'll try back again since you didn't send a 404 message". Any ideas?

Avatar
swaiba

1 April 2012 at 9:06pm Forum Moderator, 1796 Posts

Try a robots.txt...

http://www.robotstxt.org/robotstxt.html

Avatar
HeartlandTech

3 April 2012 at 3:23pm Community Member, 17 Posts

Allow: /pages/spider.php
Crawl-delay: 10
User-agent: *
Disallow: /webcal/
Disallow: /calendar/

Nope . . . still hammering it

Avatar
swaiba

4 April 2012 at 6:03am Forum Moderator, 1796 Posts

ok, you can tweak the robots file to tell various agents to ignore various parts of the site.

Maybe if you don't care for SEO just disallow all?

User-agent: *
Disallow: /