Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

General Questions /

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Googlebot issues


Reply


4 Posts   567 Views

Avatar
HeartlandTech

Community Member, 17 Posts

31 March 2012 at 1:30pm

Does anyone have a solution for getting the hits from Googlebot? It's creating a ton (literally 1476) session files on my server (causing them to be very unhappy) all with zero byte count. I tried blocking the IP and they come in on another one. They are hitting twice a minute . . . they keep trying my old calendar install. When they get the 404 Silverstripe is showing the "page missing" message and I suspect Google is going "ok, and we'll try back again since you didn't send a 404 message". Any ideas?

Avatar
swaiba

Forum Moderator, 1804 Posts

1 April 2012 at 9:06pm

Try a robots.txt...

http://www.robotstxt.org/robotstxt.html

Avatar
HeartlandTech

Community Member, 17 Posts

3 April 2012 at 3:23pm

Allow: /pages/spider.php
Crawl-delay: 10
User-agent: *
Disallow: /webcal/
Disallow: /calendar/

Nope . . . still hammering it

Avatar
swaiba

Forum Moderator, 1804 Posts

4 April 2012 at 6:03am

ok, you can tweak the robots file to tell various agents to ignore various parts of the site.

Maybe if you don't care for SEO just disallow all?

User-agent: *
Disallow: /