Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

We've moved the forum!

Please use forum.silverstripe.org for any new questions (announcement).
The forum archive will stick around, but will be read only.

You can also use our Slack channel or StackOverflow to ask for help.
Check out our community overview for more options to contribute.

General Questions /

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, Ed, biapar, Willr, Ingo, swaiba

Googlebot issues


Go to End


4 Posts   1475 Views

Avatar
HeartlandTech

Community Member, 17 Posts

31 March 2012 at 1:30pm

Does anyone have a solution for getting the hits from Googlebot? It's creating a ton (literally 1476) session files on my server (causing them to be very unhappy) all with zero byte count. I tried blocking the IP and they come in on another one. They are hitting twice a minute . . . they keep trying my old calendar install. When they get the 404 Silverstripe is showing the "page missing" message and I suspect Google is going "ok, and we'll try back again since you didn't send a 404 message". Any ideas?

Avatar
swaiba

Forum Moderator, 1899 Posts

1 April 2012 at 9:06pm

Avatar
HeartlandTech

Community Member, 17 Posts

3 April 2012 at 3:23pm

Allow: /pages/spider.php
Crawl-delay: 10
User-agent: *
Disallow: /webcal/
Disallow: /calendar/

Nope . . . still hammering it

Avatar
swaiba

Forum Moderator, 1899 Posts

4 April 2012 at 6:03am

ok, you can tweak the robots file to tell various agents to ignore various parts of the site.

Maybe if you don't care for SEO just disallow all?

User-agent: *
Disallow: /