Jump to:

23001 Posts in 11702 Topics by 2827 members

General Questions

SilverStripe Forums » General Questions » Googlebot issues

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Page: 1
Go to End
Author Topic: 429 Views
  • HeartlandTech
    Avatar
    Community Member
    17 Posts

    Googlebot issues Link to this post

    Does anyone have a solution for getting the hits from Googlebot? It's creating a ton (literally 1476) session files on my server (causing them to be very unhappy) all with zero byte count. I tried blocking the IP and they come in on another one. They are hitting twice a minute . . . they keep trying my old calendar install. When they get the 404 Silverstripe is showing the "page missing" message and I suspect Google is going "ok, and we'll try back again since you didn't send a 404 message". Any ideas?

  • swaiba
    Avatar
    Forum Moderator
    1769 Posts

    Re: Googlebot issues Link to this post

  • HeartlandTech
    Avatar
    Community Member
    17 Posts

    Re: Googlebot issues Link to this post

    Allow: /pages/spider.php
    Crawl-delay: 10
    User-agent: *
    Disallow: /webcal/
    Disallow: /calendar/

    Nope . . . still hammering it

  • swaiba
    Avatar
    Forum Moderator
    1769 Posts

    Re: Googlebot issues Link to this post

    ok, you can tweak the robots file to tell various agents to ignore various parts of the site.

    Maybe if you don't care for SEO just disallow all?

    User-agent: *
    Disallow: /

    429 Views
Page: 1
Go to Top

Want to know more about the company that brought you SilverStripe? Then check out SilverStripe.com

Comments on this website? Please give feedback.