Jump to:

23355 Posts in 17734 Topics by 2862 members

General Questions

SilverStripe Forums » General Questions » robots.txt - help

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Page: 1 2
Go to End
Author Topic: 3893 Views
  • Web Designer Perth
    Avatar
    Community Member
    49 Posts

    robots.txt - help Link to this post

    This has been asked in the past with no conclusive answer.

    What would be sensible content? Can anyone up' an example?

    Much appreciated.

  • Ben Gribaudo
    Avatar
    Community Member
    181 Posts
  • Web Designer Perth
    Avatar
    Community Member
    49 Posts

    Re: robots.txt - help Link to this post

    Generally, they do. So thank you.

    But I was seeking a SS-specific example / recommendations.

  • Ben Gribaudo
    Avatar
    Community Member
    181 Posts

    Re: robots.txt - help Link to this post

    Is there something specific you're trying to accomplish? Generally, I think a normal SS install should be fine without a robots.txt file unless you wanted to do something like http://silverstripe.com/robots.txt.

  • Sam
    Avatar
    Administrator
    679 Posts

    Re: robots.txt - help Link to this post

    Perhaps we should bundle a default robots.txt file with the installer?

    User-agent: *
    Disallow: /admin
    Disallow: /assets

  • Ben Gribaudo
    Avatar
    Community Member
    181 Posts

    Re: robots.txt - help Link to this post

    Sounds like a good idea, Sam.

    Not sure if assets/ should be included in the exclusion list, though. I can see some people liking it there because they don't want their assets showing up in search engines separately from the pages holding those assets. On the other hand, there are those who want their assets to be indexed so that they will show up in things like Google image search. The "assets/" directive would prevent that.

    Ben

  • Benedikt
    Avatar
    Community Member
    16 Posts

    Re: robots.txt - help Link to this post

    Disallowing assets doesn't make sense to me since there might be pdf or txt files which are useful content for search engines.
    It would make sense to disallow cache flushing by search engines:

    User-agent: *
    Disallow: /admin
    Disallow: /?flush

  • kcd
    Avatar
    Community Member
    54 Posts

    Re: robots.txt - help Link to this post

    I use

    User-agent: *
    Disallow: /admin
    Disallow: /?flush
    Disallow: /myotherprivatedirectory
    Allow: /

    User-Agent: Googlebot-Image
    Disallow: /admin
    Disallow: /?flush
    Disallow: /myotherprivatedirectory
    Disallow: /assets
    Allow: /

    3893 Views
Page: 1 2
Go to Top

Want to know more about the company that brought you SilverStripe? Then check out SilverStripe.com

Comments on this website? Please give feedback.