Jump to:

23483 Posts in 18976 Topics by 2878 members

General Questions

SilverStripe Forums » General Questions » robots.txt - help

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Page: 1 2
Go to End
Author Topic: 3986 Views
  • sheldon
    Avatar
    Community Member
    1 Post

    Re: robots.txt - help Link to this post

    Should any of the following lines be included in robots.txt?

    Disallow: /cms/
    Disallow: /sapphire/
    Disallow: /mysite/
    Disallow: /userforms/
    Disallow: /themes/

    Please help! Thanks.

  • itobe
    Avatar
    Community Member
    4 Posts

    Re: robots.txt - help Link to this post

    If it is still relevant...

    The CMS automatically redirects the requests you are disallowing.

    my robots.txt looks like this

    ==============================

    User-agent: *
    Disallow: /admin
    Disallow: /dev
    Disallow: /?flush
    Disallow: /assets
    Allow: /

    User-agent: ia_archiver
    Disallow: /

    User-Agent: Googlebot-Image
    Disallow: /admin
    Disallow: /dev
    Disallow: /?flush
    Allow: /

    Sitemap: http://www.mypage.com/sitemap.xml

    ==============================

    3986 Views
Page: 1 2
Go to Top

Want to know more about the company that brought you SilverStripe? Then check out SilverStripe.com

Comments on this website? Please give feedback.