Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

General Questions

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

robots.txt - help


Reply

10 Posts   4054 Views

Avatar
sheldon

27 September 2011 at 9:38am Community Member, 1 Post

Should any of the following lines be included in robots.txt?

Disallow: /cms/
Disallow: /sapphire/
Disallow: /mysite/
Disallow: /userforms/
Disallow: /themes/

Please help! Thanks.

Avatar
itobe

23 December 2011 at 10:35am Community Member, 4 Posts

If it is still relevant... :)

The CMS automatically redirects the requests you are disallowing.

my robots.txt looks like this

==============================

User-agent: *
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Disallow: /assets
Allow: /

User-agent: ia_archiver
Disallow: /

User-Agent: Googlebot-Image
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Allow: /

Sitemap: http://www.mypage.com/sitemap.xml

==============================