Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

General Questions /

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

robots.txt - help


Go to End
Reply


10 Posts   4366 Views

Avatar
sheldon

Community Member, 1 Post

27 September 2011 at 9:38am

Should any of the following lines be included in robots.txt?

Disallow: /cms/
Disallow: /sapphire/
Disallow: /mysite/
Disallow: /userforms/
Disallow: /themes/

Please help! Thanks.

Avatar
itobe

Community Member, 4 Posts

23 December 2011 at 10:35am

If it is still relevant... :)

The CMS automatically redirects the requests you are disallowing.

my robots.txt looks like this

==============================

User-agent: *
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Disallow: /assets
Allow: /

User-agent: ia_archiver
Disallow: /

User-Agent: Googlebot-Image
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Allow: /

Sitemap: http://www.mypage.com/sitemap.xml

==============================

Go to Top