This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

We've moved the forum!

Please use forum.silverstripe.org for any new questions (announcement).
The forum archive will stick around, but will be read only.

You can also use our Slack channel or StackOverflow to ask for help.
Check out our community overview for more options to contribute.

General questions about getting started with SilverStripe that don't fit in any of the categories above.

robots.txt - help

10 Posts 9270 Views

Community Member, 1 Post

27 September 2011 at 9:38am

Should any of the following lines be included in robots.txt?

Disallow: /cms/
Disallow: /sapphire/
Disallow: /mysite/
Disallow: /userforms/
Disallow: /themes/

Please help! Thanks.

Community Member, 4 Posts

23 December 2011 at 10:35am

If it is still relevant... :)

The CMS automatically redirects the requests you are disallowing.

my robots.txt looks like this

==============================

User-agent: *
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Disallow: /assets
Allow: /

User-agent: ia_archiver
Disallow: /

User-Agent: Googlebot-Image
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Allow: /

==============================