Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

General Questions

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

robots.txt - help


Go to End
Reply

10 Posts   4044 Views

Avatar
Web Designer Perth

2 July 2009 at 11:27am Community Member, 49 Posts

This has been asked in the past with no conclusive answer.

What would be sensible content? Can anyone up' an example?

Much appreciated.

Avatar
Ben Gribaudo

3 July 2009 at 12:23am Community Member, 181 Posts

Hi,

The following links may help:
http://www.robotstxt.org/
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360

Ben

Avatar
Web Designer Perth

3 July 2009 at 12:31am Community Member, 49 Posts

Generally, they do. So thank you.

But I was seeking a SS-specific example / recommendations.

Avatar
Ben Gribaudo

3 July 2009 at 2:51am Community Member, 181 Posts

Is there something specific you're trying to accomplish? Generally, I think a normal SS install should be fine without a robots.txt file unless you wanted to do something like http://silverstripe.com/robots.txt.

Avatar
Sam

3 July 2009 at 12:37pm Administrator, 685 Posts

Perhaps we should bundle a default robots.txt file with the installer?

User-agent: *
Disallow: /admin
Disallow: /assets

Avatar
Ben Gribaudo

7 July 2009 at 12:29am Community Member, 181 Posts

Sounds like a good idea, Sam.

Not sure if assets/ should be included in the exclusion list, though. I can see some people liking it there because they don't want their assets showing up in search engines separately from the pages holding those assets. On the other hand, there are those who want their assets to be indexed so that they will show up in things like Google image search. The "assets/" directive would prevent that.

Ben

Avatar
Benedikt

2 February 2010 at 12:05am Community Member, 16 Posts

Disallowing assets doesn't make sense to me since there might be pdf or txt files which are useful content for search engines.
It would make sense to disallow cache flushing by search engines:

User-agent: *
Disallow: /admin
Disallow: /?flush

Avatar
kcd

7 February 2011 at 10:00am Community Member, 54 Posts

I use

User-agent: *
Disallow: /admin
Disallow: /?flush
Disallow: /myotherprivatedirectory
Allow: /

User-Agent: Googlebot-Image
Disallow: /admin
Disallow: /?flush
Disallow: /myotherprivatedirectory
Disallow: /assets
Allow: /

Go to Top