Hi,

The following links may help:
http://www.robotstxt.org/
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360

Ben

Is there something specific you're trying to accomplish? Generally, I think a normal SS install should be fine without a robots.txt file unless you wanted to do something like http://silverstripe.com/robots.txt.

Sounds like a good idea, Sam.

Not sure if assets/ should be included in the exclusion list, though. I can see some people liking it there because they don't want their assets showing up in search engines separately from the pages holding those assets. On the other hand, there are those who want their assets to be indexed so that they will show up in things like Google image search. The "assets/" directive would prevent that.

Ben

I use