Login | Forgot password | Register
What is OpenID?
OpenID is an Internet-wide identity system that allows you to sign in to many websites with a single account.
With OpenID, your ID becomes a URL (e.g. http://username.myopenid.com/). You can get a free OpenID for example from myopenid.com.
For more information visit the official OpenID site.
General Questions
SilverStripe Forums » General Questions » robots.txt - help
General questions about getting started with SilverStripe that don't fit in any of the categories above.
|
Page:
1
|
Go to End | |
| Author | Topic: robots.txt - help | 568 Views |
-
robots.txt - help
2 July 2009 at 11:27am
This has been asked in the past with no conclusive answer.
What would be sensible content? Can anyone up' an example?
Much appreciated.
-
Re: robots.txt - help
3 July 2009 at 12:23am
Hi,
The following links may help:
http://www.robotstxt.org/
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360Ben
-
Re: robots.txt - help
3 July 2009 at 12:31am
Generally, they do. So thank you.
But I was seeking a SS-specific example / recommendations.
-
Re: robots.txt - help
3 July 2009 at 2:51am
Is there something specific you're trying to accomplish? Generally, I think a normal SS install should be fine without a robots.txt file unless you wanted to do something like http://silverstripe.com/robots.txt.
-
Re: robots.txt - help
3 July 2009 at 12:37pm
Perhaps we should bundle a default robots.txt file with the installer?
User-agent: *
Disallow: /admin
Disallow: /assets -
Re: robots.txt - help
7 July 2009 at 12:29am
Sounds like a good idea, Sam.
Not sure if assets/ should be included in the exclusion list, though. I can see some people liking it there because they don't want their assets showing up in search engines separately from the pages holding those assets. On the other hand, there are those who want their assets to be indexed so that they will show up in things like Google image search. The "assets/" directive would prevent that.
Ben
-
Re: robots.txt - help
2 February 2010 at 12:05am
Disallowing assets doesn't make sense to me since there might be pdf or txt files which are useful content for search engines.
It would make sense to disallow cache flushing by search engines:User-agent: *
Disallow: /admin
Disallow: /?flush
| 568 Views | ||
|
Page:
1
|
Go to Top |
