Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

General Questions

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Robots.txt


Reply

3 Posts   1865 Views

Avatar
OnebyOne

17 February 2009 at 6:10am (Last edited: 17 February 2009 6:11am), Community Member, 54 Posts

I used auditmypc.com sitemap generator, and i saw that assets and theme folder is getting into sitemap,
So, i added robots.txt
User-Agent: *
Disallow: /themes/
Disallow: /assets/

Almost all the cms are coming with robots.txt file after the install is finished. Could SS do the same ?
And, 2nd, will be bad for search engine indexing if i put in robot.txt
Disallow:/cms/
Disallow:/jsparty/
Disallow:/mysite/
Disallow:/sapphire/
Disallow:/modules&widgets/

Or, should i ask what folder from SS would not go under Disallow

Avatar
SalvaStripe

18 February 2009 at 12:10am Community Member, 89 Posts

ohh yes man!^^
and i think your Sitemap generator does not work with SS, because the sites are not in the folders..

here is very easy going tutorial to make sitemap in SS:
http://doc.silverstripe.org/doku.php?id=tutorial:site-map&s=sitemap

have fun

Avatar
OnebyOne

18 February 2009 at 10:47pm Community Member, 54 Posts

THX for that,

my xml sitemap is working well (added to google and is ok). Only when I created the xml sitemap with auditmypc.com other parts of the sites were crowled (like themes/images or assets).

My question was about what to putand not in robots.txt

Here is how other cms is making a robots file
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
.....................................

For SS i looked on both .org and .com sites and i saw only the Disallow: /admin is enough for preventing parts of the site like themes folder to be index by search engine or not ?