Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

General Questions /

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, Ed, biapar, Willr, Ingo, swaiba


Go to End

3 Posts   2227 Views


Community Member, 54 Posts

17 February 2009 at 6:10am

Edited: 17/02/2009 6:11am

I used sitemap generator, and i saw that assets and theme folder is getting into sitemap,
So, i added robots.txt
User-Agent: *
Disallow: /themes/
Disallow: /assets/

Almost all the cms are coming with robots.txt file after the install is finished. Could SS do the same ?
And, 2nd, will be bad for search engine indexing if i put in robot.txt

Or, should i ask what folder from SS would not go under Disallow


Community Member, 89 Posts

18 February 2009 at 12:10am

ohh yes man!^^
and i think your Sitemap generator does not work with SS, because the sites are not in the folders..

here is very easy going tutorial to make sitemap in SS:

have fun


Community Member, 54 Posts

18 February 2009 at 10:47pm

THX for that,

my xml sitemap is working well (added to google and is ok). Only when I created the xml sitemap with other parts of the sites were crowled (like themes/images or assets).

My question was about what to putand not in robots.txt

Here is how other cms is making a robots file
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/

For SS i looked on both .org and .com sites and i saw only the Disallow: /admin is enough for preventing parts of the site like themes folder to be index by search engine or not ?