Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

General Questions /

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Robots.txt


Reply


3 Posts   1899 Views

Avatar
OnebyOne

Community Member, 54 Posts

17 February 2009 at 6:10am

Edited: 17/02/2009 6:11am

I used auditmypc.com sitemap generator, and i saw that assets and theme folder is getting into sitemap,
So, i added robots.txt
User-Agent: *
Disallow: /themes/
Disallow: /assets/

Almost all the cms are coming with robots.txt file after the install is finished. Could SS do the same ?
And, 2nd, will be bad for search engine indexing if i put in robot.txt
Disallow:/cms/
Disallow:/jsparty/
Disallow:/mysite/
Disallow:/sapphire/
Disallow:/modules&widgets/

Or, should i ask what folder from SS would not go under Disallow

Avatar
SalvaStripe

Community Member, 89 Posts

18 February 2009 at 12:10am

ohh yes man!^^
and i think your Sitemap generator does not work with SS, because the sites are not in the folders..

here is very easy going tutorial to make sitemap in SS:
http://doc.silverstripe.org/doku.php?id=tutorial:site-map&s=sitemap

have fun

Avatar
OnebyOne

Community Member, 54 Posts

18 February 2009 at 10:47pm

THX for that,

my xml sitemap is working well (added to google and is ok). Only when I created the xml sitemap with auditmypc.com other parts of the sites were crowled (like themes/images or assets).

My question was about what to putand not in robots.txt

Here is how other cms is making a robots file
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
.....................................

For SS i looked on both .org and .com sites and i saw only the Disallow: /admin is enough for preventing parts of the site like themes folder to be index by search engine or not ?