Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

We're retiring the forums!

The SilverStripe forums have passed their heyday. They'll stick around, but will be read only. We'd encourage you to get involved in the community via the following channels instead:

General Questions /

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, Ed, biapar, Willr, Ingo, swaiba

Robots.txt


Go to End


3 Posts   2696 Views

Avatar
OnebyOne

Community Member, 54 Posts

17 February 2009 at 6:10am

Edited: 17/02/2009 6:11am

I used auditmypc.com sitemap generator, and i saw that assets and theme folder is getting into sitemap,
So, i added robots.txt
User-Agent: *
Disallow: /themes/
Disallow: /assets/

Almost all the cms are coming with robots.txt file after the install is finished. Could SS do the same ?
And, 2nd, will be bad for search engine indexing if i put in robot.txt
Disallow:/cms/
Disallow:/jsparty/
Disallow:/mysite/
Disallow:/sapphire/
Disallow:/modules&widgets/

Or, should i ask what folder from SS would not go under Disallow

Avatar
SalvaStripe

Community Member, 89 Posts

18 February 2009 at 12:10am

ohh yes man!^^
and i think your Sitemap generator does not work with SS, because the sites are not in the folders..

here is very easy going tutorial to make sitemap in SS:
http://doc.silverstripe.org/doku.php?id=tutorial:site-map&s=sitemap

have fun

Avatar
OnebyOne

Community Member, 54 Posts

18 February 2009 at 10:47pm

THX for that,

my xml sitemap is working well (added to google and is ok). Only when I created the xml sitemap with auditmypc.com other parts of the sites were crowled (like themes/images or assets).

My question was about what to putand not in robots.txt

Here is how other cms is making a robots file
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
.....................................

For SS i looked on both .org and .com sites and i saw only the Disallow: /admin is enough for preventing parts of the site like themes folder to be index by search engine or not ?