21294 Posts in 5734 Topics by 2602 members
|
Page:
1
|
Go to End | |
| Author | Topic: | 1550 Views |
-
Robots.txt

17 February 2009 at 6:10am Last edited: 17 February 2009 6:11am
I used auditmypc.com sitemap generator, and i saw that assets and theme folder is getting into sitemap,
So, i added robots.txt
User-Agent: *
Disallow: /themes/
Disallow: /assets/Almost all the cms are coming with robots.txt file after the install is finished. Could SS do the same ?
And, 2nd, will be bad for search engine indexing if i put in robot.txt
Disallow:/cms/
Disallow:/jsparty/
Disallow:/mysite/
Disallow:/sapphire/
Disallow:/modules&widgets/Or, should i ask what folder from SS would not go under Disallow
-
Re: Robots.txt

18 February 2009 at 12:10am
ohh yes man!^^
and i think your Sitemap generator does not work with SS, because the sites are not in the folders..here is very easy going tutorial to make sitemap in SS:
http://doc.silverstripe.org/doku.php?id=tutorial:site-map&s=sitemaphave fun
-
Re: Robots.txt

18 February 2009 at 10:47pm
THX for that,
my xml sitemap is working well (added to google and is ok). Only when I created the xml sitemap with auditmypc.com other parts of the sites were crowled (like themes/images or assets).
My question was about what to putand not in robots.txt
Here is how other cms is making a robots file
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
.....................................For SS i looked on both .org and .com sites and i saw only the Disallow: /admin is enough for preventing parts of the site like themes folder to be index by search engine or not ?
| 1550 Views | ||
|
Page:
1
|
Go to Top |

