23486 Posts in 18995 Topics by 2878 members
|Go to End|
17 February 2009 at 6:10am Last edited: 17 February 2009 6:11am
I used auditmypc.com sitemap generator, and i saw that assets and theme folder is getting into sitemap,
So, i added robots.txt
Almost all the cms are coming with robots.txt file after the install is finished. Could SS do the same ?
And, 2nd, will be bad for search engine indexing if i put in robot.txt
Or, should i ask what folder from SS would not go under Disallow
18 February 2009 at 12:10am
ohh yes man!^^
and i think your Sitemap generator does not work with SS, because the sites are not in the folders..
here is very easy going tutorial to make sitemap in SS:
18 February 2009 at 10:47pm
THX for that,
my xml sitemap is working well (added to google and is ok). Only when I created the xml sitemap with auditmypc.com other parts of the sites were crowled (like themes/images or assets).
My question was about what to putand not in robots.txt
Here is how other cms is making a robots file
For SS i looked on both .org and .com sites and i saw only the Disallow: /admin is enough for preventing parts of the site like themes folder to be index by search engine or not ?
|Go to Top|