Jump to:

23456 Posts in 18905 Topics by 2877 members

General Questions

SilverStripe Forums » General Questions » Robots.txt

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Page: 1
Go to End
Author Topic: 1843 Views
  • OnebyOne
    Community Member
    54 Posts

    Robots.txt Link to this post

    I used auditmypc.com sitemap generator, and i saw that assets and theme folder is getting into sitemap,
    So, i added robots.txt
    User-Agent: *
    Disallow: /themes/
    Disallow: /assets/

    Almost all the cms are coming with robots.txt file after the install is finished. Could SS do the same ?
    And, 2nd, will be bad for search engine indexing if i put in robot.txt

    Or, should i ask what folder from SS would not go under Disallow

  • SalvaStripe
    Community Member
    89 Posts

    Re: Robots.txt Link to this post

    ohh yes man!^^
    and i think your Sitemap generator does not work with SS, because the sites are not in the folders..

    here is very easy going tutorial to make sitemap in SS:

    have fun

  • OnebyOne
    Community Member
    54 Posts

    Re: Robots.txt Link to this post

    THX for that,

    my xml sitemap is working well (added to google and is ok). Only when I created the xml sitemap with auditmypc.com other parts of the sites were crowled (like themes/images or assets).

    My question was about what to putand not in robots.txt

    Here is how other cms is making a robots file
    User-agent: *
    Disallow: /administrator/
    Disallow: /cache/
    Disallow: /components/

    For SS i looked on both .org and .com sites and i saw only the Disallow: /admin is enough for preventing parts of the site like themes folder to be index by search engine or not ?

Page: 1
Go to Top

Want to know more about the company that brought you SilverStripe? Then check out SilverStripe.com

Comments on this website? Please give feedback.