Jump to:

17452 Posts in 4473 Topics by 1971 members

Archive

SilverStripe Forums » Archive » Dealing with robots.txt requests?

Our old forums are still available as a read-only archive.

Moderators: martimiz, Sean, biapar, Willr, Ingo, simon_w

Page: 1
Go to End
Author Topic: 2810 Views
  • 3dkiwi
    Avatar
    Community Member
    17 Posts

    Dealing with robots.txt requests? Link to this post

    Guys

    Just a quick one I hope. Have started to migrate a SilverStripe installation from a test bench system to a live system and have noticed that search engine requests for the /robots.txt file are being returned a 404 not found in the web server access log files.

    With the redirection, rewrites and other things that mangle the request on a SilverStripe installation, what is the best way to handle requests for the robots.txt file and what is a good list of URLs to deny on a SilverStripe installation?

    Cheers
    3d

  • 3dkiwi
    Avatar
    Community Member
    17 Posts

    Re: Dealing with robots.txt requests? Link to this post

    My research into this:

    Placing the robots.txt file in the root directory of the silverstripe installation seems to work.
    ie. a ls listing should show something like:

    .
    .
    .
    cms/
    robots.txt
    mysite/
    tutorials/
    favicon.ico
    .
    .
    .

    The common Disallows would be

    /images/
    /page-not-found/

    Also a Sitemap directive eg.

    Sitemap:

    Check out: http://www.sitemap.org for more details on this file that SilverStripe automatically generates for you.

    So a good starting point might be:

    ---

    # Show the way to the site map file
    Sitemap: http://www.example.co.nz/sitemap.xml

    # LinkWalker knows where to go.
    User-agent: LinkWalker
    Disallow: /

    # Archive.org was hammering site - Now knows where to get off
    User-agent: archive.org_bot
    Disallow: /

    User Agent: *
    Disallow: /images/
    Disallow: /page-not-found/

    ---

    Cheers
    3d

    2810 Views
Page: 1
Go to Top

Want to know more about the company that brought you SilverStripe? Then check out SilverStripe.com

Comments on this website? Please give feedback.