21309 Posts in 5738 Topics by 2603 members
| Go to End | ||
| Author | Topic: | 3115 Views |
-
Re: robots.txt - help
27 September 2011 at 9:38am
Should any of the following lines be included in robots.txt?
Disallow: /cms/
Disallow: /sapphire/
Disallow: /mysite/
Disallow: /userforms/
Disallow: /themes/Please help! Thanks.
-
Re: robots.txt - help
23 December 2011 at 10:35am
If it is still relevant...
The CMS automatically redirects the requests you are disallowing.
my robots.txt looks like this
==============================
User-agent: *
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Disallow: /assets
Allow: /User-agent: ia_archiver
Disallow: /User-Agent: Googlebot-Image
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Allow: /Sitemap: http://www.mypage.com/sitemap.xml
==============================
| 3115 Views | ||
| Go to Top |
