URLs Restricted by Robots.txt

When you log into your Google Sitemaps account if you see the error urls restricted by robots.txt with a number greater than zero in your summary then it is definitely worth checking out. This means you are blocking Google from indexing pages from your site. It is so easy to make mistakes with your robots.txt file and if you are unsure about any changes you make you can test them with Google’s robots.txt checker within your sitemaps account.

I have received countless emails from people saying that our sitemap generator at XML-Sitemaps.com doesn’t pick up all their pages only to find they have inadvertently blocked access to robots with their robots.txt file.

I have these errors show up all the time on one of my sites as in my case it is because I have chosen to block certain areas and pages of the website. These pages were designed specifically for individual users of the site and serve no purpose for the rest of the visitors. These pages aren’t linked to on the site anywhere and have been picked up from external links pointing to the pages generated by the users.

There are lots of reasons why you might choose to block robots from areas of your site but it is equally important to make sure you don’t block areas you want indexed too.

Posted in Search Engines