As first point of contact for my site XML-Sitemaps.com I quite often get emails from people saying “Your generator won’t find all my pages”. Because our generator works in the same manner as a search engine robot this is something that all webmasters should be concerned about.

We will always check out these concerns but haven’t as yet found any site that misses a page when there is a link to it somewhere.

The most common reason for this problem is JavaScript menus. They might look nice and fancy on the page but robots are generally blind to the links to your other pages. If they can’t find them then it goes without saying that you’re going to have a tough job getting them indexed.

Another cause is misuse of the robots.txt file. It is surprising how many people block access to search engines without realising it. Google has within their webmaster console in the Google Sitemaps program a robots checker. It allows you to put any of your pages in and it will tell you if it is being blocked.

The Google sitemaps webmaster console will also notify you of other problem areas with your site including pages not found, urls timed out, urls restricted by robots.txt, etc.

They also give you stats showing you the top search queries that are made where your site appears and also which of these bring you the most traffic.

If you have a website and you haven’t already signed up for a Google Sitemaps account then it’s definitely worthwhile and if you need a Google Sitemap then don’t forget you can get one at XML-Sitemaps.com.

Posted in Search Engines