Every time I log into my Google sitemaps account I see on the summary page for one of my sites number of URLs not found and number of pages restricted by robots.txt with a polite little exclamation mark and a note saying you might want to take a look at this.

URLs Not Found

With the URLS not found on closer inspection they are pages that don’t and have never existed on the site in my case. They are simply incorrect links that Googlebot has found while doing their crawl of the web and are either misspellings of URLS or badly formatted links.

As long as the links don’t show up as having been found in your own sitemap (in which case you should fix that) then there is little really that you can do to stop these errors from showing. It would be nice if Google at some point decided to show the originating page then at least you could contact the webmaster of the site and ask for the link to be corrected.

From a user point of view there is nothing worse than getting to a 404 page not found like below.

A good idea to stop this from happening is to create a customized 404 page for your site. That way even though the page cannot be found your visitor has some form of navigation to the rest of your site and hopefully they will delve further rather than closing the window in their browser.

It’s worth noting that if you create a customized 404 page you should check it with a header checker to make sure your server stills returns a 404 error code.

Posted in Search Engines