Studies show that web users will only read the top 15 search results for any given query. That means millions of websites gather dust, never to be seen—and what’s the point of creating that content if you’re not read?

Thus evolved the science of search engine optimization, which includes key word counts, editing metatags, link management. Webmasters can spend weeks, even months, fine-tuning these elements—but like any product, it needs a test drive. That’s what search engine spider simulators are for.

Search engine spider simulators, also known as search engine robot simulators, allow you to see the page as other web crawlers do. “Robots” is an industry term that describes how Google, et. al scour the Internet for new pages. They’re like electronic detectives, with each given a particular task. Some bots are designed to follow every link, downloading as many pages as possible for a particular index or query. Others are programmed to look out for new content.

Both these bots play a huge role in whether or not your website ends up in the top 15…or languishes at the bottom.

For example, does the bot pick up on your links? Javascript errors can also cause the bot to miss out on important links, and we all know how important inbound links are in search engine ranking.

Does it index every page of your site? It’s highly possible that a programming glitch causes the bot to skip a large portion of your content. There goes all your efforts to increase keywords or optimize titles and crossheads!

It’s also possible that the bots are basing your ranking on old versions of your website, unable to recognize the changes you have made. You might as well have not done anything at all.

You may have also made the mistake of accidentally blocking a bot from checking a section of your site. While it is important to restrict website users’ access to sensitive information—for example, those reserved for a company’s internal networks; the personal information of members who have signed up for a newsletter; or premium pages that you’d rather reserve for paying subscribers—the bot should be given free rein, if only to improve your chances of getting a higher ranking. If not, that’s just like throwing the bot with the bathwater.

It would be impossible to pick up these errors without actually recreating how the bots review your site. You can do this by using a robot simulator. Using the same processes and strategies of different search engines, these simulators will “read” your site and inform you of which pages are skipped, which links are ignored, and which errors it encounters. You can also review the robot.txt files, which will enable you to spot any problems and correct them before you submit them to real search engines.

You’ll be surprised at how many things you’ll find out about web robots, and how the bells and whistles many webmasters include on the site do nothing to improve search engine ranking. For example, search engine robots generally do not see flash based content, content that is made through javascript like javascript menus, as well as content displayed as an image. You’ll also be able to monitor how the bots will follow your hyperlinks, very crucial if you’re running a big website with sprawling content.

We’ve recently launched a search engine robot simulator in the SEO Tools section on XML-Sitemaps.com so if you want to see how robots see your site give it a try.  Our robot checks all links on your pages and will tell you if any links are restricted by your robots.txt file and also the use of the nofollow tag.

I’ve added a ROR sitemap to the site now. ROR - Resources of a Resource) is an independent XML format for describing any object of your content in a generic fashion, so any search engine can better understand that content. They are particularly useful for search engines such as Yahoo! and MSN and a good way of letting them know about all your pages with one file.

Generating ROR sitemaps is the latest new feature of my site XML-Sitemaps.com which now lets you create sitemaps in XML, HTML, TXT and ROR formats to cover all the search engines.

Google has its own sitemaps program but will also accept ROR sitemaps as well but you are probably better off conforming to their requirements if you can.

It’s always important to remember that Google isn’t the only search engine on the planet. Although they can bring in large amounts of traffic to your website and are therefore the most important to optimize for, other search engines can bring in similar amounts of traffic when combined together.

If you can make it easier for them to find and index all your pages then it is worth the effort to create sitemaps for them all.

Well I’ve finally done what I have set out to achieve. Eighteen days and 37 posts after setting myself a target of getting in the top 10 results for my name “Philip Nicosia” I have finally made the top spot in Google with this site www.philipnicosia.com.

I’m also number 1 in MSN.com for the same search term but Yahoo! is a different story altogether.

I briefly made an appearance at No. 3 in Yahoo! but have since been relegated to the Number 23 slot. What’s going on there?

The most surprising thing I see in Yahoo! is the number 1 result “Compare prices for Philip Nicosia - DVD-Video. Read dvd-video reviews and compare prices at Yahoo! Shopping.” leading to a virtually blank page with my name on it and a blank section titled “Popular Films by Philip Nicosia”.

Maybe there is a film maker named Philip Nicosia that I’ve never heard of? It’s not me unless of course you include home movies such as this one.

It will be interesting to see if my results improve in Yahoo! soon.

Added a new photo gallery to the site yesterday. There’s not much to see yet but I’ll be uploading loads of pictures over the next few weeks.

For now I have included some cute pictures of the pets and some shots taken from around where I live.

I have a memory like a sieve now so I am always forgetting important things like meetings, birthdays and generally things that I need to do. Something I always do though is read my email everyday so I thought it would be a good idea to get email reminders that would nag me enough so I wouldn’t forget anything.

Enter OnlineReminders.net.

As the name suggests once you enter in a reminder online and it will email you once a week with everything you have to do that week, the day before you need to do something and just in case that hasn’t got through to your brain it will email you the day of the event.

Not even I can forget things now. It’s also got a great Goal Planner where you can keep track of goals and see how they are going. For me its indispensable. Although this site was primarily for my use it is open to anyone who wishes to join and although it is setup to charge people for the service it is and will remain free.

So if you have trouble remembering things as well why not take a look.