Studies show that web users will only read the top 15 search results for any given query. That means millions of websites gather dust, never to be seen—and what’s the point of creating that content if you’re not read?

Thus evolved the science of search engine optimization, which includes key word counts, editing metatags, link management. Webmasters can spend weeks, even months, fine-tuning these elements—but like any product, it needs a test drive. That’s what search engine spider simulators are for.

Search engine spider simulators, also known as search engine robot simulators, allow you to see the page as other web crawlers do. “Robots” is an industry term that describes how Google, et. al scour the Internet for new pages. They’re like electronic detectives, with each given a particular task. Some bots are designed to follow every link, downloading as many pages as possible for a particular index or query. Others are programmed to look out for new content.

Both these bots play a huge role in whether or not your website ends up in the top 15…or languishes at the bottom.

For example, does the bot pick up on your links? Javascript errors can also cause the bot to miss out on important links, and we all know how important inbound links are in search engine ranking.

Does it index every page of your site? It’s highly possible that a programming glitch causes the bot to skip a large portion of your content. There goes all your efforts to increase keywords or optimize titles and crossheads!

It’s also possible that the bots are basing your ranking on old versions of your website, unable to recognize the changes you have made. You might as well have not done anything at all.

You may have also made the mistake of accidentally blocking a bot from checking a section of your site. While it is important to restrict website users’ access to sensitive information—for example, those reserved for a company’s internal networks; the personal information of members who have signed up for a newsletter; or premium pages that you’d rather reserve for paying subscribers—the bot should be given free rein, if only to improve your chances of getting a higher ranking. If not, that’s just like throwing the bot with the bathwater.

It would be impossible to pick up these errors without actually recreating how the bots review your site. You can do this by using a robot simulator. Using the same processes and strategies of different search engines, these simulators will “read” your site and inform you of which pages are skipped, which links are ignored, and which errors it encounters. You can also review the robot.txt files, which will enable you to spot any problems and correct them before you submit them to real search engines.

You’ll be surprised at how many things you’ll find out about web robots, and how the bells and whistles many webmasters include on the site do nothing to improve search engine ranking. For example, search engine robots generally do not see flash based content, content that is made through javascript like javascript menus, as well as content displayed as an image. You’ll also be able to monitor how the bots will follow your hyperlinks, very crucial if you’re running a big website with sprawling content.

We’ve recently launched a search engine robot simulator in the SEO Tools section on XML-Sitemaps.com so if you want to see how robots see your site give it a try.  Our robot checks all links on your pages and will tell you if any links are restricted by your robots.txt file and also the use of the nofollow tag.

Well I’ve finally done what I have set out to achieve. Eighteen days and 37 posts after setting myself a target of getting in the top 10 results for my name “Philip Nicosia” I have finally made the top spot in Google with this site www.philipnicosia.com.

I’m also number 1 in MSN.com for the same search term but Yahoo! is a different story altogether.

I briefly made an appearance at No. 3 in Yahoo! but have since been relegated to the Number 23 slot. What’s going on there?

The most surprising thing I see in Yahoo! is the number 1 result “Compare prices for Philip Nicosia - DVD-Video. Read dvd-video reviews and compare prices at Yahoo! Shopping.” leading to a virtually blank page with my name on it and a blank section titled “Popular Films by Philip Nicosia”.

Maybe there is a film maker named Philip Nicosia that I’ve never heard of? It’s not me unless of course you include home movies such as this one.

It will be interesting to see if my results improve in Yahoo! soon.

Added a new photo gallery to the site yesterday. There’s not much to see yet but I’ll be uploading loads of pictures over the next few weeks.

For now I have included some cute pictures of the pets and some shots taken from around where I live.

Another one of my sites is Resources.eu.com. The site is a hand edited directory for various resources that can be found on the internet.

A few years ago it seemed that Google used to list a lot of directories high in the serps so I thought I would go about making one. Quite honestly its not worth the bother. Now that the site is somewhat established it gets about 10 - 20 submissions a day which can take a lot of time to manually review. Out of these 10 - 20 submissions it isn’t unusual for only 1 or 2 to actually be approved and entered into the directory.

Anyone thinking about starting up a directory to make money with I would personally think again. Its not worth the effort.

Another one of my sites I have is Podcastblaster.com.

Podcasting is a new way of distributing audio broadcasts using the Internet. Using this method, people can subscribe to a variety of content in a certain file format (usually as MP3s). These files are then downloaded to a PC or a portable media player and can then be listened to at any convenient time. Podcasting is a new word that combines “broadcasting” and “iPod”.

A definite distinction of podcasting from other media delivery methods online is its subscription component – which basically defines the whole podcasting experience. Podcasters usually use a Really Simple Syndication (RSS) or Atom feed to push their content to subscribers. It also allows independent producers and “broadcasters” to create and self-publish their own syndicated “radio shows” – whether it be talk radio, poetry readings or even comedy shows. Podcasting also gives traditional radio programs another distribution method for their shows – one that is not confined to the airwaves and available on demand. All that a listener needs to get the podcasts is “podcatching” software that is designed to check for and download new content at a given time. Other podcatching software are even designed to synchronize the downloaded files to the listener’s portable digital player.

PodcastBlaster.com was setup to sell a unique podcasting package for beginners who wanted to enter the world of Podcasting. Everything needed to get started is included with step by step instructions, professional quality recording software for creating your podcasts together with ready to use professionally produced voice-overs, loops, background effects, stingers and Podcast-safe music.

The whole package costs only $39.95 and come with a no risk 30 day money back guarantee. In the year or so that I have been selling this software I have only had 1 refund request. That must say something for how useful people have found it.