Philip Nicosia

URLs Restricted by Robots.txt

When you log into your Google Sitemaps account if you see the error urls restricted by robots.txt with a number greater than zero in your summary then it is definitely worth checking out. This means you are blocking Google from indexing pages from your site. It is so easy to make mistakes with your robots.txt file and if you are unsure about any changes you make you can test them with Google’s robots.txt checker within your sitemaps account.

I have received countless emails from people saying that our sitemap generator at XML-Sitemaps.com doesn’t pick up all their pages only to find they have inadvertently blocked access to robots with their robots.txt file.

I have these errors show up all the time on one of my sites as in my case it is because I have chosen to block certain areas and pages of the website. These pages were designed specifically for individual users of the site and serve no purpose for the rest of the visitors. These pages aren’t linked to on the site anywhere and have been picked up from external links pointing to the pages generated by the users.

There are lots of reasons why you might choose to block robots from areas of your site but it is equally important to make sure you don’t block areas you want indexed too.

I’ve added a ROR sitemap to the site now. ROR - Resources of a Resource) is an independent XML format for describing any object of your content in a generic fashion, so any search engine can better understand that content. They are particularly useful for search engines such as Yahoo! and MSN and a good way of letting them know about all your pages with one file.

Generating ROR sitemaps is the latest new feature of my site XML-Sitemaps.com which now lets you create sitemaps in XML, HTML, TXT and ROR formats to cover all the search engines.

Google has its own sitemaps program but will also accept ROR sitemaps as well but you are probably better off conforming to their requirements if you can.

It’s always important to remember that Google isn’t the only search engine on the planet. Although they can bring in large amounts of traffic to your website and are therefore the most important to optimize for, other search engines can bring in similar amounts of traffic when combined together.

If you can make it easier for them to find and index all your pages then it is worth the effort to create sitemaps for them all.

I’ve never felt the need to buy an iPod before as the only music I tend to listen to is what I hear on the radio but now I have had a change of heart.

There are so many great podcasts available which I do try to listen to but find it distracting to listen to these on my PC. So now I’ve decided to order an iPod so I can download the podcasts onto it and hopefully I’ll have more of a chance to listen to them when I’m on the move.

Some of the podcasts I listen to regularly are:

The Best of Chris Moyles
Scott Mills Daily
Webmaster Radio

I’m sure I will be listening to more once my iPod arrives.

When I first got my Sony PSP I noticed in the menus a link to “LocationFree”. I didn’t have a clue what this was so I searched on the internet and found that you could watch TV wireless on your Sony PSP if you had a LocationFree base station. How cool is that? Unfortunately these were only available in the US at the time.

Sony have now released these in the UK a short while ago and although they were out of  stock on their online store I found one on Ebay which was also a bargain and I couldn’t resist the temptation to buy it.

Setting up my PSP to receive TV was a piece of cake and took less than 5 minutes. All I had to do once I had connected the base station to my Sky+ box was to put it into setup mode then get the PSP to find it and register itself and I was set. I have to say I was a bit dubious how well it would work but to my amazement the picture quality received on the PSP was excellent.

Next thing to do was setup the remote control by downloading the codes to my PSP. This again was a simple exercise and now I can happily control my Sky+ satellite receiver anywhere in the house or garden.

Now I had got the PSP working the next task on my list was to get it working with my Sony UX180P. This wasn’t quite as straightforward as the PSP and took a little longer. Mainly because I don’t think the instructions were all that clear or I wasn’t reading them properly.

The confusing thing to start with was that I couldn’t connect to the base station. It turns out that unless you have the base station connected into your router by cable it won’t work. It wasn’t a problem for the PSP but the pc was a different story altogether.

This is one thing that I had overlooked when reading the specs before I bought it. Being a wireless device I had thought it would connect to my router wirelessly but this isn’t the case. It needs to hard wired in with a cable. Now this was going to be a problem for me as the location of the base station was going to be in a different room to where my router is. Luckily this didn’t turn out to be such a big problem after all. I bought a Netgear wireless game adapter and plugged it into that and it worked straight away.

Picture quality on the PC was equally as good as the PSP and again was excellent. My only disappointed is that on the PSP there is a button to control the picture format so if you are feeding it with a widescreen signal you can stretch it out to fill the whole screen as it should be but this feature isn’t available yet for the PC. I only hope they support this in the future with an upgrade to the software.

Now for the tricky part. I wanted to set it up to watch TV over the internet. This involved setting up port forwarding in the router to the base station which wasn’t such an issue as I had done this before with my wireless camera. The only real problem I had was finding out how to setup the dynamic dns service in the base station. This part got me scratching my head a lot as it wasn’t that clear in the instructions. The problem I had was I was looking in the wrong place. Instead of the advanced settings it was actually in the easy setup menu. Once I had found the correct menu it was as simple as can be. You don’t need to sign up to anything and the dynamic dns service assigns you a web address to connect to the base station with.

In the specs it says that the minimum requirements are a broadband connection with upload and download speeds no less than 300kbs. As my current provider only gives me a maximum of 256kbs and probably doesn’t achieve that I was a bit dubious to whether it would actually work or not. I was pleasantly surprised to find that it did and although the picture quality wasn’t anywhere near as good as connecting directly to the base station it was perfectly viewable. I would say it was as good as watching a program recorded on an old VHS video tape.

Things can only improve when TalkTalk finally let me go and I get a different provider with 3 x the upload speed I’m currently getting.

So now I can happily watch anything on TV from home including all recorded programmes on my Sky+ box anywhere in the world where I can connect wirelessly to a hotspot or a 3G connection with my Sony UX180P.

I have to say it is pretty amazing and if anyone had told you ten years ago that you would be able to do this you would have laughed in their face. Imagine what we will be doing in another 10 years?

Pete wins Big Brother and quite deservedly so. There was no doubt in my mind that he would win and was the clear favourite right from day 1 to win the £100,000 prize.

Glyn came in second followed by Aisleyne in third, Richard in fourth, Nikki in fifth and Jenny in sixth place.

Although this is the 7th series of Big Brother it is the only one I have been able to watch. It was the most controversial series yet with Big Brother constantly changing the rules but for me that only added to the appeal of the program.

I’m sure we will be seeing a lot more of the instant celebrities from this series in the near future.