Philip Nicosia

The Robots.txt protocol, also called the “robots exclusion standard” is designed to lock out web spiders from accessing part of a website. It is a security or privacy measure, the equivalent of hanging a “Keep Out” sign on your door.

This protocol is used by web site administrators when there are sections or files that they would rather not be accessed by the rest of the world. This could include employee lists, or files that they are circulating internally. For example, the White House website uses robots.txt to block any inquiries on speeches by the Vice President, a photo essay of the First Lady, and profiles of the 911 victims.

How does the protocol work? It lists the files that shouldn’t be scanned, and places it in the top-level directory of the website. The robots.txt protocol was created by consensus in June 1994 by members of the robots mailing list (robots-request@nexor.co.uk). There is no official standards body or RFC for the protocol, so it’s difficult to legislate or mandate that the protocol be followed. In fact, the file is treated as strictly advisory, and does not have absolute guarantee that those contents won’t be read.

In effect, robots.txt requires cooperation by the web spider and even the reader, since anything that is uploaded into the internet becomes publicly available. You aren’t locking them out of those pages, you are just making it harder for them to get in. But it takes very little for them to ignore these instructions. Computer hackers can also easily penetrate the files and retrieve information. So the rule of thumb is—if it’s that sensitive, it shouldn’t be on your website to begin with or it should be in a password protected folder.

Care, however, should be taken to ensure that the Robots.txt protocol doesn’t block the website robots from other areas of the website. This will dramatically affect your search engine ranking, as the search engines rely on the robots to find and register the pages.

One misplaced hyphen or dash can have catastrophic effects. For example, the robots.txt patterns are matched by simple substring comparisons, so care should be taken to make sure that patterns matching directories have the final ‘/’ character appended: otherwise all files with names starting with that substring will match, rather than just those in the directory intended.

To avoid these problems, consider checking your pages with a robots.txt analyzer. Google has a free one within their Google Webmaster Tools.

Pete had a vision where his friend who died told him he would be the winner. He is worried that if he doesn’t win how that would affect his believe in his vision and heaven.

Pete is the nicest most genuine person in the house by far. He deserves to win and I for hope he does so his belief in his vision remains intact with his faith.

To vote for Pete:

Text PETE to 84444
(Texts cost 50p, plus std msg rate)
Call 09011 32 33 11
(Calls cost 50p, mobile and other network rates may vary)
10p from every vote will go to charity

Pete to win!

101 Ways to Build Link Popularity in 2006

There is no doubt that link building is still an important area for any website that wants to do well. The ultimate aim of any link building campaign is to increase the traffic to your site. This is achieved in 2 ways mainly, click through traffic from the links themselves and link popularity to help your search engine rankings.

Aaron Wall from SEO Book has written an article together with Andy Hagans explaining in great detail what you should and shouldn’t be doing. It lists 71 good ways of building links and 30 bad ways to build links and is a must read for anyone with a website.

To view the article go to 101 Ways to Build Link Popularity in 2006

This morning a 6 mile journey that would normally take me about 15 minutes has just taken an hour. There are road works at the entrance to Watermead where I live where they are building a new junction.

There is only 1 road in and out of Watermead and it took me 45 minutes to travel less than half a mile. This wasn’t even at rush hour and schools out so the roads are normally even quieter.

I expected there to be delays as last night it took 30 minutes to get into Watermead but this is a joke. Can’t these planners anticipate these problems and put in contingencies?

What’s worse is there is little confidence that when the new junction is completed that it will help with congestion at all and if anything the general consensus is that it will be worse.

Yet another waste of tax payers hard earned money!

When TalkTalk first started offering their “Free Broadband Offer” we thought this might be a good idea. I was already paying £14.99 to TalkTalk for my broadband connection and we already used TalkTalk for our calls so the savings could be considerable over the course of time.

Now for the small print: You need to transfer your line rental from BT to TalkTalk and take up their Talk 3 package. You are also committed to an 18 month contract. Again no big deal if things are all okay but looking into their landline services I found that they didn’t offer the call sign service where you can get another number with a different ringtone. We use this for our fax machine at home so in order for us to keep a separate number for our fax machine we would need another line.

We decided against transferring to the Free Broadband Offer and boy am I glad I did.

Shortly afterwards I started experiencing problems with my broadband connection. The speed was running at a crawls pace, I couldn’t download my emails on certain accounts and I couldn’t even get access to quite a few websites. Calling TalkTalk support was a complete waste of both time and money. For Broadband support you have to call a premium rate number and I would be held in a queue for sometimes up to an hour only for them to tell me there was nothing wrong and because I was using a router they couldn’t support me.

Now I had been using the service for 11 months with no problems and suggested that the problem was at their end which they refused to accept so it left me with no alternative but to ask to leave.

I have been trying to leave since the 11th May 2006 and was told I would have to wait up to 28 days for the all important MAC code. This eventually arrived by email on 7th June 2006 but when I tired to use it I was told it was invalid. Despite emailing requesting another one I got no replies. Eventually I called them again on 25th July 2006 insisting that they send me a MAC code straight away. I’m still waiting!

Funnily enough my broadband connection issues resolved themselves after about a week. I hadn’t changed anything at all at my end so I can only presume the problem was at their end as I had suggested before. I still want to leave as I can get a better deal and faster connection elsewhere and I have ZERO confidence in their support.

I have found TalkTalk’s customer service to be some of the worst I have ever experienced. I have been hung up before when the Indian lady in the call centre couldn’t understand my request and promised action on several occasions to get no further.

Anyone thinking about taking up their offer should think wisely as you will be tied in for 18 months. That’s okay if things work well for you but if you run into any problems my experience of their customer service has been less than favourable and you could feel like your hitting your head against a brick wall.