There is a slightly bizarre article today on Online Journalism Blog advocating that newspapers should turn off their RSS feeds and instead push their stories to Twitter (via Cybersoc). Many people have noticed that Twitter has become one of the easiest ways to disseminate content on the internet, leading some to predict the death of RSS.

There are many advantages of using Twitter to spread your message. I have written before about the fact that in some respects Twitter seems to have superseded social bookmarking sites like Delicious. The reason? Twitter has an upper hand in any activity where you want to alert people right away to something you want to share right now.

But this immediacy comes at the expense of its long-term value. Trying to find an old tweet is a nightmare; an impossibility even. You can’t tag tweets — at least without substantially eating into your stringent 140 character limit. And the use of URL shortening services necessitated by Twitter’s character limit comes with its own bucketful of problems.

So should a newspaper completely ditch RSS feeds in favour of Twitter, as Malcolm Coles seems to suggest? Hell no.

His first argument is the strangest of the lot. He points out that many RSS feeds provided by newspapers appear to have few subscribers, and maintains that this is a weakness of RSS.

Despite having virtually no users, the Mail churns out 160 RSS feeds and the Mirror 280. All so a couple of thousand people can look at them in total.

The other papers are just as bad. And while the Guardian has a couple of RSS readers with decent numbers (partly because Google recommends it in its news bundle), it has more feeds than there are people in the UK …

Never heard of the long tail? Having few subscribers to an RSS feed isn’t a weakness. In fact, it plays to the strengths of RSS feeds as the ideal way to disseminate niche content. For me, the problem with newspapers’ approaches to RSS feeds is the complete opposite. As I have written before, they don’t offer enough RSS feeds.

You can scoff at the fact that The Guardian publishes more RSS feeds than there are people living in the UK. But the cost of doing so is pretty small, especially if the feed doesn’t actually have that many takers (because then it uses up less bandwidth). Indeed, as Jon Bounds notes in the comments to the article, in a decent CMS it will take longer (i.e. be more costly) to switch an RSS feed off rather than leave it on.

What potential alternative does a newspaper have if it decides to give up on RSS? Twitter seems to be the big suggestion. Would a Melanie Phillips Twitter account run by the Daily Mail have more than 11 followers on Twitter? Maybe, but the majority of them would probably be robots advertising mucky webcam shows.

For Malcolm Coles, Twitter would be better because you can see which stories are the best by seeing what is retweeted. Retweets are extra good because they promote a newspaper’s content. But people will tweet and retweet about articles they like anyway, whether it comes from an official newspaper Twitter account or not. And to be honest, I could do without my Twitter stream being filled with yet more junky retweets.

According to Malcolm Coles, you can also provide more context in Twitter because “There’s space in 140 characters for newspapers to give some background to stories as well as the headline.” But you can provide the whole article in an RSS feed if you want to, as The Guardian (whose RSS feeds are by far the most popular) has demonstrated. The inability to provide context is in fact Twitter’s greatest weakness. Even a social bookmarking site like Delicious gives you 1,000 characters to play with, not just 140.

It is true that you can have a conversation about stories on Twitter, which you can’t do with RSS feeds. Conversation is practically the raison d’être of Twitter though, so this is not exactly a surprise. All that this underlines is the fact that Twitter and RSS are two very different kinds of tools. One cannot be comfortably substituted for the other.

Malcolm Coles says that the newspapers agree with him because they do not bother to promote their RSS feeds properly. He says that they “have already given up on RSS feeds and no longer actively promote them.”

This ignores the fact that newspapers have never actively promoted RSS feeds. Promotions of RSS feeds haven’t just recently been relegated to the footers. If anything, they have just been promoted there. My last post about newspapers’ RSS feeds outlined my exasperation over the fact that their implementation is sloppy and amateurish, and it is nigh-on impossible to find out if the RSS feed you’re looking for even exists, never mind where it is.

Perhaps, indeed, the newspapers’ failure to properly promote their RSS feeds this is the reason why Melanie Phillips only has eleven subscribers in Google Reader. Maybe Malcolm Coles sees this as a chicken-and-egg scenario, but in this case I definitely know which came first.

The real problem is not that RSS has failed for newspapers. It’s that newspapers have failed at RSS. This is demonstrated by the fact that in the comments, Malcolm Coles ends up relying on the unreliability of the Express’s RSS feeds, rather than any inherent weaknesses in the RSS format itself, in his attempts to support his arguments. If the Express’s RSS feeds are broken and poorly promoted, that’s the Express’s fault, not RSS’s fault.

Dan Thornton in the comments hits the nail on the head:

Personally, if newspapers turned off RSS, I suspect they’d never see me visit their sites again – I use Twitter as a real time stream of information, but my RSS Reader is a library of sources I’ve invested time nad effort in reading regularly and getting to know. One doesn’t replace the other – they co-exist.

Here is the full text of an article written by Grant Thoms for his Tartan Hero blog on 24 November 2007:

Wendy’s in a ’spin’ again

It should have been third time lucky for Wendy Alexander and a head of communications for the Labour Group. First, Brian Lironi left within days of Wendy’s coronation. Then Babyface Marr spectacularly resigned last week after a bout of political Tourette’s Syndrome. Now, the third man, Gavin Yates is in a spin after his blog postings were reported by the Sunday Post and Sunday Herald.

In his blog (which has since been closed down, a fine example of bolting the stable door), he praised Alex Salmond as ‘a politician at the top of his game’ and lauded the SNP Government’s achievements in it’s first 100 days. Now we shall see if this ‘journalist’ will change his tune now Labour is paying for his pipes.

Today, the Tartan Hero stable finds its door bolted firmly shut. A message simply reads: “the blog at tartanhero.blogspot.com has been removed.” His blog posts are now being reported in The Herald.

It seems as though “Tartan Hero” has become the Tartan Feartie, scared of his own views. For the man the SNP were pinning their hopes on for the Glasgow North East by-election has now withdrawn from the contest, apparently afraid that his blog “would return to haunt him”.

We have seen this sort of thing before of course. As Tartan Hero’s post says, one of Wendy Alexander’s spin doctors, Gavin Yates, closed down his blog and deleted it. As I pointed out at the time, if you want to hide your blog then deleting it is pretty futile. You leave traces of yourself all over the place, and deleting your blog only brings attention to the fact that you might have something to hide.

In the case of Gavin Yates, I was still able to access all of his archives which were sitting in my Google Reader account. Anyone can access old RSS feeds in Google Reader as long as they were subscribed to the website while it was still being published.

This week The Herald says that “traces” of the Tartan Hero blog have been retrieved by Mr Thoms’s political opponents. In my Google Reader account I have found a bit more than “traces”. I have access to the full content of 684 of his articles. I think this is a very substantial proportion of his archives.

In the words of Lallands Peat Worrier, he has been “Indygalled“! We can add his name to the list which includes Gavin Yates (whom, ironically, he gloated about), “Indygal” Anne McLaughlin and Kezia Dugdale.

Anne McLaughlin’s blog made the news when she became an MSP. Journalists trawled her archives looking for anything vaguely juicy, and they found a few interesting comments about (and a few photographs of) other politicians, but not much more. After some of the offending content was deleted, and a brief hiatus, she continued blogging and the whole thing blew over.

Kezia Dugdale also took some time off her blog after deciding it was “far too risky a past-time”. I think she got in hot water a couple of times about some of the things she published. Now with a promise that she will “be a bit smarter” with her blogging activities, it remains one of the very best Scottish Labour blogs going.

Tartan Hero was not among my personal favourites (although I guess I should be grateful to him for once rather inexplicably deciding that this was the second best Scottish political blog!). But it was clearly a very popular blog and appeared to attract quite a wide audience. His opinions didn’t do him any harm then.

For what it’s worth, I don’t think there is anything in Tartan Hero’s archives which is worth getting too excited about, which makes the deletion all the more strange in my view. The Herald hints at worries about this views on gay rights and Catholic schools. Jeff (apparently with the scoop!) also pinpointed Catholic schools as a potential issue.

The thing is, Tartan Hero was always had quite a provocative style. The views were not particularly extreme, but they were forthright and strongly expressed.

It seems strange to me that a politician would get cold feet over political views they so vehemently expressed just a year or two ago. It can’t be a surprise that his writing would find itself in the spotlight. Indeed, that was surely the intention.

It is true that in the rough-and-tumble world of party politics, one’s character and history faces a different type of scrutiny, and the game is not often played very fairly. But Grant Thoms is surely an intelligent person who has presumably had his sights set on becoming a Parliamentarian for a while now. None of this can be a surprise to him and he will surely have seen it coming.

So the deletion of his blog does make me scratch my head a bit. Moreover, it looks particularly silly given what he wrote when Gavin Yates deleted his blog.

As I said the last time I tackled this issue, no doubt if someone tried hard enough they’d find plenty of material on my blog to use against me. After all, as a mere 22 23-year-old scamp who has been blogging since 2002, I have left a fairly thorough record of my opinions going back to the age of 16.

It’s not that my opinions as a 16-year-old were particularly invalid or wrong, but a lot of them will have changed. It is not beyond the realms of possibility that I have written something in the past that could be taken out of context and used against me.

I’d like to trust people to be responsible about it, but I wonder if it’s possible. Certainly, it is a sad reflection of the state of politics that astute bloggers feel the need to cover up their writing for fear of it being used against them and thwarting their political careers.

At least Anne McLaughlin and Kezia Dugdale have not been put off for good and have been able to continue blogging in the long run. I wonder if one day soon a modified version of Tartan Hero will return to the blogosphere.

If you are a user of the internet (which you are) then you have to be really careful what you publish on it. Even if you think you can’t delete something, you can’t. While on the screen it looks like it’s disappeared, in reality there is a huge probability that all of the data will remain somewhere.

The most well-known example is Google Cache, which is the copy of each document on the web that Google uses to assess whether or not it is helpful to your search. This can also be used by anyone who wants to see what used to be on a page that has since been deleted or changed, if Google’s bot visited the page at the right time.

Another example is the Wayback Machine which literally visits web pages and archives them. Forever. Internet users can — to a point — browse the web as it was even in the mid-1990s. For example, here is the Microsoft website as it was in 1996.

But it is not just huge central databases like Google that can thwart self-censors. Everyday internet users are doing it all the time. Thanks to the popularity of RSS, there is now little chance that anyone who publishes an RSS feed will ever be able to hide their content. Anyone who is subscribed to your RSS feed has access to that content for as long as they want.

If you use a desktop-based RSS reader the files will actually be on your computer. But I use Google Reader, and I have access to every single blog post written by Gavin Yates since the 29^th of May 2007. It looks like the Sunday Herald have as well, and possibly more.

Gavin Yates is Wendy Alexander’s new head of communications, a job which seems to be somewhat of a poisoned chalice. Brian Lironi left the post just a few days after Wendy Alexander took office, seemingly because he was fed up with the new Scottish Labour leader.

Then last week Matthew Marr was given the heave-ho after a drunken performance at the Scottish Politician of the Year Awards (a bit oxymoronic if you ask me). Particular attention was given to the fact that Mr Marr called Alex Salmond a cunt. While a lot of bloggers pointed out that he was probably right, it’s not very good conduct and you would expect much better behaviour from such an important Labour official.

And while Mr Marr was keen to point out that the incident was “entirely out of character”, bloggers lined up to say that it was in character. Mr Eugenides was “reliably informed”, while Osama Saeed and Mark McDonald have both been at the receiving end of one of Marr’s verbal outbursts.

Now Gavin Yates has run into difficulties before he’s even started. Some of what he wrote on his blog has been less than flattering about Labour and Wendy Alexander, as the Sunday Herald story points out.

The thing is, this needn’t be a problem. Surely a bit of reality, a bit of honesty, is what Wendy Alexander and Scottish Labour really need. I do wonder, though, if the culture within the party means that only yes-men are tolerated. This is presumably what drove Gavin Yates to delete his blog. Yet, as usual, it is the cover-up rather than the original ‘crime’ which makes this an embarrassing episode for Labour.

As Will P points out, Gavin Yates’s excuse does not make sense.

My comments have been taken out of context. I wrote them as a journalist in July and they do not reflect my own views. I think Wendy Alexander is a winner as is Andy Kerr.

A blog that doesn’t reflect your own views? Whose views do they reflect then? Nor was this just a few posts in July as he tries to make out. He was critical of Labour as recently as September.

But as I say, there really is nothing particularly damning about the blog posts themselves. Skimming through the archives of his blog, the criticisms he made of Labour were mostly sensible and constructive. He didn’t say anything that is truly embarrassing.

Like I say, it could have been seen as a much-needed dose of reality for the Scottish Labour elite. In fact, his blog posts demonstrate that he has a pretty good idea of the SNP’s strengths as well as Labour’s weaknesses. This ought to bode well for him in his new post.

It is the fact that Gavin Yates felt the need to delete his blog that makes it the story. It has become the forbidden fruit. But in this day and age, once you publish something on the web, there is no going back. I alone have access to 48 of his posts, just by making a few clicks in Google Reader. By deleting his blog, Gavin Yates has created a lot of interest in what he wrote — and access to it is by no means impossible.

Another blogger, Kezia Dugdale, filled the post on an temporary basis. She made the very wise decision of keeping her blog going (although there is seemingly never any danger of her going off-message). Gavin Yates should have taken note. Keeping your blog up there will do less harm than trying to remove it — because actually removing it is impossible.

It is amusing to think that for all the hype about bloggers and their ability to scrutinise, it could be your own blog, rather than other people’s, that is the most dangerous.

(Yes, every post I write about RSS must contain the hilarious “‘RSS’ sounds a little bit like ‘arse’” pun.)

I have a request for those people who publish RSS feeds. Make them full feeds!

I know there is a supposedly a debate about whether partial or full feeds work best. Well, that is not really the right way to put it. Everybody knows that full feeds work better than partial feeds. I mean, it is like saying that a sandwich is better than the crumbs. It’s just obvious.

But some website owners are, for some reason, sniffy about full feeds. Some people publish partial feeds for relatively superficial reasons, for instance because they can’t bear for any readers to be reading it in an environment other than their lovingly handcrafted web page design. Others have more serious suspicions: that full feeds rob them of page views and rob them of advertising revenue.

Earlier this year, the rather good Freakonomics blog moved to The New York Times website. At the same time, the full feeds were snatched away from the blog’s many readers. Apparently, it is NYTimes policy.

Immediately there was an angry reaction from readers. It (mostly) wasn’t from readers concerned about NYTimes itself or even due to the fact that the URLs had changed, that there was an entirely new navigation system to accustomise to, or anything like that. They were almost all from people who were angry that the full feed had overnight turned into a partial feed. Many readers even said they were unsubscribing.

The comments to the initial post were just the start of it. Several subsequent threads descended into similar “outraged of Bloglinesville” mobs, and it has become a recurring topic on the blog ever since. This is one plus side — at least the authors are open about the problems and the reasons why they can no longer offer a full feed.

While I wouldn’t go so far as to get angry, I would guess that I have read a lot less of the Freakonomics blog since the move. This is entirely down to the fact that it no longer offers a full feed.

I am aware that a lot of people simply cannot believe that (or understand why) full feeds generate as many clickthroughs as (or sometimes even more clickthroughs than) partial feeds do. It doesn’t seem to make sense, right? If people can read the entire content without leaving their RSS reader, why on earth would they visit the website?

But it doesn’t work like that. FeedBurner say so — and they would know. To me, it is just common sense. I have been reading RSS feeds for a few years now, so I think I have a pretty good idea of the reasons why partial feeds just do not work.

Think about why people use RSS feeds as opposed to visiting the different web sites all the time. It’s obvious: people who use RSS feeds do so because it makes it easier and quicker to read everything they want to read.

So immediately we have run into the problem with partial feeds — they do the precise opposite of what the reader wants. They make it more difficult and slower to read what you want to read. If you have begun reading and want to read the rest of the content, it involves clicking through and waiting for the (probably bloated) web page to load. It is a needless, unwanted, time wasting, inefficient hassle.

That explains why readers generally don’t like partial feeds. But what about the clickthrough rate? First of all, it is worth pointing out that page views are falling out of favour as a meaningful web metric thanks to the increasing use of Ajax and other kinds of magic. In a funny way, more page views usually means it’s a worse website. (Ask users of MySpace and Facebook about the navigation of those sites, and see which site has the happiest users.)

But let us say that page views (and certainly visits) are a good thing. So why should you use full feeds? Once again, for me it is down to convenience. I use RSS feeds because it allows me to squeeze more reading into a shorter space of time. Imagine sitting there in front Google Reader. You have a list of items waiting to be read. So you get on with it and start scrolling through, scanning for anything interesting.

By now, you may have realised why partial feeds do not automatically generate clickthroughs. It is because there is less of the content for me to scan-read and evaluate. Typically, a partial feed will contain the headline and the first couple of dozen words. This simply is not enough to give me as a reader an idea of how good the rest of the article is. Neither is it long enough for the author to sell the article.

There is one site that falls victim to this more than any other if you ask me. Tim Worstall, one of the most widely-respected British bloggers. His RSS feeds simply do not do his blog justice.

I will sit there with Google Reader and scroll through the many posts he has written that day, and all too often I find myself not being enticed by a single one of them. That is not because they are not interesting. It’s because his partial feeds simply do not give me any confidence that clicking through to read the rest of the post will be worth my time.

If Tim Worstall writes ten posts in a day (which is my conservative estimate of what he averages), he is asking me to read ten summaries, click ten times, wait for ten web pages to slowly load, then read ten full posts. What a waste of time!

This is especially annoying if the partial feed stops in the middle of a sentence, which is almost every time. When the partial feed stops at the end of a sentence, then there is the confusion over whether I had read the full post (just a really short one), or if it was just a fluke that the feed finished in a neat position.

If Tim Worstall provided full feeds in the first place, I could have just read them all there instead of going through all of that hassle. Who knows, I might even have clicked through and left a comment. I might have bookmarked one of his posts in Delicious, letting other people know how good the post is. I might even have blogged about it. I might even have clicked on an advert!

As it is, I just scroll through the summaries and ignore them all. I have, in the past, unsubscribed from his blog because of the frustration over this. I recently subscribed again, but can’t say I read a good deal more of his blog as a result.

Some other blogs provide “summaries” instead of partial feeds. This is where, instead of the first few words of the post, the author has instead specially written a summary designed for the feed. The problem with this is that sometimes it is made up of a random paragraph taken from the middle of the article. Even worse, it might give away the conclusion before I have even read what it was the conclusion for!

If I am enticed by such a summary, I will click through and find myself reading the post and thinking, “This isn’t what I thought I was reading.” Then I will come across that paragraph in the middle. Ah, and that introduction in the summary? I have found out that it was actually a conclusion. It is like forcing somebody to read the last page of the novel before reading the rest of it!

There is another more fundamental reason why people should offer full feeds. It is just plain rude not to. RSS subscribers are your most dedicated readers. They are people who have decided that your content is good enough to have it effectively delivered straight to them on a regular basis.

Yet, how are these dedicated readers paid back? By getting a mangled fraction of the content that they asked for. It is like subscribing to your favourite magazine only to find the publisher sending out cuttings rather than the whole magazine. What a way to treat your regular readers!

I can hear the howls already: “What about all of the beautiful adverts that I have lovingly placed on my blog / newspaper / whatever? If I offer full feeds, nobody will look at the adverts and I won’t make any money!” Again, there are several responses.

I have already explained why full feeds do not lead to a reduction in clickthroughs. So people will see your adverts just as much as they always did.

There is an even more obvious answer: what is stopping you putting adverts on your feed? Plenty of big websites already do this. It is perfectly possible. People who are refusing to offer full feeds because “they don’t contain my adverts” are simply shoving their heads in the sand.

Even if there was a legitimate concern about adverts, it has to be remembered that your regular readers (the sort who would subscribe to your RSS feed) are the very people who are the least likely to click on the adverts anyway.

Let us not forget also that a lot of adverts are not even designed for human eyes as much as they are designed for SEO. These kinds of adverts would not even mind not being seen (just as long as Googlebot sees it).

Maybe you are concerned about stats. Let’s face it, as bloggers we all are. We want to know how many people are reading. What would be the point if you had no way of knowing if people were reading or not. Gordon McLean (whose recent post on RSS is an interesting read) falls into this group.

Admittedly, this is one downside to RSS as it becomes impossible to find out precisely how many people are reading. Mind you, web stats are not generally the most reliable things anyway. Run four different stats counters and you are bound to get four different — sometimes wildly varying — figures. RSS further muddies the waters.

As it happens, I recently moved over to having this blog’s feeds provided by Feedburner (combined with the absolutely vital FeedSmith WordPress plugin), partly because it would give me some fairly accurate (but not precise) statistics. I was pleasantly surprised to find that around 140–150 people are subscribed to this blog. (Hello to you good people. I hope you are enjoying the full feed!)

Beforehand I had vague ideas of who was reading this blog’s webpages and why. But I had no idea of how many people were actually subscribed to this blog’s RSS feed. But now I do have some fairly interesting and meaningful stats about my RSS feed. So even the stats issue with RSS feeds is resolved to an extent.

All of this is not to say that partial feeds do not have their place. For instance, they are perfect for news websites. This is because of the way they work. We are used to just scanning through a front page containing only a headline and a (very) brief summary of each story. From here we choose which stories we want to read. This is how news websites work, and partial feeds can reflect this.

Blogs, however, do not work in this way. Very few blogs offer just a summary of each post on the front page. The blog format does not usually lend itself well to this approach. Rather, the vast majority of blogs’ front pages contain either the full content of the most recent posts, or at least a huge chunk of them.

As far as I can see, there is no reason why the vast majority of web sites should be forcing their most dedicated users to put up with shoddy, sub-standard partial feeds. For me, the fears that website owners have surrounding full feeds are mostly unfounded.

After an astonishingly tinker-free summer, I have made a few changes on the blog.

Perhaps the most important is the reintroduction of the Best of page, which I hinted at a couple of months ago. This is just to attract attention to some of the notable posts I have written in the past. There is a huge number of archived posts here, so it is impossible for people to find the really good ones. Hopefully the Best of page will make it a bit easier.

The first section of the Best of page is what was there before I took it down. These are not posts that I have chosen, but posts that have attracted some kind of attention beyond what is normal. They have either been mentioned in the mainstream media, the Britblog Roundup (or a similarly prestigious blogging showcase) or used as a citation in Wikipedia. So if you’re a relatively new reader, why not check out some of these older posts?

If you scroll beyond that list, you will see something absolutely brand spanking new. After years of dithering about it, I have finally installed a post ratings plugin (WP-PostRatings).

I was looking for something a bit more like a thumbs-up / thumbs-down system, but I couldn’t find one, so we have a star ratings system (although I’m using squares because the stars look crap against the dark background). So please feel free to rate my posts as you read them — it only takes a click.

To encourage people, I have rated the posts on the front page, but I will probably step back from rating my own posts in the long term. If this system gives good results, I might place the list on the sidebar rather than in the middle of the Best of page.

The only problem with the ratings plugin is that it adds substantially to the clutter at the bottom of the post. I do make efforts to keep the clutter to a minimum in general, but I can’t think how else I can add the ratings system without cluttering it up.

I needed something better than what I had before, which was the ‘most popular posts’ list. This is calculated mostly on page views (but also things like comments and trackbacks). Unfortunately, this means that far from highlighting the best posts, it actually merely shows the posts that Google likes the most. This means that some of the posts on the list are not only not-good, but they are actually actively bad. I will keep the feature there, but it’s not a very good list.

The same goes for the ‘most commented on’ posts. Comments are great, but any threads that get more than about 15 or 20 soon descend into crazy flame wars, loon magnetism, and generally generate more heat than light. It’s probably not the side of this blog that I should be putting out there.

I have also finally got round to redesigning the 404 page. Unfortunately it is cluttered with adverts, which I’m not sure about. I can’t really be bothered getting rid of them yet, so I might just leave it as it is. But at least now it actually matches with the rest of the pages on the blog!

I still have not got round to designing a theme for people who do not like the current one (I plan to let visitors choose which they prefer between two). I might not get round to it in the end.

I probably have not stopped tinkering, because tinkering is like eating Pringles and the itch that gets worse the more you scratch it. But I thought I would just point out what I have done so far because [moment of honesty coming up] I can’t be arsed writing about anything else at the moment.

Update: I knew there was something I forgot! A few weeks back I changed the links page so that it automatically contains every blog that I read in Google Reader. So if I’m reading your blog, it’s on the links page. It’s quite good to not have to worry about updating the links manually now. The internet truly is making us a bunch of lazy bastards.

If you’re wondering how it works and want to add it to your own blog, all of the information is here.

(Incidentally, if you are wondering about Scottish political blogs, I keep most of them in a separate folder for me to concentrate on for the roundup. So many of these blogs will not appear on my links page yet, even if I read them.)