I removed something from my blog today. Well, from the sidebar, actually. All right, all right — that is hardly the Times relaunching, I am aware of that, but it is still noteworthy for two reasons : one, it was the most recent addition there. Two, I removed it as a matter of, well, exorcism will have to do.
The item I removed was a pretty inconspicuous link to the aideRSS ranking of my blog’s feed items. I you missed Kit Meredith’s post extolling its virtues, aideRSS is a free web service that will swallow your blog’s feed (any feed, really, it doesn’t need any kind of subscription) and, after some rumination, spit out a ranking of your posts, which it calls PostRank. The FAQ tersely states that « PostRank™ is a scoring system that we have developed to rank each article on relevance and reaction [my emphasis]. ». The idea is to define sub feeds of, say, the top 10 % posts, so people can subscribe to these instead of the whole feed. Which sounds rather neat.
So what’s wrong with it ?
Nothing at first sight, which is exactly why I included the link in the sidebar (the much more informative widget provided is unavailable for wordpress.com hosted blogs, as it requires JavaScript to work). After all, if it helps my readers, it’s a good thing.
What made me wonder if that was the thing to do was what I discovered when I had a look at what aideRSS considers my « top 20 » posts (click on the screenshot for a larger picture) :
I mean, I can more or less agree on the inclusion of four posts among the top five. The reaction numbers (which aideRSS computes from the number of comments, Google blog search hits, Diggs and del.icio.us bookmarks linking to your post — although oddly enough, its count is slightly off from the ones the services themselves provide) are mostly corroborated by the reader statistics of wordpress.com. The fifth one, my interface rant, is the odd man out. Obviously, that is one case where aideRSS does its magic computing relevance. Independently from any feedback numbers.
So why remove the link ? Was I miffed by some patent pending, trademarked Google-ish algorithm showing me it knows more about my posts’ relevance than I, as the author, do ?
Wish it was that.
I removed the link because I was frightened — frightened to death by seeing what aideRSS considers the seventh most relevant post on my blog. Ever. See for yourself :
Rheta’s World 
well I enjoyed that post, and was very relevant for my days ahead 😛
That comment rates a clean 10.0 at SarcasmRank™.
Rheta, you’ve raised some very good points. A few comments on how we do the ranking:
As you’ve pointed out, ‘reaction’ is our primary source of analysis – we collect meta data such as comments, bookmarks, etc., and then based on these metrics single out stories that have received the most attention. Now, as you can imagine, attention can be good and bad. It may be the case that the article has generated a great discussion, or it could be ‘flame bait’. We do not do any magic to distinguish between these cases, and simply suggest to the reader: “hey, this is what people are paying attention to”. After all, a big debate is information in itself, even if negative.
To address the relevance point: we don’t go any further than measure the rank of the story with respect to the past history of the feed / web-site. While our algorithm takes in a whole set of parameters, we do try to be as transparent as we can be about it. In fact, we’re working on a white paper to explain our logic, and ranking methodology.
Hope this answers some of your questions. Let me know if anything else comes up!
Ilya
Ilya, thank you for actually commenting my little jibe. Do I understand you right that « relevance » means « reaction in light of past history of the blog » ? Meaning posts that gather an uncommonly high or low number of reactions for the blog’s average get ranked up or down ? So you didn’t actually magically infer how typo prone I am ? Oooooohhh… 🙂
On a more serious note, this might go a long way towards explaining some oddities in the ranking, and I think you should indeed document this as clearly as possible, because it runs a bit counter to what one would expect (viz. my own reaction).
Rheta, yep, we collect a lot of meta data about the content, it’s structure, etc., but the primary weight of the PostRank is given to the ‘reaction’ component as you’ve described.
We’re working supporting material – white paper and microsite – which will go into much more detail on how we do the ranking, just as you’ve suggested. 🙂
/me runs to subscribe to SarcasmRank™ !!!
Ah the magic of what a machine finds relevant as opposed to a computer. reminds me of an article (sorry no reference link) about automated scanning of various websites to PREDICT community/national/global trends in attitudes on various topics… I suppose it’s a matter of finding that you felt it neccessary to post a humorous warning that it picked up on…. a post about subjective feeling, rather than the content… or maybe it’s just the image link 😉 PS I’m still stuck for the moment, but doing fine, hope all is well with you
Ilya : maybe you should post a slightly more illuminating explanation straight on the FAQ. What there is in it now is so terse as to be useless. A sentence or two about how reaction is computed, and another one or two about how reaction history makes for the relevance factor in the vein of what you so nicely explained in these comments would be really helpful for those who, like me, wonder what is going on 🙂 — you can always link to a more detailed explanation after that, but I think not too many people will want to read something as technically sounding as a « whitepaper ».
On another tack, and judging from my own rankings, you might consider dampening the impact of flukes on the relevance part of your computation — filtering out the top and bottom 10 %, say, before computing the historical average of the blog reactions. Take my « Est-ce une révolution », for instance, whose huge exposure was both entirely unmerited and only tangentially related to my usual blogging : this is a classic fluke, and the way PostRank downrates all my subsequent postings in the light of that exposure is more skewed than helpful. I could imagine this holds true of other cases than mine, too.
Eidur : maybe I should trademark SarcasmRank and offer a ranking engine ?
Void : I’m so glad to hear from you, especially to hear you’re doing fine. Whatever you miss, I know I miss you.
Rheta, good points. We’ll update the FAQ as you’ve suggested. On flukes and top 10%, PostRank does take these cases into account – I’ll try to elaborate on that as well in our FAQ. Thanks a lot for all the suggestions!