The how's and why's of rating addons

[RisingFury]

For a while I've been observing the addon voting system on OrbitHangar and it seemed to me like people didn't put enough thought into the vote the assign to an addon, or that in some cases, they vote dishonestly, because they don't want to offend the author.

So I decided to write guide that will hopefully provoke some thought, before a rating is given. This has been a long time coming.

I initially intended to write a blog, but I soon realized that the amount of content and images were too much for the blog system. Even a new thread couldn't handle it all, so I decided to write it into a PDF.

It's quite long, spanning 19 pages (don't worry, lots of pictures) and it details a system of rating that I think is objective and fair. So without further ado:

The how's and why's of rating addons

Feedback is appreciated.

 

[Wishbone]

The notion of objectivity and equal weights of various components of a rating formula don't fare well with each other. There is nothing objective in saying "I assign weights equal to 1 to all the parts because I think they should be equal". There's no law of nature that prevents people from using a non-linear rating, either.

 

[Artlav]

Basically, it's a guide on how to be reasonable.
As such, it is of limited use - it may make considering people consider carefully, but it won't make wow-cool-addon people and this-sucks people consider anything else.

It would be useful to have separate ratings for different scales as described, rather than what is like measuring both mass and size in seconds.

It could also be better to bring back the old rate & explain-why-so system.

4 DGs for effort, in any case

 

[RisingFury]

The notion of objectivity and equal weights of various components of a rating formula don't fare well with each other. There is nothing objective in saying "I assign weights equal to 1 to all the parts because I think they should be equal". There's no law of nature that prevents people from using a non-linear rating, either.

You're absolutely right about the scale. I considered using a non linear scale. It would certainly make sense because rating 5 DGs in bugs and rating 5 DGs in features aren't equivalent.

I decided to go with the linear scale as a tradeoff between accuracy of rating and people having to do the math. We've had numerous examples on O-F where people would ask for help in solving the most simplistic linear equations and sticking more factors in there would make things confusing for people and they'd just revert to the "THIS ADDON ROCKS" rating system.

The guide shows an example of a rating system that is more sophisticated than how people rate addons today, but if you come up with your own system that rates more accurately, please use it!

Thanks for the comment

Basically, it's a guide on how to be reasonable.
As such, it is of limited use - it may make considering people consider carefully, but it won't make wow-cool-addon people and this-sucks people consider anything else.

It would be useful to have separate ratings for different scales as described, rather than what is like measuring both mass and size in seconds.

It could also be better to bring back the old rate & explain-why-so system.

4 DGs for effort, in any case

From the start I've known that this guide will be of limited help. Only a small percentage of orbonauts will actually read it. But I figured I had to do something and this is better than standing still and ignoring it.

Rate-and-explain is also a good system because at the very lease you'd see how many people took the time to look at the entire addon instead of just writing what their first impression was. That way you'd get a measure of how many ratings a credible.

Thanks for your comment

 

[Izack]

4/5 for effort and interesting concepts, with the main cons being already mentioned above. Other than a few typos (you told me to nitpick!!) no additional complaints.

 

[RisingFury]

4/5 for effort and interesting concepts, with the main cons being already mentioned above. Other than a few typos (you told me to nitpick!!) no additional complaints.

Could you point out the typos, please? I'd very much like to fix them. Yea, nitpicking is good.

 

[Eli13]

Could you point out the typos, please? I'd very much like to fix them. Yea, nitpicking is good.

The only one I saw was on page 16 when you said "Also please not..." instead of "Also please note..."
Other than whats said above I can only add another 4/5 DG's for the aforementioned points.
Overall, an interesting read, thanks for sharing it.