The how's and why's of rating addons

RisingFury

OBSP developer
Addon Developer
Joined
Aug 15, 2008
Messages
6,290
Reaction score
120
Points
153
Location
Among bits and Bytes...
For a while I've been observing the addon voting system on OrbitHangar and it seemed to me like people didn't put enough thought into the vote the assign to an addon, or that in some cases, they vote dishonestly, because they don't want to offend the author.

So I decided to write guide that will hopefully provoke some thought, before a rating is given. This has been a long time coming.

I initially intended to write a blog, but I soon realized that the amount of content and images were too much for the blog system. Even a new thread couldn't handle it all, so I decided to write it into a PDF.

It's quite long, spanning 19 pages (don't worry, lots of pictures) and it details a system of rating that I think is objective and fair. So without further ado:

The how's and why's of rating addons

Feedback is appreciated.
 

Wishbone

Clueless developer
Addon Developer
Joined
Sep 12, 2010
Messages
2,421
Reaction score
1
Points
0
Location
Moscow
The notion of objectivity and equal weights of various components of a rating formula don't fare well with each other. There is nothing objective in saying "I assign weights equal to 1 to all the parts because I think they should be equal". There's no law of nature that prevents people from using a non-linear rating, either.
 

Artlav

Aperiodic traveller
Addon Developer
Beta Tester
Joined
Jan 7, 2008
Messages
5,560
Reaction score
35
Points
138
Location
Moscow
Website
orbides.org
Basically, it's a guide on how to be reasonable.
As such, it is of limited use - it may make considering people consider carefully, but it won't make wow-cool-addon people and this-sucks people consider anything else.

It would be useful to have separate ratings for different scales as described, rather than what is like measuring both mass and size in seconds.

It could also be better to bring back the old rate & explain-why-so system.

4 DGs for effort, in any case :)
 

RisingFury

OBSP developer
Addon Developer
Joined
Aug 15, 2008
Messages
6,290
Reaction score
120
Points
153
Location
Among bits and Bytes...
Wishbone;bt3373 said:
The notion of objectivity and equal weights of various components of a rating formula don't fare well with each other. There is nothing objective in saying "I assign weights equal to 1 to all the parts because I think they should be equal". There's no law of nature that prevents people from using a non-linear rating, either.

You're absolutely right about the scale. I considered using a non linear scale. It would certainly make sense because rating 5 DGs in bugs and rating 5 DGs in features aren't equivalent.

I decided to go with the linear scale as a tradeoff between accuracy of rating and people having to do the math. We've had numerous examples on O-F where people would ask for help in solving the most simplistic linear equations and sticking more factors in there would make things confusing for people and they'd just revert to the "THIS ADDON ROCKS" rating system.

The guide shows an example of a rating system that is more sophisticated than how people rate addons today, but if you come up with your own system that rates more accurately, please use it!

Thanks for the comment :)



Artlav;bt3374 said:
Basically, it's a guide on how to be reasonable.
As such, it is of limited use - it may make considering people consider carefully, but it won't make wow-cool-addon people and this-sucks people consider anything else.

It would be useful to have separate ratings for different scales as described, rather than what is like measuring both mass and size in seconds.

It could also be better to bring back the old rate & explain-why-so system.

4 DGs for effort, in any case :)

From the start I've known that this guide will be of limited help. Only a small percentage of orbonauts will actually read it. But I figured I had to do something and this is better than standing still and ignoring it.

Rate-and-explain is also a good system because at the very lease you'd see how many people took the time to look at the entire addon instead of just writing what their first impression was. That way you'd get a measure of how many ratings a credible.

Thanks for your comment :)
 

Izack

Non sequitur
Addon Developer
Joined
Feb 4, 2010
Messages
6,667
Reaction score
1
Points
0
Location
The Wilderness, N.B.


4/5 for effort and interesting concepts, with the main cons being already mentioned above. Other than a few typos (you told me to nitpick!!) no additional complaints. :)
 

RisingFury

OBSP developer
Addon Developer
Joined
Aug 15, 2008
Messages
6,290
Reaction score
120
Points
153
Location
Among bits and Bytes...
Izack;bt3376 said:


4/5 for effort and interesting concepts, with the main cons being already mentioned above. Other than a few typos (you told me to nitpick!!) no additional complaints. :)

Could you point out the typos, please? I'd very much like to fix them. Yea, nitpicking is good.
 

Eli13

Fish Dreamer
Joined
Mar 5, 2011
Messages
1,562
Reaction score
0
Points
0
Location
Somewhere, TN
RisingFury;bt3377 said:
Could you point out the typos, please? I'd very much like to fix them. Yea, nitpicking is good.
The only one I saw was on page 16 when you said "Also please not..." instead of "Also please note..."
Other than whats said above I can only add another 4/5 DG's for the aforementioned points.
Overall, an interesting read, thanks for sharing it. :)
 
Top