How to calculate mean based on number of votes/scores/samples/etc?

Question

For simplicity say we have a sample set of possible scores {0, 1, 2}. Is there a way to calculate a mean based on the number of scores without getting into hairy lookup tables etc for a 95% confidence interval calculation?

dreeves posted a solution to this here: How can I calculate a fair overall game score based on a variable number of matches?

Now say we have 2 scenarios ...

Scenario A) 2 votes of value 2 result in SE=0 resulting in the mean to be 2

Scenario B) 10000 votes of value 2 result in SE=0 resulting in the mean to be 2

I wanted Scenario A to be some value less than 2 because of the low number of votes, but it doesn't seem like this solution handles that (dreeve's equations hold when you don't have all values in your set equal to each other). Am I missing something or is there another algorithm I can use to calculate a better score.

The data available to me is:

• n (number of votes)
• sum (sum of votes)
• {set of votes} (all vote values)

Thanks!

Answer 1

You could just give it a weighted score when ranking results, as opposed to just displaying the average vote so far, by multiplying with some function of the number of votes.

An example in C# (because that's what I happen to know best...) that could easily be translated into your language of choice:

double avgScore = Math.Round(sum / n);
double rank = avgScore * Math.Log(n);

Here I've used the logarithm of n as the weighting function - but it will only work well if the number of votes is neither too small or too large. Exactly how large is "optimal" depends on how much you want the number of votes to matter.

If you like the logarithmic approach, but base 10 doesn't really work with your vote counts, you could easily use another base. For example, to do it in base 3 instead:

double rank = avgScore * Math.Log(n, 3);

Which function you should use for weighing is probably best decided by the order of magnitude of the number of votes you expect to reach.

You could also use a custom weighting function by defining

double rank = avgScore * w(n);

where w(n) returns the weight value depending on the number of votes. You then define w(n) as you wish, for example like this:

double w(int n) {
    // caution! ugly example code ahead...
    // if you even want this approach, at least use a switch... :P

    if (n > 100) { 
        return 10; 
    } else if (n > 50) {
        return 8;
    } else if (n > 40) {
        return 6;
    } else if (n > 20) {
        return 3;
    } else if (n > 10) {
        return 2;
    } else {
        return 1;
    }
}

Answer 2

If you want to use the idea in my other referenced answer (thanks!) of using a pessimistic lower bound on the average then I think some additional assumptions/parameters are going to need to be injected.

To make sure I understand: With 10000 votes, every single one of which is "2", you're very sure the true average is 2. With 2 votes, each a "2", you're very unsure -- maybe some 0's and 1's will come in and bring down the average. But how to quantify that, I think is your question.

Here's an idea: Everyone starts with some "baggage": a single phantom vote of "1". The person with 2 true "2" votes will then have an average of (1+2+2)/3 = 1.67 where the person with 10000 true "2" votes will have an average of 1.9997. That alone may satisfy your criteria. Or to add the pessimistic lower bound idea, the person with 2 votes would have a pessimistic average score of 1.333 and the person with 10k votes would be 1.99948.

(To be absolutely sure you'll never have the problem of zero standard error, use two different phantom votes. Or perhaps use as many phantom votes as there are possible vote values, one vote with each value.)