Newsgroups: rec.arts.int-fiction
Path: nntp.gmd.de!news.ruhr-uni-bochum.de!news.rhrz.uni-bonn.de!news.rwth-aachen.de!news-koe1.dfn.de!news-han1.dfn.de!news-ber1.dfn.de!fu-berlin.de!unlisys!blackbush.xlink.net!howland.erols.net!news-peer.gsl.net!news.gsl.net!ix.netcom.com!netcom.com!erkyrath
From: erkyrath@netcom.com (Andrew Plotkin)
Subject: Re: Review of Russ Bryan's Review of Andrew Pontious's Capsule Reviews of Contest Games
Message-ID: <erkyrathE223JB.5C5@netcom.com>
Organization: NETCOM On-line Communication Services (408 261-4700 guest)
X-Newsreader: TIN [version 1.2 PL1]
References: <5826jv$1h8@news.thorn.net> <5841as$k5h@vixen.cso.uiuc.edu> <Pine.SUN.3.91.961204120437.6292A-100000@nebula.phy.duke.edu> <ant052249868M+4%@gnelson.demon.co.uk> <58afbp$et7@nntp5.u.washington.edu>
Date: Sat, 7 Dec 1996 18:33:11 GMT
Lines: 66
Sender: erkyrath@netcom2.netcom.com

Dan Shiovitz (scythe@u.washington.edu) wrote:
> >I agree about criteria: aesthetic judgements are not so easily
> >legislated for.  But we might slightly normalise the scoring
> >system, perhaps?  Perhaps each voter should have just 100 points
> >to award, with a maximum of 10 to any one game?  I wonder what
> >the variation in average scores per game was.

> I agree with what I think you're saying, which is that it's silly that
> some people awarded scores like "6 7 8 10 10 9 6 8" and some people did 
> "4 3 5 8 3 4 2 6" and some people did "2 8 3 7 7 4 10", which I'm sure 
> must make the whole averaging thing quite odd.  The way you say seems
> to be an ok workaround, but it doesn't deal with the problem of people
> judging a variable number of games, or with the problem that they might
> happen to really like all the games they get, and so they finally decide
> to give game A a 7 and B an 8, whereas another person hates game A and 
> so has enough points to give game B a 10.  But I'm not sure what a better 
> solution is .. (is there a statistician in the house?)

I was thinking about this months ago, when the rules for this contest 
were being debated. 

The draconian solution is to normalize everybody's set of scores -- in 
the statistical sense, normalizing both mean and deviation. This means, 
roughly, that you take each set of scores and shift it up or down until 
the average is exactly 5, and then you spread or narrow the range until 
it's, er, exactly the same as everyone else's range in a statistical 
sense which I don't feel like explaining.

So if you rate only three games, assigning scores "2, 3, 4" winds up being
the same as assigning "2, 6, 10".

This seems a little too much. (For one thing, if you play only one or two 
games, you might as well not send in your scores at all.)

We could normalize the mean but not the deviation. Shift the scores up and
down so that the average is 5, and that's all. (Now, if you rate three
games, assigning "2, 3, 4" is the same as assigning "8, 9, 10", but
different from "1, 5, 9". The first two options say that the three games 
are close together in quality, and the third says that they vary a lot. 
You could also meaningfully say "1, 9, 10" or something like that.)

This latter scheme is similar to what Graham suggested. Actually it's
morally identical to a variant of Graham's scheme: "Each voter is given a 
pool of N points, where N is 5 times the number of games he played. He 
can divide the points up among those games however he wants, assigning no 
less than 0 and no more than 10 points per game." You can see that this 
forces the average to be 5.

If you want the ranges to be comparable, without strictly normalizing 
them, you could add a rule that the worst game you play must be a 0, and 
the best game a 10. This is still harsh on someone who players only two 
games -- they would have to make a "0, 10" split. And of course it's 
always impossible to rate one game in any scheme that normalizes the 
mean. But I don't think it's a big deal to require people to play at 
least three games. 

All of these schemes are complex, and I don't actually recommend any of 
them. I think the current scheme isn't really any better, but at least 
it's simple.

--Z

-- 

"And Aholibamah bare Jeush, and Jaalam, and Korah: these were the
borogoves..."
