Is one judge better than a jury at choosing a literary prize winner? What does the math say?

“My literary taste is unorthodox by today’s standards. I believe this is correct and do my best to defend it in my criticism. But I have never had the courage to imagine that my literary judgments amount to objective, provable truths. However, I don’t consider them negotiable either. So the prospect of sitting down with a couple of strangers and haggling over our literary tastes seemed radically unappealing to me.”

– “Judge a literary prize?” No thanks, this is all a waste of time,” David Free, 9 July 2021, Sydney Morning Herald.

David Free’s essay raises an interesting question: Assuming we need to rank literature, how should we do it? David Free is an Australian literary critic (and novelist), so I think he’s doing well when it comes to literary ratings. But he is definitely against a group of literary figures handing out literary prizes.

Why? He doesn’t trust juries. He wouldn’t mind having one judge make the decision; at least the subjectivity would be obvious. Even desirable. But the jury’s consensus, he argues, will invariably gravitate towards the mediocre and safe options. Often it’s not the book at all, but the author.

I’ve been on a few prize discussions and yes, I always found it quite strange. This could be serious melodrama. I’ve seen books excluded for the most flimsy of reasons – or so it seemed to me. The author’s biography is important, although it is considered bad manners to mention this. There is also no consistency in the way we talk about or evaluate literature. Some judges wanted to grade books as homework; others seemed to see it as an opportunity to make the world a better place. My own bias was that I valued originality above all else, so I invariably chose asylum cases. And so on and so forth. Overall a pleasant experience when your friends suddenly spank you with a wet bath towel. Damn it then, but later, haha.

I have concluded that if the goal is to reward representativeness, then a jury is best. For example, if age values ​​visceral content over plot, then the jury’s decision will reflect that value.

This explains why the author’s biography has come to play a role in the jury’s selection of prizes. Our age increasingly values ​​morality or manifestations of morality. This also explains why it matters so much whether an author belongs to one group or another: our age has come to value representation of the underrepresented in all fields. The jury’s decision will certainly reflect this. This is what juries do: they select the most representative candidate from an imaginary set of shared values.

On the other hand, if the goal is to reward originality rather than representativeness, then it is better to have one judge, preferably an insane one. The trouble is that useful madness is hard to find, so original books often go unread and unawarded.

By the way, let me be clear: I am 100 percent in favor of being given literary prizes. Any prize, really. My first prize was the Eric Knight Prize. Lassie, go homereceived at the age of seven for winning a frog jumping competition.

From a mathematical perspective (i.e. feel free to stop reading here) there is a much deeper problem with All group evaluations, not just literary evaluations. Suppose we think of authors as athletes, and of literary criteria as athletic competitions in the decathlon. Each judge evaluates the performance of the athlete (author) in each event and awards 1st, 2nd, 3rd,… places. Now we need to somehow combine/match these ratings into one overall rating. What are the minimum conditions for this matching procedure?

Well, we don’t want any judge to be able to suppress any other judge (ie no dictators). But if everyone prefers author A to author B, then the final ranking should reflect this preference (Pareto principle criterion). Also, can we agree that when we try to decide between authors A and B, we don’t want any other author C to be relevant to the discussion (relevance criteria). And, of course, each rating must be acceptable (unlimited scope criteria). For example, if some judge wants to rank Barbara Cartland over James Hadley Chase over Samuel Beckett, then fine, whatever.

Here’s the question: Can we have a matching/aggregating/merging process that takes into account each judge’s ratings and produces an overall rating that satisfies these four very reasonable criteria: no dictators, Pareto principle, relevance, and unlimited volume?

The answer is no. Kenneth Arrow’s famous impossibility theorem states that, in general, there is no matching scheme that can simultaneously satisfy all four criteria. For this he received the Nobel Prize. Amartya Sen became famous (and won a Nobel Prize) for developing Arrow’s work.

The point? Panel literary evaluations are always at risk because they select winners that no judge wanted; they risk having one judge overrule all the others; they risk considering irrelevant alternatives; and they risk excluding certain preferences.

Mathematics is to blame.

---

This was originally posted by Anil Menon on his page Facebook profile.