The AnimeSuki community has already entered the voting phase of their annual community anime awards. It’s a silly popularity ballot to celebrate the titles the community is fond of, but I always found it enjoyable to be part of. This year I was bored enough to cast my votes in the nomination phase of the tournament. After the nominations were closed, the community started deliberating possible changes to next year’s awards, one of which were changes to the nominee acceptance threshold. It’s a numbers problem, so naturally I was interested.
For this year, the threshold was set to four, meaning the four titles with the most points received during the nomination phase would move on to the more casual voting phase. The nomination phase is needed to cut down on the number of award candidates, to make the voting phase seemingly more competitive and fun. But four is a pretty arbitrary number, is it not? Users have suggested lowering the threshold, so that extra titles that actually have a legitimate chance of winning or making things more interesting will have a place in the voting phase.
Before I get to the statistics part, how does the nomination phase play out? Each participating user gets a two-point vote and two one-point votes for each category. For example, I gave Mobile Suit Gundam Unicorn two points, whereas Kill la Kill and Sword Art Online II were my two runner-ups in the category of Best Visuals & Animation, each receiving one point. You can abstain from voting in any category and you can choose not to nominate all three titles. The points get tallied and the four titles with the most points in each category advance.
So, where should we set the cut-off point? Keep it at four? Increase it to five? Six? Eight? Ten? I decided to test this with a machine learning approach. The goal of this exercise would be to calculate the ideal cut-off point for each category based on voting data. I looked at the voting data and counted for each nominee in each category how many points it received, the percentage of the vote it got, and how many of those votes were two-point votes. I then fed the data to a clustering algorithm called K-Means. K-Means calculates the distances between data points for each title and forms a K-number of most compact clusters of data points. To get a better sense of what I’m talking about, look at the night sky and try to draw circles around groups of stars based on how close they appear to be. That’s generally what the algorithm does.
I ran the algorithm at K=2 and K=3 on each category, meaning I wanted to see how the algorithm would separate titles in two or three groups. I would then take the highest-ranked group and set its membership count as the threshold. The results?
As we can see, the K-Means clustering algorithm makes a strong case for setting the threshold differently for each category. The most clear-cut example would be the Gundam Unicorn OVA in C2. The algorithm throws it into a group of one, because the voting stats for it are so above the rest of the competitors it would set a bad precedent to put any of them in its group. However, taking into account it’s fairly useless to set the threshold below four for the purposes of the competition, we can conclude the current threshold of four is actually pretty good, with a few exceptions. At K=2 (being the more permissive parameter), roughly ten categories would step over it. At K=3, only five would, with only D6 (Best Credits Theme category) overstepping it considerably.
Whether the cut-off point should be set a bit higher, possibly at five, is debatable. A consensus is forming that D6 should be treated differently. However, we should first look at voting trends from past awards to judge whether certain categories truly deserve special treatment. Generally, you can’t go wrong with four, it’s a good fit. My exercise only confirms that the organizers have so far had a very good idea how many nominees on average actually matter in the final voting phase.
In a perfect world, thresholds should change yearly because the amount of notable anime each year increases or decreases. However, going down this path would present a problem for the integrity of the voting process and the transparency of the process for choosing nominees, because the thresholds would have to be set after the competition starts. It might also raise questions of favoritism in case of variable thresholds.
That said, there have been years when I thought certain deserving anime titles were missing from the nominee pool. It was just a gut feeling, but I did ask myself very briefly whether the nomination process was somehow skewed, given there were many notable anime titles missing from the plate. That’s really the only reason why I would appreciate a bigger nominee pool.