By Evan Miller
February 16, 2016
In the previous episode, I analyzed the steady-state behavior of the Splatoon ranking system and showed that it is heavily biased towards producing A and A+ players (at the expense of the C and B ranks). In the long term, regardless of actual skill level, almost three-quarters of Splatoon players will maintain the rank of A− or higher, with over half having a rank of A or A+, and over one-third of players ranked at A+.
Because so many players were able to reach the highest nominal ranks, there were very large skill disparities within the A ranks, making Splatoon battles less challenging for very high skill players, and more depressing for players of moderate skill who stumbled into the upper echelons of competitive play. As the A ranks became flooded with players of lower and lower skill levels, it became apparent to players, and to Nintendo, that something needed to be done.
Unfortunately, while that “something” addressed the visible symptom of having too many “A” players, the patches thus far have failed to address the underlying cause. Purely in terms of balancing the number of players in each rank, the situation is even worse than it was before the patches.
Version 2.0: A Cure Worse Than The Illness
To address the rampant grade inflation, last year the masters of Splat introduced two new ranks beyond the A+ rank: S and S+. With the Version 2.0 patch in August 2015, these new ranks were accompanied by the following reward structure:
| Rank | Current Points | Reward for Victory | Penalty for Defeat |
|---|---|---|---|
| S | 0-39 | +5 | −4 |
| 40-79 | +4 | −5 | |
| 80-99 | +3 | −6 | |
| S+ | 0-39 | +3 | −6 |
| 40-79 | +3 | −6 | |
| 80-99 | +2 | −8 |
On its face, the new reward structure appears to address the problem of having too many players at the top. Once you achieve a rank of S and reach 40 points, the penalty/reward structure requires you to win significantly more games than you lose in order to maintain your point level, and so you might expect only players with the highest skill to reach S and S+.
But did the introduction of S and S+ really solve the grade inflation problem? Using the mathematical tools developed in the previous article, we can solve for the steady-state behavior of the new ranking rules. Here is the long-run distribution of ranks under the Version 2.0 rules:
| Score | Steady state | Sum |
|---|---|---|
| C− | 0.002% | 0.4% |
| C | 0.048% | |
| C+ | 0.35% | |
| B− | 1.29% | 8.53% |
| B | 3.11% | |
| B+ | 4.14% | |
| A− | 5.52% | 22.7% |
| A | 7.36% | |
| A+ | 9.81% | |
| S | 67.9% | 68.36% |
| S+ | 0.46% |
One glance at the above table should tell you that the Version 2.0 ranking rules flooded the game with S players. While it achieved the goal of having S+ exist only for ultra-elite players (less then 0.5%), the cost was that nearly two-thirds of players were eventually pushed into S, creating a vast skill disparity within the second-highest rank.
It’s not too hard to see the source of the problem: in addition to promotion/demotion asymmetry that caused the previous grade inflation (see previous article), the new S rank injects even more points into the system for players with 0-39 points. For these players, a single win (+5 points) more than offsets a single loss (−4 points), so this region tends to pull in recently-promoted A+ players and never sends them back. The combined effect of promotion/demotion asymmetry and positive bias in the S/0-39 zone is a massive glut of players with a rank of S.
Not too surprisingly, Nintendo attempted to patch the patch as quickly as possible.
Version 2.2: The Patient Will Live, But The Prognosis Isn’t Good
In an attempt to shrink the swollen ranks of S, Nintendo tweaked the reward structure in the Splatoon Version 2.2 patch, which was released in October of 2015, a mere two months after the Version 2.0 patch. The new reward table looked like this:
| Rank | Current Points | Reward for Victory | Penalty for Defeat |
|---|---|---|---|
| S | 0-39 | +5 | −5 |
| 40-79 | +4 | −5 | |
| 80-99 | +4 | −6 | |
| S+ | 0-39 | +4 | −4 |
| 40-79 | +3 | −5 | |
| 80-99 | +2 | −5 |
You can see here that Nintendo eliminated the point-injection locus at S/0-39 by balancing the reward and penalty. (In Version 2.0, it was a +5 reward, −4 penalty; with Version 2.2, it is a +5 reward, −5 penalty.) So far so good. But how does the steady state look under the new rules? We can solve for it:
| Score | Steady state | Sum |
|---|---|---|
| C− | 0.0032% | 0.65% |
| C | 0.078% | |
| C+ | 0.57% | |
| B− | 2.08% | 13.8% |
| B | 5.02% | |
| B+ | 6.7% | |
| A− | 8.93% | 36.71% |
| A | 11.91% | |
| A+ | 15.88% | |
| S | 45.71% | 48.84% |
| S+ | 3.12% |
While it’s not the lopsided fiasco of the Version 2.0 patch, the grade inflation is still quite bad: over 45% of Splatoon players will end up with the rank S. In other words, the second-highest rank will roughly encompass skill level ranging from the median player all the way up to players in the 97th percentile.
The skill disparities within S will tend to leave many players in limbo between A+ and S, that is, constantly being promoted to S and then demoted back to A+. The net effect is to inject a steady stream of points into the system and accelerate progress toward the steady state. Anecdotal evidence confirms the existence of these A+/S limbo players:
@SplatoonNA forever between A+ and S- :')
— Juliana Chen (@fivepaninis) February 8, 2016
@SplatoonNA Currently A+. I keep going up and down between A+ and S. Still haven't ever gotten to S+ yet.
— ET (@etthenerdling) February 8, 2016
@SplatoonNA A+ my rank always goes up and down lol
— Fly Me To The Moon✨ (@sea_salt_prince) February 8, 2016
Another effect of the Version 2.2 changes was to make S+ more accessible: whereas before, less than 0.5% of players could be expected to have the S+ rank in the steady state, the figure is now about three percent of players. Reasonable people can disagree whether this particular development is a good or bad for the chi of Splatoon, but there you have it.
Balancing The Ranks
I will reiterate the advice to Nintendo found my previous article: the simplest way to fix Splatoon’s widespread grade inflation is not to introduce new ranks, but to eliminate the asymmetry that exists between promotion and demotion. (To be demoted, you must lose 31 points relative to your last promotion; to be promoted, you must gain 30 points relative to your last demotion.)
The steady state with a "demote at zero" rule in place is as follows:
| Score | Steady state | Sum |
|---|---|---|
| C− | 0.016% | 2.34% |
| C | 0.3% | |
| C+ | 2.02% | |
| B− | 6.65% | 29.33% |
| B | 11.34% | |
| B+ | 11.34% | |
| A− | 11.34% | 34.01% |
| A | 11.34% | |
| A+ | 11.34% | |
| S | 32.14% | 34.33% |
| S+ | 2.19% |
Under this distribution, roughly a third of players will have rank B(±), a third rank A(±), and a third rank S(+), which is considerably more balanced than the Version 2.2 regime (or any that preceded it).
Even with this fix in place, a glut exists at the unadorned rank of S. Whereas only 11% of players will have rank A+, fully 32% will have a rank of S. One way to address this is to tinker with the reward table a little bit further. For example, having a reward of +4 and a penalty of −5 for all players of rank S, and keeping the rest of the reward structure constant, yields the following distribution:
| Score | Steady state | Sum |
|---|---|---|
| C− | 0.02% | 2.81% |
| C | 0.36% | |
| C+ | 2.42% | |
| B− | 7.98% | 35.2% |
| B | 13.61% | |
| B+ | 13.61% | |
| A− | 13.61% | 40.83% |
| A | 13.61% | |
| A+ | 13.61% | |
| S | 19.94% | 21.16% |
| S+ | 1.22% |
That simple change reduces the number of S players by about 50% while preserving the character of the rest of the distribution.
In any event, Nintendo’s failed attempts to fix Splatoon’s ranking system should serve as a warning to designers of competitive multiplayer games. Like the butterfly-hurricane trope, subtle asymmetries in point systems can have massive, long-term effects on the distribution of player ranks, and by extension, the overall game experience. Before implementing the next set of changes and suffering any unintended consequences, I strongly suggest that Nintendo — as well as other designers of online multiplayer games — perform a proper steady-state analysis, perhaps with the help of a script or two.
You’re reading evanmiller.org, a random collection of math, tech, and musings. If you liked this you might also enjoy:
- Evaluating Splatoon’s Ranking System
- Adventure Games and Eigenvalues
- Deriving the Reddit Formula
- Ranking News Items With Upvotes
- Bayesian Average Ratings
- Ranking Items With Star Ratings: An Approximate Bayesian Approach
Get new articles as they’re published, via LinkedIn, Twitter, or RSS.
Want to look for statistical patterns in your MySQL, PostgreSQL, or SQLite database? My desktop statistics software Wizard can help you analyze more data in less time and communicate discoveries visually without spending days struggling with pointless command syntax. Check it out!
Wizard
Statistics the Mac way
Back to Evan Miller’s home page – Subscribe to RSS – LinkedIn – Twitter