Deviation Index (A metric for assessing performance quality)

April 16, 2020 - 5 minutes

Introduction to Deviation Index

The goal of this blog is to introduce statistics called the deviation indicies. Deviation indices give an assessment of a team’s performance in a given game relative to the entire season. This can serve multiple purposes including but not limited to:

  1. Determining the reason for a specific result
  2. Determining how much a record is due to well-timed strong performances or poorly-timed poor performances
  3. Determining a given team’s typical result
  4. Determining if a game is a true upset

Briefly, the deviation index for a given game is calculated using the following formula:

\[ DeviationIndex = log(\frac{GameAverage}{SeasonAverage}) \]

Game average is the average performance against an opponent relative to all other teams who have played them. Season average is the average of the same performance statistic across the entire season. For a complete explanation of these statistics see the following link. At its simplest, higher absolute values of deviation indices indicate more out of the ordinary performances. Positive performances are indicated by positive values on offense and negative values on defense. The opposite holds for negative performances. We can calculate a complete summary of performance using the following formula:

\[ TotalDeviation = DeviationIndex_{Off} + (1-DeviationIndex_{Def}) \]

To make it even more concrete we show the top 5 best deviation indices from the 2019-2020 season.:

and the top 5 worst:

“Tot Dev” indicates the total deviation. As we can see 3 out of the 5 games are shared between the two tables. This is often the case because stronger defense than usual on one side can be driven by poorer than usual offense on the other. These games are generally blowouts, some with unexpected results. For instance, the highest deviations in the negative and positive direction both happened in the same game. It is no coincidence this game ended in one of the largest blowouts in NCAA history. Also of note is the unexpected blowout of Memphis by Tulsa and Oklahoma State’s big win against Ole Miss.

How deviation indices can be used to identify true upsets and why a team wins

Total deviation can also be used to discern between true upsets and “false upsets”. It can can also be used to determine which team is responsible for a result. An example of a game where responsibility can be assigned to a single team is Clemson’s (RR 77, KenPom 72 ) upset of Florida State (RR 18, KenPom 15) this year. Based on how relative rankings work, it is expected that both teams playing to their typical ability would result in a Florida State win by a reasonable margin. In this particular game Clemson played relatively close to their average (TotalDeviation = 0.06, 12th best performance) while Florida State played extremely poorly (TotalDeviation = -.15, 2nd worst performance). This was enough for a 1 point Clemson win. In their only other matchup of the season, Florida State had their 4th best performance of the season, winning 72-53.

Now for a true upset, we can look to Evansville’s (RR 273, KenPom 294) early season upset of Kentucky (RR 22, KenPom 29). In this particular game, Evansville put up a TotalDeviation (0.30) in the top 95% of all games and Kentucky put one up in the bottom 5% (-.34). Despite this, Evansville won by just 3 points. This result, highlights how unlikely the Evansville win truly was. On a side note, Evansville consistently had similar quality performances early in the season before tanking after an injury to Deandre Williams. One of their worst pre-injury performances saw them nearly beat SMU. It would be hard to argue that Evansville with Williams wouldn’t be near the top of the MVC.

On the flipside, a fake upset can be thought of as a game considered an upset but where neither team performs exceptionally different than average. Examples are a little more difficult to find because oftentimes, even if teams are closely matched, one significantly outperforms the other. Additionally, if teams are closely matched according to conventional rankings then the game will not be considered an upset. A rather loose example (Loose because Georgetown were favored at the time of the game) is Georgetown’s (RR 78, KenPom 67) January 83-80 win against Creighton (RR 32, KenPom 12). In this particular game, Georgetown (TotalDeviation = 0.07) and Creighton (TotalDeviation = 0.04) both performed close to expected but the lower ranked team won.

Games where both teams play at their typical level give good comparative insights

While high absolute values of TotalDeviation gives insights about upsets, small values allow us to benchmark teams. When both teams have low absolute values then we can make a true comparison of quality. One example is this years California team (RR 217, KenPom 153). They had a near zero deviation game (-0.02) vs. Harvard (RR 105, KenPom 110) who also had an similar deviation (-0.03). In this particular game, the much higher ranked (at least in RR) Harvard won 71-63. Another example Georgia Southern (RR 155, KenPom 134) vs. Coastal Carolina (RR 153, KenPom 190) who are similarly ranked in RR and had a game with identical deviations. In that particular game, Coastal Carolina won 70-67. As expected, close ranked teams play close games when performing similarly.

We can also use the metric to lend credibility to high quality mid-majors. San Diego State (RR 5, KenPom 6) and Utah State (RR 16, KenPom 41) provides a good benchmark for the current season. Dayton (RR 3, KenPom 4) either played better than usual or worse than usual against big conference teams and therefore can’t be used. San Diego State’s lowest TotalDeviation different game was against Iowa (RR 38, KenPom 23) and they won 83-73. This result shows that San Diego State, playing at their typical ability are capable of blowing out a solid Big 12 team playing at their typical ability. Conversly, Utah State beat LSU (RR 19, KenPom 30) and Florida (RR 41, KenPom 32), 80-78 and 65-62 respectively. The difference in TotalDeviation between the teams in the games were both neglible. Based on this, it would not be a stretch to say Utah State would be a contender in the SEC.