Somebody please explain the computer models' ridiculous rankings

[SilentsAreGolden]

Coming into this weekend, we were 14th and UL was 5th (Kenpom). We beat the then #78 team by 22 pts. UL beats the 89th team by 8 and we fall to 16th and they stay 5th. Our rankings are similar in the Sagarin and BPI too, so I know it's not just Kenpom.

 

[kats23]

Not entirely sure but it probably has to do with what all the other opponents, future and past did over the weekend. If the ACC did well and the SEC didn't, it could keep them steady and drop UK

 

[KWilt43atbuzz]

Have you factored in the anti UK conspiracy?

 

[Smashcat]

Sagarin has UK 3-1 vs. its top 50 and UL 0-2 against its top 50 yet UL is rated far higher. Right now the computer folks absolutely loooove Nova, UL and Xavier. Those teams can do no wrong, and I'll eat my hat if any of those three makes a FF. LOL

 

[UKnCincy]

Coming into this weekend, we were 14th and UL was 5th (Kenpom). We beat the then #78 team by 22 pts. UL beats the 89th team by 8 and we fall to 16th and they stay 5th. Our rankings are similar in the Sagarin and BPI too, so I know it's not just Kenpom.

Not sure how Sagarin or BPI work, but Kenpom is driven by a team's efficiency ratings. UK's average offensive and defensive efficiencies are not that impressive because there have been some dramatic swings that water our ratings down. Other teams, such as Lousiville, have been very consistent in their efficiencies from game to game.

For example, against USF we scored 1.2 points per possession (extremely high number). In the next game we managed only .96 points per possession, which is atrocious. We still won big however, because our defensive efficiency improved and offset the offensive lapse. However, when you average these types of swings together, the result is a less than impressive number for both offensive and defensive efficiency.

You also have issues like the UCLA game. UCLA averages around 1.08 points per possession, so they aren't exactly worldbeaters on offense. However, we allowed them to score over 1.2 points per possession, which is an elite number. Anytime your defense allows a team to perform dramatically above their average, you're going to take a hit when the adjustments are made for opponent quality.

When you add all of these types of things together, the computers end up rating UK lower. I wouldn't sweat this too much right now. UK has a high degree of performance variance due to their youth and this inconsistency results in lower ratings. If they become more consistent, you'll see the ratings rise.

 

[jockstrap_mcgee]

This is what I think of computer rankings.

 

[fatguy87]

Coming into this weekend, we were 14th and UL was 5th (Kenpom). We beat the then #78 team by 22 pts. UL beats the 89th team by 8 and we fall to 16th and they stay 5th. Our rankings are similar in the Sagarin and BPI too, so I know it's not just Kenpom.

Wins and losses are irrelevant. Margin of victory is what's important to those systems.

Our rating is lower than UL's because we've had a few stinkers. We played poorly against overmatched teams like Wright State, Albany, and Illinois State. Throw in our two losses, and that's over 1/3 of the schedule where we looked like a NCAA tournament bubble team or worse.

Louisville has played a week schedule, but they've blown out all the Wright State's of the basketball universe by 20+ generally. Looking at their schedule, the only game I can point out that they played poorly in may have been the game tonight. A system like Kenpom and Sagarin's predictor will actually see UL's 2 point loss against UK as evidence that UL is better than UK due to a home court adjustment.

These margin of victory based systems are built to predict future results, not judge a team's win/loss accomplishments. That's a common point of confusion.

 

[KentuckyRLD]

Computer models are nothing more than a set of programs that analyze the available data and spit out what the person who wrote the program has designed it to spit out. Most of the people who put these things together claim that the projections will be more accurate later in the season because more data is available. The truth of the matter is that the people who put these programs together are simply not smart enough to figure out how to construct a program that accurately weighs the available data.

 

[JeffGoodmansVestigialTwin]

When March rolls around those computer rankings will go down the crapper....especially this year.

 

[wildcatsboston1984]

The computer rankings, like Pomeroy, are all based on offense and defensive efficiency, adjusted for pace. Rather you win or lose a game doesn't really matter for Pomeory, it's how you won or lost the game that matters.

 

[SilentsAreGolden]

Thanks for the responses. It's actually more clear to me now. I just hate that their fans get to pretend they are Top 5 program when they haven't beaten anyone of consequence yet. If they win at NCState, I'll be impressed.

 

[mj2k10]

UofL has found the Wisconsin "cheat Pomeroy" hole this year.

Wisconsin was legit the last 2 years, but prior to that, almost every single season Wisconsin was wildly overrated in Pomeroy's system. You'd see things like them having double digit losses and still being top 5.

The key to that was scheduling. Yeah, Pomeroy has limiting factors built in for blowouts, but there's only so much to that limitation. Wisconsin would beat the crap out of all the dogs they played, putting up tremendous efficiency numbers, then they would keep almost all the games against quality opponents close (even if they lost a lot of them). It added up to really good efficiency numbers against an overall OK schedule, and the Pomeroy formula loved it, even if Wisconsin ended up a 5 seed or somewhere in that range.

 

[.S&C.]

It's all bullsh*t. Dorks have tried for years to take the game off the court and onto a frequency table. Don't buy it.

Reality is the outcome of the games.

 

[JonathanW]

As a statistician, I have to disagree with much of the above saying computer rankings are BS.

I even created my own computer ranking system about 10-15 years ago, although I haven't ran it this year (new job = less free time). But the more information/data a system gets, the more reliable/accurate it is. It still can't account for match-up issues, so they aren't good to use in trying to compare how 2 teams will play vs each other.

Until December, computer rankings are useless. Largely based on the authors biases or pre-season assumptions. I would say until most teams get 10 games under their belt, I wouldn't even look at a computer ranking. And even then, they don't start getting accurate until after about 2 weeks (4 games) of conference play.

But to the OPs original question, any legit ranking system will at a minimum look at not just how well you do, but how well your opponents do. So perhaps our 11 opponents thus far had a bad weekend compared to UL's prior opponents. Also was UL's game on the road or at home like ours (that should be a factor).

My system (I think) has 6 iterations, so it looks at opponents of opponents of opponents of opponents of opponents of opponents. The reason for that is to calculate a probability of winning (assuming no match-up issues) vs each of the 300+ D1 teams, and it takes 5-6 iterations after about 10 games to be able to link every team with every other team.
The factors my system uses include:
- scoring margin of each game
- takes home court into account (based on historical conference data, if I remember correctly the home team wins about 2/3 of the time, I use the exact %)
- considers if game was played home, road, neutral, or semi-neutral court
- weights game outcomes based on how recent (so a game last week affects your ranking more than one a month ago, because things change over time)
- considers if a game went into OT (5 pt OT loss different than a 5 pt loss)
- considers opponents and opponents of opponents, & so on
So the outcome is, ignoring match-up issues, if 2 teams played on a neutral court, team X would win with probability Y, so if it was UK would beat UF with probability 0.7 then we should expect to win 70% of the time we play on a neutral court.

 

[JonathanW]

Thanks for the responses. It's actually more clear to me now. I just hate that their fans get to pretend they are Top 5 program when they haven't beaten anyone of consequence yet. If they win at NCState, I'll be impressed.

State isn't that good this year, and the arena isn't that intimidating (not like Reynolds was). I think State will be in that group finishing 7th-12th this year in the ACC.

 

[BBUK]

I delve into statistics a little but am not a statistician. (I do it for a different reason but not nearly as in depth.)

I will use the KISS method for those inclined for a more simple reason as to why the computer models seem so far off from a simple smell test;

Garbage in, garbage out...

 

[SilentsAreGolden]

State isn't that good this year, and the arena isn't that intimidating (not like Reynolds was). I think State will be in that group finishing 7th-12th this year in the ACC.

Yeah I know, and any team coached by Gottfried is already suspect, but it is on the road, and they should be good enough to make that a game, so I have to give UL some credit if they win there.

 

[mj2k10]

As a statistician, I have to disagree with much of the above saying computer rankings are BS.

I even created my own computer ranking system about 10-15 years ago, although I haven't ran it this year (new job = less free time). But the more information/data a system gets, the more reliable/accurate it is. It still can't account for match-up issues, so they aren't good to use in trying to compare how 2 teams will play vs each other.

Until December, computer rankings are useless. Largely based on the authors biases or pre-season assumptions. I would say until most teams get 10 games under their belt, I wouldn't even look at a computer ranking. And even then, they don't start getting accurate until after about 2 weeks (4 games) of conference play.

I agree that they aren't "BS", but wouldn't you agree that there are always ways to essentially game any system?

I specifically have a team like Wisconsin 11-12 in mind. That Wisconsin team finished 4th in the Big 10, lost in the 2nd round of the Big 10 tourney, and lost in the Sweet 16 to Syracuse (as a 4 seed) . They were 5-8 against ranked teams. Yet, even after the NCAA Tournament, they finished 7th in the Pomeroy ratings, at 26-10 (and I'm pretty sure they were higher than that prior to the tourney). If you look at their schedule, they had early wins by scores of 85-31, 68-41, 69-33, 77-31, 66-43, 70-42, 66-33, and 79-45 (all games against teams that were 150 or lower overall, and Wisconsin's non-conference SOS was 271). They kept their losses close for the most part, with 7 of them being within single digits.

In a case like that, it seems obvious that Pomeroy's system was overvaluing those early season massacres. Wisconsin built up such a huge rating's edge by beating the crap out of dogs that it became almost impossible for them to drop significantly, no matter how many times they lost (especially since the Big 10 was very highly-rated that year).

 

[nssdigitalchumps]

 

[JonathanW]

I agree that they aren't "BS", but wouldn't you agree that there are always ways to essentially game any system?

I specifically have a team like Wisconsin 11-12 in mind. That Wisconsin team finished 4th in the Big 10, lost in the 2nd round of the Big 10 tourney, and lost in the Sweet 16 to Syracuse (as a 4 seed) . They were 5-8 against ranked teams. Yet, even after the NCAA Tournament, they finished 7th in the Pomeroy ratings, at 26-10 (and I'm pretty sure they were higher than that prior to the tourney). If you look at their schedule, they had early wins by scores of 85-31, 68-41, 69-33, 77-31, 66-43, 70-42, 66-33, and 79-45 (all games against teams that were 150 or lower overall, and Wisconsin's non-conference SOS was 271). They kept their losses close for the most part, with 7 of them being within single digits.

In a case like that, it seems obvious that Pomeroy's system was overvaluing those early season massacres. Wisconsin built up such a huge rating's edge by beating the crap out of dogs that it became almost impossible for them to drop significantly, no matter how many times they lost (especially since the Big 10 was very highly-rated that year).

I wouldn't use the phrase "game the system", but certainly any predictive model (which is what a computer ranking is) is never 100% accurate. There are always assumptions made, some of which may not be completely true, for example how much should margin of victory weigh in, should a game played last week factor in more than one played 3 months ago and if so how much more.

SOS is a pet-peave of mine. It is generally calculated as the average Rank of a team's opponents. That is NOT a good way to determine SOS. In the simple case Team A could play teams ranked #10 & #290, and Team B play teams ranked #100 & #110. Most SOS calculations would give Team B the better SOS. But I think it should be relative to where your team ranks. If you are a top 20 team, then I think Team A has the tougher 2 games, but if your team ranks say #80 or higher then Team B may have the tougher 2 games. This is one reason (along w/ the SEC being weak) that UK's SOS is not better, despite us playing 3-4 top 10 teams nearly every non-conference year, those 5-6 games vs bottom 50-100 teams kill our SOS. But for a top 20 team, whether you're playing #200 or #300 shouldn't matter, they are both teams you should always beat.

 

[catntn]

They use IBM XT's for their analysis

 

[BigBlueFanGA]

Computer rankings aren't worth the paper they're written on.

 

[SilentsAreGolden]

I wouldn't use the phrase "game the system", but certainly any predictive model (which is what a computer ranking is) is never 100% accurate. There are always assumptions made, some of which may not be completely true, for example how much should margin of victory weigh in, should a game played last week factor in more than one played 3 months ago and if so how much more.

SOS is a pet-peave of mine. It is generally calculated as the average Rank of a team's opponents. That is NOT a good way to determine SOS. In the simple case Team A could play teams ranked #10 & #290, and Team B play teams ranked #100 & #110. Most SOS calculations would give Team B the better SOS. But I think it should be relative to where your team ranks. If you are a top 20 team, then I think Team A has the tougher 2 games, but if your team ranks say #80 or higher then Team B may have the tougher 2 games. This is one reason (along w/ the SEC being weak) that UK's SOS is not better, despite us playing 3-4 top 10 teams nearly every non-conference year, those 5-6 games vs bottom 50-100 teams kill our SOS. But for a top 20 team, whether you're playing #200 or #300 shouldn't matter, they are both teams you should always beat.

I agree. It's sort of a built-in defect isn't it? The model depends on data, which in the beginning is subjective, and that subjectivity affects future data which may or may not be accurate. I don't see how any model could be entirely objective because they all weigh results based on initially subjective rankings do they not?

 

[JonathanW]

I agree. It's sort of a built-in defect isn't it? The model depends on data, which in the beginning is subjective, and that subjectivity affects future data which may or may not be accurate. I don't see how any model could be entirely objective because they all weigh results based on initially subjective rankings do they not?

You could, and some may do this, run regression modeling (or some other type of modeling) on historical data, and base how they weight different factors on that. That is something I wanted to do with margin of victory, using about 10 years worth of data, but never got around to doing it.
But even those are averages. A good example is how recent a game was. Team A plays all 30 games with the same 10 guys, Team B loses 2 mid-season with a knee injury (or adds a transfer). Of course Team B should weight their more recent (post-injury/transfer) games more heavily than Team A. Models can't do that. And even if they could, how much would they make them differ?

 

[michnittlion]

As a statistician, I have to disagree with much of the above saying computer rankings are BS.

I even created my own computer ranking system about 10-15 years ago, although I haven't ran it this year (new job = less free time). But the more information/data a system gets, the more reliable/accurate it is. It still can't account for match-up issues, so they aren't good to use in trying to compare how 2 teams will play vs each other.

Until December, computer rankings are useless. Largely based on the authors biases or pre-season assumptions. I would say until most teams get 10 games under their belt, I wouldn't even look at a computer ranking. And even then, they don't start getting accurate until after about 2 weeks (4 games) of conference play.

But to the OPs original question, any legit ranking system will at a minimum look at not just how well you do, but how well your opponents do. So perhaps our 11 opponents thus far had a bad weekend compared to UL's prior opponents. Also was UL's game on the road or at home like ours (that should be a factor).

My system (I think) has 6 iterations, so it looks at opponents of opponents of opponents of opponents of opponents of opponents. The reason for that is to calculate a probability of winning (assuming no match-up issues) vs each of the 300+ D1 teams, and it takes 5-6 iterations after about 10 games to be able to link every team with every other team.
The factors my system uses include:
- scoring margin of each game
- takes home court into account (based on historical conference data, if I remember correctly the home team wins about 2/3 of the time, I use the exact %)
- considers if game was played home, road, neutral, or semi-neutral court
- weights game outcomes based on how recent (so a game last week affects your ranking more than one a month ago, because things change over time)
- considers if a game went into OT (5 pt OT loss different than a 5 pt loss)
- considers opponents and opponents of opponents, & so on
So the outcome is, ignoring match-up issues, if 2 teams played on a neutral court, team X would win with probability Y, so if it was UK would beat UF with probability 0.7 then we should expect to win 70% of the time we play on a neutral court.

This is a good post. I play around with a computer model of my own, and I go back-and-forth quite a bit on whether margin of victory should be included. As a positive: (a) it undeniably increases the predictive value of your model, and (b) it's a fairly straight-forward way of showing that a 3-point-win is most likely different than a 13-point-win.

But on the other hand: the ultimate objective of any game is simply to WIN. There are no "style points" given next door to a team's won-loss record. You either win or lose. "Style points" don't carry over in the NCAA tourney either, a 1-point-win is equivalent to a 50-point win.

The model I ultimately use now accounts for everything that you have, except margin of victory and the recency of the game. There are pros and cons to "recency of the game" as well, I guess I prefer to view a team's resume as a whole.

One extra thing I have: games among teams with high winning percentages get weighted significantly higher versus other games. If two teams are real good, does it really matter if one beat #216 (currently Austin Peay in my model), and the other beat dead-last #351 (currently Liberty in my model)? Big picture, they both beat bottom-half teams that they should have. What should really matters is the differentiating factors against GOOD teams.

FWIW, I currently have Kentucky #19, which at least passes a sniff test. The Ohio State loss was not a particularly great loss, and UCLA is bumbling around a bit themselves. Just went 0-2 on a Washington road trip, losing to teams that have home losses to Oakland & USC on their own resumes.

Top 10 is Oklahoma, MSU, Xavier, Virginia, Providence, Villanova, Iowa State, Arizona, SMU and Kansas. Louisville is #29. The strangest ranking of all right now is Chattanooga (?!?!?!?!?) at #16. Sometimes even the best algorithms throw strange things out. They did win at Dayton, that's a good win. But also a road-loss to Louisiana-Monroe, who is 4-7 vs. D-1 competition.

 

[marshalfan]

We did not beat UL by more than home court advantage. We lost to mediocre teams in OSU and UCLA, Louisville has not had any big wins but their 2 losses were to very good teams on the road and by less the home court advantage

 

[SilentsAreGolden]

We did not beat UL by more than home court advantage. We lost to mediocre teams in OSU and UCLA, Louisville has not had any big wins but their 2 losses were to very good teams on the road and by less the home court advantage

Yeah, everybody knows that. Other people already answered with specifics that make sense. UL not beating anyone but losing to the two good teams they've played does not warrant a #5 ranking though, no matter how well the system is explained.

 

[marshalfan]

Yeah, everybody knows that. Other people already answered with specifics that make sense. UL not beating anyone but losing to the two good teams they've played does not warrant a #5 ranking though, no matter how well the system is explained.

I don't disagree but by year's end the computer rankings other than RPI are usually pretty good. I did not read all the replies and certainly not a UL fan but I can certainly see how at this point they have higher computer ranking than us.

 

[UKWildcats#8]

I don't disagree but by year's end the computer rankings other than RPI are usually pretty good. I did not read all the replies and certainly not a UL fan but I can certainly see how at this point they have higher computer ranking than us.

THe computers basically reward beating bad teams, and having good losses over having 2 quality wins but losing to two bad/mediocre teams. That is pretty much I see it. No way the committee would agree with that if the NCAA Tourney was announced tonight so I'm fine with these meaningless computer rankings saying UL is better. No intelligent person would argue they would be seeded higher today.

 

[SilentsAreGolden]

And the BPI is really a joke. They have KU at 8 and UL at 3.