Barnhardt's not alone in that area, by a long shot. Take a look at the revenue and expenses of just about every other P5 athletic department in the country, and you will see those programs doing exactly the same thing.
http://ope.ed.gov/athletics/#/
The UK athletic department has more than enough money to field a championship level football team. The plain fact of the matter is that for too long, UK fans only demanded a winner in basketball. It's the reason why Adolph Rupp stayed while Bear Bryant left.....
Just my opinion (as always), but I think the UK-UofL football series helps to "set the bar" for the minimum level of football success that UK fans will tolerate. For far too long, UK football fans were willing to accept a lot of losses in the SEC, because it is year-in and year-out the best college football conference in America. UK fans won't accept losing to UofL, however (regardless of the sport). If the UK coach and AD aren't doing everything they can to beat UofL on the football field, then UK fans will demand improvement - that's the bottom line.
Barnhardt had better get a head coach who can win that game more often than not - or he's going to continue to get a ton of criticism.