I would say it is possible. I am old enough to remember when boxing and horse racing were huge national sports. Even Major League Baseball has dropped off in popularity to some degree mainly because of competitive imbalance.Your scenario is possible, but I'm not convinced it will happen. The passionate fanbases in college football are unmatched in the world of sports, except for maybe some professional soccer teams in Europe and South America. I agree it is likely that some more casual fans may check out, kind of like the few thousand NFL "fans" that quit watching because of kneeling, but I would not bet money on a catastrophic loss of fans.
Once it becomes obvious that the teams ponying up massive sums of money year in and year out are going to constantly win, and win easily, that's gonna put those passionate fan bases to the ultimate test