So you've already given up on Lee improving at all? After 2 years and limited playing time, you think you've seen all that Lee has to offer? I disagree.
You specifically said Swanigan has higher upside, which I can't agree with at all. While Swanigan is a much more skilled player at this point than Lee, his upside is similar to that of a Jarnell Stokes - shorter, big-bodied post players with very good rebounding abilities. Lee, on the other hand, is a long, freakish athlete that lacks offensive basketball skills. If he improves simple things (free throws, post moves), he will be highly coveted by almost every NBA team.
Lee is similar to WCS in a lot of ways. Both grew up playing other sports and they both are freakish athletes. WCS has improved, especially in his Junior year, so to think Lee can't is kind of silly. Remember, Lee has been on stacked rosters where his role was probably to provide energy, rebounding and defense while in the game.