First, the public wanted this memorial to be moved because it celebrated Lee's role with the Confederacy, an army formed primarily for the continuation and legitimization of slavery.
Second, Lee could have not taken up arms at all, that seems a pretty easy way to not fight your fellow Virginians. Instead, he fought, and led, an army dedicated to the preservation of slavery.
Third, Lee himself opposed Confederate monuments stating that such memorabilia would continue to divide the nation. If you want to honor that part of his legacy, it seems you would support the removal of his statue donned in Confederate uniform.
But this argument is less about the statue, more about the fact that white nationalists and Nazis are so incensed on keeping memorials honoring segregation, racism, slavery, and oppression of minorities. The 'changing of history' argument is a hollow one, as removing these memorials is finally confronting history.