I oppose erasing history. "Those who fail to learn from history are doomed to repeat it" - George Santayana
If you rename something you erase history, partially or completely.
The pavilion was put there as a memorial and it's name is Confederate Civil War Pavilion and not The One Human Family Pavilion.
Everybody has a different look on history. History can be good and it can be bad but it tells what happened whether we like it or not.
I know you say it's just a name and it's not the same as removing a structure. If it's just a name then maybe we shouldn't change it. And if it's just a name then you should be ok with giving a new name to The City Hall naming it "Enemies Of The People Hall", because I'm not happy with the decisions you make by controlling people, ruining businesses and peoples lives