People have asked Zwift to implement some basic follower management features. They’ve basically not responded or found some excuse for a particular feature can’t be implemented (e.g. allowing you to sort your follower list by gender, age, location, etc will violate their right to privacy).
I am not exactly pleased. Admittedly there isn’t necessarily a huge point to having followers. I just maintain a follower list to give out ride ons, really. I bet some of them probably do the same for me. It’s cool. But otherwise, maybe the unofficial solution is just ignore your list, seeing as they aren’t able/willing to help you do anything to manage it.