There are different privacy settings for your profile and for your activities. It sounds like you set the default activity privacy to followers, but I think you want to set your profile privacy to approve follow requests:
https://support.zwift.com/en_us/updating-your-privacy-settings-in-zwift-companion-rkegDuVB
There has also been a recent discussion about harassment and blocking riders which Zwift says they’re going to work on: