I have written a sauce 4 zwift mod that can give you a bulk unfollow of users that are not following you. It might be possible to modify it to figure out those that have not given you a ride on in the last X days, but it would be clunky. The information is only available from the activities on the web site.
I was looking to clean up those two lists since there are plenty of people there whom I have no idea anymore who they are of have stopped Zwifing long ago.
Now I went looking but it seems there no such list on the website on your profile but it seems to be only done from the Zwift Companion App. But it goes one by one by one.
And every one I want to remove I have to :
Click profile
Click Followers
Click a name
Click the 3 dots
Click remove follower
Click remove follower
Press the back arrow which brings me back to the main screen so I have to do this all over again for the next.