It would be great if you can do a workout with just one or two friends the same features as group workouts just with who you want to so you don’t drop them if your stronger.
I think there are already a few request for this, if they match what you are suggesting make sure you vote them up: https://support.zwift.com/hc/en-us/search?utf8=%E2%9C%93&query=private+workouts&commit=Search
https://support.zwift.com/hc/en-us/search?utf8=%E2%9C%93&query=private+group&commit=Search