Often in leader led group rides leader references to the elapsed time. If you join late or you get drop-out and rejoin you elapsed time is not same as for the leader.
Create a Zwift time zone, like “Zwift Standard Time” (ZST) and have a clock with this time added in the game. Same time for all riders independent of location or elapsed time in group ride.