This is a long-standing issue with Zwift and Elite trainers (Direto and Drivo). Search through the forums for “Direto ERG” or “Drivo ERG” or “Elite ERG” and you will see people complaining for a long time (eg: ERG Workout Delay )
There are also many people complaining on the Elite forums, stating this is just a Zwift thing, as it work better on other platforms