Inability to ride together

When I ride with a colleague who is the same weight and wattage and therefore W/KG and we do the same structured workout in erg mode, inevitably at some point in say an hour ride one of us (and it varies) zooms ahead in a very brief time not allowing us to continue to ride together.  If we are riding the same course, doing the same workout and are the same W/KG why does this happen and how do we keep it from happening again. It kind of kills the idea of working out together.  Thank you