If you click on a rider you should stay with that rider until they leave the game.

 When watching another person on Zwift, you will get swapped to another person if that person stops. An example would be watching on one PC and riding on another or setting up to record a ride on one computer while riding on another, It is particularly annoying when connecting to ios on an iphone and watching on a PC so you can have it on a bigger screen.