I have a similar problem!
I do not think it needs to be in two different worlds. When I triggered this the riders I was looking for and me were both in Watopia.
Twice today when I fan-viewed a friend, once about 30 kilometers into a ride with Coco and another about 1 kilometer in (also with Coco), most riders (but not all) disappeared. Random riders would reappear and disappear again and again.
When I do not ‘fan view’ anyone I can ride for hours without the issue.
This also happened on Monday when I got away from Coco and tried to ‘fan-view’ to see where she is. Most riders, but not all, disappeared from view and would randomly blink in an out of existence after that.
Before this it happened on two consecutive Thursdays when I tried to find where a group of friends was on Tempus Fugit so I can make a u-turn at the right time to join them.
I am not sure if doing a fan view triggers this 100 percent of the time, but I rarely, if ever, have network issues when I ride Zwift in the last 3-plus years.
Can someone else chime in and see if this happens to them too?