Since there are so many riders now, how about a way to just show riders in your group. It wouldn’t remove other riders from that ride, but basically “block out” other riders other than who is riding in your group at that given time. Maybe like a exclude others from your group ride option. This way you wouldn’t have 100+ other riders on the sidebar, it would just show your group.
Zwift social graph is coming. You’ll be able to ‘follow’ other riders which will allow us to explore options for displays.