It is mysterious. In my mind (which is also hard to fathom at times) I’ve explained it away by deciding every section of route in Zwift has a direction attribute. If you’re going one way it’s Up. The other way is Down. Not tied to elevation or gradient, btw. Fun fact: this is how direction works on railways (in the UK at least)
So, if you got a jersey on a sprint segment that was in the Up direction and you’re then pedalling in the Down direction on a different segment, you’ll stop wearing the jersey until you change to a different segment where you’re going in the Up direction.
Thank you for coming to my TED talk.
Fun fact: I have absolutely no idea if Zwift actually works this way. 