Why do I have to improve in order to rank down?

Possibly due to weight averaging. You’re probably right on the border.