I hear you loud and clear. I often use my iPhone to do some misc tasks like changing bikes, kit, etc… While I do not have all of the problems many here complain of, I definitely agree that the remote could use some love from the coding team. While it works for me, it could def be better in so many ways.
If I remember correctly, the Apple TV interface is due for a refresh in January. Hopefully, that includes a better remote experience as well.