The apple remote is very finicky with the Zwift app. Sometimes it takes several minutes of flicking around on the remote just to select a menu item. I’ve tried the touch of remote and the regular remote and both have issues.
This is a well-known ‘issue’. You should search the forum and vote up some of the long-standing threads about the Apple remote.