Holding same watts significantly harder with larger gear

I can hold 180 watts easy if it’s in workout mode with an easier gear, but once I’m in free ride or a race where I have to use a larger gear for the same watts and cadence it becomes noticeably more challenging.
However, going up a hill using a smaller gear creates the same effect as erg mode, and using a larger gear in erg mode increases the difficulty (same watts).
I have no clue what’s going on and would really like to fix this since it really effects my workouts and races.

I think it’s just biomechanics. Many (most?) of us find it harder to hold a given power on the flat compared to going up hill. e.g. I find 250W up hill “easier” than pushing 250W on the flat.

Erg mode is different again because your trainer will hold a given power regardless of your cadence or gearing, so you can do whatever’s easiest for you.

That wouldn’t really explain the difference between the difference on a flat between a Erg mode and races/free ride

It does, to me at least. Erg mode is easier on the flat because it prescribes the power. Free ride doesn’t and you’ve less to ‘push against’.