Numerous factors already mentioned play a part in this. Wind being an important one, traffic (lights) being another important one, and your own technical skill/nerves especially downhill and in cornering being another one.
But let’s also not forget rolling resistance (by both tire as well as road surface), bike weight (especially on the uphill and for acceleration purposes) as well as climate. Zwift rolling resistance doesn’t change based on surface as far as I’ve noticed.
Real road feel might have come into play, however speed wise I’m not noticing an impact between driving on asphalt, wooden planks or cobblestones.
Bike weight… I’m not sure how and if Zwift factors this in. However, an important element of bike weight will most certainly be factored out in Zwift. Water and possible small repair tools and food as well. On longer trips I start off my rides with 1.5 L of water and several bars, spare tire and multitool. All in all, a quick rough 2kg of additional weight right there. The impact will not be astonishing, but also count for some.
The climate factor. Indoor with my fan(s) on I can control my own temperature. Outside it’s often either (too) cold or (too) warm for comfort. That does have its bearing on at least my performance.
Another factor ingame is the easy competitiveness. Many players are roughly going in the same direction. As such, when seeing someone going passed me, there’s this urge to keep up to them. Out on the road, most roads here are relatively short before the next turn arises. People often head out into directions of no interest to me. That little sense of competition is quickly lost there.
For best comparisons though, use a pedal based power meter both inside as well as outside. Disregard speeds entirely both indoors as well as outside. It’s a rather meaningless value for comparing performance over time. The only moment speed counts is during a race, compared to your direct competitor.