Explain the conductor work for a noob and tell me what I'm missing

I've been watching symphonies/orchestras for a decade or so and still it does make me confused. I have no musical training whatsoever. I just like classical music a lot

I do know that their work is PARAMOUNT for the musicians (I'm not questioning that) - and I notice the musicians while playing their instruments always keeping an eye on the conductor.

But why what I see that is been played seems to me that has no relation to what I hear?

The last ones that I really tried to put an effort in observing the conductor's gestures was Dudamel conducting Dvorak's new World and Celibidache conducting Bruckner's 7th.

What am I missing??