How to make gravel go faster?

I have a stock Specialized Diverge E5, which is sort of slow on roads, I get passed by basically everyone - MTB, city bikes and moms. At this point I don't understand if problem is the bike or my physique.
I'm not really experienced in bike equipment, so any help would be appreciated. I'm thinking maybe it is enough to change some of the parts instead of getting a road bike.
Would anyone have any recommendations how to make this bike go quicker? I understand changing to narrower tyres (700x30? 700x32? 700x28? 🤦‍♂️) might help a bit, but maybe anything else? Or isn't it worth it and a road bike is the key?