Sleeping together too soon?

I (25F) have been talking to a guy (25M) for the past few months. He lives a 5 hour drive away. We texted / called for about a month before meeting the first time, where we hung out for a whole weekend. Since then, we’ve been calling on a nearly daily cadence. I met him briefly for coffee 2 weeks after when he happened to be flying through my town. A week after that, we met up for about a week when I had some time off, where things escalated and we ended up sleeping together.

i’m having mixed feelings about sleeping together. It was the third time we met, about two months since we started talking but really 3 weeks since we started talking seriously. We both had a good time, but I am worried about the sex causing intimacy to artificially inflate, without allowing the emotional intimacy to first strongly develop. I know I personally have a bad history of letting sexual compatibility or chemistry cloud other relationship compatibility which matter longer term.

I’m considering discussing holding off on sex so we can focus on building the relationship first. Is this a weird reason to hold off? Or is it better to just be mindful of compatibility, regardless of the sex ?