Am I rushing through the story too fast?

I've had the game for a couple of days and already rebuilt the Pokemon center and opened the gate.

I haven't really built all these elaborate houses and stuff like I see a lot of people on this subreddit. So idk if I am just rushing the story too much? I guess I am worried that by doing only the mandatory missions I might end up finishing the main storyline too quickly and just be left with the building sandbox elements earlier than if I took a different approach.