Is it worth buying a bigger home?

We live in a home we adore. But with two kids and one on the way, it’s feeling snug. We also have an au pair that lives downstairs. It’s a 4 bed (one in basement), 4 bath, 3,500 sq ft. We could make it work. We have a good income where we’d likely be able to pay this home off in 10 years. Or we could afford a big, nice, luxurious home. But, should we? I kind of like living in a mid sized home in a regular neighborhood. Just wondering as it’s a big topic in our lives currently.