Should I obey my Pastor?

My family is having its 3rd kid in January. We are going from condo to house. The house that’s our top choice is right by my Pastor. He’s cool with us, I think, but he told my wife when she asked that the house is too close. We have a lot of people over from church and he doesn’t want all that. He thinks it’s inappropriate and wants us to live further. I said I don’t think he has Biblical right to tell us where to live and he can move if he’s unhappy with it. I wanna honor my Pastor, but I don’t care that much if he doesn’t like where I live. Thoughts?