AITAH for not showing our house

We bought a house last year which has been undergoing a full renovation and is currently empty. The builder we hired to do the works did the renovation at my in laws home as well as my husbands uncles house, so came well recommended. And because he has done work at a lot of the family homes, he has become friends with my FIL (father in law) and his wife is now friends with my MIL (mother in law). Despite this however, we only have interactions with him as our builder and don't know him well personally.

A couple weeks ago our neighbour messaged my husband telling him that he saw the builder along with 3-4 other men coming out the side passageway of our house. He recognised one of the men was a neighbour a few houses down so asked him what they are doing at our house? He responded that his brother bought a house nearby and is looking for a builder so our builder is showing him the work he has done on our house. Our neighbour asked whether we, the owners were informed and they had our permission to which they responded yes, they are here with the builder. My husband and I were shocked as we were not aware of this and would not have been okay with this even if we were as the house has already been broken into twice and I am already a little worried that I will be at the house alone most of the day with our young baby while my husband is at work.

So yesterday I called the builder to ask him when he is coming to the house to finish the little outstanding works and he said next week. He then asked if we are at the house at the moment and I told him yes, thinking he was probably going to say he will come over now and I can point out what jobs are outstanding. Instead he said 'I have a couple customers on the way to your house now and I will be there soon too, to show them your kitchen as they want to see my work'. I was actually shocked and decided now was the right time to bring up that we are not happy he had strangers in our house without our permission before and was now going to do the same thing again without asking us. I told him we were happy for him to take pictures and videos but not for strangers to be in our home. He responded 'well they're in their way now and they don't want to see pictures and videos as they can be deceiving so they want to see the kitchen in person. Do I tell them to turn back or not?' I don't know why I didn't stand my ground and say no, but my husband and I reluctantly agreed as we were under the impression it was just the kitchen.

The builder and these strangers arrived at the house and my husband took them straight to the kitchen. They started treating the house like it was a show room and opening drawers and cupboards. The builder then said they want to see the rooms upstairs and the bathrooms upstairs and I said there's a toilet downstairs they can see but I am not comfortable with them going upstairs. My husband said he would send them pictures and videos instead to which they agreed. Once they left, the builder was fuming with us saying we were rude to them and they came to see the house etc. I explained to him that he only told me they wanted to see the kitchen but regardless, we are not happy with the fact that he is inviting strangers without our permission into our home. I kindly explained that the house had been broken into before and I was worried about my safety so I don't want strangers in the house. Instead of understanding, he said he knows these people as he has done work for their family members etc and that he doesn't associate with bad people. I said that was fine but we don't personally know these people so I don't want them in my house. He returned the house keys to us and I said what about the outstanding works? He said he will liaise with my FIL and come to finish the work when he is able to let him into the house (my husband isn't able to as he has to go to work and I can't drive to the house as we share a car). I could tell he was angry and offended so I apologised if I said anything wrong but all we wanted was for him to ask for permission and I explained again about worrying for my safety. He was not willing to understand, saying we gave permission for these people to come and see the house but then didn't let them go upstairs. I clarified that no, you said kitchen only but I am not okay with having them go upstairs to our rooms etc. he was like so what if I said kitchen and now they want to see the whole house? They were already here so you should have let them. We were at an impasse so I just left it.

When we got home, I explained what happened to my MIL. She saw a missed call from the builder on her phone and called him back. I couldn't hear what was said but could just hear my MIL agreeing with him and not once saying anything to make him understand mine and my husbands point of view about safety etc. when she got off the phone, she said he is now saying he will not be coming back to the house to finish the work as I was rude to him and that he didn't think he needed permission from us because he thinks of us as family and his own children so thought it was okay to bring his customers in to see his work, especially because the house is empty. My point is that if the house wasn't empty and we were living there, he would ask for our permission so why is it any different just because we aren't living there yet? And also that if these people aren't strangers to him and he has done work for their family members then why isn't he showing their houses instead of ours? Also, while my MIL and FIL have a personal relationship with him, me and my husband only interact with him as a builder so have never given him the impression that we are like his children and he can treat the house as if it's his own home. But Now my MIL is pissed at me too because I've apparently ruined their relationship. So AITAH for not wanting strangers in our house without our permission?