Changing social plans

I'm curious if this is an introvert thing, an infj thing, or just a me thing.

I get really disappointed and sometimes stressed out when friends I have plans with randomly invite others to join our plans.

I think it's partially because it changes the dynamic that I was looking forward to and prepared for , and partially because for me when I have plans with people it's meaningful to me, I'm looking forward to getting to know them more and it means a lot that they want to spend time with me. I have a lower social battery, so agreeing to plans means it's something I think is worth spending social energy on, and I also appreciate when people choose to spend their social energy on me.

So when friends randomly tack on other people to the plans, now I feel like that makes our hanging out less meaningful, like it's no big deal who comes along cuz they are more focused on what activity we are doing not who is joining..

Example : I hadn't seen a friend in a long time and was going to be in her town, so we made plans to get dinner. I was looking forward to catching up. Then she messages me last minute saying another mutual friend wanted to join.. this mutual friend I don't know as well, and would thus limit the time and deep convos I could have with the good friend​. It also turned out the additional guest was much more extroverted than the friend I originally had plans with they tended to take over the conversation etc. So I didn't get to hear as much from the more introverted friend.

I feel like this kind of thing happens to me all the time, but others feel like it's no big deal .. maybe I'm just too sensitive and need to realize for most people social stuff just isn't that deep :)