Your spouses play partner is unbearable

Ok, what happens when your spouses play partner is just the most annoying person and being around them just make you want to scream? If so do you let it affect the 4 way dynamic? The scenario is that a few months ago we played with a couple for the first time and my wife really did not like my play partner and got really shitty with her during play time, to the point where that couple did not want to play with us again. I was kind of pissed because my play partner was fun and we were having a good time, but apparently she rubbed my wife the wrong way.

Fast forward to present day and now my wife has a play partner that she really likes playing with but he is a narcissistic asshole and I can’t stand him (he has also done some pretty shitty stuff by breaking some boundaries between himself and his wife). The wife (my play partner) is good and a lot of fun but her husband negates any urge I have to play with her. We are supposed to meetup with them again in a month or so and I am just cringing at the thought of it.

My wife had no issues with putting a stop to it when she felt the way I do, so I feel that I should have the same level of power to end it with this couple. My concern is that my wife will not be happy? I would like to think that I can just go along with the situation as it is but I can’t stop feeling like I will be pissed the entire play session and it will not be enjoyable for me. Am I being selfish or justified in calling it off?