How to Get Better At Rejecting/Excluding People Who Are Bad For Me?

So, I know this sounds bad but hear me out. Growing up neurodivergent, I know how painful peer rejection can feel. Even just recently I experienced it and it hit like a truck. At the same time though, I understand that some people are just bad actors, and I've had a really bad habit of hanging onto peer relationships which were bad for me in the long run where I had been taken advantage of simply because I didn't want to hurt their feelings.

I've been playing DND with a small group of friends, and my buddy wanted to invite an old Elementary/high school acquaintance who I've always found really annoying. I've never been impolite to this person but I've also never really been friendly. I heard from another friend that he drunk the Tate brothers' koolaid, so there's that.

Am I a hypocrite for this? Like I said, I know from firsthand experience how much rejection can hurt, but I'm also at a point in life where I don't really want to put up with people I only tolerate.