AITA for not meeting BF's friend he once slept with?

I (26F) have been dating my BF (25M) for a few months now, we have known each other peripherally for quite a few years before that. I am generally happy in the relationship but have noticed that he has a bit of a past with other girls that keeps coming up. There one particular that keeps getting brought to my attention recently.

My BF lives about two hours away (plans to move closer in about a month) and has a social circle in that area that I haven't really interacted with yet, I have met one or two friends so far. He has mentioned there is one friend of his that in the past they would be "cuddle buddies" or sorts. I later learned from him this was the implication that not only would they cuddle/make out but they did sleep together (I believe once or twice). He followed it up by saying it wasn't good and he isn't interested in her romantically. I also learned she is a stripper/dancer and is very free with her relationships.

I understand people have pasts, we all do. I also respect friendships between different genders, I have friends who don't all identify as women and I would be hard-pressed for a partner to tell me that I can't have them in my life. That said, I haven't slept with those friends, so it feels a bit different. While I am not thrilled about them still hanging out and him occasionally giving her rides to go deliver letters/etc., not only does it not feel right to manage them but I do not fear him cheating on me so that isn't the major issue (he was cheated on in the past and it made him staunchly against cheating. I realize that you can't always know but I can't imagine him doing it and I trust him).

Luckily, they live in another city so I haven't had to be involved very much. Until recently. She has invited both of us to a birthday camping trip that is happening at the end of the summer which he brought up to me a few weeks back. It is planned to happen a few months after he will be moved to my area and supposedly not as enmeshed with those friends. I made the comment "oh nice," to try and avoid and move on from it and hope nothing would come of it (I realize this stems from bad habit of avoidance and passive/aggression, not proud). I had hoped that would be it but he brought it up again yesterday the exact date and time of it to see about planning. I responded that I would be busy and can't go, but that he is welcome to go on the trip. He seemed bummed about it but didn't push me on the topic or discuss any more. Incidentally I happen to have made tentative plans to see an artist with friends that weekend; even if I hadn't, I still would have said I was busy. I don't want to be around her because I know I will feel weird and not be comfortable, which sucks because I am sure she is cool. I don't love him going but I'm not his manager.

AITA? I keep wondering if I should bring it up to discuss with him but don't want to overstep and i am not sure if I am being overly sensitive. Helpful advice welcome, sorry for long story.