AITA for going on vacation with my ex?

We want to go camping with some friends from work. The thing is that I slept about 3 times with one of them. This happend last year when we were both drunk. My current boyfriend said it was some sort of shock when I told him I want to go on the trip. But after we talked about it he said I should go and have fun. But now I started thinking about it and reconsidering it cause it kinda sounds fucked up to go on vacation with an ex affair. Especially after he told me he would never go even with normal female friends.