Need advice on my work situation and my boyfriend for a leisure trip. Need help.

I am 25 F and in a relationship with 25M for 1.5 years. My work friends are going on a trip and one of the boys has spoken ill about me in the past behind my back. I know about it but he doesn’t know that i know. My boyfriend hates that guy. Doesn’t want me to work with him. But there is this trip going (not a work trip) which one of my female work friend organised. Initially, it was supposed to be a girl trip only but now the boys have joined in too. My boyfriend wants me to cancel the trip. I don’t know what to do. I really want to travel cause i hardly go anywhere and i am getting at-least some company. I also don’t want the fights after the trip if i happen to go. Even if i don’t go , i will not be completely happy with my decision and my boyfriend.