Dilemma

Hello, my boyfriend's family lives in another state to me that requires a plane ride. I went to visit them once freshman year of college, but it was just for dinner and shopping, then I spent the rest of the time with other family. That was the only time I visited them, other times they have come to my state for a week to visit me and move my boyfriend back in after the summer.
The past three summers, they, my boyfriend especially, have been wanting me to go visit his home town, but this would be my first time travelling alone because I was with my sister the last time. My sister is my comfort person. Each of those summers I have been putting it off because I am deathly anxious of being driven around and sleeping over at someone else house. There town has no hotels, so I would be staying there, while also meeting the rest of the family and hanging out with his family more casually for the first time. I also get a little anxious when playing board games with people because I feel trapped, same with going to restaurants or eating together at a table without my comfort person/people. 
I feel like the tension between me and my boyfriend is growing the more I end up saying I dont think I would be able to go. I know the answer is to just get it over with and do it, but it is hard when I have built up and dreaded dealing with my anxiety on this trip for so long. There is nothing I can think of that would be fun about the trip because I have catastrophised it so much. I also feel like my boyfriend doesn’t totally understand my agoraphobia no matter how many times I explain it to him, we have been dating four years.
What should I do? Has anyone else dealt with this kind of situation before? I used to take Zoloft but couldn’t keep up with taking it, so now i am only on hydroxyzine.