AIO for being upset that my bf says we were never together?

Burner account, for privacy reasons. I apologize for how long this post is. This is partly a rant. I apologize.

I (24F) feel like I'm losing my mind and I genuinely don't know if I'm the one who completely misunderstood this relationship. I met my bf (25M) about a year ago after we finished college. We weren't really the type of couple who immediately put labels on everything, but we spent almost every. single. day. together. He basically LIVED at my place. We cooked together, went out together, met each other's friends, and everyone around us considered us a couple. This is random, but he even helped supervise the contractors at my parents' home one time.

He's always been a VERY reserved person and is not someone who talks about his feelings easily, but was different with me. He opened up so much to be about things he doesn't normally share. He would say things like I was the only person who really understood him. He wasn't OVERLY romantic, but when he showed affected it felt very real.

A few months ago, we slept together for the first time on a camping trip... won't get into all the details here, but it rained unexpectedly so we had to stay indoors most of the trip. After that, I honestly thought we were officially together, especially since sleeping with a guy is a very big step for me. Maybe we never had the big conversation where we said we were bf and gf (my mistake, I know now for the future...) but everything about our lives looked like a relationship and we acted like a couple.

Then last week, I woke up in the middle of the night and noticed he wasn't in bed. I found him OUTSIDE LOADING HIS CAR. I asked him what he was doing, and he just TOLD ME HE WAS LEAVING. Apparently his mom has been pressuring him to move back home and help with some stuff. He told me he had to go and didn't really have a choice. I understand family obligations, but it was heartbreaking to see him leaving in the middle of the night without telling me -- it's hard to even type this out.

When I confronted him, he admitted he knew it would hurt me and that he didn't know how to say goodbye. I asked him how he could just leave after everything and if our relationship even meant anything to him. This is where I got completely blindsided. He told me that he cares about me, but that we were never actually in a relationship! He said we never agreed to be bf / gf. His exact words: "We were never actually together." I kind of went crazy after this, I have to admit, since I didn't understand how he thought he could just leave like this, how he could pack his car in the middle of the night and expect me to not feel betrayed. He just told me it was his family again and told me, "I'm going home not of my own free will." Honestly don't even know what to say. He's also just completely shut down, he's calm, as if he's already moved on. I feel like I'm the only one grieving here.

I want to ask my sister to reach out to him and talk some sense into him. She's close with both of us. AIO for asking my sister to contact him and try to get him to come around?