AITA For wanting to go to my brother’s graduation?

Hello, I created this throwaway account because I’m so conflicted. I have anxiety every morning just thinking about it. I need to know if acting irrationally so I can make things right.
I‘m a 24F live with my boyfriend 30M . We have been together for 4 years. although times are wonderful. We have things we disagree on. One thing is family. He’s okay and supportive when I go see my parents on occasion it’s always Saturdays. I’m off work and he’s not. I have had to miss my parents birthdays, brothers’ birthdays if they are not on Saturday. Why? Because he thinks our time is more important. I always felt so guilty that I ended up always being there for him because he says he struggles a lot with some fears and insecurities. I always thought it was such an emotional torture because I felt always guilty no matter what. Guilty for being there and not with my family or guilty for being with my family and not supporting him

My brother is about to graduate. Hes my best friend. We have always been so close. Although my relationship has put a little distance and I feel like is once again my fault for not balancing things correctly.
I want to go to his graduation I don’t want to miss it for anything. I’m telling myself I’m not missing for anything. i Wanted to go with my parents they plan to spend a whole week there. My bf planned to take time off 1 week after the week of my brother's graduation. I feel like I’m going crazy, he knew my brother‘s graduation was that date he knew for a long time. I have this gut feeling he did that on purpose to make me choose again. Without going into too much detail he says that a week is a long time and he wants to break up if I go. I do love him. I understand at some point I have to grow up and have a family of my own I don’t know if I’m being immature for thinking so much about my parents and brothers.

I don’t want to miss my brother’s graduation even if it means losing my bf. I feel like I have bent to his will too much I need this it’s important to me but this physiology torture is driving me insane I’m questioning if I’m being irrational?