I dont know how to deal with my friend anymore

The start of this goes all the way back to high school. Me and my own family had somewhat of a troubled relationship. So I would spend a lot of time at my best friend's house, we'll call him Ben. Ben's family were some of the best people I had ever met, they just completely welcomed me with open arms. It became like a house for troubled children. At one point I was basically living there. They had a kid with autism who got kicked out of his own house who also went to our high school, and then another younger girl that came from a very tough home. They never once asked me to leave, they included me in all their family activities, I just became incredibly close with this family and them with me. They always referred to me as their bonus child. Life was great. Unfortunately though Ben got into this really really toxic relationship that lasted many years and basically pissed off everyone that was close to him. He just kind of shut everyone out to be with this terrible person and after they broke up no one was nearly as close with them as they were before. Around this time Ben also really leaned into this kinda "troubled artist" persona. He'd paint these really weird kinda messed up paintings, talk about philosophy all the time and it really became his entire personality. It might also be important to note that he is incredibly lazy, gets fired from every job he's worked on, gets out of bed usually around 1 in the afternoon. Suffice to say he was really not in a good head space but at the same time he was treating everyone close to him like shit. So Ben has recently decided that the best thing he can do is to just “isolate” from everyone so he can focus on himself. Which personally I think is a terrible idea, he just gets into his own head and I think it makes him spiral more. So he asked me not to come over as much, which I respected. I went from being at his house every other day to not going over for nearly four months. Ben never reached out to hangout like he said he would and never responded to any of my texts. So I went over there one time yesterday and spent most of the time hanging out with his dad while Ben was upstairs. But when I leave he sends me a text telling me that he's still “isolating” and doesn't want me to come over unannounced. Now at this point Ben and I’s friendship has been on the rocks for years but I really want to see his family. Ben's mom was trying not to cry asking me why she hadn't seen me in so long. I really really miss them and just being over there, and they miss me too. Part of me just wants to tell Ben to just fuck off, and part of me thinks that I should respect his wishes and just keep my distance just because its his biological family and this is what he thinks is best for his mental health. What do you guys think? Should I respect his wishes or is he being selfish?