SMs with good relationships with their SKs- advice please

I’ve posted here several time but for context - been an SM for 5 years to 8 year old SS. Great kid, very cordial coparenting structure (each house does it‘s own thing really without bothering the other) and BM is a good mom.

Initially had a good relationship with SS for the first 2 years but now it’s respectful and decent at best. I have reduced a lot of my involvement (went in too hard initially lol) and now enjoying my OD 8 months and the life DH and I are building. The relationship with SS shifted a lot because only up until a few months ago, SS struggled a lot with being in our home, said he hated me and DH etc. He seems fine now.

Thing is, I believe that SS and I could potentially have a better relationship however my issue is that I feel emotionally unsafe around him as an indivisual. I don’t dislike him or anything like that. I recently learnt that as a person I really value privacy and trust in my own relationships and the fact that we are in a structure where he has his own min and world outside of our home makes him feel unsafe to me. I know we can’t control anyone so I don’t want to stop him from sharing freely in trying to reconcile his world. I get that and have much compassion for him but it doesn’t change the fact that I feel uncomfortable around him and feel like I can’t be myself because that could be shared outside of my home with people I have no business with. Does this make any sense? Ultimately I’m asking how you overcame the feeling of being potentially discussed in BM’s world and your SK probably contributing to those conversations/narratives. I’m stuck because in a normal situation, I would never try to connect with someone I feel emotionally unsafe around.