How to handle the stress?

My friends and family are not accepting of me maintaining a relationship with my fiancé since his incarceration. I keep things like visiting him on the down low and omit that information from conversation. I do this so I'm not dug into emotionally and feel stressed by family and friends. It's stressful enough having my fiancée away but for family and friends to be so mean and cruel about it absolutely sucks. My choice of staying does not effect them in anyway other than their judgment of me. They have no clue how it feels and don't understand the depression and stress that is exacerbated by them isolating me and constantly putting me down for it. They say they want me to be happy. Why can't anyone see they are the biggest stresses in my life, not my fiancé?