I feel like a bad person for being scared of the homeless men in my city

I won't discuss the traumatic events but men in general are a trigger for me. For a long time years ago I actually wasn't able to even be around men I didn't know without panicking. Nowadays, I am generally uncomfortable around all men. I'm hyper aware and vigilant even around my boss or friends of friends and friends of family. It's not so bad, a lot of the time I can chat and make eye contact after a couple of minutes even if the discomfort doesn't really go away. However, with men I don't know, I often avoid interaction as much as possible. The grocery store, the dentist, the doctor, the pharmacy. I avoid our paths crossing as much as possible. I just feel terrified. I really wish I didn't. There are great men out there. Men who wouldn't hurt or scare me. I love my male friends so much and I really like the men I work with. I feel terrible.

I feel especially bad about the homeless men in the city I work in. I feel so awful that I'm terrified of them, I feel so awful that it gets worse when they ask me for things and I say no and the panic gets worse. I want to be nice and sometimes I want to get them food, but the panic part of my brain tells me that if I give a man anything, even just kind words or acknowledgement, he will take it as an invitation to hurt me.

I love people, I want to help people and be nice. I used to be someone who, though I didn't have money, would chat with anyone.

One of them who I see a lot, I said good morning to today. I've been trying to say good morning to people more on the streets to try and get rid of the fear. He called me back and asked if I could get him food. I was immediately scared of prolonging the interaction, scared of having him walk close to me, scared that if I gave him something he would hurt me. I know it's irrational. I feel fucking horrible because I never assume the worst in everyone, especially not people I know are suffering and wanting and who have been abandoned by society, they are people still and I see them as such. But I'm fucking scared and I hate it so much.

I walk the long path to my job to avoid a very nice florist who is genuinely a sweetheart but I just can't interact with him much without discomfort. I cross the street usually to try and avoid one of the vendors. I felt so bad the other day because I went on a date and couldn't even look at the extremely friendly bartender. I feel so horrible for being so scared, I hate the idea that I come off as rude or mean or uncaring.

I hate the constant discomfort then immediate, consuming guilt of assuming the worst. But it's not even really assuming the worst. I'm not conciously choosing these things, I'm not thinking of how they might hurt me. It's just that primal, instinctual fear.

I really hope I don't hurt people's feelings. I'm really a nice, happy person... I just have RBF, am shy, and genuinely terrified a lot of the time. UGH

I'm just upset because I have to leave work soon and I'm dreading another interaction... I might just start walking all the way around downtown just to avoid everything but that would be a 20-25 minute walk which is nice but sometimes I run late or have to deal with traffic. :/