Freeze up / feel weak when people get angry at me, 41M struggling with this

I’m 41M and I keep running into this issue where if someone snaps at me or gets angry (coworkers, strangers, shop staff, etc.), I immediately feel small and “less of a man” inside.

Even if I stay calm on the outside, internally I get anxious and start thinking:

I look weak

I should’ve reacted better

What if it escalates and I can’t handle it

Afterwards, I often avoid those people/places and keep replaying the situation in my head.

It gets worse when I’m with my wife and daughter because I feel pressure to be “strong” and handle things properly.

Part of me wants to stay calm and composed, but another part of me feels like I need to prove I’m not weak by reacting strongly, which I know isn’t helpful.

Has anyone dealt with something similar? How do you stop taking these situations as a reflection of your self-worth?