Did anyone have difficulty accepting their gender because of past experiences?

I first transitioned to male and considered myself exclusively a man for quite a few years, and only in the past few years I've realized there's not just a man part of myself but also a woman part of myself. I'd much rather not be limited to one or the other and be able to live as both.

It's been really complicated because the "woman" thing was forced on me for so long. Girls and women treated me with hostility and cruelty back when they viewed me as "one of them" and became much friendlier once they started perceiving me as a cis man. When I think back on it, I remember the way girls would criticize my body out of nowhere as if I was repulsive (although in my opinion, I just looked average and normal). And the women in my family clearly thought I was too weird and unfeminine, like an alien more than a girl or boy, and they acted inexplicably weird about my body too, just like the high school girls. The entire "girl" experience felt like a punishment I was being subjected to for not abiding by the rules of a secret unwritten manual no one would explain to me.

Now that no one is forcing me into that anymore and I've spent a significant portion of my life in both roles, I feel like I can engage with it on my own terms without it feeling like a punishment, but I've had a ton of trepidation and fear surrounding it because of the "bad taste" it left in my mouth from the earlier part of my life. Did any of y'all deal with similar situations, and how did you handle it?