How do I get over my fear of women?

Maybe fear is a bit too far. I can interact and build basic relations with women as acquaintances but I'm always to hesitant to push forward as friends, let alone anything more than that.

I'm not particularly antisocial, I've come along way since I was younger, but women is still the one issue I have trouble with.

I'm certain my problem stems from the lack of seeing any positive relationships growing up, my dad and mom divorced multiple times over my life. (mom had remarried and divorced before she died from a drug OD; my mom was my dad's second marriage. He had then remarried twice, one with a terrible witch of a woman who tried to coddle me and abused my brother, and the other who is one of the best people I know) My brother also has had nothing but bad relations and nowadays, I don't get along with him.

The only examples of good relationships between both genders I've had have been my grandparents and my dad's current wife.

I'm still young, just breaking into the early twenties, and I've started tackling the issues that have plagued me as a child, but this is one issue I'm not sure how to handle.