Is there a reason that so many people use women’s rights to justify transphobia?

When I see transphobia the most common thing I see used to justify it is feminism. Like, things ranging from women having their own spaces to the existence of trans women making the term female meaningless and somehow making it impossible for cis women to discuss their issues and experiences. Why is this such a common thing? Like I don’t get it other than to lend validity to the idea that what they are doing is actually to benefit women rather than to detriment trans people. But it all falls apart under the tiniest bit of scrutiny. Like… how can people take transphobia seriously, especially if they are a feminist? Is there a specific thought process? Does the feminism come first and the way they see it, it should be trans exclusionary or is it they are transphobic and try to justify it with their feminism?

Also, I noticed that almost the entirety of transphobia is centred on trans women. Is this related to patriarchy or misogyny at all? Or is it just the fact that acknowledging trans men results in their arguments falling apart?