Another ignorant question: Trans, gender-neutral bathrooms, and safety

So, I'm wondering -- From what I can tell, radical feminism is happy with gender neutral bathrooms, but they also want female sexed bathroom, and a male sexed bathroom for issues of safety.

Considering how prevalent sexual assault against woman is, why is it controversial to desire a female-sexed bathroom, if something like a gender neutral bathroom is offered for those who desire it?

In this question I am not trying to equate trans with assailant,

but as it's own point, recognizing that a fair amount of females might feel unsafe with the loss of a private space - having been assaulted by a person with a penis in their personal history.

• also taking into account, many crossdressers are in fact heterosexual self-identified men/male

so to breakdown/reiterate: Trans people deserve to feel/be safe. Female sexed people deserve to feel/be safe. Male sexed people deserve to feel/be safe

Perpetrators are always looking for cracks/loophopes to take advantage, so questioning if the fear can be discussed of men who may take advantage of this to find a new vantage point of assaulting women, and the fear of loss for recourse for this?