If trans woman should be allowed to compete against cis women in sports, why not just do away with gender segregated sports entirely?

The main reason sports is segregated by gender in the first place is because of the physical advantages men have over women. If a trans woman is free to compete against cis women, she will have a physical advantage over them even if she's taking hormone blockers.

If we're going to ignore the physical advantages for trans women, why not ignore all physical advantages between men and women and just have them compete against each other?