When to switch bathrooms

i've been on t for about 10 months now and it's getting to the point where i don't know what bathroom to use. i always use gender neutral bathrooms if available and i use the mens if it's a single person bathroom. if neither are available i use the women's because up until recently i did not pass at all but it's now getting to the point where i occasionally get weird looks when i have to use the women's. i don't know if im passing to the point where i wont get weird looks or possibly harmed if i use the mens.

tldr: what point did you start using the men's bathroom?