Constantly being called a man, I’m a girl

There’s this new guy in my school and he has a crush on me, and I have a crush on him. He says that everyone told him that I’m a man and I’ve overheard the comments myself as well. Some say that I look like a man, which is one thing. But others say that I am a man and that is different because it’s definitely untrue, I have never been a part of the LGBTQ+ community (I support, no hate) so there’s no past of being a man, and I am very much a woman. They all know that as I swim with some of them and it’s obvious if someone has a penis or not.

He says that he trusts me but he doesn’t know who to believe, and I just don’t know what to do because it keeps happening, my school know about the incident but it keeps happening. Not only does he think I’m a freak now, but it doesn’t make me feel great either. Any help would be appreciated.