Looking for advice -- do you bully the bully?

I've been experiencing being the target of "transinvestigation" in my neighborhood/city -- people spreading rumors that I am trans even though I am cis. I even heard from discussions around me that there was a website about me.

I know it's malicious because I know they know that I am cis, but they are using "Trans" as an insult. And the worst part for me is how few people have stood up against it, or called it out or even told me about it, they just let it fester.

The way it happens is that people say it as I pass by them in the street, saying that I am trans, or that I am a man. I am not sure why or when it started, but in the past few years, I've lost weight, became more slim, and I've become more vocal against white supremacy/racism, so it could be any of those things. But at the beginning, I noticed people taking photos of me in public.

I've noticed that the rumor is being spread mainly by women, and I've started calling it out when I hear it. However, most of these people deny it when I directly confront them. So I've started saying things around them like, "It's always the most busted people who spread rumors" or "Get a mirror before you speak about other people's looks."

What do you all think of this? Do you think it's fair to bully them back? Or should I report it as harassment/hate speech? I feel like I have to wait around and record them to get evidence. It's getting to the point where I am thinking of hiring a private investigator.