Jack Lalanne said something like this. There's nothing wrong with being a health nut. If you're not neglecting your kids or doing pullups off a hobo's beard you're alright.
People call attention to you because refusing food is not culturally appropriate. Exercising every day is an addiction. They are frustrated for some reason.
It doesn't mean you're better than them. You're just respecting your health.
Jack lived to his nineties and worked out for 2 hours the day before he died. What it's really about tho is a whole life of the freedom to do what you want to do.