how do I stop getting annoyed when something I like gets popular?

okay this is kind of self explanatory, but often times I like a certain show or hobby or just interest that I hear relatively few people talking about, and then a couple months later it becomes really popular and mainstream.

I know im no better than anyone and that we're all allowed to like the same things (and that this very phenomenon happens to many people), but i can't help getting annoyed because it feels like a part of what I associated with my personal identity, is now being coopted by very loud people who will often claim to have some superior knowledge or authority on the topic.

since I know I can't control what becomes popular, and it's a futile game to try seek nicher and nicher media, how do I avoid feeling so misanthropic over my interests.