Why am I never my cat’s “person”?

I adopted two cats from the same litter four years ago, and one’s affection for me is completely dependent on who I live with. When I lived alone, I was his absolute favorite, but as soon as I moved in with roommates, he completely ditched me for another person in the house. I thought this was a weird, one-off attachment to another person in my house, but the pattern has held every time I’ve moved. He’s all about me initially, but as soon as he’s comfortable with another housemate, he switches his attention to him/her.

I recently moved in with my boyfriend, and the exact same thing happened. He was my guy for two weeks, and then nothing. He follows my bf around the house, sits with him, meows for attention, pets, and snuggles, giving all of his attention to my bf and giving me no time of day.

The only commonality in the household is me, so is it something I’m doing? Is it just my cat being a cat?

I don’t know if this detail matters, but my other cat has always been my girl. She snuggles, sits, and sleeps with me only.