My (23F) boyfriend (25M) has feelings for a mutual friend

Like the title says. We’ve been together for 5 years. He met the friend (26F) through his previous job about three years ago. We usually see her when we do group hangouts. Recently he told me he has feelings for her, and has for a few months, but that he didn’t want to pursue anything with her or for our relationship to change.

The issue is that I can very clearly see the difference in behavior when he’s talking to me versus her. He’s generally down and depressed around me, and he says it’s because his current job is overworking him (which they are). But he could work a 10 hour day and if we hang out with the group he’s all smiles and laughing, specifically with her. He stares at her, and they always make an effort to sit across from each other if we’re out at a bar. She likes to get really close to him and talk, and the look in his eyes when he talks to her is nothing like when he talks to me.

I feel like my presence annoys him more than anything, but he says he still wants to be with me, and says he feels guilty for even having feelings for her in the first place. I know that crushes in long term relationships are common, and maybe I’m just too sensitive, but the idea of sitting next to him while he gazes at her in future group hang outs makes me feel sick.

I don’t know what to do. He’s not going to want to end their friendship, and I would feel shitty asking him to, but I can’t stand seeing her lean against him while they’re walking down the street, knowing he likes her, while he almost always refuses to show me any kind of affection in public. My gut feeling is that I need to end it for my own sanity, but am I being irrational?