Why does this always happen to me?

To start, I'm in a long-term relationship. Things are great, I'm attracted to him and I'm very much in love. I wouldn't call our dynamic poly, but we'll occasionally try to find another lady to hang out with/date and be physically intimate with. We don't pursue men in that way, but often I find myself looking at them and fantasizing about them. Sometimes I think it's a close proximity attraction thing, because it's often been coworkers. I have no intentions of going outside of what me and my guy and I agreed on because I don't want to break trust. It's just not worth it. But the temptation is overwhelming at times, and I can't help but feel like I shouldn't indulge in these feelings when I'm... you know. Alone with myself. What should I do? Is this wrong?