feeling guilty like i'm betraying my wife, but why? and how to move past these feelings?

i am a mostly straight guy in a heterosexual marriage, but i have always had sexual attraction towards men every now and then and while my wife knows this and ultimately doesn't care, i can't help but feel guilty fantasizing about men. it feels like a betrayal, because i don't ever fantasize about sex with other women, because i just simply have amazing sex with my wife instead. i have no need or desire to fantasize about other women. however, there is a part of me that desires men that she can't necessarily satisfy, and that's why i fantasize sometimes.

i am fully monogamous and i only desire to actually have sex with my favorite person in the world and the woman i married, but i can't stop these urges from surfacing sometimes, i have tried. and logically, simply fantasizing about it seems like it should be harmless, so why do i feel so guilty? how do i get over this guilt? because i can't stop myself from being bi, is this just a curse that monogamous bisexual people have to bear?