Internalized acephobia got so bad I ended up dating an allo

That's it. Sometimes I doubt myself so bad I don't even take my asexuality seriously. This time I ended up dating an allo guy who's clueless about asexuality and thinks taking meds will make me like sex. He's not inherently queerphobic since he never treated me (non binary) as a fetish or something to fix, but this time we discussed about sex and I told him I don't care about it and only do it to make him happy, now he's seemingly spiraling thinking I don't find him attractive or something lke that

I don't know what to do right now, I mean I like him but if sex is this important to him I fear we'll have to break up for the sake of our mental health