At what point is amazing sex just not worth it?

I’m (47F) happily divorced. No acute guy on the street organically and we’ve been sleeping together on and off for about four months now.

I’m not looking for a husband and the sex is the best I’ve ever had in my life, but this man is a pain in the ass and in so many ways.

At what point is it just not worth it even if the sex is fucking incredible?