My husband only gets aroused when I see my lover

First off a disclaimer, I dont really consider myself much of a "hotwife" and I actually dont know that much about the hotwife lifestyle. I have read a little about it, mostly for my husband, but I am by no means an expert.

My husband, 38 M, and I 32 F, have been in this lifestyle for a little over a year now. It started with him asking me to flirt with other men, then it gradually escalated to me sleeping with my lover. I have been seeing my lover pretty consistently since then, and he is the only one Ive been seeing (Im open to others, but I enjoy what I have and am too busy to go out and meet more men)

At first, this arrangement made my sex life with my husband so much better. He was constantly aroused, especially when I had recently been with my lover. But over time he had slowly got to be less aroused during our normal routine and now its to the point where he is only aroused after Ive had sex with my lover.

I was wondering if any of you have experienced this before, if so, what did you do to fix it? My needs are being met physically but I also want my marriage to remain healthy.