How to deal with Hot And Cold Behavior from AP?

We been on and off for a while. I usually broke it off when I felt like I was being too emotional and my emotional needs weren't being met .or out of guilt.

In the beginning he told me that hes mainly in it for sex. I never had an extramarital affair and I realised quickly that i cant just have sex with someone without emotional connection. I told him this and he said hes fine with the emotional connection . His affairs usually have been mainly sexual but Hes willing to build a connection beyond that and make me happy. We should just remember to not fall in love and get into deep because none of us will leave our SOs.
The last time we met before lockdown I was feeling pissed because he really wanted to get in my pants and I wasnt up for it. I enjoyed making out but didnt want more. So it was awkward when we said goodbye. That awkward situation was redeemed by his willingness to put more effort in our thing. He appeared attentive and was making our communication his priority. He messaged me even when he was in a family problem, while he was with his wife etc... I didnt expect him to do this but since he did I got kinda used to that.

I broke it off "finally" one month ago and when I returned things havent been quiet the same. His responses are mostly short and take ages. I then told him that theres no need to proceed this if hes no longer interested and he responded that he was but just that currently he couldn't invest as much as he used to. I asked him what he meant by that. Havent gotten a response to that question ever. All he said was he really wanted to meet. I left him hanging abit until he begged and pleaded that's when I gave in and agreed to meeting up. Prior before meeting up that day he seemed very eager. Sex was not on the cards so thatz why I was in awe that he still made that much of an effort.

We met and we talked like buddies..I thought he would be all over me but he wasnt. And I was okay with that because I did want a bit of a balance and not just talk about sex all the time. So when it was time for us to part ways he asked when do we meet again and I shrugged, he said he will message me in regards to that but surely this Thursday or friday!

What has he sent me since we last met on wednesday ? Just a " Hello" that's it!! I replied 12 hours later with a question mark, no response. I'm so mad at myself for thinking that we are getting back on track. When hes just pulling me along for reasons I cant understand. If it was mainly for sex then he wouldnt have made all that effort to meet knowing he wont get any or am I wrong?

Well now I'm frustrated. His behavior has been very hot and cold. Before.the break up he would text me all day. But now he doesn't do that. Does he not want to keep the infatuation going? I want to be straight and call him out for his hot and cold behavior without appearing too needy.

I understand that we both married so maybe I cant expect the same transparency like singles? I dont know but I really have the urge to say something. I want him to know that I dont want to start these.games where I ignore him for a week just to get him to understand what he did wrong.