Difficult to stick to my plan

For back story: Had been seeing a guy (we'll call him Sam) on and off for a few years, casually really but I had feelings and that was fine, I knew what I could accept. Saw each the most this year and I didn't bother seeing other people (not because I thought things with Sam could go anywhere - just didn't make effort in dating others) but at the beginning of last month I cut things off because my energy wasn't being reciprocated and I was done. I had broken/tried to break things off a couple times over the last few years but always would find ways to initiate conversation within a few days or weeks. So now my vent: I've started dating again and am finding it difficult. Dates are fine for the most part but a date I had on the weekend I made a joke (thought it was pretty clear it was a joke) and this guy thought I was being serious and just about freaked at it. And my first thought after the date was Sam would have gotten that right away and think it was hilarious. Am I destined to have to decide between either having a humourless relationship or no relationship at all??? It's making it difficult for me to not find a reason to start conversation with Sam again