Thinking about being petty

This married man I work with has done me so dirty. A few months ago he pursued me HARD and charmed me into getting involved with him (sort of). We only had phone sex twice and met up once but never had sex. Like made me fall in love with him almost. Yes, I realize I never should’ve entertained it. I normally consider myself immune to any married man’s advances, and I’ve never been involved with anyone who was married before. And never will again.

Now he is spreading rumors about me at work. I basically stopped talking to him because he kept flaking yet always kept me on the hook by texting me just enough to maintain a connection. I’m really pissed at him because he pursued me, slipped me his number, kept hitting on me and texting me. Yet now that we stopped communicating outside of work I feel like he is telling people I’m the one who pursued or is pursuing him. He of course acts fake and nice when we talk because he is trying to stay friends and make his work life not awkward. Need some suggestions on what to do. He is lucky I don’t just tell his wife and son the bullshit he is doing behind her back.