What should I do..

My ex and I broke up 2 months ago because my family heavily disapproved of his background and got way too involved. Right after, I made the mistake of seeing someone else. He found out, and even though I quickly ended it with the other guy because I wasn't in love, the damage was done.
Now, my ex and I are back to talking, but we’re doing Friends with Benefits because that's all he wants. The problem is he constantly brings up the past and guilt-trips me. He keeps saying things like, "We used to be perfect," "I loved you more than anything," and "You hurt me, you'll never have me again."
If he only wants FWB, why does he keep throwing the past in my face? On top of that, he barely wants to meet up because he says he doesn't want either of us getting attached or falling in love again.
It’s making me feel incredibly weird and confused about where we stand. I still want him back, but I don't know what to do or think anymore. Is there any way to fix this, or am I just hurting myself?