My EX BF is coming back to town and he want's me back... I don't know what to do

First of all, I am a 19 year old Bisexual guy with Aspergers Syndrome, now we have that out the way I shall begin :')

2 years ago I dated a guy(Lets call him James, hes 21) It was a good relationship at first but the spark didn't last too long(Probably due to the fact the relationship was more sexual than anything else), which caused us to drift apart and I ended up leaving him for a girl I had recently met...

A month or two after he moved away to go live with family, and thus ended that chapter of my life, or so i thought...

He recently(last week) messaged me on facebook and said he is coming back to town and he wants me back, he constantly bugs me when i sign on until i either reply or go offline.

I really don't know what to do about this, because i know when he gets back to town the first thing he will do is come to my house, and because of my Aspergers I don't say no to anyone(I find it impossible), He was okay as a friend but anything more and he just gets creepy(think overly attached girlfriend creepy)

What should i do about this? should i just do what i normally do in this situation and just accept it, knowing that me, in myself doesnt have the heart or the ability to say no .... or do I somehow tell him no and cut him out of my life? I really have no idea what to do :(

If anyone has any questions I will reply, anything goes so ask away if it helps