Invited a guy I can't stand to my birthday party

Hi, I went through a manic episode a while ago during which I invited literally everyone I could think of to my birthday party, including people I'd just met that same week, which is also worrying but not the main problem.

The main problem is that I invited this one guy who annoys me so much with every fiber of his being. He's autistic and laughs and talks extremely loud which just pisses me off every time I'm around him. He also very obviously has a thing for me and always follows me around and tries to start conversations with me even when I'm very much not in the mood, he also used to write me terrible poems for every single holiday for some reason and could never understand that I was absolutely not into it. He's very sweet and kind-hearted, but I just definitely do not enjoy his company.

I can't believe I fucked up this bad by inviting him, I hate how I get myself into situations like these because of my illness and how little I am able to do about it. I was looking forward to this party too, it's my first time really having a big celebration. To make it even worse, he doesn't know anyone there so he'll just follow me around all the time and I won't be able to have a good time at all (my fault, not his).

I was considering maybe texting him that some plans have changed and I won't be able to invite him, but maybe suggesting meeting up just us two instead? Maybe that wouldn't be as much of a shitty move, especially since he wouldn't know anyone at the party and would probably have a better time just hanging out with me. (I'm sorry that makes me sound awful but it's just the truth).

Help 😭