AITAH for blocking a friend over their actions and rude comments

My ex-friend, I'll call him Drew (20M), used to be in a couple of the classes I (22F) was taking for my degree in college. Drew and I became friends through these classes and we were good friends for two years up till December of 2023.

For context, Drew was autistic and knew that I was Lesbian through us being friends. Drew would off-handily call me "Token Lesbian" in conversation or would make inappropriate and uncomfortable comments of me "jokingly" using an electric toothbrush as a vibrator in a bathroom. I never said anything and often just brushed it off as if it was funny, when it really was not.

This problem of him saying these things came to ahead in, as mentioned, December of 2023. I had invited Drew to a music store a few cities away for the afternoon. I had to pick him up at his house, where he proceeded to try to connect his spotify to my car without asking; additionally, he brought a bunch of cords and headsets to listen to his music. He ended up blasting his metal music in my car from a CD he purchased and brushed me off while we were walking around the mall in the area. When I was dropping him off at his house, he made a comment regarding a dream he had with me in it the night before, saying, " I had a dream you and I made out in your car."

I actively avoided Drew after that on my college campus. The last time I spoke with him in person I made a joke about being Bisexual since Baldur's Gate 3 had just come out and I thought Gale was cool; however, I now know it was gender envy and not attraction. I realized making this comment probably made him think he could ask me out or try something along that line. I blocked him earlier this year after he kept texting me asking to talk every other week during the summer.

People I have told this to think I was in the right for blocking him; however, I have also met people who say I am an asshole because he was special needs and the guy didn't have many friends. AITAH?