AITAH for not talking to this guy in my class?

I (24F) am on my very last semester of college. Ever since my first day of school, I’ve been somewhat keeping distance from this guy in my class, a 20yo-something dude, and for a very clear reason. Two years before I started college, I was severely ab*sed by a young man. He s*xually and physically ass*ulted and harassed me for over almost half a year. It got so bad that I eventually had to contact the police and he got sent to jail (for much less time than he should have) and when the war with him ended, I had a new battle to face- PTSD.
I’ve been going to therapy, tried medicine and stopped because of side effects. I’m doing everything in my power to get better.
On my first day of school I saw him- a guy who looks and talks EXACTLY like my ab*ser, only a bit shorter in height. The panic attacks arrived as expected but one of my goals in therapy was to not let my PTSD manage my life, so I decided to be nice to this guy and hope that it makes the panic attacks go away eventually when my body realizes that he is not a threat to me.
A few months passed and I’ve only been talking to this guy maybe a few minutes every week on bus rides on the way back home. When Valentine’s Day arrived, he gave me a gift as a way to make a move on me. Not only was I already in a relationship, I also showed zero interest in him prior.
I thanked him for the gift but decided to take some cautious distance from him, as I would do with any man who is interested in me romantically that I don’t feel the same towards.
Eventually I got more and more anxious around him. The panic attacks returned and I could barely spend an entire lesson together with him in the same classroom so I decided that for my own sanity, I’m avoiding him whenever possible.
I thought that my arrangement worked, until a few days ago when my friend got into this really weird argument with me about how badly I’ve been treating this guy. I was kinda shocked because not only does she know my reason for keeping my distance, I’ve also not made any sort of contact with him personally in years. I did talk to her lately about how obnoxious he’s gotten, because he has, but only to my friends in private and in a harmless conversation, I’ve never caused him any actual harm.
As we kept talking I found out that he’s been recently talking to her about me! And that he claims that my cutting of contact with him three years ago has created this kind of domino effect which is now ruining his life. 
I thought that it was insane and tried to resonate with her that I haven’t been causing him any harm by not talking to him and that she shouldn’t be supporting his way of thinking, because she should know me better than this.
My PTSD is now stronger than it’s been in a very long time because now I’m terrified that this white man who is using many dr*gs is going around telling people that I ruined his life. I’m genuinely scared and I don’t feel like I could have done anything differently about any of this.