Harsh words remembered years later?

Many years ago I briefly got involved with a man I knew at college. He was 4 years older than me. He seemed like a nice guy. Here's the tricky part. Everyone told me he was nice and I should go out with him. They said he really really liked me a lot. I liked him too. After our first date he started acting distant. We did not go out on another date. Then a few weeks later called me up on the phone and said insulting things to me in anger. It made me feel awful and I was in a new place away from home and didn't know many people at the time. My grades went down and I left and transferred somewhere else. From time to time this awful man pops back up in my head and I do not know why.