What should I do if my friend’s ex likes me and I kind of like him back?

Before he and my friend started dating, I thought he was quite attractive. In fact, I tried to date him, but he liked my friend and started dating her. I immediately stepped aside and forgot about him. Anyway, I continued to think he was attractive, but I never tried anything else or had feelings for him. I just thought that if he ever flirted with me, I would flirt back. I was convinced he would never notice me. A few weeks ago they broke up, and he texted me that same day. His messages were pretty weird. The next day, he was following me everywhere and constantly trying to start a conversation with me. However, after the rumor spread that he liked me, he told me straight out that he didn't have feelings for me and that he knew I didn't have feelings for him either, so he would stop talking to me to avoid making things awkward. I thought about telling him that I did have feelings for him, but I didn't want to betray my friend or be rejected. He stopped talking to me for a while, but recently he started talking to me again and we have become a little closer. I like his company and that he tries to get close to me, then I don't do anything to push him away. I feel guilty, because, literally, he is my friend's ex. I don't know what to do. I know he was shitty to my friend when they were in a relationship, and I hate his empty personality, it's the complete opposite of my type. But his physical appearance is too attractive to me. That's why I have a hard time pushing him away. I don't know what to do with my feelings or how to approach this situation. Please, I need opinions:(