I anoy my enfp somehow

I have a huge crush at my enfp (M) coworker. We have age difference of 5 years and because of that I didn't pursue anything and me being INFJ doesn't help either (shy). He was always super sweet with me and maybe even flirty, but lately he is anoyed by me for no apparent reason. It's like everything I do it bothers him. Help I don't want to lose him, even as a friend.