Is it bad if I’m attracted to a married man?

I want this married man who posed as single. I know it won’t go anywhere but I’m extremely sexually attracted to him. I said no when I found out he was married- but I think I want him now. He’s still active on dating sites so it’s not like me saying no will make him loyal to his wife. I know he only wants me sexually too. Is that so bad?