Is it wrong to be with someone if you know they have a bf/gf?

I am seeing this girl at the moment who has a boy friend. I have been with her a number of times before she had a boy friend but I have been away for work. Now that I am back I am going out with her for dinners and we are hooking up. Is it really something I should feel bad about at all? I don't know the guy at all and I feel like I am not doing anything wrong because of this. I feel like she is responsible for that mess. Thoughts?

edit: I am not doing it for a relationship. I'm just doing the sex.