I should have ended things after this but I didn't

There was this guy I dated, and I told him about a really inappropriate joke a coworker made.

I was in a meeting, sitting next to my boss, and this coworker was like, "put your hand on his thigh, he likes that". Everyone in the room was appalled and my boss said I would be justified in making an HR complaint.

But when I told the guy I was dating about it, he was like, "yeah, well, you did say your family is super uptight about these things, so it makes sense that you are". I had no idea what to say to that, so I didn't say anything. But I'm wishing I had ended things in that moment. There were a lot of moments where I could have ended things, but didn't, and that's one of them.