Relationship Troubles

My (F24) Bf (M23) and I have been together almost 4 years. He had sex with me when I didn’t want to have sex. He didn’t intentionally do it but I can’t get over it. Me and him had a quasi free use agreement. We agreed he could have sex with me whenever he wanted, provided I wasn’t in a bad mood or wasn’t feeling well. I agreed to this because I don’t really enjoy sex and it takes off some of the pressure because I can just let my mind go elsewhere and not have to be present. Anyway, a couple nights I was in a bad mood and he had sex with me but I didn’t let it be known, so I don’t blame him at all. However, one night, I kept doing weird bits and trying to distract him when he wanted to have sex, but he still had sex with me. He’s apologized, admitted fault but It’s still hanging over our relationship. We’ve been together almost 4 years I don’t want to throw the relationship out, but I don’t want to do. I know it’s my fault, I know I should have just told him, but I didn’t and now I’m lost. Where do I go from here, how can we get our old dynamic back?