Does it count as SA if my boyfriend asked for consent?

When I (ftm) was 17, and not fully aware of being trans yet, I dated a guy who was one year older than me. He was liked by everyone, and he had many girlfriends before me (he said there were seventeen of them). When we started dating he said that one of his girlfriends accused him of SA, and that he was awful for lying like that, and for that reason he said he wanted to make sure that I am fine with everything we did.

When he first touched me in a sexual way, i only remember that he asked if he can take my clothes off, and when i said no, he just put his hands under my clothes.

Then, he would often ask me if we can do a thing he was into. i wasn't sure, so i always said no, but he would try to do it anyway, every twenty minutes when we were doing anything physical. He never actually did it, but he was really close. Once he almost did it, he got very excited and was telling me "see? it didn't feel so bad, we should do it fully".

There is this one thing that bothers me the most, which is that once we were sitting in complete darkness (there were people around, but it was so dark you couldn't see anything), and he held his hand on my thigh. Then he moved it up, and rubbed. I didn't react, trying to ignore it. After that, with his other hand he rubbed my lips, and then pushes his thumb into my mouth. i just remember my lips shaking a lot as he did that for at least ten minutes. Only then he asked if it's fine, and I think I noddes, because he kept doing that. He told me to suck, but i couldn't, because my lips were shaking so much.

I am sorry if it is not written coherently enough, English is my second language, and the memories are a little blurry, because it was a few years ago.

I broke up with him, saying I was aromantic (since then I diacovered I am not, but I think I was scared of saying that I didn't love him. I also felt like i am incapable of feeling love towards anyone ever). His first question was "Does it mean we can still f*ck?". I said yes, but I just started avoiding him after that day.

After all this, I met many new people and I later found out that all of them though i was SA'd, even though i wouldn't even mention kissing my ex when we talked about relationships. I mostly would say that we broke up because "he was annoying".

Hearing that all my friends think that about me, made me think of this relationship. I am not reacting well to unexpected touch (I freeze for a second, and suddenly feel cold), but I think part of it is how as a trans person I feel about my body.

i would like to know how this situation looks from an outaide perspective.