If I said I’m not comfortable coming to his house but he invites me again what do I do?

I’ve seen this guy for several months and I’d say once per week. Sometime at 1.5 months he took a really long roadtrip where he was traveling with friends, and at times just him and 1 other female friend. So he told me about this arrangement from the first time that we met. He said we were off dating apps at the time of his leaving. He also moved in with a woman around this time.

When he came back I asked what we are doing in the future and he said well you can be my girlfriend. And ever since then I feel like things have gone downhill. He’s often talking about himself, which I notice more as I’ve spent more time with him. He re tells certain stories and doesn’t remember what I tell him it seems.

Anyway he didn’t even compliment me or really flirt. Just sends me shorts and then looks at me a bit, hugs me and holds my hand then he kisses me. It feels like I have a friend who I kiss. I get relationships can be that way but he’s completely stopped replying as frequently to me. Sometimes he’ll be more inquisitive and nice but there was a time he was so vague and just low effort on text.

I think the reason for it was since we went „official” he asked me to come to his place to hang out because he had an injury and wanted to rest. I said no. He asked me a few times before while we were dating too. The next time I see him we go to a cafe. He bought ice cream and said he’s gonna leave it at home we can both stop by. I told him I’m not stopping into his house.

He seemed to get upset and said let me just drop it off. So we go… and I don’t go in. I stand outside of his actual home. He wanted me to come up. We go on a walk, and then he said wanna come back to my place to chill? I said no. And he said ok I’m sore from the gym… bye!

The next time something similar happened. So I confront it. I told him I don’t want him to do that. It feels like he’s pushing me to his house and I can’t trust it. And then he said: ok if you want we can just go to my place to actually chill. I wanna have sex with you but we don’t have to yet but I’m a bit sad because I prefer to have sex earlier it makes me feel close to my girlfriend.

We are making our next plans. Why after this talk about how I’m not ready to come over or I don’t feel like it’s the time for that, he says: we can go to my house and I’ll show you this boardgame. I’m not opposed to coming over. My friends go to their bfs houses and they literally just hang out. But the way he’s been doing this I’m scared