I feel like he expects sex, but I don't know how to say I'm not ready. Help?

So I'm a virgin adult and for a few reasons I don't feel like I'm ready for sex right now. Partly anxiety. I've been seeing this guy for a little bit, we're talking about what would be the fifth date. Neither of us lives alone so I didn't feel any pressure to like take him home or anything before, but he's now suggested us staying the night somewhere and I don't know how to respond. I feel like the implication is that he wants sex but we have never even talked about that or come close. I swear the most sexual thing that has happened is making out and him putting his hand on my knee. Verbally there has been nothing other than like winking emojis. I don't know what to do! I feel like it's awkward to say "you're implying you want to have sex, right?" and then give him a no to something that hasn't even happened yet. I am attracted to him and could see myself sleeping with him eventually I just don't think I have that kind of trust yet and I have some baggage to unpack. We're pretty awkward physically.

I think I have internalised some pretty bad ideas about sexuality, like it makes me feel guilty to turn someone on or imply future sex and then not follow through (i.e through dating). I don't know how to express my preferences. Last time I was in a sexual situation, I wasn't assertive about my wants and just decided to "go along with it" because of my approach to dealing with anxiety in other areas of my life (I try to do things I feel nervous about). I would like to go on the date he is suggesting, even the overnight part, but I feel like accepting will be saying yes to something implicit. We also have both told each other we're not looking for a relationship but didn't establish what we are looking for. Is it normal for people to beat around the bush that much about sex? I get the impression this guy isn't super experienced either just based on what he's told me about his dating history, so maybe that's why he's being super PG, but I'm really nervous to reply to his text, idk if I should just be like "yeah that sounds fun" and crush his expectations later if and when things heat up, or be gauche and say "yeah but no sex tho" when he never even remotely brought it up. What should I do? Is agreeing to an overnight date saying yes to sex and is it rude to not follow through?