First date went well… until he got way too touchy. Now I feel bad for not replying.

Hello everyone. So I (22F) went on a date last night with a guy (33M) He seemed great on paper, successful, confident, easy to talk to and I actually felt comfortable at first. But during dinner, he started getting really touchy. Like rubbing my back and legs, whispering things, even trying to kiss my neck, all on a first date. I pulled away a couple of times and tried to keep things light, but inside I just felt super uncomfortable. He later apologized and said “physical touch is my love language,” but honestly, it still didn’t sit right. He didn’t even ask if I got home safe or anything. Now he’s texted saying he wants to see me again, and I haven’t responded. Part of me feels guilty for not saying something directly, but another part of me feels like… I don’t owe an explanation to someone who couldn’t respect boundaries because honestly I didn’t feel a connection he made me really uncomfortable and I don’t want to go out with him anymore. My friends think I shouldn’t respond or go out with him anymore as well but it’d be nice to hear other people’s opinions. Am I overreacting by just ghosting him? Or is that behavior enough to walk away from without needing to explain? Thank you!