How do I tell someone I no longer want to talk to them after exchanging numbers?

I gave my number to a customer who I’ve seen often and talk to all the time at work. He usually tells me about his time at work and updates me on life and I do the same with him. It’s like that classmate you have in university that you always sit next to in class, but never meet outside of it.

Yesterday, he came in to where I work and we started talking. He then asked for my number since he’d love to not just have our conversations when he comes in and I told him sure. We had been so hyped up in a conversation on video games. I’ve known him for a while and didn’t think twice about it.

However, once he left, the awkwardness set in and I realized I gave him my number. The man I gave my number to is almost 20 years older than me. I couldnt rub away the feeling it gave me and talked to my bf about it. He’s never been interested in a relationship with me and knows I have a bf. But, something definitely feels OFF. He said that it’s weird for a man that age to be asking a 20 year old for their number.

He’s never given me red flags in our numerous in person convos we’ve had over the past 2 years. So why am I feeling weird now?

How should I communicate that I’m no longer comfortable talking to him over the phone but will still like to talk in person whenever we see each other?