Do I need to be scared?

Hey guys,
Yesterday I rode home from work on my bike, like I always do. There’s a long bike path, and since it was already 11:45 p.m., I wanted to get home quickly, so I was riding pretty fast and passing people.

Suddenly, a guy I had passed earlier while he was riding really slowly flew past me about a minute later and said, “Too slow.” I thought it was funny and kind of took it as a challenge, so I passed him again and waved goodbye as I reached the area where I park my bike.

I locked my bike and was about to cross the street to my house. That’s when I noticed he had been waiting there. He said, “Hey, I think you’re really attractive. Can I get your number?” There was this Middle Eastern guy who was about twice my size, both in height and build. It was dark and there was no one around, so I got scared and I said yes and gave him a fake number.
He looked so happy, smiling like a little kid, and said, “Really? Today is my lucky day! Where do you live?” I told him it was a few streets away. Then he said, “I’ll text you later. See you.”

My entrance was right there, but I didn’t want him to know where I lived, so I kept walking a few more streets until I was sure he wasn’t around anymore.
Now he’s probably realized I gave him a fake number.

Do you think he’ll wait for me there to confront me, or maybe even do something worse? Or will he probably just let it go?

I know I’m probably overthinking it, but I honestly got scared. At first I thought the whole bike thing was just funny, and I didn’t realize he might think I was flirting or interested. When he asked for my number, I didn’t feel safe saying no.