Veteran shopper asked me if I could leave his turf

I’ve been doing a mall area recently with a nice Sephora and some restaurants but I did eventually notice one other shopper who’s always there. Long story short I noticed that if are both there and nobody has any special priority I will always beat him to the punch on accepting orders. I’m assuming I just have a faster phone service giving me a few seconds edge. Anyway I guess he finally got fed up today because it’s July 4th and I was there even earlier than usual and he walked up to my car and asked if we could speak “ man to man”. And basically because he’s partially disabled he can’t do really difficult orders and this has been ”his spot” for a long time and he basically asked if I could find a different place. I told him I’d think about it ( I’m not really considering it obviously) but how should I handle it?