Important question on panhandling/strangers asking money

As a Buddhist I have a hard time finding the right way to approach panhandling or even random people coming up to me to ask for money. My default reaction is usually to say I don't have any cash on me and proceed to walk away without helping them. This is mostly due to not knowing if the person needed help authentically and that they might be abusing it and using the money for things other than what they're asking for.

As a working class who is not rich nor poor, I do not like giving my money away simply to someone who do not want to work for it. Now the that thing gets me is how do I tell if this person really needed the help or not. Because if the person authentically needed help and I wrongfully denied helping them, this would guilt me deeply. So the question is, would the Buddha say to openly help all people? Knowingly that the person is lying and might be using the money for harmful reason (i.e. Cigs, alcohol, drugs, personal gain)? Or would it be ok for me to be wary of help depending on how the person come off as? I know it is hard to tell but I just don't feel like it is right to just give my hard earned money to anyone that comes up to me and ask, just because I am a Buddhist. Thank you for your time reading this, and I would appreciate any insight on this topic.

As of late, I have been mostly saying I do not have cash to anyone that comes up to ask for money, which is the truth- although I would be able to pay for what they are asking for through credit card.