Advice about aggressive panhandlers

I work downtown and get stopped by aggressive panhandlers on a regular basis. They literally stand in front of me and wave their arms until I stop. One even shouted obscenities at me as I tried to walk away until some random dude stepped in and yelled at him to leave me alone. (Whoever you were, thank you for helping out a 30 year old woman who was just trying to get to her office in peace!)

I do absolutely nothing to cause this. I don't look at or talk to anyone. I mind my own business. I don't wear my nice professional clothes outside the office (I commute in a nondescript tee shirt and jeans, and change into my work clothes at the office). I don't have an expensive purse or anything that says "I have money." I don't smile and I have a massive case of RBF. Yet somehow, in convinced I have an invisible target painted on me that I don't know how to remove.

Does anyone else have this problem? If so, what do you do?

EDIT: Here's a little more context. I work at Sinclair. I do not drive myself to work because my husband works farther away so he has the car during the day. I take the bus (way less expensive) so I walk from the Hub to campus down either 3rd or 4th. The first two blocks are always the worst with panhandlers; once I get close to campus, they disappear.