Under what laws or justifications do (US) Towing companies hold your vehicle "hostage" in return for payment?

The entire concept irks me a bit. Go ahead and tow an illegally parked vehicle, I understand the need for that. Charge the person a fee for doing so, I too understand that. However, what I don't understand is how they can refuse to return your vehicle to you until you pay them the fee (Between $100-$200 USD, normally).

It seems to go against all reason, really. Considering a vehicle is normally worth at least fifty times that of the towing charge, this would be like missing a minimum payment on a credit card, to have the company immediately seize, and literally hold under lock and key everything that you bought with that money.

At the end of the day, your vehicle is still owned by you, and while you, after towing, have a debt to them, how can they "ransom" its return?