Libertarians: what's the connection between libertarianism and personal morality?

Just to be clear, I'm not asking what you think about abortion and other social issues, or how you think moral issues should be decided in the abstract. I'm asking about how the ideas about right and wrong you acquired from your family and personal experiences influenced your decision to be a libertarian.

I'm asking this out of curiosity. A lot of left-liberals often view libertarianism as a philosophy of selfishness and egoism, but I don't think this is entirely fair. At the same time, I grew up in a left-liberal family with a comfortable, if not lavish, income, and somewhat patrician social attitudes (think FDR) and I find libertarian attitudes really hard to comprehend: it seems like there is there is this constant hanging fear that the government will come and deprive you of things, or that the poor will seize control of the government and have it deprive you of things, and that this is not just wrong, but a sort of general statement about how evil in the world always works, and that Evil is Real. Whereas I grew up in a big urban center with a lot of disenfranchised people, but relatively comfortably personally, and always kind of felt the point of politics was to Make The System Work For Everybody and with a lot of faith in altruism against human misfortune.

For this reason, I tend to imagine libertarians as coming from not very affluent backgrounds but above what you could call 'poverty,' usually self-employed and in the western states. I also wonder about the role of religion in all of this: the idea of Evil I see in a lot of libertarian stuff seems very calvinist/protestant in inspiration. I don't know if this is accurate, though.

So are there particular things that make libertarianism make sense to you that you've experienced in your lives?