How do you feel aboutt the "it's taken out of context" argument?

I personally am annoyed by it because even if it is out if context, it still doesn't make it acceptable. I often see this argument when people try to defend verses that obviously sound horrible as if that would automatically excuse the verse.