Where did it all go wrong for Lucius Malfoy?

So in the earlier movies, he was an ass, but a very confident and arrogant one.

Then at some point in the last few movies, he's just a mess. A wormy, unshaven mess.

I'd love to know how it all went wrong for him. Thanks!