The Emperor, why the hate ?

He's considered a vilain by most, but from my point of view, he didn't do anything bad or evil like the 3 gemstone holders.

He also not power greedy like them, because he's ok if you decide to keep the gemstones (and transform to an Illithid).

He's just a man who got caught and had bad luck.

So what part of the story am I missing ? (finishing the 2nd run hopefully atm)