Why did Scott Patterson really leave?

I'm not really sure exactly what his issues were, other than the stated character direction, which could mean several different things. Was he unhappy with Sully's direction or Liam/Maggie or some other combination?