Book about a magical house with magical doors?

I think it was the first of a series, all about this house with these doors which could lead to crazy places. I think actually the title of the books/series was something quite descriptive of this - like house of magic or house of doors or something. I remember the series was a bit dark. There was a plot line in which the main guy (who is master of the house) has all these siblings who all fight for power/dominion, and his sister sneaks off through one of the doors where you’re never supposed to go because it’s too dangerous but she thinks she can handle it, and by the time she’s found she’s been basically tortured to within an inch of her life. There was also an oblique reference to a semi-incestuous relationship between the main guy and his sister?