What are the negatives of owning a city or temple holding?

Do you get the taxes from it? Do your vassals get angry?

p.s I play with no demesne limit, not sure if that matters whether or not having a wrong holding affects me.