Is there a way to limit a lord's ability to magically sprout a hundred men immediately upon being ransomed?

This is getting ridiculous. I can't hold them without receiving an honor penalty once they get ransomed, but they just zerg my castle and town to the point that I can't do anything but defend against them.