Question Relating to Madness

Is there a lore-related reason for why deer, eagles, and every other animal aside from rats seem to be completely unaffected by the Towers of Madness? I've always found it sort of fascinating that in areas such as where you encounter Vyke, the deer there are completely normal.