Velify1

The reason is the damage bonus which means that your characters take twice the damage they otherwise would. Casters have spells that mitigate damage with buffs like mirror image, stoneskin, ironskin, or protection from magic weapon, being some of the best. Those buffs don't care for incoming damage, they are equally effective against any attack.

This means that the casters are much more durable in this context. Fighters can get hit for 60-80 damage at most when dealing with the real heavy hitters while casters can tank them just as well as they tank their core-rules equivalents. It's especially grim when dealing with certain spells which can just one-shot the unprotected tanks.