Issue with casting spells as a mage

Hi,

First time playthrough (PC) so still on the "no idea what I'm doing" stage.

I've gotten to the Queen of Knives and can pretty much dodge everything effectively, but what I've found is that I can't cast my damage spell. I know there's a minimum stamina required and fatigue can drop it below it, but my fireball takes maybe 20% stamina, and I'm unable to cast it once my stamina bar reducing to about two thirds through fatigue. Any idea why?