Wow. Am I just too early in the game or is fast travel crazy exspensive?

Just unlocked the spell and it costs four freaking mana. So I am only halfway through my first summer but even so. If I’m using fast travel I imagine I’m using it multiple times in a day. Is it just more useful later on? Or does Mana just become a non issue?