How long are party members content being left at camp?

I'm just trying to work out who to gather in my party for venturing forth.

Shadowheart seems pretty critical to the plotlines so I guess she is staying. I want to keep Wyll to romance, he is smokin' hot. But Gale is also totally intriguing. But that means I won't be playing out the Githyanki storylines. Can you trade people in and out when storylines featuring them are completed - without sacrificing their XP? Or should I stick to a party and just figure I can play the others next time.

Signed, an overthinker.