All allies except Simon are greyed out on the swap screen?

I can't for the life of me figure out how to change companions. I'm stuck with Simon right now and it seems as if everyone is greyed out in the swap ally screen.