Help with assigning companions

Trying to play multiplayer for the first time with someone and we each want to control ourselves and one companion. When the host tries to switch companion to other player, it says "each player must control at least one avatar" and wont let us put the companion with that player.