Stealing from Grave

We're playing a 3 man magic game. An opponent played [[Sepulchral Primodial]] stealing a card from each of the other 2 players graveyards.

One player argues that if the player who stole our cards loses the game, because the creatures are on the board they should go to our fields. I argue that they would just go back to the graveyard.

What is the correct ruling on this and can you post which rule this is? Thank you in advance.