Soul gem clarification

So I was playing legendary with the forge the infinity gauntlet scheme and I was wandering if with the soul gem if you put the card back could you keep the shards you put on it and if when you have shards on the soul gem if you could spend them