I am confused with "Intro to Felt Surrogacy"

So here is my confusion.. Pierce was not there at the table whenever they were retelling the story correct. I understand his part was probably thrown together by the rest of the group. BUT what about his secret? If no one knew(or remembered) anyone else's secret how could they have retold Pierces secret?