What happens to 'Unassigned' Faces in Protect?

I didn't realize how 'Reassign' works in Protect and accidentally reassigned all of my son's faces to my daughter's "profile". I tried to 'Unassign' all of my son's incorrectly assigned faces, but doing so just causes them to disappear. I now can't find the faces that UI had previously detected of my son..... Any help?

And yes, I know I'm making very little sense. My apologies, not sure how else to describe the issue. I'm hoping someone else has experienced this and knows the solution :P