Do you think it would ruin that dynamic if Joel really was Ellie’s father?

Like imagine if during those 20 years, he hooked up with a random woman who was really Ellie’s mother Anna, and he just took off, never expecting her to be pregnant. Nobody has ever even once said a word about what happened to Ellie’s bio father.