He won't be with me because he's afraid of losing his kid