A Major Mistake in Adventure Time (I think)

Sorry if this was said already.

Ice king does not know his past (being Simon), so in Holly Jolly Secrets, how does he know that was him when he says "NOW YOU KNOW! I USED TO WEAR GLASSES." How does he know that was him in the video?