When people respond with "How come I've never heard of that!?"

Obviously, that is a rhetorical question, but I feel like responding with "How the fuck would I know!?". Most of the time we are talking about a topic that the person has already expressed a lack of knowledge of.