How do you stop evals from becoming a cheat sheet for the prompt?

We kept tuning a prompt against the same small golden set until every check passed. But then the paraphrases failed in ways the score never predicted... Most synthetic cases shared one template, near duplicates leaked across train and holdout and the scorer rewarded memorized formatting more than instruction following. NGL that green check was hot steaming trash.

I am pushing for semantic deduplication before the split, a holdout set that the prompt author cannot inspect and adversarial slices built from fresh failures. Pass at k also looks more honest for tasks with several acceptable outputs, however it can hide a weak first answer if we are not careful. Confidence intervals matter too because a five point gain on forty near identical cases is just decoration.

How do you keep an eval set useful after the team has already seen all the cases? Do you keep a hidden holdout, add fresh production examples, use paraphrases or treat it more like a real train/test split?