LLM-as-Judge for redaction quality: what biases should I worry about?

I'm using pairwise LLM judging (MT-Bench style) to compare two input redaction strategies. Same prompt, two variants, judge scores on 4 criteria.

One thing I noticed: when the judge model is the same as the response model, presentation order matters. In one run, showing variant B second gave it a +8.2 mean advantage, but showing it first gave only +1.7. In a second run with a stronger model, the gap nearly disappeared (6.6 vs 6.8).

I randomize order and track position_swapped per prompt so I can split the analysis, but it made me wonder what other people do:

• Do you use a completely separate model for judging?
• Has anyone found that certain model families are more position-biased as judges?
• Is there a sample size where you stop worrying about this and just trust the aggregate?

Sharing because I haven't seen much practical discussion on bias in LLM-as-Judge setups outside the original papers.