If an AI judge picks the training winners, whatever it's systematically wrong about gets quietly taught to the model.

This explores what happens when an LLM acts as the preference annotator inside an on-policy alignment loop. The setup that makes this concrete is online AI feedback: instead of a fixed offline preference dataset, the model samples two fresh responses from itself each iteration and an LLM judge picks the preferred one, which the literature finds beats both offline DPO and RLHF and reduces reward over-optimization Can online AI feedback make preference alignment truly on-policy?. The catch is that the judge's preferences become the gradient. Whatever the judge systematically rewards, the policy learns to produce more of — so the judge's biases stop being measurement error and become training signal.

The most direct hazard is the family of exploitable, semantics-agnostic biases. LLM judges score responses higher for fake authority signals (invented citations, credentials) and for rich formatting, independent of whether the content is any good — and these are zero-shot attacks needing no model access Can LLM judges be fooled by fake credentials and formatting? Can LLM judges be tricked without accessing their internals?. In an on-policy loop nobody has to mount an attack: the model will simply drift toward verbose, authoritatively-styled, prettily-formatted answers because that's what wins, producing a policy that performs competence rather than having it.

The subtler and more corrosive bias is self-preference. LLM judges pick LLM-generated arguments as winners far more often than humans do (62% vs 39%) even after controlling for quality, and this bias sits downstream of component scoring so it contaminates the whole pipeline Do LLM judges systematically favor arguments from other LLMs?. On-policy alignment is precisely an AI judging AI's own distribution — so a self-favoring judge rewards the model for sounding more like itself, a positive feedback loop that pulls the policy away from human preference rather than toward it, narrowing rather than aligning.

Two deeper findings suggest the bias isn't easily trained out. Cognitive biases are mostly planted during pretraining and only modulated by finetuning Where do cognitive biases in language models come from?, and alignment training tends to mask biases rather than remove them — implicit-association-style probes surface stereotypes the model refuses to admit under direct questioning Can psychology methods reveal what alignment training conceals?. A judge built on the same pretrained backbone as the policy shares its blind spots, so the loop can launder a hidden bias into reinforced behavior while looking clean on the surface. Alignment procedures already produce uneven results across dialects and global viewpoints from upstream annotator and task-design choices How does LLM alignment affect representation across dialects?, and an LLM judge inherits and compounds those choices.

The corpus also points to escapes. Training judges to reason through an evaluation — converting judgment into a verifiable problem — substantially cuts authority, verbosity, position, and beauty bias Can reasoning during evaluation reduce judgment bias in LLM judges?. And a panel of smaller judges from different model families beats a single large judge, because ensemble diversity cancels family-specific bias at a fraction of the cost Can a panel of smaller judges outperform one large judge?. The thread connecting both fixes: a single same-family judge is the worst case for an on-policy loop, since its idiosyncratic and self-preferring biases face nothing to cancel them before they become the policy's reward.