How do pseudo-relevance labels enable training without ground truth relevance judgments?

This question is really about a workaround: ranking and retrieval models normally need humans to say "this document answers that query," but those judgments are expensive and scarce. Pseudo-relevance labels sidestep that by treating some cheaper, automatically-generated signal as if it were ground truth. The corpus doesn't house a single paper named for this trick, but it circles the same territory from several angles worth stitching together.

The clearest working example is distillation. At Walmart, an LLM was used to label query–product pairs at scale, and a smaller BERT cross-encoder was trained on those machine-generated labels — no human relevance judgments in the loop Can smaller models outperform their LLM teachers with enough data?. The striking result is that the student didn't just inherit the teacher's noise; trained on a large enough augmented set, it *outbeat* the teacher, because the teacher's soft predictions smoothed a much broader slice of the query distribution than any human annotator would have covered. That's the optimistic case for pseudo-labels: the proxy signal generalizes better than the sparse gold standard it replaces.

The same self-supervision logic shows up where models generate their own training targets. Consistency training uses a model's own clean-prompt responses as the labels for teaching it to ignore irrelevant prompt wrapping — the supervision comes from the model, not from an annotator Can models learn to ignore irrelevant prompt changes?. And bidirectional RAG goes further, letting a system feed its own generated answers back into its retrieval corpus as if they were trusted documents Can RAG systems safely learn from their own generated answers?. Both make the central bet of pseudo-labeling explicit: machine-produced signal can stand in for ground truth *if* you guard the quality.

Which is exactly where the corpus issues its warnings. The write-back system only works because it gates every candidate through entailment verification, source-attribution checks, and novelty detection — without that filter, hallucinations would quietly poison future retrievals. The failure mode the gate is defending against is named directly elsewhere: vector embeddings measure *semantic association*, not *task relevance*, so a naive automatic relevance signal will happily score a semantically-close-but-wrong document as a match Do vector embeddings actually measure task relevance?. Pseudo-relevance labels built on raw similarity inherit that confusion wholesale.

So the synthesis is this: pseudo-relevance labeling works not because the proxy is correct, but because a good proxy plus a quality filter beats a tiny pile of human labels. The interesting frontier the corpus hints at is *which* proxy to trust — a calibrated model's own uncertainty turns out to be a more reliable internal signal than external heuristics for deciding when retrieval even matters Can simple uncertainty estimates beat complex adaptive retrieval?, suggesting the best pseudo-labels may come from the model's self-knowledge rather than from similarity scores it was never designed to produce.