Do retrieval models actually follow natural language instructions?

LLMs follow long, complex instructions, and IR models increasingly use LLM backbones — yet nearly all retrievers still take only a query, with no instruction defining what relevance means for this task. FollowIR builds a benchmark from the TREC tradition (where human annotators receive narratives — detailed instructions — to judge relevance) by altering those annotator instructions and re-annotating, then measuring whether IR models adjust their relevance decisions accordingly. The finding: nearly all retrieval models do not follow instructions, with exceptions only for very large (3B+) or instruction-tuned LLMs not typically used for retrieval. But it is learnable — a training corpus teaches instruction-following, and FollowIR-7B improves on both standard retrieval metrics and instruction-following.

The keeper is that retrieval is stuck in an ad-hoc keyword paradigm while the rest of NLP moved to flexible instructions: relevance is treated as a fixed property of the query rather than something an instruction can redefine on the fly. Closing that gap would let users specify complex information needs in natural language.

This connects the vault's retrieval thread to instruction-following. It complements Can question features alone predict when to retrieve? (when to retrieve) by addressing what counts as relevant — both are limits of the query-only retrieval paradigm the RAG-gap note diagnoses.