How can affordance become a primary retrieval signal instead of a filter?
This explores whether affordance — what a robot can actually *do* with an object — could drive retrieval from the start, rather than being bolted on as a re-ranking filter after visual similarity has already chosen the candidates.
This explores whether affordance — what a robot can actually *do* with an object — could drive retrieval from the start, rather than being bolted on as a re-ranking filter after visual similarity has already chosen the candidates. The corpus's anchor here is AffordanceRAG, which today keeps affordance in the *second* seat: it retrieves objects by visual similarity, then reranks by physical executability so the robot only acts on things it can actually grasp or use Can visual similarity alone guide robot object retrieval?. The catch with any filter-stage design is that it can only choose among what the first stage already surfaced — if similarity never retrieves the executable object, no amount of reranking recovers it. So 'affordance as primary signal' really means: can the *recall* step itself be task-grounded?
The most suggestive path in the collection is to retrieve through *description* rather than raw embedding similarity. SignRAG shows that turning an image into a natural-language description and then matching against a text-indexed database bridges a gap that direct visual embeddings miss Can describing images in text improve zero-shot recognition?. Affordance is fundamentally a verbal-relational property ('can be poured from', 'affords sitting'), so describing candidates in affordance terms and indexing on those descriptions would let the *first* retrieval pass already be about action, not appearance — collapsing the two stages into one.
A second route is to build affordance into the item's identity. TransRec found that pure-ID or pure-text identifiers each fail, but fusing numeric IDs, titles, and attributes into one structured identifier gives distinctiveness, semantics, and grounded generation at once Can item identifiers balance uniqueness and semantic meaning?. Read across to robotics, that's a template: an object identifier that carries affordance attributes as a first-class facet would make executability part of what's matched, not a score applied afterward.
There's also a reason to keep *some* of the two-stage shape, just relocated. Identity-sensitive matching argues that a cheap recall pass followed by a learned verifier operating on full token-interaction patterns catches 'structural near-misses' that compressed similarity vectors wave through Can verification separate structural near-misses from topical matches?. The lesson isn't 'filters are bad' — it's that the discriminating signal should run on rich structure, not a squeezed vector. Affordance is exactly the kind of structural property a downstream verifier handles well, which is why it landed in the filter slot to begin with. Making it primary means giving the recall stage access to that same structure.
The deeper move the corpus hints at: stop treating retrieval as a fixed pipeline at all. Aspect-aware recommendation retrieves differently depending on which aspect matters to the user, rather than running one generic similarity pass Can retrieval enhancement fix explainable recommendations for sparse users?. The analog for robots is an affordance-conditioned retriever: the *task* ('I need to pour') selects the retrieval lens, so similarity and executability stop being sequential stages and become a single query that already knows what 'relevant' means. That's the real shift from filter to signal — affordance stops being the gate at the end of the corridor and becomes the question you asked walking in.
Sources 5 notes
AffordanceRAG reranks visually retrieved objects by affordance scores, ensuring the robot selects only physically executable actions. This architectural shift from similarity to task-grounded ranking prevents plans that fail at execution time.
SignRAG demonstrates that describing an unknown image via vision-language model, then retrieving known designs from a text-indexed database, eliminates the need for recognition model training. Natural-language description bridges the visual-reference gap better than direct embedding similarity.
TransRec shows that combining numeric IDs, titles, and attributes into structured identifiers solves three problems simultaneously: distinctiveness from IDs, semantics from text, and generation grounding from structural constraints. Neither pure IDs nor pure text alone achieves all three.
A two-stage pipeline—pooled-cosine recall followed by a small Transformer verifier operating on token-token similarity maps—reliably rejects structural near-misses that MaxSim-style late interaction cannot. The verifier succeeds because it operates on full token interaction patterns rather than compressed vectors.
ERRA combines model-agnostic review retrieval with personalized aspect selection to address data sparsity that embedded methods cannot solve. Retrieval augmentation provides richer signal when user history is sparse, while aspect personalization ensures explanations match user context rather than generic defaults.