Can attention mechanisms improve on Wide & Deep's static feature crosses?
This reads the question as: does attention's dynamic, context-dependent weighting beat the hand-engineered, fixed feature combinations of the classic Wide & Deep recommender — and the honest answer is that this corpus speaks to attention's core advantage and its limits, but not to recommendation systems directly.
This explores whether attention mechanisms can outperform Wide & Deep's static feature crosses — where Wide & Deep relies on manually specified, fixed combinations of features and attention instead learns which signals to weight on the fly. Up front: the collection doesn't contain recommendation-system papers on Wide & Deep or feature-cross engineering specifically, so there's no direct head-to-head here. What the corpus does give you is a sharp picture of *what attention buys you over any static structure* — and, just as usefully, where that advantage frays.
The central appeal of attention over static crosses is that it activates differently depending on context rather than being baked in ahead of time. The clearest evidence is that only a tiny slice of attention — less than 5% of heads — does the heavy lifting of pulling the right item out of a long context, and those 'retrieval heads' switch on dynamically based on what's actually present, not on a fixed wiring diagram What mechanism enables models to retrieve from long context?. That's exactly the property a static feature cross lacks: a hand-built cross fires the same way regardless of context, while attention re-decides per input. Translate that to recommendation and the intuition is that attention can discover interactions a human never enumerated.
But the corpus also delivers the counter-story, and this is where it earns its keep. Soft attention is not a neutral learner of importance — it's structurally biased toward whatever is repeated or prominent, regardless of whether it's relevant, creating self-reinforcing feedback loops Does transformer attention architecture inherently favor repeated content?. A static feature cross is dumb but predictable; attention is flexible but carries its own systematic distortions that you'd have to correct for (the same note describes regenerating context to strip irrelevant material as a fix). So 'dynamic beats static' isn't a free lunch — you trade one set of failure modes for another.
The cost dimension matters too. Attention's flexibility historically came with a quadratic price, and the collection shows that's no longer the clean tradeoff people assume: larger sparse-attention models beat smaller dense ones at equal compute, making sparsity a Pareto improvement rather than a quality sacrifice Does sparse attention trade off quality for speed?. And treating the attention-to-MLP ratio as a tunable architectural variable yields real efficiency and accuracy gains Can architecture choices improve inference efficiency without sacrificing accuracy?. The lesson for anyone weighing attention against static crosses: 'how much attention, applied how sparsely' is itself a design knob, not an all-or-nothing switch.
One lateral thread worth your time: some problems attention handles badly aren't capacity problems at all but missing training signal — models follow 'what to attend to' instructions but never learn 'what to ignore' until explicitly taught Why do language models engage with conversational distractors?. That reframes the original question. The interesting comparison may not be 'attention vs. static crosses' but 'can attention learn which feature interactions to *suppress*' — something a fixed cross can't express and an untrained attention layer won't do on its own. If you want the recommendation-specific evidence, this corpus won't supply it; what it gives you is a clear-eyed sense of when attention's adaptivity is a genuine win and when it's just a more expensive way to be wrong.
Sources 5 notes
Less than 5% of attention heads across all model families function as retrieval heads, are intrinsic to short-context models, dynamically activate by context, and are causally necessary for factuality. Pruning them causes hallucination despite information being present in context.
Transformer soft attention systematically over-weights repeated and context-prominent tokens regardless of relevance, creating a positive feedback loop that amplifies opinions and framing before RLHF acts. System 2 Attention—regenerating context to remove irrelevant material—can interrupt this mechanism.
The Sparse Frontier benchmark shows that at equivalent compute cost, larger sparse-attention models outperform smaller dense models on long-context tasks. Sparsity lets you train bigger models within the same budget, making it Pareto-improving rather than a pure trade-off.
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
Fine-tuning on just 1,080 synthetic dialogues with distractor turns significantly improves topic resilience, revealing that the gap is not model capacity but absent training signal. Models learn to follow what-to-do instructions but not what-to-ignore instructions.