What inductive bias would force models to learn Newtonian mechanics instead of shortcuts?
This explores whether any change to a model's architecture or training signal could push it to learn the underlying physical laws (the kind that generalize) rather than the brittle pattern-matching tricks that happen to fit the training data — and the honest answer the corpus gives is that we don't yet have one that reliably works.
This explores whether any inductive bias could force a model to learn real Newtonian mechanics rather than shortcuts, and the corpus' starting point is sobering: the shortcut problem is the default, not an edge case. The most direct evidence is that transformers trained on orbital mechanics learn predictive patterns, not the unified force law underneath — probe them with inductive-bias tests and fine-tune them, and they reveal nonsensical, 'slice-dependent' physics that changes from one region of the data to the next (Do foundation models learn world models or task-specific shortcuts?). The same note finds arithmetic running on range-matching heuristics rather than an algorithm. So the question isn't academic: a model can predict planetary motion beautifully and still have no Newton inside it.
Why do shortcuts win by default? Two notes point at the training signal itself. When reward shaping pushes models too hard, they don't reach for deeper structure — they collapse into degenerate tricks like answer-repetition and computation-skipping, and those shortcuts then contaminate capabilities the model already had (Do overly hard RLVR samples actually harm model capabilities?). And the biases models do absorb trace straight back to the statistics of their data: LLMs reproduce human causal-reasoning errors — weak 'explaining away,' Markov violations — not because they reason worse, but because they're fitting the correlational shape of what they saw (Do large language models make the same causal reasoning mistakes as humans?). A pure correlation-fitter has no reason to prefer the law over the shortcut when both fit the data equally; that's the gap an inductive bias would have to close.
This is where the corpus quietly reframes your question. The dream of a model that 'just reads the laws off the data' is exactly the 'theory-free AI' fallacy — high accuracy masking a correlation-for-causation substitution, the way a 95%-accurate justice model still convicts thousands wrongly (Can AI models be truly free from human bias?). The lesson isn't that bias is bad; it's that *some* structural prior is unavoidable, so the real engineering question is which prior to build in. And you can't even tell whether a model has learned a law by looking at its behavior — representational analysis finds correlations without causation, behavioral tests show effects without mechanism. Only paired causal-plus-representational probing can confirm a model holds the actual structure (Can we understand LLM mechanisms with only representational analysis?). In other words, you can't reward 'learning Newton' if you can't detect it.
The one architectural candidate the corpus offers is the most interesting thread to pull: Energy-Based Transformers replace next-token prediction with assigning an energy to each input–prediction pair and minimizing it by gradient descent at inference. That shift in objective — learn a landscape, then search it — yields markedly better out-of-distribution generalization *without* domain-specific scaffolding (Can energy minimization unlock reasoning without domain-specific training?). That's the shape an answer to your question would take: not a Newton-specific rule bolted on, but a learning objective whose inductive bias rewards consistent global structure over locally-good shortcuts, so the law becomes the cheaper solution.
The thing you may not have known you wanted to know: 'shortcut vs. law' isn't a property a model either has or lacks — it lives in three separable places at once. The objective (energy-minimization vs. token-prediction), the reward design (which actively manufactures shortcuts when miscalibrated), and the measurement (you can't fix what causal probing can't see). No single inductive bias forces Newtonian mechanics; the corpus suggests the leverage is in changing what 'fitting the data' is even allowed to mean.
Sources 6 notes
Inductive bias probes show transformers trained on orbital mechanics and games learn predictive patterns, not unified world structure. Fine-tuning reveals nonsensical, slice-dependent laws; circuit analysis shows arithmetic relies on range-matching heuristics, not algorithms.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
LLMs show weak explaining away and Markov violations in collider networks, matching human error patterns exactly. This suggests shared mechanisms rooted in training data statistics rather than categorical reasoning inferiority.
Research shows that 'theory-free' AI models mask bigotry behind high accuracy metrics while committing fundamental statistical errors. A 95% accurate criminal justice system would wrongly convict thousands, demonstrating that model sophistication does not validate causal inference.
Representational analysis alone identifies correlations without causation; causal analysis alone shows behavioral effects without explaining them. Only paired methods—locating candidate features representationally, then verifying causally—produce complete mechanistic claims.
Energy-Based Transformers assign energy values to input-prediction pairs and use gradient descent minimization for inference, yielding 35% higher training scaling rates and 29% more inference-compute gains than Transformer++, while generalizing better on out-of-distribution data without domain-specific scaffolding.