Which code verification tasks still require execution instead of reasoning?
This explores where verifying code can now be done by reasoning about it (structured 'thinking' instead of running it), and where actually executing the code is still the only reliable check.
This reads the question as a frontier-mapping problem: the corpus has been steadily moving tasks out of the "must execute" column and into the "can reason" column — so the interesting answer is which tasks have resisted that move. The headline result is that a lot less requires execution than you'd expect. Semi-formal reasoning — natural-language templates that force an agent to lay out premises, trace each code path, and check evidence — hits 93% accuracy verifying whether two patches are equivalent, crossing the reliability bar needed to use it as a training reward signal Can structured reasoning replace code execution for RL rewards?. Those templates work by importing the *discipline* of formal verification (no skipped cases, no unsupported claims) without the symbolic machinery, catching subtle bugs like function shadowing that free-form thinking sails past Can structured templates replace formal verification for code reasoning? Can structured templates make code reasoning more reliable than free-form thinking?. So patch equivalence, fault localization, and policy-compliance checking are increasingly reasoning tasks, not execution tasks.
Sources 8 notes
Semi-formal reasoning templates enable execution-free patch equivalence verification at 93% accuracy on real agent code, crossing the reliability threshold needed for RL reward signals. This makes execution-free verification viable for certain task classes like fault localization and code reasoning.
Semi-formal reasoning using natural-language templates enforces the discipline of formal methods without formalizing language semantics. Templates prevent case-skipping, unsupported claims, and confirmation bias—capturing the verification benefits of formalism through forced completeness scaffolding rather than symbolic rigor.
Semi-formal templates requiring explicit premises, code-path traces, and evidence checks improved patch equivalence accuracy from 78% to 88%, catching cases like function shadowing that free-form reasoning missed. Templates act as completeness certificates without formal verification.
Models confined to text-only generation cannot execute multi-step procedures at scale, even when they know the underlying algorithm. Tool-enabled models solve problems beyond the supposed reasoning cliff, suggesting the bottleneck is procedural execution bandwidth.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
DGM replaces formal proofs with empirical benchmarking and maintains an evolutionary archive of agent variants, achieving 2.5× improvement on SWE-bench and 2.2× on Polyglot by discovering capabilities like better code editing and context management.
Research shows code uniquely enables agents to externalize reasoning, execute policies, model environments, and verify progress through its simultaneous executability, inspectability, and statefulness across task steps.
Reliability for long-trace reasoning comes from checking intermediate states and policy compliance during generation, not from scoring final outputs. Adding intermediate verification raised task success from 32% to 87% because most failures are process violations, not wrong answers.