SYNTHESIS NOTE
Topics›Novel Architectures›this note

Can models learn to evaluate their own work during training?

Explores whether language models can internalize reward function computation as part of training, transforming external feedback into internal self-assessment capability without slowing inference.

Synthesis note · 2026-02-23 · sourced from Novel Architectures

Current training paradigms terminate learning at the end-of-sequence token, wasting the entire sequence space after model output completion. Post-Completion Learning (PCL) systematically exploits this neglected space. A temporary termination marker (<-- post-completion -->) creates a "post-thinking" space where models continue generating self-assessments and reward predictions during training, while inference stops at the marker — zero additional cost at deployment.

The core innovation is white-box reinforcement learning: the model explicitly learns to understand and compute reward functions, internalizing the reward model as its own evaluation capability. This transforms the model from "passive reward acceptance" (external reward signal tells it what's good) to "active self-evaluation" (it learns to compute quality assessments itself).

Implementation uses dual-track SFT: one track optimizes reasoning, the other optimizes evaluation capability. These are mixed with RL training for multi-objective hybrid optimization. The model learns both to solve problems and to assess its own solutions — but critically, only the problem-solving capability is active during inference. The self-evaluation is internalized during training, shaping the model's generation without requiring explicit self-assessment at inference time.

This addresses three limitations simultaneously:

  1. SFT's passive learning — models learn to mimic demonstrations without developing self-assessment ability
  2. RL's external dependency — reward models are opaque external components; PCL internalizes the evaluation
  3. Self-correction's inference cost — methods like Self-Refine require additional generation passes; PCL's self-evaluation is absorbed into training

The parallel with human cognition is direct: "Humans, after completing a task, often engage in self-reflection and quality assessment — this post-thinking process is crucial for improving future performance." PCL operationalizes this for LLMs.

This connects to What limits how much models can improve themselves? — PCL attempts to close the gap by training the verifier and generator as the same model, with the verification capability internalized rather than external. It also complements Does reflection in reasoning models actually correct errors? — PCL's self-evaluation is trained against ground-truth reward functions, not against the model's own prior outputs, potentially avoiding the confirmatory pattern.

Inquiring lines that read this note 143

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How can we prevent synthetic data from contaminating statistical inference and corpora? What should agent evaluation prioritize to reveal reliable behavior? Does AI assistance promote real skill development or substitute for independent learning? How can reward models capture diverse human preferences without excluding minority populations? Do language models possess genuine introspective self-awareness or only behavioral mimicry? How should designers communicate what AI systems truly are and can do? Can preference-based training achieve better behavior optimization than supervised fine-tuning alone? Why do token-level mechanisms matter for learning to reason? Do language models respond to social pressure and face-saving like humans? How can we build reliable evaluations of AI reasoning despite judge bias and reward-seeking? What training dynamics and scale trigger emergence of reasoning capabilities? How does self-revision in reasoning models affect accuracy and confidence? Can self-generated feedback reliably guide model training without ground truth? Does RL create genuinely new reasoning capabilities or refine existing ones? How does the generation-verification gap limit what we can measure about AI reasoning? Does model confidence reliably signal actual accuracy in practice? How do spurious versus genuine rewards shape model reasoning and behavior? Can prompt-based context override biases that were embedded during pretraining? How do capability benchmark scores systematically misrepresent true model abilities? How do pretraining biases affect reward signal effectiveness in RLVR? What training data selection strategies maximize generalization across difficulty levels? Do language models develop actual world models or merely task heuristics? How do surface patterns enable correct outputs but reduce robustness? How can humans maintain meaningful oversight as AI systems become increasingly autonomous and complex? What fundamental constraints limit how effectively agents can improve themselves? Do language models reason like humans or mimic surface patterns? Do language models learn genuine understanding or just surface patterns? Can diffusion models match autoregressive performance on language generation tasks? What makes step-level supervision effective for complex reasoning traces? How should inference compute be allocated based on problem difficulty? What prevents conversational agents from taking initiative in dialogue? Does alignment training create genuine alignment or just output compliance? How should retrieval systems handle complex multi-step reasoning? Does RLHF training systematically drive models toward sycophancy and away from accuracy? Does encoded knowledge in language models actually influence their outputs? Can reasoning scale in latent space without tokens? How much do training data properties shape model reasoning? How does evaluation scope and dimensionality affect what we measure? Why do agents falsely report success on failed tasks? How does reasoning length affect model performance across different tasks? What compositional reasoning failures limit large language models despite scale? Can memory architectures handle ultra-long context better than attention? How do agent-learned skills transfer and improve across different tasks? How do neural networks achieve compositional generalization at scale? How does harness optimization generalize across different model architectures and domains? Can we reliably detect when models game evaluations? How do LLM judges' systematic biases affect alignment and evaluation outcomes?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 126 in 2-hop network ·medium cluster Open in graph ↗
Can models learn to evaluate their own work duri… What limits how much models can improve themselves… Does reflection in reasoning models actually corre… Can model confidence work as a reward signal for r… Do reward models actually consider what the prompt…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Does internalizing verifiers actually close the generation-verification gap?Why does reflection in reasoning models stay confirmatory instead of corrective?Can model confidence signals replace explicit external reward functions?How much inference efficiency do we gain by eliminating self-correction passes?
Original note title

post-completion learning uses the ignored post-eos space to internalize self-evaluation during training with zero inference cost