Can latent thought vectors scale language models beyond parameters?
Explores whether explicit latent thought vectors with dual-rate learning create new scaling dimensions independent of model size. This matters because it suggests alternatives to simply building larger models.
Latent-Thought Language Models (LTMs) propose a different scaling strategy than larger parameters or longer contexts: explicit latent thought vectors that follow a prior model in latent space and guide autoregressive token generation. This creates additional scaling dimensions — higher sample efficiency by increasing training compute per token, with further gains by trading model size for more inference steps.
Architecture. Latent thought vectors represent an abstract representation of the entire sequence, controlling the decoder's generation of each token. Training uses variational Bayes with a dual-rate process: fast learning of local variational parameters for the posterior distribution of latent vectors (adapting quickly to specific inputs) coupled with slow learning of global decoder parameters (gradually accumulating general knowledge).
Cognitive inspiration. The dual-rate scheme parallels established cognitive models:
- Declarative-procedural model (Ullman 2004): latent vectors and local parameters parallel declarative/episodic memory; global decoder parameters parallel procedural memory
- Fast-slow learning (Kumaran et al. 2016): fast episodic learning and slow schematic learning interplay
- Language of thought (Fodor 1975): latent thought vectors as "words" of an internal thought language
Scaling properties. LTMs demonstrate superior sample and parameter efficiency compared to conventional autoregressive models and discrete diffusion models. They significantly outperform on validation perplexity and zero-shot language modeling. Emergent few-shot in-context reasoning capabilities scale with both model size and latent size — providing two independent scaling dimensions.
The connection to existing latent reasoning approaches is important but distinct. Can models reason without generating visible thinking tokens? describes depth-recurrent architectures that iterate in latent space at inference time. LTMs use latent vectors differently — as sequence-level abstractions that guide token generation rather than per-token iterative computation. The dual-rate learning provides a training-time mechanism that depth-recurrence does not.
The Titans parallel is also notable: Can neural memory modules scale language models beyond attention limits? separates fast attention (short-term) from slow memory (long-term). LTMs separate fast local adaptation from slow global learning. Both architectures implement the fast-slow cognitive distinction but at different levels — Titans for memory, LTMs for generation.
Inquiring lines that read this note 64
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does chain-of-thought reasoning reveal genuine computation or imitate patterns? Why do embedding systems fail to capture task-relevant relationships? Can reasoning scale in latent space without tokens?- What makes internal embeddings useful as multimodal input for language model training?
- How does this compare to trained autoencoder approaches for thought sharing?
- Do latent sequence vectors outperform per-token latent iterative computation for reasoning?
- How does LatentQA differ from predefined concept steering like representation engineering?
- Can latent space represent reasoning dimensions that text cannot?
- Can continuous latent reasoning match discrete chain-of-thought without training modifications?
- Can language models generate plausible latent thoughts without human annotation?
- Why does recursion on latent state drive generalization better than hierarchy?
- What makes looped latent computation more efficient than scaling attention capacity?
- Can latent reasoning scale test-time compute without verbal tokens?
- How much explicit verbal signal must latent chains retain to perform well?
- Does latent manipulation outperform token-level prediction for efficiency?
- Can latent reasoning scale test-time compute without verbalized tokens or special training?
- Can autoregressive models be trained to produce more cataphoric text?
- Do diffusion language models learn differently than autoregressive models?
- Does scaling model size solve compositional generalization problems?
- Why do language models fail at iterative numerical optimization despite scale?
- Do pretrained language models carry reusable computational scaffolding for length handling?
- Why do language models plateau at constraint satisfaction regardless of scale?
- What is the comprehension-generation asymmetry in language models?
- How do parameter scaling and latent vectors interact in language models?
- Can latent recurrence and energy minimization both escape the same computational depth constraints?
- Can latent recurrence overcome the trainability costs of depth?
- Why does recursion on latent states improve generalization more than scale?
- Do representations in models causally influence text generation?
- Do language models show functional splits between conscious and automatic processing?
- How does dual-rate learning separate episodic and procedural memory in neural networks?
- Can fast-slow separation improve both memory and generation in language models?
- Do long-term memory modules outperform consolidation into fast weights?
- What are the concrete efficiency gains of linear-attention state-space models?
- Can fixed-size latent states losslessly store arbitrary input context?
- Does the prediction unit shape what language models actually learn?
- How can language models extract more value from fewer demonstrations?
- What non-parametric methods could replace latent factors for inductive learning?
- How do encode-decode contractive biases create stable attractors in latent space?
- Why does scaling data and model size improve compositional generalization?
- Why is latent-level prediction more sample-efficient than token-level prediction?
- How do latents at the same hierarchy level become more correlated than tokens?
- What prevents representation collapse in latent-prediction world models like JEPA?
- What scaling laws govern the compute efficiency of latent prediction versus token prediction?
- How do static embeddings and contextualized representations divide semantic labor?
- Why does textual chain-of-thought avoid the representational drift problem automatically?
- How does training distribution shape what language models understand best?
- What are the scaling law differences between vision and language learning?
- Why should scaling laws be understood as properties of data distribution rather than training in general?
- How does concept vocabulary size affect the efficiency gains from joint training?
- Do base models contain latent reasoning that minimal training can unlock?
- How much training data is truly necessary to unlock latent model reasoning?
- Can minimal training signals unlock latent reasoning capability in base models?
- What latent reasoning capability do base models already possess before training?
- Can language models execute iterative numerical methods in latent space?
- What empirical evidence supports the Learning Law on real language models?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can models reason without generating visible thinking tokens?
Explores whether intermediate reasoning must be verbalized as text tokens, or if models can think in hidden continuous space. Challenges a foundational assumption about how language models scale their reasoning capabilities.
different latent approach: per-token iterative computation vs sequence-level latent vectors
-
Can neural memory modules scale language models beyond attention limits?
Can separating short-term attention from adaptive long-term memory allow models to efficiently handle context windows exceeding 2M tokens while maintaining competitive performance?
Titans implements fast-slow at memory level; LTMs implement it at generation level
-
Can inference compute replace scaling up model size?
Explores whether smaller models given more thinking time during inference can match larger models. Matters because it reshapes deployment economics and compute allocation strategies.
LTMs demonstrate this: model size can be traded for inference steps
-
Can computational power accelerate scientific discovery itself?
Does the pace of research breakthroughs scale with computing resources, like model performance does? ASI-ARCH tested this by running thousands of autonomous experiments to discover neural architectures.
LTMs provide new dimensions for architecture search
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Scalable Language Models with Posterior Inference of Latent Thought Vectors
- Reasoning to Learn from Latent Thoughts
- Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
- Learn from your own latents and not from tokens: A sample-complexity theory
- Scaling Laws for Neural Language Models
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
Original note title
latent-thought language models introduce additional scaling dimensions beyond parameters by incorporating explicit latent thought vectors with dual-rate learning