Can deterministic computation actually create new information in data?
This explores whether a fixed, repeatable computation can generate genuinely new information — or whether it only rearranges what was already present in the inputs, the model, or the observer.
This reads the question as a deep one: not 'can a program produce new output' (obviously yes) but 'can deterministic computation create information that wasn't already latent in its inputs?' The corpus leans toward no — but in a way that's more interesting than a flat denial, because it relocates where 'new information' actually comes from.
The sharpest anchor is the argument that computation presupposes an experiencing mapmaker Can computation arise without a conscious mapmaker?. Its claim is that no amount of added algorithmic complexity can manufacture meaning; a conscious agent who carves continuous physics into discrete symbols has to logically precede the computation. On that view, deterministic computation never originates information — it only transforms symbols whose significance was assigned beforehand. A complementary, more formal version of this shows up in epiplexity What can a bounded observer actually learn from data?, which defines 'information' as the structure a computationally bounded observer can actually extract from data. Information here isn't a property of the bits alone — it's relative to who's reading them. That reframes the question: deterministic computation can surface structure an observer couldn't otherwise reach, even if it adds no entropy.
The determinism-specific evidence is blunt. Zero-temperature, fixed-seed LLM runs Does setting temperature to zero actually make LLM outputs reliable? reproduce the same output every time — but that output is still just one draw from a probability distribution. Determinism buys you repeatability, not new content; consistency is not the same as having learned or generated anything additional. And when researchers add randomness to recursive reasoners, naive stochasticity yields nothing either Does adding randomness to recursive models actually help reasoning?. The gains from stochastic latent reasoning Can stochastic latent reasoning help models explore multiple solutions? come not from injecting noise but from a variational framework that learns where to branch. So neither pure determinism nor pure randomness is the source — structure is.
The most provocative counterpoint comes from systems that look like they create information from nothing. Asymmetric self-play Can language models improve themselves without any external training data? lets a model improve with no external data at all, a proposer and solver bootstrapping a curriculum between them. Seedless and instance-seed synthetic data methods Can we generate synthetic data without any seed examples? Can synthetic data replace seed examples in task generation? generate training data with no seed examples. These feel like information creation — but read against the corpus, they're better understood as redistribution: the model's existing latent knowledge is recombined, decomposed taxonomically, or pressure-tested into explicit form. No new entropy enters; what changes is which structure becomes extractable, exactly the epiplexity move. The same theme runs through prompting being Turing-complete Can a single transformer become universally programmable through prompts?: a fixed transformer can compute any computable function, but the function's information was always specifiable — the prompt selects it, it doesn't invent it.
So the synthesis: deterministic computation doesn't create information in the strong sense — it doesn't add entropy or originate meaning. What it does, and what most of these systems exploit, is make latent structure newly legible to a bounded observer. The interesting frontier isn't 'creation from nothing' but reuse and reconstruction — see memory-amortized inference Can cognition work by reusing memory instead of recomputing?, which casts intelligence itself as navigating and reusing prior inference rather than recomputing. The thing you might not have known you wanted to know: the deepest accounts here don't locate novelty in the computation at all, but in the observer who can finally read what was always there.
Sources 10 notes
Computational systems depend on a conscious mapmaker who alphabetizes continuous physics into discrete symbols. No increase in algorithmic complexity can generate this agent; it must logically precede the computation it makes possible.
Epiplexity formalizes the structural information a computationally bounded observer can extract from data, separating learnable regularity from time-bounded entropy. This task-free measure correlates with out-of-distribution generalization and explains why some datasets enable broader transfer than others.
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.
GRAM's ablations show naive stochasticity added to existing recursive models yields no improvement. Gains come specifically from amortized variational inference, which couples sampling to a principled generative objective and learns where to branch rather than injecting undirected noise.
GRAM replaces deterministic latent updates with stochastic sampling, enabling models to represent distributions over solutions rather than single predictions. This allows handling of ambiguous problems and multiple valid strategies that deterministic designs cannot represent.
SQLM uses a proposer-solver framework where the proposer generates calibrated problems and the solver learns via majority-vote verification. Both agents improve through RL alone, creating an automatic curriculum that scales without human labels or ground-truth answers.
Simula separates global coverage from local diversity, using taxonomy construction for coverage and agentic refinement for complexity. This architecture makes all three desiderata—quality, diversity, complexity—controllable simultaneously without requiring seed data.
TarGEN generates synthetic data using atomic task elements (instance seeds) instead of full input-output examples, achieving 1-3 point improvements on SuperGLUE tasks. The approach works by constraining label generation after seeding inputs, enabling data creation for domains with no prior examples.
Research proves a single finite-size transformer exists that can compute any computable function given the right prompt, achieving complexity bounds nearly matching unbounded models. However, standard training rarely produces models that learn to implement arbitrary programs this way.
Memory-Amortized Inference proposes intelligence arises from structured reuse of prior inference paths over topological memory, inverting RL's reward-forward logic into cause-backward reconstruction. This duality explains energy efficiency and suggests memory trajectories form the substrate of adaptive thought.