Can reinforcement learning improve models during general pretraining?

RLVR's reach is capped by a data-wall: it needs domain-specific verifiers to label samples, and RLHF needs reward models and can only train limited steps before reward hacking. PretrainZero attacks the wall by moving RL into pretraining on a general corpus, with two characteristics. Active pretraining: a unified reasoning policy actively identifies reasonable, informative content from the corpus and reasons to predict it — mimicking human active learning rather than passively predicting every token. Self-supervised: no verifiable labels, no pretrained reward model, no SFT — it pretrains reasoners (3–30B) directly on Wikipedia via RL, breaking the verification data-wall for general reasoning.

The keeper finding is that even Wikipedia — already seen during base pretraining — yields further gains under reinforcement active learning, beating continued pretraining, SFT, and random/entropy-based reinforcement pretraining. The lever is the active selection of what to reinforce, not new data.

This extends the reinforcement-pretraining family along the active-learning axis. It is the active-selection sibling of Can next-token prediction become a reasoning task with RL? (RPT, which reframes every next-token as a verifiable reasoning reward) — PretrainZero adds that which content to reinforce should itself be chosen for informativeness and not-yet-mastery, and it complements Can chain-of-thought reasoning be learned during pretraining itself? on the information-gain motive.