NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

Abstract We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level autoregressive generation. NCP-ArchPreview builds a latent space by constructing a product-quantized concept vocabulary directly from its hidden states, and subsequently learns to predict future concepts via a dedicated Concept Module. These predicted concepts are then fed back to the token level to guide subsequent generation, with NTP and NCP trained jointly end-to-end. We scale this architecture to 8.9B parameters and train it on 5.73T tokens from the Dolma-3 dataset, marking the largest demonstration of a latentspace language model to date. Remarkably, by consuming only 51.3% of the total training tokens, NCP-ArchPreview achieves the final pretraining loss of OLMo-3-7B. Following full pretraining, it outperforms OLMo-3-7B by 2.45 points on the downstream macro-average, including a notable 5.99- point gain on GSM8K.

Introduction. Recent progress in generative AI has demonstrated that the representation space in which a model learns can matter as much as its parameter scale. In visual synthesis, latent diffusion shifts generation from raw pixels into compact continuous representations, dramatically boosting modeling efficiency and scalability [Blattmann et al., 2023, Rombach et al., 2022]. Modern language models similarly induce high-level abstractions, such as semantic concepts and latent world representations, in their hidden states [Gurnee and Tegmark, 2024, Li et al., 2023, Park et al., 2024]. Under standard Next Token Prediction (NTP), however, these abstractions arise purely as an indirect byproduct: supervision is strictly confined to granular tokens, lacking explicit objectives that guide how semantic structure unfolds across multi-token spans. To bridge this gap, we introduce NCP-ArchPreview, a novel latent-space foundation model architecture that incorporates the direct prediction of discrete concepts spanning over multiple tokens into the pretraining stage.

Discussion / Conclusion. This work establishes that latent representations learned by a language model can serve as first-class prediction targets at trillion-token scale. NCP-ArchPreview turns this idea into an 8.9B-parameter architecture that jointly models tokens and learned discrete concepts while preserving standard autoregressive generation. After pre-training on 5.73T tokens, it reaches the final loss of OLMo-3-7B with only 51.3% of its training tokens and improves the downstream macro-average by 2.45 points, including a 5.99-point gain on GSM8K. Controlled experiments show complementary gains from the latent architecture, hierarchical routing, and NCP; the complete model approaches a parameteraligned 40-layer Transformer with only 85% of its computation, while scaling-law experiments indicate a 1.74× improvement in compute efficiency.