What scaling exponent would audio or other modalities require in a truly multimodal system?
This explores whether audio (and other modalities) would need its own distinct scaling exponent in a single multimodal model — and what the corpus's work on vision-vs-language scaling implies about that question.
This explores whether audio would demand its own scaling exponent in a truly multimodal system, the way vision already does relative to language. The honest answer up front: no paper here measures audio's exponent directly. But the corpus is unusually clear about the *shape* of the answer — and it's more interesting than a single number.
The foundational finding is that vision and language don't just differ in degree, they scale on fundamentally different curves Why do vision and language scale so differently?. Under IsoFLOP analysis, language sits near the Chinchilla compute-data balance while vision is far more data-hungry. That's the key reframe for your question: a 'scaling exponent' isn't a property of a model, it's a property of how much information a modality packs per token and how that information density grows with scale. So the real question becomes — is audio more like language (compact, near-balanced) or more like vision (data-hungry)? Given that raw audio is a dense continuous signal closer to pixels than to discrete words, the corpus's logic predicts audio would land in its own data-hungry regime, distinct from both.
The more useful insight is that the field has largely stopped treating the exponent as a fixed obstacle. Modality competition turns out to be architectural, not inherent Can we solve modality competition through architectural design? — it comes from rigid dense capacity allocation and distributional shift, not from modalities being incompatible. Mixture-of-Experts dissolves the problem by allocating capacity per token and routing to modality-specific experts, which lets language and vision coexist at their own optimal points inside one model. Extend that and the answer to 'what exponent would audio need?' is: it doesn't need to share one. A truly multimodal system gives each modality its own effective scaling regime through routing, rather than forcing them onto a common curve where one starves the other.
There's a deeper reason audio resists language's exponent at all. Speech self-supervised models don't learn discrete phonetic categories — they infer the continuous causal physics of how the vocal tract produces sound Do speech models learn language-specific sounds or universal physics?. That's the same gap text models suffer from the other direction: text is a lossy abstraction that strips the physics, geometry, and causality of reality Are text-only language models fundamentally limited by abstraction?. Audio carries articulatory and temporal dynamics that have no compact symbolic form, which is exactly why its information-per-token — and therefore its scaling behavior — should differ from language's.
Two caveats worth carrying. First, optimizing one modality's objective can actively hurt another: verbose chain-of-thought helps text reasoning but degrades fine-grained visual perception because it trains the wrong bottleneck Does verbose chain-of-thought actually help multimodal perception tasks? — a warning that a single shared training signal mis-serves modalities with different exponents. Second, in practice multimodal systems often sidestep joint scaling entirely by synchronizing modalities at inference: temporal-aware retrieval keeps visual, audio, and subtitle evidence aligned at the same moments without retraining How can video retrieval handle multiple modalities at different times?. So the field's working answer is less 'find audio's exponent' and more 'build architectures where each modality gets to live on its own.'
Sources 6 notes
IsoFLOP analysis shows language scales near Chinchilla balance while vision is significantly more data-hungry. Sparse MoE shifts language toward the data-hungry regime, enabling both modalities to coexist optimally in one model by routing tokens to modality-specific experts.
Modality competition arises from caption distributional shift and rigid dense capacity allocation, not from vision and language being fundamentally incompatible. Mixture of Experts resolves the architectural bottleneck by allocating capacity per token, enabling modalities to coexist without competing.
Self-supervised speech models learn the language-agnostic physics of how the vocal tract produces acoustics, not language-specific phonetic categories. This explains their multilingual transfer and predicts their downstream task performance better than phonetic probing.
Text strips the physics, geometry, and causality present in reality, forcing language models to manipulate symbols without grounding in their source dynamics. This creates predictable failure modes in physical, geometric, and causal reasoning that multimodal training could address.
Long rationales and text-token RL help reasoning but hurt fine-grained perception tasks because the actual bottleneck is visual attention allocation, not verbalization. Standard CoT optimization trains the wrong policy target.
TV-RAG ranks retrieved text by temporal proximity and selects key frames via entropy-based sampling, not uniform stride. This keeps visual, audio, and subtitle evidence synchronized at the same moments, enabling video LLMs to reason across modalities without retraining.