How do you partition LLM experts by domain versus by time?
This explores two different ways to carve up 'expertise' inside an LLM — by subject area (law, medicine, code) versus by time period (recent vs. historical) — and what the corpus knows about each as an engineering and a failure-mode problem.
This reads the question as contrasting two axes for splitting up what a model knows: domain (a subject-matter slice) and time (an era slice). They turn out to be very different kinds of problem. Domain partitioning is something you can deliberately engineer; time partitioning is mostly something that happens to you, as an accident of what your training data over-represented.
The cleanest example of *deliberate* domain partitioning is Branch-Train-MiX Can asynchronous expert training beat synchronized distributed LLM training?, which trains separate domain experts in parallel — no synchronization between them — then stitches their feed-forward layers back together as mixture-of-experts modules and learns a router to pick which expert handles each token. The appeal is practical: experts can be grown independently and merged, beating the overhead of synchronized distributed training. But carving by domain has a cost the corpus is blunt about. Domain specialization buys depth at the price of a 'capability cliff' How do you build domain expertise into general AI models? — over-specialized models fail catastrophically the moment a query steps outside their lane, while under-specialized ones produce confident nonsense in high-stakes settings. And the adaptation techniques you'd use to build those experts each have a narrow sweet spot with hidden costs: gains in domain performance often come with quiet degradation in reasoning faithfulness and the ability to transfer skills elsewhere How do domain training techniques actually reshape model behavior?.
Time partitioning is the stranger axis, because the model already partitions itself by time whether you want it to or not. Legal-reasoning benchmarks show clear era sensitivity: models do measurably worse on historical Supreme Court cases than modern ones, and the root cause is simply that recent cases are over-represented in the training corpus, leaving older precedent with shallower internal representations Why do language models struggle with historical legal cases?. So 'partition by time' isn't usually a design choice — it's a bias you inherit, where recency in the data becomes competence in the model.
Where time *is* handled deliberately, the corpus points toward workflow design rather than separate experts. In forecasting, the trick isn't a time-specialized model but a workflow that separates numerical reasoning from contextual reasoning — split those two and the model's latent forecasting ability surfaces; keep them in one monolithic prompt and it stays hidden Can LLMs actually forecast time series better than we think?. And in personalization, the temporal slice that matters most is a user's *history of outputs* rather than their queries — past outputs alone match or beat full profiles, because what carries over time is style and preference, not semantic content Do user outputs outperform inputs for LLM personalization?.
The thing worth taking away: domain partitioning is an architecture you build (parallel experts, routers, merges), with a known depth-versus-breadth tradeoff; time partitioning is mostly a bias you mitigate, and when you do handle it on purpose, you do it by reshaping the *workflow* — separating numeric from contextual reasoning, weighting recent over historical signal — rather than by minting a 'time expert.' Same word, 'partition,' but one is a wiring diagram and the other is a data-distribution problem.
Sources 6 notes
Branch-Train-MiX trains domain experts in parallel without synchronization overhead, merges their feed-forward parameters as MoE experts, and learns token-level routing, achieving better accuracy-efficiency tradeoffs than synchronized training or routing-free merging.
Research shows that over-specialized models fail catastrophically outside their domain, while under-specialized ones produce confident-sounding errors in high-stakes settings. The tension is structural, not solvable through technique alone.
Research shows every adaptation method—from parameter-efficient tuning to knowledge graph curricula—has optimal conditions tied to specific domains. The key finding: visible benefits like performance gains often come with hidden degradation in reasoning faithfulness, capability transfer, and format flexibility.
Supreme Court overruling benchmark (236 pairs) reveals era sensitivity: models perform worse on historical cases than modern ones. Root cause is training corpus over-representation of recent cases, creating shallower representations of older precedent.
LLMs have stronger intrinsic forecasting ability than recognized, but only when workflows separate numerical reasoning from contextual reasoning. Monolithic prompting obscures this capability; structured decomposition surfaces it.
Research shows that user profiles built from outputs alone match or exceed performance of complete profiles across multiple tasks, while input-only profiles degrade performance. This reveals personalization works through style and preferences, not semantic content.