Do KANs maintain their advantages in deep architectures and large-scale training?

Straight answer first: nothing in this library studies KANs directly, so I can't tell you whether their function-approximation advantage holds up deep and large — that's a gap worth flagging rather than papering over. What the corpus does have is a recurring story about *which* architectural advantages scale and which are artifacts of being small, and that story is the useful thing to take away.

The collection's clearest signal is that depth and scale don't reward architectures uniformly — they reward the ones whose mechanism is *compositional*. Does depth matter more than width for tiny language models? finds that for sub-billion-parameter models, going deep-and-thin beats going wide, because layers compose abstract concepts rather than just adding capacity. The advantage isn't the depth per se; it's that depth lets a good primitive stack. That reframes your KAN question: a learnable-activation network will likely keep its edge at depth only if its spline-based units compose cleanly layer over layer, and lose it if the gains came from overfitting a shallow function.

Several notes push back on the assumption that scale is what generates capability at all. A 7M-parameter recursive network out-generalizes billion-parameter models on hard puzzles (Can tiny recursive networks outperform massive language models?), and a 27M-parameter hierarchical model clears reasoning ceilings that fixed-depth transformers can't (Can recurrent hierarchies achieve reasoning that transformers cannot?). Both make the same point an alternative architecture like KAN implicitly bets on: the *right structure* can beat brute parameter count. The catch is that these wins came from recursion and effective depth — structural mechanisms — not from a novel unit being intrinsically better. An architecture has to earn its scaling story mechanistically.

There's also encouraging evidence that good structure becomes *more* reliable with scale, not less. Do neural networks naturally learn modular compositional structure? shows that pretraining sharpens modular structure rather than dissolving it — the bigger and more trained the model, the more consistent its decomposition into clean subnetworks. If a KAN's edge is genuinely about cleaner functional decomposition, this is the pattern you'd hope to see it follow. And Can neural memory modules scale language models beyond attention limits? is a case study in an alternative-architecture advantage that *does* survive scale-up — neural memory that holds its benefit out to 2M-token contexts where the standard mechanism's costs explode.

The thing you didn't know you wanted to know: the corpus suggests the real test for any alternative architecture isn't "does it work in a small demo" but "is its advantage a *composition* property or a *fitting* property." Composition properties (depth stacking, modular decomposition, recursion, separated memory) tend to strengthen with scale; fitting advantages tend to wash out. To answer the KAN question properly you'd want a note benchmarking learnable-activation networks at depth — and that note isn't here yet.