LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization

Large language models (LLMs) have emerged as powerful tools capable of accomplishing a broad spectrum of tasks. Their abilities span numerous areas, and one area where they have made a significant impact is in the domain of code generation. Here, we propose using the coding abilities of LLMs to introduce meaningful variations to code defining neural networks. Meanwhile, Quality-Diversity (QD) algorithms are known to discover diverse and robust solutions. By merging the code-generating abilities of LLMs with the diversity and robustness of QD solutions, we introduce LLMatic, a Neural Architecture Search (NAS) algorithm. While LLMs struggle to conduct NAS directly through prompts, LLMatic uses a procedural approach, leveraging QD for prompts and network architecture to create diverse and high-performing networks. We test LLMatic on the CIFAR-10 and NAS-bench-201 benchmarks, demonstrating that it can produce competitive networks while evaluating just 2, 000 candidates, even without prior knowledge of the benchmark domain or exposure to any previous top-performing models for the benchmark. The open-sourced code is available in https://github.com/umair-nasir14/LLMatic.

Introduction. A major challenge in deep learning is designing good neural network architectures. Neural Architecture Search (NAS) is the generic term for various approaches to automating this design process [50]. The idea is to formulate an objective, such as maximum accuracy on a classification problem with a given budget of parameters and training cycles, and cast the problem as a search for the architecture that maximizes the objective. Every test consists of training the candidate network architecture using some form of gradient descent on the chosen benchmark dataset to measure its performance. This typically means that many thousands of architectures are tested and discarded in the process. Two common algorithmic approaches to NAS are reinforcement learning and evolutionary computation. Reinforcement learning approaches to NAS [20] train a controller (typically another neural network) that outputs network architectures; these network architectures are tested and their performance is used as a reward signal.

Discussion / Conclusion. To conclude, we present LLMatic: a novel neural architecture search (NAS) algorithm that harnesses the power of large language models (LLMs) and Quality-Diversity (QD) optimization algorithms. LLMatic successfully finds competitive networks that are diverse in architecture. We show empirically that LLMatic can find more than 20 competitive networks in CIFAR-10 and near-to-optimal networks in NAS-bench-201, using only 2000 evaluations. LLMatic decreases the max population size per generation to only 100. LLMatic achieves this while relying on a 6.1B parameter language model. Furthermore, we show that each component in LLMatic is necessary. We conducted an extensive ablation study and found that LLMatic finds the network with the best accuracy among other variants. LLMatic achieves this with many constraints in hand. Firstly, we use CodeGen-6.1B code generation LLM, which is a smaller language model when compared to existing LLMs.