Automatic Prompt Optimization with "Gradient Descent" and Beam Search

Large Language Models (LLMs) have shown impressive performance as general purpose agents, but their abilities remain highly dependent on prompts which are hand written with onerous trial-and-error effort. We propose a simple and nonparametric solution to this problem, Prompt Optimization with Textual Gradients (ProTeGi), which is inspired by numerical gradient descent to automatically improve prompts, assuming access to training data and an LLM API. The algorithm uses minibatches of data to form natural language “gradients” that criticize the current prompt, much like how numerical gradients point in the direction of error ascent. The natural language gradients are then “propagated” into the prompt by editing the prompt in the opposite semantic direction of the gradient. These gradient descent steps are guided by a beam search and bandit selection procedure which significantly improves algorithmic efficiency. Preliminary results across three benchmark NLP tasks and the novel problem of LLM jailbreak detection suggest that Automatic Prompt Optimization can outperform prior prompt editing techniques and improve an initial prompt’s performance by up to 31%, by using data to rewrite vague task descriptions into more precise annotation instructions.1

Introduction. Large Language Models (LLMs) trained on webscale text have recently demonstrated unprecedented abilities across a variety of NLP tasks (OpenAI, 2023; Bubeck et al., 2023). These LLMs use prompt inputs to follow human instructions. Writing prompts in natural language remains a manual trial-and-error process requiring significant human effort (Jiang et al., 2022) and expertise (Reynolds and McDonell, 2021; Zamfirescu-Pereira et al., 2023). Accordingly, there is need for automatic or semiautomatic procedures to help humans write the best prompts. This would help reduce manual effort, improve task performance, and produce interpretable descriptions of a cognitive decision process. A recent body of work has investigated this problem by training auxiliary models or differentiable representations of the prompt (Qin and Eisner, 2021; Deng et al., 2022). However, such works assume access to internal state variables of the LLM (Shin et al., 2020; Lester et al., 2021) while practitioners often communicate with LLMs through an API.

Discussion / Conclusion. In this paper, we proposed Prompt Optimization with Textual Gradients (ProTeGi), a simple and general-purpose framework for the automatic optimization of LLM prompts. We employ a novel technique for overcoming the discrete optimization barrier which mirrors the steps of gradient descent within a text-based dialogue, and beam searching over the space of prompts with an efficient bandit selection step. Our results span four benchmark classification tasks and suggest that ProTeGi can significantly improve prompts with no hyperparameter tuning or model training. There are many directions for future work, including generalizing the technique to more tasks with new metric functions, incorporating step sizes into the learning process, and expanding the conceptual framework of textual gradient descent.