LLMs can implicitly learn from mistakes in-context

Learning from mistakes is a fundamental feature of human intelligence. Previous work has shown that Large Language Models (LLMs) can also learn from incorrect answers when provided with a comprehensive rationale detailing why an answer is wrong or how to correct it. In this work, we examine whether LLMs can learn from mistakes in mathematical reasoning tasks when these explanations are not provided. We investigate if LLMs are able to implicitly infer such rationales simply from observing both incorrect and correct answers. Surprisingly, we find that LLMs perform better, on average, when rationales are eliminated from the context and incorrect answers are simply shown alongside correct ones. This approach also substantially outperforms chain-ofthought prompting in our evaluations. We show that these results are consistent across LLMs of different sizes and varying reasoning abilities. Further, we carry out an in-depth analysis, and show that prompting with both wrong and correct answers leads to greater performance and better generalisation than introducing additional, more diverse question-answer pairs into the context.

Introduction. A crucial aspect of human cognition is the ability to learn from mistakes (Metcalfe, 2017). Analogously, LLMs have been shown to benefit from observing incorrect answers in their context (Madaan et al., 2024; Shinn et al., 2024) and even their training data (An et al., 2023; Paul et al., 2024), provided

Discussion / Conclusion. Our results demonstrate that LLMs perform better across several mathematical reasoning tasks when they are prompted for implicit learning, even over CoT prompting and providing the models with additional information through rationales. To minimise any risk that spurious correlations may be influencing these results, here we provide further, in-depth analysis of our findings, their robustness and implications.