Cognitive Effects in Large Language Models

Abstract. Large Language Models (LLMs) such as ChatGPT have received enormous attention over the past year and are now used by hundreds of millions of people every day. The rapid adoption of this technology naturally raises questions about the possible biases such models might exhibit. In this work, we tested one of these models (GPT-3) on a range of cognitive effects, which are systematic patterns that are usually found in human cognitive tasks. We found that LLMs are indeed prone to several human cognitive effects. Specifically, we show that the priming, distance, SNARC, and size congruity effects were presented with GPT-3, while the anchoring effect is absent. We describe our methodology, and specifically the way we converted real-world experiments to text-based experiments. Finally, we speculate on the possible reasons why GPT-3 exhibits these effects and discuss whether they are imitated or reinvented.

Introduction. Over the past year, Large Language Models have established themselves as the most influential and widely-adopted AI technology to date. Such models are very large-scale machine learning models designed to process and produce realistic human text. The standard way in which these models are trained is to present them with essentially all the digital text available in the world – this text is typically scraped from the World-Wide Web, social media, and indeed every other source of digital text that the developers can obtain. Because of their inherently opaque nature, concerns have been raised about possible biases and hidden goal structures that such models may acquire. For example, there is a huge amount of racist and misogynistic content on the World-Wide Web, and while designers might try to filter out the most obviously toxic content, the scale of the training data means this must inevitably be an imperfect process.

Discussion / Conclusion. We found that four out of five effects that we tested are presented by GPT-3: namely the priming, distance, SNARC, and size congruity effects. Although it is conceivable that some patterns in GPT-3 could perhaps exist due to pure randomness, we believe this result suggests more than a coincidence. To begin, we note that it is likely that GPT-3 was trained on papers describing the effects we have tested for. However, it is unlikely that these papers have text with formats similar to our queries, and it would be absurd to expect GPT-3 to generalize such as “people respond faster to words recognition when words following associated words, and so I should assign such cases higher probabilities”. Hence, we conclude that GPT-3’s knowledge about the effects is not the origin of their presentation within GPT-3 itself, and so we consider some alternative options. We have shown that some cognitive effects, namely the priming, distance, SNARC, and size congruity effect are presented in GPT- 3, while the anchoring effect was absent. We have presented our methodology in detail, and some analogies that served us when turning real-world experiments to text-based.