Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models

In the recent past, a popular way of evaluating natural language understanding (NLU), was to consider a model’s ability to perform natural language inference (NLI) tasks. In this paper, we investigate if NLI tasks, that are rarely used for LLM evaluation, can still be informative for evaluating LLMs. Focusing on five different NLI benchmarks across six models of different scales, we investigate if they are able to discriminate models of different size and quality and how their accuracies develop during training. Furthermore, we investigate the extent to which the softmax distributions of models align with human distributions in cases where statements are ambiguous or vague. Overall, our results paint a positive picture for the NLI tasks: we find that they are able to discriminate well between models at various stages of training, yet are not (all) saturated. Furthermore, we find that while the similarity of model distributions with human label distributions increases with scale, it is still much higher than the similarity between two populations of humans, making it a potentially interesting statistic to consider.

Introduction. Before the state-of-the-art (SoTA) in NLP was constituted almost exclusively by large language models (LLMs), a popular way of evaluating models’ understanding of natural language was to consider their ability to perform natural language inference (NLI) tasks (most famously, Bowman et al., 2015; Williams et al., 2018). Motivated by the idea that concepts such as entailment and contradiction are central to many aspects of language meaning (Bowman et al., 2015), in NLI tasks, a model is asked to judge the relationship between the meaning of two sentences, typically chosing between entailment, contradiction, and no relationship. Included in the then widely-used natural language understanding benchmark GLUE (Wang et al., 2019b), the NLI benchmark Multi-Genre Natural Language Inference (MNLI, Williams et al., 2018) was up until relatively recently one of the most popular benchmarks to evaluate language models, and is – with over 600 citations to date in 2024 – well-cited even in the recent past.

Discussion / Conclusion. In this work, we revisit natural language inference (NLI) benchmarks and investigate if they may still play a role in LLM evaluation, both during and after pre-training. We consider five different NLI benchmarks – αNLI, ANLI, HANS, and MNLI – and evaluate them across six different models of two model families: Llama 3.1 8B, 70B, and 405B, and Mistral 7B, 8x7B, and 8x22B. Furthermore, we consider how the benchmark behave during the training of two Llama-style 8B and 70B models. We find that, with the exception of αNLI, all benchmarks are able to discriminate between models of different qualities, and in particular ANLI is challenging even for the largest models. Furthermore, we find next to no effect of contamination for the benchmarks. Considering the benchmark ChaosNLI (Nie et al., 2020b), containing 100 human annotations for over 4500 samples of three of the benchmarks we consider, we also find that the differences between human label distributions and model label distributions – as measured with Jensen Shannon Divergence (JSD) – has decreased for the new generation of models.