Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMs

Reasoning encompasses two typical types: deductive reasoning and inductive reasoning. Despite extensive research into the reasoning capabilities of Large Language Models (LLMs), most studies have failed to rigorously differentiate between inductive and deductive reasoning, leading to a blending of the two. This raises an essential question: In LLM reasoning, which poses a greater challenge - deductive or inductive reasoning? While the deductive reasoning capabilities of LLMs, (i.e. their capacity to follow instructions in reasoning tasks), have received considerable attention, their abilities in true inductive reasoning remain largely unexplored. To investigate the true inductive reasoning capabilities of LLMs, we propose a novel framework, SolverLearner. This framework enables LLMs to learn the underlying function (i.e., y= fw(x)), that maps input data points (x) to their corresponding output values (y), using only in-context examples. By focusing on inductive reasoning and separating it from LLM-based deductive reasoning, we can isolate and investigate inductive reasoning of LLMs in its pure form via SolverLearner.

Introduction. Recent years have witnessed notable progress in Natural Language Processing (NLP) with the development of Large Language Models (LLMs) like GPT-3 (Brown et al., 2020) and ChatGPT (OpenAI, 2023). While these models exhibit impressive reasoning abilities across various tasks, they face challenges in certain domains. For example, a recent study (Wu et al., 2023) has shown that while LLMs excel in conventional tasks (e.g., base-10 arithmetic), they often experience a notable decline in accuracy when dealing “counterfactual” reasoning tasks that deviate from the conventional cases seen during pre-training (e.g., base-9 arithmetic). It remains unclear whether they are capable of fundamental reasoning, or just approximate retrieval. In light of this, our paper seeks to investigate the reasoning capabilities of LLMs. Reasoning can encompasses two types: deductive reasoning and inductive reasoning, as depicted in Fig. 1.

Discussion / Conclusion. This study aims to explore a less-investigated aspect of LLMs: within LLM reasoning, which presents a greater challenge — deductive or inductive reasoning? To investigate the inductive reasoning capacities of LLMs, we introduce a novel framework called SolverLearner. By concentrating on inductive reasoning while setting aside LLM-based deductive reasoning, SolverLearner can scrutinize the pure form of inductive reasoning in LLMs. Our findings unveil remarkable inductive reasoning prowers in LLMs through SolverLearner, achieving near-perfect performance with an ACC of 1 in most cases. Surprisingly, despite their strong inductive reasoning abilities, LLMs often exhibit weaker deductive capabilities, particularly in tasks involving “counterfactual” scenarios. LLMs cannot perform inductive reasoning over all the tasks In our inductive learning setting, LLMs are provided with only a limited number of contextual examples. The goal is to infer the function that accurately maps inputs to outputs based solely on this constrained dataset. In order to solve this problem, it is significant that we can find a unique function satisfied given these examples.