Investigating abstract reasoning in humans and machines

State-of-the-art AI models have matched or exceeded human performance on reasoning benchmarks such as the Abstraction and Reasoning Corpus, a prominent benchmark for conceptual understanding and analogy. But does high accuracy on this benchmark mean that these models understand and reason with the humanlike abstractions intended by the task creators? In this talk, Melanie Mitchell will describe an evaluation methodology, inspired by experimental methods in cognitive science, to assess and compare the abstraction abilities of AI “reasoning” models and human participants.

Mitchell’s evaluations show that, while some models match or exceed human accuracy, their reasoning is frequently based on surface-level patterns or spurious associations, and thus lacks generalizability. She will speculate on what remains to be done to capture human-like abstract reasoning in AI models. Mitchell will be hosted by Chris Lynn.