A procedurally generated and programmatically verified benchmark for evaluating spatial reasoning capabilities in LLMs
Large language models (LLMs) are showing remarkable results on solving complex reasoning problems across domains—from mathematical proofs and logical puzzles to graduate-level science and engineering questions. On the other hand, their spatial reasoning capabilities are less understood, even though such reasoning underlies many everyday tasks. We set out to test frontier models in this domain.
Spatial reasoning is everywhere — from navigating a city map, to assembling furniture, to understanding how molecules interact in a chemical diagram. It’s a foundational cognitive skill that allows us to make sense of space, position, and movement.
But when it comes to LLMs, how well do they handle tasks that require tracking objects moving around in space? Can they mentally “rotate” a shape or keep track of an object as it moves and turns through a sequence of actions?
To answer these questions, we built SnorkelSpatial — a new benchmark that pushes LLMs to reason about space, movement, and orientation.
SnorkelSpatial operates in a simple yet rich environment: a 2D grid world. Think of it like a small virtual board game. On this board, a few particles sit on tiles. The board and the particles can all move – forward, backward, left, or right — and they can rotate. Every action of the board affects the particles on it. So when the board shifts or rotates, the particles rotate with it.
In this world, each problem in the benchmark contains:
The model must mentally simulate the world to answer questions like:
For each problem, we generate a programmatically verified ground truth answer, making the evaluation precise and unambiguously repeatable.
The complexity of the problems in the benchmark is adjusted by changing parameters such as board size, number of particles, allowed actions, and the number of actions performed. For SnorkelSpatial, we fix several of these variables and control the problem complexity via the number of actions.
Each problem begins with a sequence of actions — movements or rotations applied to the board or its particles. These actions change the spatial configuration over time.
For example, if the board facing north rotates 90° counterclockwise, it now faces west — and every particle on it rotates too. The complexity arises as multiple movements and rotations accumulate, forcing the model to mentally “simulate” the world step by step.
After these transformations, the model must answer questions designed to test spatial reasoning from both allocentric (absolute) and egocentric (relative) frames of reference; a fundamental dichotomy that stems from cognitive psychology.
The queries fall into several categories:
Each query probes a slightly different type of spatial understanding — and together, they create a comprehensive test of how well LLMs can “think in space.”
For the benchmark, we fixed the board size (20×20), randomly placed three particles, and generated 330 problems, each with 10, 20, 50, 100, or 200 actions. Intuitively, we expect that more actions make the task harder.
Each model response is evaluated against the verifiable ground truth answer. We attempt each question 10 times, and report accuracy@1 across all 330 questions in the results below. When a model fails all 10 attempts, we treat it as incorrect.
Overall accuracies: The models particularly tuned for reasoning tasks are the strongest performers here. Only a handful of models, grok-4-fast, o3, gpt-5, and gpt-oss, exceed 50% accuracy, while several other models, particularly older generation models, fall significantly short.
Accuracy vs. number of actions: Intuitively, we expect that the longer the list of actions, the harder the problem. This breakdown of the top 5 models shows that the results hold true to our expectation.
Different question types reveal different strengths and weaknesses:
i) Orientation queries are easiest — likely because there are only a few possible directions (north, south, east, west).
ii) Tile queries are the most challenging for the top models: Compared to the location and orientation queries, finding the tile on which a particle is located involves additional reasoning. We can see in the following results that the top 5 models have the lowest performance on these types of queries.
iii) Absolute (allocentric) vs. relative (egocentric) queries: Since relative queries depend on two objects (particles or board), they require accurate assessment of the states of the both objects and thus are more difficult than the absolute queries. The following figure shows that models perform worse on the relative queries compared to the absolute queries for the same attributes.
Spatial reasoning sits at the crossroads of language, logic, and visualization — and understanding it will be key to building more capable and trustworthy AI systems.
SnorkelSpatial provides a systematic framework for evaluating spatial reasoning in LLMs through procedurally generated, programmatically verified problems that test both allocentric and egocentric reasoning across varying complexity levels. Our results show a wide variance in performance among the most popular models available, with decline in accuracies with increasing problem complexity.
In follow-up work, we plan to explore LLM capabilities in solving these problems by generating code or doing visual chain-of-thought reasoning.
If your project relies on high-quality, expert-verified data or you’re building models that need to reason across complex domains, we’d love to collaborate. Come talk to us — and let’s push the boundaries of reasoning together.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。