Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models

TL;DR We introduce POVBench, a benchmark for evaluating contextual observer grounding, and show that mapping observer-relative language to object locations remains a challenging problem for state-of-the-art VLMs.

A robot explores a house and remembers it from its own viewpoint. The resident describes the mug from theirs. Resolving left means first working out where they stood and what they could see.

Overview of contextual observer grounding in POVBench — Overview of contextual observer grounding. Models infer the observer's location (*operational reasoning*), reconstruct their perspective (*observer grounding*), and localize the target (*relational reasoning*). Conditions *A: Inferred*, *B: Stated*, and *C: Given* require three, two, and one of these steps, respectively.

POVBench for evaluating contextual observer grounding

One scenario, three ways to describe it

The three conditions share the same target placements but provide different levels of information about the observer's viewpoint, enabling direct comparison across conditions.

POVBench construction and the three conditions — Exploration images are collected along a robot trajectory in ProcTHOR. Directional targets are defined around the anchor *from the observer's viewpoint*.

47houses

288observer-centric scenarios

813instances per condition

2,439total

Contextual observer grounding is challenging for VLMs

Explicitly identifying the observer offers limited benefit

Contextual observer grounding remains challenging, with the best-performing model reaching a mean normalized L2 error of 0.294 under Inferred. Meanwhile, the Inferred–Stated gap is at most 0.012 across all evaluated models.

We evaluate ten VLMs spanning proprietary and open-weight general-purpose models as well as dedicated pointing models.

Model	Inferred	Stated	Given
MolmoPoint-8B	0.294	0.282	0.237
GPT-5.4	0.315	0.311	0.189
Qwen3-VL-8B (Transformer)	0.372	0.369	0.296
Qwen3-VL-8B (vLLM)	0.359	0.356	0.291
Qwen3-VL-32B (vLLM)	0.449	0.440	0.488
InternVL3-38B	0.372	0.364	0.232
Gemma-4	0.463	0.464	0.367
Gemini-2.5-Flash	0.489	0.487	0.434
Gemini-Robotics-ER	0.535	0.538	0.507
RoboPoint	–	–	0.209
Llama-3.2-Vision	–	–	0.433

Mean normalized L2 error, lower is better, averaged over three runs and spatial directions. Qwen3-VL-8B is tested with both Transformer and vLLM implementations. RoboPoint and Llama-3.2-Vision are evaluated only under Given, as they did not produce valid responses in the multi-image settings.

Localization error by spatial direction — Error by spatial direction for Gemma-4, GPT-5.4, and Qwen3-VL-8B (Transformer).

Decomposing the reasoning improves localization

We test four interventions against the basic prompt on Qwen3-VL-8B: marking the observer, marking the anchor too, chain-of-thought, and Spatial-CoT.

Intervention	Inferred	Stated	Resp.
Basic	0.372	0.369	100%
O-Plot	0.398	0.401	100%
O&A-Plot	0.342	0.343	100%
CoT	0.322	0.333	79%
Spatial-CoT	0.294	0.302	85%

Qwen3-VL-8B (Transformer), same protocol as above. Resp. is the response rate.

Spatial-CoT achieves the lowest L2 error under both Inferred and Stated, reducing error from 0.372 → 0.294 and 0.369 → 0.302, respectively.

The difficulty persists on real-world scenes

We construct ten observer-centric scenarios with 102 instances from two house-tour videos and evaluate them manually.

Standard versus Spatial-CoT prompting on real house-tour footage — Standard versus Spatial-CoT prompting on frames from CC BY house-tour videos.

Model	Inferred	Stated	Given
GPT-5.4	0.29	0.21	0.41
Qwen3-VL-8B (Transformer)	0.03	0.09	0.24
Gemma-4	0.41	0.38	0.47
Gemini-2.5-Flash	0.09	0.09	0.21
Gemini-Robotics-ER	0.29	0.24	0.24
Qwen3-VL-8B + Spatial-CoT	0.15	0.15	–

Manually evaluated success rates, higher is better, pooled over two house-tour videos.

Under the basic prompt no model exceeds 50% in any condition. Spatial-CoT also improves Inferred and Stated on Qwen3-VL-8B, suggesting that its benefit may extend beyond simulation.

Paper Abstract

Reasoning over language instructions in embodied tasks such as robotics often requires understanding spatial relations from a speaker's situated perspective. Humans infer such perspectives from shared environmental knowledge, activity context, and commonsense. Recent vision-language models (VLMs) appear capable of spatial reasoning, but their ability to infer a speaker's viewpoint from contextual cues and interpret situated spatial relations from that viewpoint remains unclear. We call this capability contextual observer grounding. To study this capability, we construct the Point-of-View Benchmark (POVBench), a dataset of 3D scenes and queries that disentangles Inferred, Stated, and Given forms of observer grounding in natural embodied communication. Given multi-view observations and a natural-language sentence, models must localize unseen or underspecified targets from situated spatial and contextual cues. Across state-of-the-art VLMs, localizing targets from directional language remains challenging, even when observer grounding is made explicit. We find that explicit breakdowns of observer-relative spatial reasoning improve target localization.

BibTeX

@inproceedings{shirasaka2026povbench,
  title     = {Contextual Observer Grounding: Evaluating Situated Spatial
               Reasoning in Vision-Language Models},
  author    = {Shirasaka, Mimo and Zhang, Haochen and Bisk, Yonatan},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}