Open-o3-Video-v2: Seek, Ground, and Reason with Verifiable Spatio-Temporal Evidence

Jiahao Meng*, Zhongyuan Bao*, Yue Tan, Haiquan Wen, Renrui Zhang,
Anna Wang, Jason Li, Yanwei Fu, Yunhai Tong, Ming-Hsuan Yang
Open-o3-Video Team:     PKU     FDU     NTU     UC Merced
* Equal contribution
ArXiv Paper Code Model Data V1 Project Page
Open-o3-Video v1 and v2 comparison, showing adaptive evidence seeking and benchmark results

TL;DR: From grounded reasoning over fixed frames to an agentic system that actively seeks, grounds, and reuses visual evidence. Open-o3-Video-v2 identifies missing clues, inspects relevant clips, anchors observations to moments and objects, and carries that evidence forward to refine its reasoning and produce a verifiable answer.

Abstract

Reliable video reasoning requires models to produce correct answers and expose supporting visual evidence. Open-o3-Video enables verifiable reasoning by embedding timestamps and object-level bounding boxes into reasoning traces, but is restricted to frames from an initial sparse sampler. We introduce Open-o3-Video-v2, which formulates video reasoning as a unified agentic evidence-grounded process. The model identifies missing information, acquires additional observations, grounds the moments and objects supporting its reasoning, and writes this evidence into the trajectory before continuing or answering. To support this behavior, we construct STGR-v2-CoT-135K, including 52K multi-turn agentic grounded reasoning trajectories, and STGR-v2-RL-13K for reinforcement learning. We further propose Clue-Seek GRPO (CS-GRPO), which tracks how tool calls change accumulated evidence, rewards clue coverage and step-wise gains, and penalizes excessive, repetitive, or uninformative retrieval. On V-STaR, Open-o3-Video-v2-4B achieves 40.4% mAM and 56.3% mLGM. It also delivers consistent gains across general video understanding, long-video reasoning, and temporal grounding benchmarks. These results show that active evidence acquisition and verifiable object-level grounding are complementary for reliable video reasoning.

What’s New in Open-o3-Video-v2?

  • Agentic evidence-grounded reasoning. V1 grounds its reasoning in timestamps and object boxes from initially sampled frames, but cannot recover missed evidence. V2 identifies what is still unknown, retrieves a relevant clip, grounds the useful moments and objects in that observation, and uses them as explicit premises for the next reasoning step. Seeking, grounding, and reasoning form one iterative trajectory.
  • Multi-turn grounded data. V1 uses single-response grounded traces. V2 constructs STGR-v2-CoT-135K, including 52K tool-use QA trajectories that connect sparse observations, targeted cropping, returned frames, object evidence, and final answers. STGR-v2-RL-13K supplies informative rollouts for policy optimization.
  • Clue-Seek GRPO (CS-GRPO). V1 optimizes answer, format, and grounded thinking in a single response. V2 additionally scores how the accumulated temporal clue evidence changes across tool calls: it rewards final clue coverage and useful gains, and penalizes excessive, repeated, fragmented, or uninformative retrieval. Answer correctness and object-level grounding remain part of the joint objective.

Demos

Open-o3-Video-v2 reasoning demo, case 3
Open-o3-Video-v2 reasoning demo, case 5
Open-o3-Video-v2 reasoning demo, case 4
Open-o3-Video-v2 reasoning demo, case 6
Open-o3-Video-v2 reasoning demo, case 1
Open-o3-Video-v2 reasoning demo, case 2

Data Construction

STGR-v2 data construction and dataset composition

The 52K multi-turn grounded QA trajectories are built in four steps:

  1. Filter for useful clues: remove questions consistently solved from eight sparse frames and questions that remain unanswerable even with the annotated clue clip.
  2. Annotate evidence: identify question-relevant objects and time spans, then use SAM3 to generate object boxes on clue frames.
  3. Rewrite the reasoning trajectory: connect the initial observation, targeted tool calls, returned frames, grounded reasoning, and final answer.
  4. Check quality: verify boxes, timestamp–object matches, tool intervals, and whether each reasoning statement is supported by the cited visual evidence.

Training corpora: STGR-v2-CoT-135K combines these 52K trajectories with single-turn grounded QA, auxiliary tool-use, temporal and spatial grounding, general video QA, and dense captions. STGR-v2-RL-13K retains questions with both correct and incorrect sampled rollouts, giving reinforcement learning informative comparisons.

Model Training

Open-o3-Video-v2 multi-turn reasoning and Clue-Seek GRPO overview

Stage 1 — SFT: Train on STGR-v2-CoT-135K in a conversation format that interleaves assistant reasoning, optional crop_video calls, returned frames, grounded evidence, and a final answer. Only assistant-generated tokens receive the next-token training loss.

Stage 2 — CS-GRPO: Optimize complete interaction trajectories on STGR-v2-RL-13K. For each call, CS-GRPO measures the temporal overlap between all clips retrieved so far and the annotated answer-relevant clue intervals. This accumulated evidence state captures both clue coverage and irrelevant footage.

  • Reward useful acquisition: score final clue coverage and give step-wise credit when a later call improves the evidence state by more than 0.1.
  • Discourage wasteful calls: penalize calls beyond three, near-duplicate intervals (temporal IoU above 0.7), runs of adjacent fragments, and calls that reduce the evidence state by more than 0.1.
  • Optimize the whole trajectory: combine the tool-use score with answer accuracy, output-format, and grounded-thinking rewards. The tool score guides where to observe; the thinking score checks how retrieved moments and objects support the reasoning.

Experimental Results

Original V-STaR results table from the paper, without its caption

V-STaR. The table evaluates both answer correctness and the alignment of cited moments and objects. At the same 4B scale, v2 raises answer accuracy from 59.7 to 62.8, mAM from 34.6 to 40.4, and mLGM from 47.3 to 56.3 compared with v1. These combined-metric gains support the benefit of active clue seeking for evidence-grounded answers.

Original general video understanding and temporal grounding results table from the paper, without its caption

General video understanding and temporal grounding. The v2-4B model reaches 53.2 on LVBench, 48.0 on Video-Holmes, and 49.8 mIoU on Charades-TimeLens. The v2-8B model leads the listed methods on LVBench, VideoMMMU, VideoMME, VideoMME-v2, and Charades-TimeLens. These results show that the grounded, multi-turn approach transfers to long-video reasoning and temporal localization.

BibTeX

@article{meng2026openo3v2,
  title={Open-o3-Video-v2: Seek, Ground, and Reason with Verifiable Spatio-Temporal Evidence},
  author={Meng, Jiahao and Bao, Zhongyuan and Tan, Yue and Wen, Haiquan and Zhang, Renrui and Wang, Anna and Li, Jason and Fu, Yanwei and Tong, Yunhai and Yang, Ming-Hsuan},
  journal={arXiv preprint arXiv:2610.17656},
  year={2026}
}