HVLR: Unifying Heterogeneous Visual Reasoning in Expressive Latent Space

Example Video

Overview

Abstract

Visual reasoning increasingly relies on heterogeneous intermediate representations, including textual rationales, regions, scene graphs, segmentation, depth, and visual manipulations. Because these representations differ substantially in structure and modality, they are typically modeled with separate reasoning interfaces, which scale poorly as more reasoning forms are incorporated. This raises a fundamental question: can heterogeneous forms of visual reasoning be unified in a single shared latent space? We investigate this question through HVLR (Heterogeneous Visual Latent Reasoning), which unifies eight distinct reasoning forms within a common autoregressive latent interface. To reconcile heterogeneous supervision, we first render all reasoning representations into a shared pixel space and encode them with the same vision encoder. We then use a two-stage distillation procedure to translate these representations into autoregressive latent states and internalize them into a closed-book model that requires no explicit intermediate representation at inference time. We conduct a systematic analysis with HVLR and uncover three main findings. First, heterogeneous reasoning forms can be effectively unified: combining all eight improves Qwen2.5-VL-7B from 64.66% to 69.97% average accuracy across nine benchmarks. Second, the shared latent space consistently benefits from reasoning diversity. Under a fixed 40K-example budget, increasing the number of reasoning forms from 1 to 3, 5, and 8 improves performance from 66.65% to 67.67%, 68.61%, and 69.97%, respectively, with positive trends across all nine benchmarks. Third, latent reasoning outperforms explicit intermediate decoding while reducing intermediate computation from roughly 177 generated tokens to only 8 latent steps, suggesting that continuous latent representations provide a more expressive and efficient medium for heterogeneous visual reasoning.

Introduction

One of the foundational ideas in computer vision is that visual understanding is rooted in intermediate representations that make useful properties of the scene explicit. Edges expose boundaries and shape; segmentation exposes objects and regions; depth exposes geometric structure; and scene graphs expose objects and their relations. Recent visual reasoning methods exploit many of these representations, but they typically assume a particular representation for a particular reasoning mechanism. Yet there is no “magic” representation for all visual reasoning problems, and different tasks may benefit from different representations, or combinations thereof.

HVLR uses these perceptual representations as priors that regularize how a model learns to reason. The model is exposed to multiple representations during training and reasons without them at inference, using a visual reasoning scratchpad, an internal workspace composed of a sequence of latent states. Each prior is expressed visually, so all priors are processed by the same vision encoder and mapped into the scratchpad in a unified manner. A two-stage training procedure first learns the mapping from the priors to the scratchpad, then distills the scratchpad into a model that has no access to the priors as inputs. Training with eight perceptual priors improves Qwen2.5-VL-7B from 64.66 to 69.97 average accuracy across nine benchmarks, and increasing the number of priors from 1 to 3, 5, and 8 increases average performance from 66.65 to 67.67, 68.61, and 69.97. For priors that admit textual serialization, latent reasoning achieves higher accuracy while reducing intermediate computation from roughly 177 generated tokens to 8 latent steps.

Collection of Heterogeneous Visual Reasoning Data

To provide a common supervision interface, we rasterize every reasoning representation into an image and process it with the same vision encoder. This allows linguistic, symbolic, spatial, and perceptual reasoning signals to supervise the same latent space without dedicated prediction heads. We collect eight representative reasoning forms. Text CoT renders textual rationales on a blank canvas; Scene Graph visualizes objects and relations as node–edge diagrams; BBox Crop and BBox Highlight isolate or emphasize question-relevant regions; Segmentation, Depth, and Edge expose complementary dense visual structure; and Interleaved reasoning (“Helper Image” in Figure 1) provides one to three intermediate images from multi-step visual reasoning traces. After curation, the full corpus contains 1.31M examples; unless otherwise specified, all experiments use the same balanced 40K subset, with 5K examples each form.

Two-Stage Training

Training overview. Stage 1 trains a latent teacher MLLM that reads the image, question and helper image, generates teacher latents that are aligned with encoded helper features, and re-inserts them so the answer only sees the helpers through the latents. Stage 2 distills a closed-book student MLLM that generates student latents from the image and question alone. — Figure 2: **Training overview.** Stage 1 trains a teacher that summarizes the helper images into latent slots. Stage 2 trains a student to generate these latent slots from the image and question alone, supervised by the features exported from the Stage-1 teacher.

Teacher Model

Each helper is compressed into a block of L = 8 latent slots, and each slot reuses the preceding last-layer hidden state as the next input embedding. Each training step uses two forward passes. In the first, the teacher observes the helper images and autoregressively generates the latent slots. In the second, these slots are reinserted into the sequence, while an attention mask prevents the answer positions from directly attending to the helpers, so information from the helpers must be mediated through the latent slots. To preserve the helper content, the latent states are aligned with representations extracted from the same helpers by a frozen base model.

Student Model

The student is initialized from the base model and has no access to the helpers. It learns to reconstruct the teacher’s latent states from the original image and question alone, using targets precomputed by the frozen teacher at each latent position and layer. At inference, the student receives only the image and question, generates the latent reasoning states, and then decodes the answer. No helper image, rationale text, or dedicated reasoning decoder is required.

Main Results

Table 2: Main results on nine benchmarks. Bold = best per column among same-protocol rows. Frontier-model rows (gray) are reported numbers taken from published papers.

Model	V^⋆	HR4K	HR8K	MME-RW	BLINK	POPE	RWQA	CV-B	QSp++	Avg.
base (Qwen2.5-VL-7B)	75.92	68.88	65.25	45.49	55.65	86.58	69.15	73.40	41.58	64.66
Latent Reasoning
Monet-7B	80.85	70.38	68.88	47.06	41.56	87.75	61.05	51.64	31.68	60.09
LVR-7B	81.18	70.75	66.12	49.56	53.45	84.76	68.10	75.09	24.75	63.75
CoVT-7B	78.53	71.00	69.50	50.34	56.35	84.58	70.07	78.75	53.47	68.07
SkiLa-7B	82.72	71.12	64.50	46.74	54.87	87.39	69.93	76.19	40.59	66.01
Mirage-VSP	79.06	70.12	66.88	48.31	56.78	85.29	69.93	75.07	42.57	66.00
Open-source VLM
InternVL3-8B	70.16	69.00	59.00	47.68	55.18	89.00	70.07	80.98	53.47	66.06
LLaVA-OneVision-7B	72.25	65.12	58.13	48.41	50.55	88.20	70.07	76.32	51.49	64.50
Tool Using
DeepEyes-7B	82.20	73.62	70.88	52.42	44.29	87.74	69.41	78.59	44.55	67.08
PixelReasoner-7B	78.53	70.62	64.50	45.75	45.08	85.97	69.67	79.09	45.54	64.97
Frontier Closed-Source Models
GPT-5^‡	74.87	74.6	71.62	57.0	71.0	—	82.8	—	63.37	—
GPT-4o^‡	62.8	68.0	63.9	51.6	60.0	86.9	75.4	79.4	61.06	—
Gemini-2.5-Pro^‡	79.2	87.3	83.62	58.30	70.6	—	78.0	86.3	40.59	—
HVLR (Ours)	81.70	71.75	70.88	54.30	56.85	89.19	72.03	78.53	54.46	69.97
Δ vs. base	+5.78	+2.87	+5.63	+8.81	+1.20	+2.61	+2.88	+5.13	+12.88	+5.31

Table 2 evaluates three model families. Our model improves on its base and attains the highest average score among all same-protocol rows, +1.9 above the best latent-reasoning baseline (CoVT-7B). It also exceeds the best tool-using agent, DeepEyes, with a single closed-book pass instead of a multi-pass zoom-in loop.

Finding ❶

More Forms, Stronger Model: Capability Scales with Latent Diversity

The capability of latent reasoning scales with the diversity of the content the space must host: packing more forms of thought in one latent space yields a stronger model.

Design. We fix the budget at 40K and vary only the number of forms it spans: K=1 gives the whole budget to a single form (8 models, one per form); K=3 and K=5 each draw 8 random combinations with 40K examples in total; K=8 is the model of Table 2.

Three plots: (a) average score rises with the number of reasoning forms K from 1 to 8; (b) all 25 ablations plus base ranked by average score, with the 8-form model first and base last; (c) the per-benchmark slope against K is positive on all nine benchmarks.

Figure 3: Modality-count ablation. (a) average score vs. number of forms K: tier means (solid) rise monotonically; OLS +0.483/form, Spearman ρ=0.88. (b) All 25 ablations + base ranked by average score: ours ranks #1, the eight random K=5 subsets occupy #2–9, and base ranks last. (c) Refitting the slope on each benchmark separately: all nine are positive.

Finding ❷

Latent Reasoning Is More Effective and More Efficient than Decoded Text

Keeping intermediate thought in the latent space dominates decoding it into text: carried in vectors, the same reasoning reaches higher accuracy at a fraction of the inference cost.

Text CoT and scene graphs can be written out faithfully as text, which lets us test whether a form is better internalized in the latent space or emitted as text.

Three plots comparing latent and decoded-text reasoning: (a) average score of variants that supervise text CoT, scene graph or both as explicit text, under free routing, forced latent and forced text decoding, all below the all-latent model; (b) the same runs against think steps; (c) think tokens per question: Ours 9, SkiLa 29.3, Monet 129.1, PixelReasoner 159.5, DeepEyes 179.6.

Figure 4: Latent versus explicit decoded-text reasoning. Three variants supervise the text-CoT portion, the scene-graph portion, or both as decoded text while all other forms stay latent; each is evaluated with free routing, forced latent or forced text decoding. The dashed line is our all-latent model. (a) Average score over the nine benchmarks. (b) The same runs against think steps. (c) Think tokens per question across methods, measured as in Fig. 5.

Inference efficiency: (a) wall-clock latency per question relative to ours, from 1.00x for ours to 5.42x for SkiLa-7B; (b) decode TFLOPs, 0.294 for ours; (c) five-benchmark accuracy against latency, where every other method is slower and less accurate than ours.

Figure 5: Inference efficiency on V^⋆, HR-Bench 4K/8K, RealWorldQA and QSpatial++. (a) Latency per question, with the ratio to ours. (b) Decode FLOPs computed from token counts; all methods share one backbone. (c) Mean accuracy over the five benchmarks against latency; the shaded region marks methods that are both slower and less accurate than ours.

Finding ❸

The Gains Transfer across Backbones and Recipes

The benefit of heterogeneous latent supervision transfers across model scales, backbones, and training recipes.

The same data and training stages also improve a smaller (Qwen2.5-VL-3B) and a newer (Qwen3-VL-8B) backbone (Table 3), as well as a modified LVR-style recipe (Table 4).

Table 3: Cross-backbone transfer. Other backbones trained with the same data and training pipeline also improve over the base models.

Qwen2.5-VL-3B

Model	CV-Bench	MME-RW	QSpatial++	POPE
base	59.22	42.31	52.48	86.31
ours	69.91	48.93	58.42	87.96
Δ	+10.69	+6.62	+5.94	+1.65

Qwen3-VL-8B

Model	QSpatial++	MME-RW	V^⋆	HR8K
base	51.49	49.82	80.10	70.25
ours	64.36	55.97	83.77	71.88
Δ	+12.87	+6.15	+3.67	+1.63

We adapt LVR into a pipeline that shares only the data and the latent interface with ours (Fig. 6). Training is a single teacher-forced stage without distillation: the rendered helper’s visual features fill the latent slots as inputs, and a trainable prediction head predicts each slot from the hidden state at the preceding position, trained with an alignment loss, the answer cross-entropy and a contrastive term that keeps the latents sample-specific. At inference the helper is absent and the head generates the latent slots autoregressively. Trained on 500K examples of our corpus, this pipeline gains 3.47 points of average score over its base and surpasses the publicly released LVR checkpoint by 5.03 points, most notably on QSpatial++ (Table 4).

LVR-style training pipeline. Top: the image and question go into the MLLM, which produces latents z1 to zm and then the answer; the helper image is encoded by a visual encoder into helper features that supervise the latents. Bottom: like token prediction from a hidden state through the LM head, an LVR head (GLU) predicts each latent from the preceding hidden state and is trained with an alignment loss against the helper feature.

Figure 6: LVR-style training pipeline. Top (Overview): Helper image is encoded by visual encoder as ground truth to supervise latent. Bottom (Details): latent is predicted from preceding hidden state like a text token.

Table 4: Cross-recipe transfer. A modified LVR-style recipe trained on our dataset, against its base and against the publicly released LVR checkpoint.

Model	V^⋆	MME-RW	BLINK	POPE	RWQA	CV-Bench	QSpatial++	Avg.
base (Qwen2.5-VL-7B)	75.92	45.49	55.65	86.58	69.15	73.40	41.58	63.97
LVR-7B	81.18	49.56	53.45	84.76	68.10	75.09	24.75	62.41
LVR-style + our data	82.72	49.40	56.13	87.21	70.85	78.26	47.52	67.44
Δ vs. base	+6.80	+3.91	+0.48	+0.63	+1.70	+4.86	+5.94	+3.47
Δ vs. LVR-7B	+1.54	−0.16	+2.68	+2.45	+2.75	+3.17	+22.77	+5.03

@article{yang2026hvlr,
  title   = {Unifying Heterogeneous Visual Reasoning in Expressive Latent Space},
  author  = {Yang, Ruoliu and Li, Yijiang and Luo, Haozheng and Wang, Yixuan and Vasconcelos, Nuno},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}

HVLR

Unifying Heterogeneous Visual Reasoning
in Expressive Latent Space