InSpatio-World: Real-Time 4D World Simulation via Spatiotemporal Autoregressive Modeling

The ability to simulate a 4D world — one that evolves in time and can be viewed from arbitrary perspectives — is a foundational capability for autonomous driving, robotics, and embodied AI. Existing video generation models produce visually compelling sequences but lack spatial consistency when the camera moves. 3D reconstruction methods achieve geometric fidelity but struggle with dynamic scenes and real-time performance. InSpatio-World bridges this gap through a spatiotemporal autoregressive (STAR) architecture that combines the strengths of both paradigms.

This article provides a detailed technical analysis based on the paper (arXiv:2604.07209) and the open-source implementation.

Interactive Demo

The following viewer shows the complete pipeline output for a circular orbit trajectory. Three videos play in sync: the original input, the geometric rendering condition, and the predicted novel view.

Controls: Play/pause all videos simultaneously. Drag the timeline to seek. Speed control: 0.5x–2.0x. Keyboard: Space = play/pause, Arrow keys = frame step.

1. The Core Problem: Why Not Just Generate Video?

Video generation models (Sora, Wan, CogVideo) produce temporally coherent frames but have no notion of 3D geometry. When you ask them to “move the camera left,” they hallucinate plausible-looking motion that is not geometrically consistent with the underlying scene.

InSpatio-World identifies three specific failure modes in existing autoregressive world simulators:

Spatial Persistence Degradation: As the autoregressive rollout extends, the model “forgets” the original scene geometry. Objects drift, textures blur, and structural coherence decays.
Synthetic-to-Real Gap: Training on rendered (synthetic) data provides precise camera control but produces artifacts. Training on real video produces realistic frames but lacks control signals. Neither alone is sufficient.
Insufficient Control Precision: Existing trajectory-conditioned models fail to accurately follow user-specified camera paths, especially for large rotations.

2. Architecture: STAR (Spatiotemporal Autoregressive)

The STAR architecture generates video in blocks of $N_f$ frames (default: 3), each conditioned on three types of information:

The denoising process for block $i$ is:

\hat{z}_i = \text{Denoise}_\theta(z_i, \sigma \mid z_{<i}, z_{\text{ref}_i}, [z_{\text{warp}_i}, m_i])

2.1 Implicit ST-Cache: The Global Spatial Anchor

The reference latent $z_{\text{ref}}$ is extracted from the source video and injected into every block as a persistent spatial anchor. This solves spatial persistence degradation by ensuring the model always has access to the original scene appearance.

In the implementation, this works through a KV cache mechanism:

1
2
3
4
5
6
7
# Concatenate reference + history as context frames
context_frames = torch.cat([ref_block, last_pred_padded], dim=1)
# Reference block is prepended to every denoising step
denoised_pred, _ = denoise_block(
    noisy_current, context=context_frames,
    render_block=render_condition, ...
)

A critical implementation detail: position encoding anchoring. The RoPE position indices for the reference block, history block, and current block are each anchored to fixed absolute positions, preventing the position encoding from drifting as the sequence length grows during autoregressive rollout.

2.2 Explicit Spatial Constraint: Depth → Point Cloud → Render

The explicit geometric pipeline operates in three stages:

Depth estimation: Depth-Anything-3 (DA3) estimates per-frame depth maps and camera poses from the source video.
Point cloud reconstruction: Each frame’s depth map is unprojected into a 3D point cloud (one PLY per frame).
Trajectory-conditioned rendering: Given a user-specified camera trajectory, the point cloud is re-projected to the novel viewpoint, producing render_offline.mp4 and mask_offline.mp4.

The render video provides a coarse geometric guide for where objects should appear from the new viewpoint, while the mask indicates which pixels have valid geometry. The DiT learns to refine this coarse render into a photorealistic frame.

2.3 Trajectory Specification

Trajectories are defined as simple text files with three lines: pitch angles (degrees), yaw angles (degrees), and displacement scale factors. The sphere2pose function converts spherical coordinates to 4×4 camera-to-world matrices:

1
2
3
4
# x_y_circle_cycle.txt
0 0 ... 30 30 ... 0 0 ... -30 -30 ... 0 0
0 0 ... 45 45 ... 90 90 ... 45 45 ... 0 0
1.0 1.0 ... 1.0

Keyframes are interpolated using scipy.interpolate.UnivariateSpline for smooth trajectories. The system adaptively adjusts frame count based on total angular change (0.3–0.8 degrees per frame).

3. JDMD: Solving the Synthetic-Real Gap

Training on synthetic data (rendered point clouds) provides precise camera control but produces visual artifacts. Training on real video produces beautiful frames but lacks control signals. InSpatio-World’s solution: train on both simultaneously.

The JDMD (Joint Distribution Matching Distillation) loss:

\mathcal{L}_{\text{JDMD}} = \mathcal{L}_{\text{vis}} + \lambda_{\text{ctrl}} \cdot \mathcal{L}_{\text{ctrl}}

$\mathcal{L}_{\text{vis}}$ : Standard flow-matching loss on latent space, applied to both branches.
$\mathcal{L}_{\text{ctrl}}$ : Control precision loss, computed only on the V2V branch, measuring how well the generated video follows the specified camera trajectory.

This dual-branch training ensures the model inherits both geometric accuracy (from synthetic data) and visual realism (from real data).

4. Inference Pipeline

The complete inference pipeline has three steps:

Step 1: Caption Generation

Florence-2 generates a text description from the source video. This caption provides semantic context for the T2V component of the model.

Step 2: Depth Estimation + Geometric Rendering

DA3 estimates depth maps and camera poses. The depth maps are unprojected to point clouds and re-rendered from the target trajectory viewpoints, producing the geometry condition videos.

Step 3: Autoregressive Inference

The Causal DiT generates the novel-view video block by block, with each block conditioned on the reference latent, the history cache, and the geometric render.

1
2
3
4
# Run the complete pipeline
bash run_test_pipeline.sh \
  --input_dir ./test/example \
  --traj_txt_path ./traj/x_y_circle_cycle.txt

Key inference options:

Flag	Purpose
`--relative_to_source`	Combine trajectory relative to initial view (for driving)
`--rotation_only`	Pan/tilt only, ignore translation
`--freeze_repeat N`	Freeze time, repeat frame N times
`--use_tae`	Tiny AutoEncoder for faster inference
`--compile_dit`	torch.compile acceleration

5. Performance

Metric	Value
Model size	1.3B parameters
FPS (H-series GPU)	24
FPS (RTX 4090)	10
WorldScore-Dynamic	68.72 (SOTA among real-time methods)
Camera control precision	81.51
RE10K-Long FID	42.68
RE10K-Long FVD	100.55

The model achieves real-time performance while maintaining competitive quality against offline methods. The block-wise causal architecture enables streaming output — the first few frames are available before the entire sequence is generated.

6. Connection to Autonomous Driving

InSpatio-World has a natural connection to autonomous driving planning. The project includes integration documentation for DrivoR, a Transformer-based E2E planner that achieves PDMS 93.7 on NAVSIM-v1.

The key insight: use InSpatio-World not as a planner, but as a future observation generator. Given a candidate trajectory from DrivoR, InSpatio-World can render what the ego vehicle would see if it followed that trajectory, enabling:

Future-consistency scoring: Add a feature to the DrivoR scorer that evaluates whether the predicted future observation is consistent with the planned trajectory.
Counterfactual data augmentation: Generate training data for rare scenarios by rendering novel views along hypothetical trajectories that differ from the ground truth.
Trajectory-conditioned world simulation: Combine DrivoR’s trajectory output with InSpatio-World’s rendering to create a closed-loop simulation environment.

This points toward a broader trend: the convergence of world models and planning models in autonomous driving, where the world model provides the “what would happen” and the planner provides the “what should I do.”

7. Limitations and Open Questions

Long-range consistency: While the ST-Cache mitigates degradation, extremely long rollouts (hundreds of frames) still show gradual drift.
360-degree roaming: The current architecture handles moderate viewpoint changes well but struggles with full panoramic exploration.
Dynamic objects: The explicit geometric pipeline (point cloud re-projection) treats objects as static; handling moving objects in the scene remains an open challenge.
Sim-to-real gap for driving: Although JDMD helps, the gap between rendered and real driving scenes is larger than for general video, due to complex reflections, transparent surfaces, and fine textures.

Interactive Demo#

1. The Core Problem: Why Not Just Generate Video?#

2. Architecture: STAR (Spatiotemporal Autoregressive)#

2.1 Implicit ST-Cache: The Global Spatial Anchor#

2.2 Explicit Spatial Constraint: Depth → Point Cloud → Render#

2.3 Trajectory Specification#

3. JDMD: Solving the Synthetic-Real Gap#

4. Inference Pipeline#

Step 1: Caption Generation#

Step 2: Depth Estimation + Geometric Rendering#

Step 3: Autoregressive Inference#

5. Performance#

6. Connection to Autonomous Driving#

7. Limitations and Open Questions#

References#