# Grounded World Model: Latent Planning with Language Goals

Authors: Quanyi Li, Lan Feng, Haonan Zhang, Wuyang Li, Letian Wang, Alexandre Alahi, Harold Soh.

This is a text companion to the [GWM project page](https://quanyili.github.io/gwm-wiser/). The [paper](https://arxiv.org/abs/2604.11751) provides the full methodology, experimental protocols and ablations.

- Paper identifier: arXiv:2604.11751; DOI: 10.48550/arXiv.2604.11751.
- Status: arXiv preprint, first submitted April 13, 2026; version 2 revised September 30, 2026.
- [PDF](https://arxiv.org/pdf/2604.11751)
- [Code](https://github.com/QuanyiLi/gwm-wiser)
- [Models](https://huggingface.co/Shady0057/GWM)

## What is a Grounded World Model?

GWM predicts the visual consequences of candidate robot actions in a latent representation. A frozen video-language embedding model connects those predictions with language, allowing a planner to select actions toward a goal described in words without a goal image. The project contrasts this with goal-image planning in DINO-WM and LeWM.

## How is GWM trained?

The current frame and rendered robot poses provide the predictor's input. Action frames are rendered from the robot's URDF along a target joint trajectory using the scoring camera's calibration. The predictor learns to match the visual features of the recorded future with an MSE loss. The vision encoder and the video-language embedding model remain frozen; only the predictor learns.

The real-data model uses Qwen3-VL-Embedding as its frozen base and trains on DROID and MolmoB0T. Predictor training uses videos and actions without rewards or task-language labels. No demonstrations are collected for the target tasks. Language alignment comes from the pretrained base model; the full stack is not trained from scratch without language.

## How does language guide planning?

With the current frame fixed, GWM predicts futures for many candidate action sequences. Each predicted future is scored against a language goal using cosine similarity through the frozen video-language embedding model. Changing the instruction changes the cost landscape and the preferred action. The same cost can rank recorded motions, guide cross-entropy method (CEM) search, or select trajectories proposed by grasp and geometric motion planners.

## What do the experiments show?

### Pointing and pushing

Pointing tests whether language-guided search identifies the named image. Pushing tests whether candidate trajectories make precise contact with the named cube. These probes characterize semantic grounding and spatial precision. The CEM visualizations show the search distribution narrowing over successive optimization rounds.

### Isaac Sim and a real robot

The simulated tabletop evaluation includes 14 tasks with five trials per task. The project page provides an HTML results table and task instructions. The GWM-based manipulation system combines the GWM cost with language-blind trajectory proposals and geometric motion planning.

On Franka hardware, the GWM system completes 55 of 60 sub-tasks, compared with 52 of 60 for TiPToP. These are system-level results. GWM inference runs on an RTX 5090 using up to 24 GB of CUDA memory, while TiPToP uses a frontier Gemini model.

The real-robot experiments use different objects, instructions and camera viewpoints. Candidate motions are rendered and scored from calibrated views without retraining. The model retains some ability to reason about spatial relations, visual attributes and object affordances.

### WISER controlled testbed

WISER is a controlled simulation testbed that separates semantic grounding from learning new motions. On 288 held-out tasks, GWM reaches 87% success. Ten fine-tuned policy baselines, including the world-action model Motus, average 22%; the strongest test baseline, InstructVLA, reaches 47%. GWM training success is 92%, compared with a 90% average for the ten policies. These comparisons are specific to the WISER testbed.

In a separate comparison using the same candidate trajectories and scoring readout, the pixel-based world model DreamDojo reaches 17%. This is not the same comparison as the ten-policy bar chart.

## Features, limitations and future work

- Camera poses: the same model scores motions from different calibrated camera poses. Rendering must match the scoring view.
- Embodiment transfer: in simulation, a model trained on a Franka Panda transfers to an xArm6 using its own URDF and candidate trajectories without updating the model.
- Planner compatibility: the same language-based cost can be used with different sources of candidate trajectories.
- Prospective video-only training: segmenting robots, arms or human hands from video could supply motion inputs and enable mixing data from different embodiments without recorded actions. This is a future extension, not a claim that the reported model was trained on video alone.
- Base-model limitation: the planning cost inherits the frozen video-language embedding model's strengths and weaknesses. If the base model cannot recognize a behavior or connect it to an instruction, the score can be unreliable. Stronger video-language embedding models are a future research direction.

See the [project's features and limitations](https://quanyili.github.io/gwm-wiser/#features) and [paper discussion](https://quanyili.github.io/gwm-wiser/#paper).
