A useful action representation has to satisfy two awkward requirements at once: the same action must look the same across different executions, and it must look the same across different robots, rooms and body shapes. Most latent action models fail the second one — they entangle the action itself with the visual noise around it, which is why their policies transfer badly.
DreamX, posted to arXiv on 3 November, attacks that problem by repurposing a tool nobody would have thought to use for it: a pretrained video diffusion model.
The Move
Instead of training an action model from scratch, DreamX extracts compact latent actions from the video model’s rich spatiotemporal features. Each latent video frame is compressed by 300x, from 1,200 tokens down to 4. The compression is not a side effect — it is what makes the representation usable as an interface.
The second ingredient is the generative prior. Because the backbone knows what physically plausible motion looks like, the latents it produces tend to capture action-relevant dynamics disentangled from visual appearance. A person reaching for a cup and a robot gripper closing on a cup can land in the same place in latent space, because appearance is no longer what the representation is tracking.
What It Buys
The result is a shared action interface spanning embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX reports state-of-the-art performance among latent action models. More tellingly:
- the latent space supports any-to-any video transfer — an action encoded in one setting still works in another
- zero-shot action transfer across diverse embodiments and environments
- policy learning happens in a compact latent action space, which cuts the computational cost of video-diffusion-based policies
That last point is the practical lever. Video diffusion is an expensive way to train a policy; DreamX keeps the expensive part in the loop only as a teacher of representation, then does the actual policy learning in a space small enough to be cheap.
The open question is how much of this survives outside the pretraining distribution of the particular diffusion model used. A latent action space is only as general as the videos the backbone was trained on, and “any-to-any” claims are the first thing to break when the embodiments get strange.
Paper: arXiv:2611.05377