Egocentric human data offers scalable supervision for robot manipulation. However, behavior cloning entangles transferable content—like objects, scenes, and task semantics—with non-transferable factors—like human morphology, head motion, and behavioral style.
We study whether World Action Models (WAMs) provide a better training signal by requiring policies to predict not only actions, but also how the scene evolves. The central question is what world representation best enables human-to-robot transfer.
We hypothesize that an effective world target should abstract appearance, capture agent-invariant physical effects, and separate camera motion from environment change.
We introduce EgoWAM, a controlled human-robot co-training framework that fixes the policy backbone, action head, and data mixture while varying only the world prediction target, comparing Pixel, DINO, and 3D motion flow.
Across three real-world bimanual tasks, WAM co-training scales more effectively with in-the-wild egocentric human data than behavior cloning. Pixel-based prediction transfers weakly, while DINO and 3D flow yield substantial gains: DINO improves out-of-distribution object and scene generalization by up to 4×, and 3D flow improves in-domain performance by 20–30%.
Method.
Mode
both heads & backprop active
ℒaction
Action Head Flow Matching Decoder
ℒworld
⇆ swappable✕ off at inference
World‑Model Head 3D Flow
z
Action Tokens
Obs. Tokens
Future Tokens
TRANSFORMER
Ego Vision Stem
Proprio Stem
Wrist Vision Stem
Action TokensLearned Context
Future TokensLearned Context
Human & RobotRobotForward PassBackpropagation
Architecture.EgoWAM builds on a Heterogeneous Pretrained Transformer (HPT) backbone with embodiment-specific stems feeding a shared trunk. Two read-out heads—a shared action head (conditional flow matching), conditioned on the trunk's action-token outputs, and a swappable world-model head, conditioned on the future-token outputs and trained to reconstruct the future observation in its target space—let us vary only the world target while holding the trunk, action head, and data mixture fixed; the obs-token outputs serve as a shared condition for both heads. Use the tabs to swap the world-model head.
Data Gallery.
We co-train on three data sources per task: robot teleoperation, in-domain human (same scenes/objects, unmatched viewpoint and behavior), and in-the-wild human (diverse scenes, objects, and demonstrators). Use the buttons to switch tasks; each panel scrolls through clips with the arrows.
Real-world rollouts comparing co-training recipes. Switch the task and the In-Domain / OOD split; each column is a different method.
Performance.
Real-world rollout results across the three tasks. Switch the task, the In-Domain / OOD split, and the metric (success rate or normalized score). Each method (BC, Pixel, Pixel-PT, DINO, 3D Flow) is shown for three training regimes—Robot Only, + In-Domain Human, and + EgoVerse—with the percentage labels giving the gain over Robot Only. OOD averages the unseen-object and unseen-scene splits.
Alignment Ablation.
How robust is each approach when the human co-training data is misaligned with the robot? We compare co-training on in-domain (aligned) human data against deliberately unaligned human data, each measured against the method’s robot-only baseline. Behavior cloning degrades sharply under unaligned data—dropping below its robot-only baseline—while 3D-Flow world-model co-training stays robust.
Different grasping strategy and larger head motion
In-Domain Human DataUnaligned Human Data
Bag Grocery
Human-aligned-robot action and static camera
In-Domain Human DataAligned Human Data
Cup on Saucer
WAM co-training resists misalignment and scales with aligned data. Under unaligned human data, BC collapses to 20% (below its 40% robot-only baseline) while 3D Flow holds at 75%. Under aligned co-training, every world target improves over the in-domain setting, and 3D Flow is both highest and most stable (85% → 85%).
Representation (UMAP).
UMAP of the learned trunk representations, colored by data source. Toggle BC ↔ WAM to watch the embeddings morph. Under BC, robot and human data stay in separate clusters; WAM co-training pulls them into a shared space.
UMAP of learned representations. BC (HPT action features) isolates robot and human embeddings under unaligned actions, whereas WAM (3D-Flow co-training, concat features) aligns them into a shared space through task-relevant dynamics supervision.
World Prediction.
What each world target predicts from the same observation. Switch between Pixel (VAE), DINO (RAE), and 3D Flow, and compare a robot-only model against the co-trained (robot + in-domain human) model side by side.
Takeaways.
WAM co-training unlocks human-data scale. Where action-only BC stalls (or degrades) under misaligned human data, predicting future dynamics turns the same data into reliable gains.
World representation is the next critical axis. Pixels transfer weakly; abstraction is what matters—DINO drives object and scene generalization, while 3D flow grounds in-domain spatial gains.
A recipe for the target. An effective world representation should abstract appearance, keep effects consistent across embodiments, and factor out ego-motion—positioning human data as a flywheel for scalable robot learning.
Acknowledgements.
This work was supported in part by the Toyota Research Institute through the TRI University 3.0 program. We thank the members of the Robot Learning and Reasoning Lab (RL2) for helpful discussions and hardware support.
