§18.3 Video-pretrained action models

Why video is hard to learn actions from

A robot demonstration is a sequence of observations paired with the actions that produced them: image, then the joint command the teleoperator sent, then the next image. Behavior cloning (§6.1) needs that pairing, because it learns a map from observation to action, and the action is the training target. A YouTube video of someone slicing an onion has the observations, every frame, and none of the actions. You see the knife move; you do not see the torques the wrist applied, and you could not read them off even if you tried, because the human’s joints are not the robot’s joints. This is the label gap, and it is why decades of robot learning could not simply ingest video the way NLP ingested text.

There are two ways around the gap, and the field has tried both. The first is to use video only to learn representations, not actions: train a visual encoder on video so it produces features that make downstream action learning cheaper, then attach a small action head trained on the scarce robot data. R3M (Nair et al., 2022) was an early, clean version of this, and it worked, in the modest sense that a video-pretrained encoder beat one trained from scratch on the robot data alone. The limitation is that it only ever moves the visual front end; the action policy still needs robot demonstrations, so the data bottleneck is loosened, not broken. The second way is more ambitious and is where the recent excitement lives: learn a world model from video, a model of how the world evolves, and then use that model as the substrate an action policy plugs into, so that most of what the system knows about physics and object behavior came from video and only the last mile came from robot data.

The concrete result: V-JEPA 2 and V-JEPA 2-AC

For most of this book’s writing, video pretraining for control was a promising direction with thin evidence. That changed with V-JEPA 2 (arXiv:2506.09985), and it is worth being precise about what it showed, because the result is the strongest existing argument that the video bet pays off. §9.4 introduced V-JEPA as a video-prediction world model that learns by predicting in a latent space rather than reconstructing pixels, which §9.2’s latent-dynamics discussion motivated: predicting whether the representation of the next state is right is a better learning signal than predicting every pixel, most of which are irrelevant. V-JEPA 2 scaled that recipe on internet video, over a million hours, learning a world model with no robot data and no action labels at all.

The action-conditioned variant, V-JEPA 2-AC, is where control enters. Meta post-trained the video world model into a policy using a small amount of robot interaction data, under 62 hours of unlabeled robot video, and the resulting system did zero-shot pick-and-place on a real robot: reaching and grasping objects it was not explicitly trained to manipulate, in a new lab, without task-specific demonstration. The number that made people look twice was the comparison: V-JEPA 2-AC reportedly reached around 80% success on a held-out pick-and-place task where Octo (arXiv:2405.12213), a strong conventional generalist trained the usual way, managed roughly 15%. Take the exact figures as provisional, the way §18.1 said to take LingBot’s morphology count, and the shape of the claim is still the important thing. A world model built almost entirely from video, with a thin layer of robot data on top, beat a policy built the conventional way by a wide margin on a task neither was trained on. That is the video bet working, at least once, on real hardware.

What the result does and does not prove

Be careful about what to conclude. V-JEPA 2-AC did zero-shot pick-and-place, which is the short, forgiving end of the task spectrum, precisely the tasks §18.2 said the benchmarks over-reward. It did not do a two-minute dexterous task, and there is no result yet showing video pretraining closes the long-horizon or fine-manipulation gaps. What the result proves is narrower and still important: that a world model learned from action-free video transfers to control, that the physics and object-permanence and motion structure absorbed from watching are usable by a policy, and that the amount of robot data needed to convert the world model into a controller is small. That last point is the one that matters for the data bottleneck. If 62 hours of robot video can turn a video world model into a working policy, the economics of building a new robot’s policy change, because 62 hours is a weekend of teleoperation, not the fleet-years Open X-Embodiment represented.

The open questions stack up from there. Does the advantage hold on contact-rich tasks, where the video shows a hand doing something the robot’s gripper cannot copy? Human video has a worse embodiment gap than robot video, since a human hand is not a robot hand, and the label problem returns in a new form: even if you learn what happens from video, mapping it onto the robot’s own actions is unsolved for anything beyond gross reaching. And there is the architecture argument from §9.5, still unresolved, about whether the world model should be a separate module a policy plans against or whether the policy should absorb it, which V-JEPA 2’s separation-then-post-training approach answers one way and the end-to-end VLAs answer the other.

It helps to be concrete about why the pixel-versus-latent choice matters, because it is the design decision that made V-JEPA 2 work where earlier video models stalled. A pixel-reconstruction world model spends most of its capacity predicting things a controller does not care about: the exact texture of a wall, the flicker of a shadow, the leaves moving outside a window. None of that helps a robot decide how to grasp, and a model graded on reconstructing it wastes its learning on detail. Predicting in latent space, the §9.2 idea, changes the grading: the model only has to get the representation of the next state right, so it is free to throw away the pixels that do not matter and keep the structure that does, object identity, rough position, the fact that a pushed cup slides rather than teleports. That is why a latent video model transfers to control and a pixel one mostly does not, and it is the single most important thing to carry out of this section if you are choosing an architecture rather than just tracking the field.

Genie and the interactive-video direction

V-JEPA 2 is the prediction-and-planning branch. There is a second branch worth naming, because it attacks the label gap from the opposite side. Genie (Bruce et al., ICML 2024), introduced in §9.4, learns a generative interactive environment from video: it infers a latent action between consecutive frames, so it recovers a made-up action space from action-free video by asking what latent variable best explains the transition from this frame to the next. Where V-JEPA 2 learns a world model and adds real actions later, Genie invents an action space during pretraining and lets you act inside the learned world. For control, the appeal is that the inferred latent actions might align with real ones well enough to bootstrap a policy, turning action-free video into pseudo-labeled data. The line from Genie to a deployed robot policy is longer than V-JEPA 2’s and less demonstrated, but it is the cleaner statement of the dream: recover actions from video that never recorded them.

Video pretraining moved from a slide labeled “future work” to a real result on real hardware in the span of this book’s writing, which is the best reason to take it seriously and the best reason to be careful about it. The V-JEPA 2-AC number says the ocean of video is usable; it does not yet say it solves the hard tasks, and the embodiment gap between a human hand and a robot gripper is a real limit the pick-and-place result was allowed to sidestep. What the section leaves you with is a shift in where the advantage sits: for years the answer to “we need more robot data” was “collect more,” and video pretraining is the first serious argument that the answer might instead be “watch.” The next section turns to a different resource the field is chasing, not more data but more deliberation: letting a model reason in language before it acts.