§2.4 When it works and when it does not

What success looks like

Run the §2.3 script unchanged on libero_object, task index 0, with a fixed seed. On a single GeForce 4090 the episode takes about 50 to 70 seconds of wall clock. The instruction is “pick up the alphabet soup and place it in the basket.” The Panda arm moves down and slightly forward, the fingers close on the soup can after roughly 60 to 90 control steps, the arm lifts, translates above the basket, and releases. reward jumps from 0.0 to 1.0 around step 180 to 220, done becomes True, the loop breaks. This sequence happens on roughly nine out of ten seeds; Kim et al. (2024, arXiv:2406.09246) report 88 to 90% success on libero_object for the released OpenVLA-7B checkpoint after their LIBERO-specific fine-tune, and that number reproduces within a couple of percentage points on a clean install.

Two things about that success run look unremarkable until they stop happening. First, the gripper closes once. It doesn’t chatter, doesn’t open and close every other step. A jittery gripper means the seventh-axis bin sits right at the 0.5 threshold and noise in the policy keeps flipping it, a sign of distribution mismatch rather than a working policy. Second, the arm’s motion stays smooth across timesteps even though each step gets decoded independently. OpenVLA doesn’t predict an action chunk; it predicts one 7-vector per inference call. The visual continuity of the resulting trajectory is an emergent consequence of the model having seen smooth demonstration trajectories during pretraining, not of any explicit smoothness loss. When a VLA’s trajectory looks jagged, that’s the second distribution-mismatch tell.

The three silent failures

A failure is silent when the loop keeps running, the tensors keep having the right shapes, and no exception gets raised, but the robot has quietly stopped doing the task. The script in §2.3 has exactly three of these built in, and each one is a one-character or one-keyword change away.

The image flip. Remove the [::-1] from obs["agentview_image"][::-1]. MuJoCo’s OpenGL convention puts pixel row 0 at the bottom of the image; OpenVLA’s vision encoders (SigLIP and DINOv2) were pretrained on web images where row 0 sits at the top. Without the flip, the model sees a world where the table hangs from the ceiling. The actions it produces stay syntactically valid 7-vectors, and the Panda still executes them, but the arm drifts away from the can and either swipes through it or stops at a height that has nothing to do with the actual geometry. Success rate on libero_object task 0 drops from around 90% to single digits. If you instrumented the loop with the five-line print from §2.3, the diagnostic is that img_mean looks fine (rendering works), but the trajectory itself makes no sense. Dump the agent-view PNG once per episode and the flip becomes obvious to the eye.

The prompt template. Change the prompt from "In: What action should the robot take to {instruction.lower()}?\nOut:" to anything else, even a seemingly equivalent rephrasing like "What action should the robot take? {instruction}". The model still emits seven action tokens, since the output layer is unconditional on prompt format. But OpenVLA was fine-tuned with that exact In:/Out: scaffold (see the OpenVLA repo’s prompt_builder and the discussion in arXiv:2406.09246 §3), and the language tower’s attention pattern over the visual tokens depends on it. Without the scaffold, the seven generated tokens drift toward the action-bin centers (128 is the bin mapping to roughly the middle of every per-axis range), and the commanded actions hover near zero. The arm twitches and never reaches the object. The diagnostic: np.round(action, 2) returns values clustered near zero, or clustered at the per-axis extremes, every step, regardless of the scene.

The unnorm_key. Change unnorm_key="libero_object" to unnorm_key="bridge_orig". Both keys exist in OpenVLA’s dataset_statistics.json; bridge_orig is the Bridge V2 embodiment from Open X-Embodiment (O’Neill et al., 2023, arXiv:2310.08864), which used a WidowX 250 arm with a much larger workspace and faster commanded velocities than LIBERO’s Panda. The model is doing the right thing in token space, still picking bins that correspond to “move down and forward,” but the bin-to-meters mapping is wrong by a factor of 3 to 5 on translation axes. The Panda’s controller saturates, the arm slams toward joint limits, and LIBERO either terminates the episode with an error flag in info or the simulation goes unstable. The diagnostic: magnitudes in your action print are obviously too large, |dx| exceeding 0.1 m per step for an embodiment that should be commanding 2 to 5 mm steps. This failure also makes a useful object lesson: action detokenization is embodiment-specific even when the policy backbone is shared, and the choice of unnorm_key is the cleanest example of a non-policy parameter that determines whether a VLA works at all. Chapter 11 returns to action tokenization in more depth; for now, treat unnorm_key as a required argument with no safe default.

These three failures aren’t exhaustive, but they’re the ones you’ll hit first, and each corresponds to one of the three transformation boundaries inside the loop: pixels into encoder, instruction into tokens, tokens into meters. A useful rule for debugging any VLA wrapper is to check each boundary in turn before suspecting the policy itself.

When it does not work even with the script right

Now make the script correct again and change the task instead. Switch from libero_object to libero_10, the long-horizon suite. Pick task 2, which involves picking a plate from one fixture, moving it to a stove, then returning to retrieve a separate item. Run the same loop. On the released OpenVLA-7B, success drops from around 90% to roughly 50 to 55% (Kim et al., 2024). Nothing is wrong with the loop; it’s identical to before. The policy is simply worse at sequencing two subtasks than at executing one. You’ll watch episodes where the plate gets grasped correctly, placed correctly, and then the arm never returns for the second item, instead re-approaching the now-empty plate location and stalling. This is a behavioral failure of the model itself, not an integration bug, and there’s nothing to fix on the integration side. The fix lives in Chapter 14, where dual-system architectures (Helix, GR00T N1; arXiv:2503.14734) introduce a slower high-level planner alongside the fast policy.

Distribution shift is the second category. LIBERO randomizes object positions on each reset but doesn’t randomize lighting, camera angle, or distractor objects. The moment you deviate from LIBERO’s defaults, adding a second can of soup to the scene, dimming the lighting, rotating the camera by 15 degrees, success on that same libero_object task 0 falls sharply. OpenVLA’s pretraining mix (Open X-Embodiment plus LIBERO fine-tuning data) covers a great deal of visual variation, but the specific combination of LIBERO’s renderer, camera, and lighting is exactly what the LIBERO checkpoint was tuned on. Visual robustness, and how to measure it, is a Chapter 15 problem.

Language brittleness is the third category. task.language for libero_object task 0 reads exactly “pick up the alphabet soup and place it in the basket.” Rewrite the instruction in the prompt to “put the soup can in the bin,” semantically identical, lexically different, and success drops by 10 to 30 percentage points depending on the seed. This isn’t a bug in the model. It’s a measurement of how much of OpenVLA’s language conditioning is verbatim memorization of training prompts versus genuine semantic understanding. The numbers are public (Kim et al., 2024, Table 4), and the gap is large. RT-2 (Brohan et al., 2023, arXiv:2307.15818) and π0 (Black et al., 2024, arXiv:2410.24164) close it partially by relying on larger language backbones. Neither closes it fully.

The fourth category has no satisfying fix at this layer of the stack: the model can fail at a task inside its training distribution simply because the per-episode physical configuration is unfavorable. The soup can spawns leaning against the basket lip, the gripper-can contact angle is bad, or the friction parameters MuJoCo sampled for this rollout cause the can to slip mid-lift. These failures are stochastic at the level of the simulator. They contribute to the roughly 10% of episodes that fail even on the easiest LIBERO suite, and they aren’t a sign anything is actually wrong.

Why one rollout does not tell you anything

The corollary of stochastic failure is that a single rollout carries almost no information. Run task 0 once and it succeeds, you still can’t distinguish “the model is at 90% success rate” from “the model is at 10% success rate and you got lucky.” The standard evaluation protocol for LIBERO, and for VLA work generally, runs 50 episodes per task across all 10 tasks in a suite, with seeded initial conditions, reporting mean success rate plus standard error. The OpenVLA paper uses 500 episodes per suite; reproducing within ±2% needs at least 100. On a single 4090, 500 episodes at roughly 60 seconds each comes out to about 8 hours of wall clock per suite, a useful planning number for the rest of the book and for §2.5, which covers what to chase and what to skip.

The instrumentation from §2.3, step, img_mean, action, reward, becomes especially useful during a 50-episode sweep. Dump it to a .jsonl file per episode; the file stays small (a few hundred KB per episode at 20 Hz), human-readable, and analyzable after the fact. A useful descriptive statistic is the step at first nonzero gripper command: across successful episodes it’s tightly distributed (gripper closes between step 60 and 100), while across failed episodes it’s bimodal (either never closes, or closes at step 5). That single statistic catches roughly 80% of the failure modes above without watching a single video.

With OpenVLA working on libero_object, three named failure modes debugged, and an honest read on where the model breaks, the remaining question is what the rest of the book does with this baseline. §2.5 lays that out.