§16.4 Sim-to-real fine-tuning loops

Two jobs sim can do, and one it cannot

Simulation plays two distinct roles in a fine-tuning pipeline, and conflating them is where people go wrong. The first role is data augmentation: generating additional training episodes in sim to supplement a small real dataset, usually with domain randomization (Tobin et al., 2017) that varies textures, lighting, object poses, and physics parameters so the policy cannot lean on any single simulator artifact. The bet, the same one §7.5 laid out, is that if the policy sees enough randomized variation, the real world looks like just another sample from the distribution, and the gap shrinks to something the fine-tune can cross. The second role is evaluation, the SimplerEnv (§15.4) use: running the fine-tuned policy in a sim built to correlate with real success, as a cheap screen before you spend hardware time.

The job sim cannot do is certify. A high sim number is evidence, not proof, and §15.4 was blunt that even SimplerEnv’s correlation is a property of the policies it was validated against, breakable by a new architecture that games the visual match. So the loop below uses sim to decide what to test on hardware and to cheaply kill obviously-broken fine-tunes, and it uses hardware to decide what actually works. It never lets a sim number stand as the final answer.

The loop

The loop is the §15.6 procedure turned into a training cycle. One turn of it looks like this. You fine-tune on your current dataset, real plus any sim augmentation. You screen the result in sim across several seeds, computing the confidence interval from §15.x rather than trusting one number, and if it is clearly broken you fix it before touching the robot. You then run a small, honest real evaluation on the survivors, logging every trial as a full episode with a failure mode. You read the failures, and, the move that closes the loop, you collect or generate the next batch of data targeting exactly those failures, whether that means more real recovery episodes (§16.2) or more randomized sim episodes in the failing condition. Then you turn the crank again.

Two properties make this work. It spends real robot time only on policies that already survived the cheap screen, so hardware evaluation is confirming, not exploring. And it grows the dataset toward the policy’s demonstrated weaknesses, which is the DAgger intuition (Ross et al., 2011) from §16.2 operating at the level of whole training rounds: the states where the policy fails are the states you add data for. A fine-tune that stalls is almost always one where this loop was skipped, where someone trained once, tested once, and concluded the method does not work.

Three failure modes, and how to tell them apart

Fine-tuning fails quietly. The loss goes down, the training curves look healthy, and the robot still does not work, because a low training loss measures how well the policy fits your data, not whether it does your task. Three failure modes account for most of what goes wrong, and the reason to name them is that they look similar on the robot and demand opposite fixes.

Over-specialization is the fine-tune that learned your dataset too literally and nothing around it. On the robot it presents as a policy that works beautifully in the exact conditions you collected, the same object, the same spot, the same light, and falls apart the instant anything shifts. This is catastrophic forgetting (§16.3) and narrow data (§16.2) showing up together: the model overwrote its general competence and your data did not cover the variation to replace it. The tell is a large gap between success on in-distribution starts and success on shifted ones. The fix is on the data and the method side, not the training-longer side: widen the coverage of the dataset and lower the LoRA rank or reduce training steps so you change the base model less. Training harder makes over-specialization worse, which is the counterintuitive part people get wrong.

Mode collapse is the policy that has stopped representing the multimodality the task actually has. Many manipulation tasks admit several valid solutions, grasp the mug from the left or the right, and behavior cloning on a dataset containing both, trained with a loss that rewards matching the average, can produce a policy that commits to the mean of the two: reaching straight down the middle and grasping nothing. On the robot it shows as a policy that freezes, jitters between options, or takes a blended action that satisfies no mode. This is exactly the failure §10.5 said continuous multimodal heads (diffusion, flow-matching) exist to prevent, so one fix is architectural, use or fine-tune a model whose head can represent multiple modes. The data-side fix is to reduce the ambiguity the policy has to resolve, by conditioning more precisely, sharper language or an added goal, so that each state maps to one clear action rather than an average of several.

Action-token mismatch is the convention bug from §16.2 wearing its consequences. If your logged actions are in a different convention, units, or frame than the base model expects, or if OpenVLA’s normalization statistics do not match your robot’s action range, the model trains happily on numbers that mean the wrong thing, and the loss drops because it is predicting your (mislabeled) actions accurately. On the robot it presents as confident, smooth motion to the wrong place, or motion at the wrong scale, over- or under-shooting consistently. It is the easiest of the three to prevent and the most embarrassing to diagnose late, because the fix is the open-loop replay check from §16.2 that you should have run before collecting the full dataset. If a fine-tune produces coordinated-but-wrong behavior, suspect the convention before you suspect the model.

The diagnostic that separates them is cheap and worth building into the loop. Evaluate on three sets: the exact training conditions, mildly shifted conditions, and a replay sanity check. A policy that fails the replay check has a token mismatch. A policy that passes training conditions but fails shifted ones has over-specialized. A policy that fails even the training conditions with frozen, blended, or hesitant motion has collapsed a mode. Same symptom on the surface, three different root causes, and reading the pattern saves you from applying the wrong fix and making things worse.

Co-training and the ratio that matters

When you do use sim data alongside real, the mixing ratio is a knob, and the safe default is to keep real data well represented rather than letting a flood of cheap sim episodes drown it out. A common pattern is co-training, sampling each training batch from both sources with real weighted up relative to its raw count, so the policy grounds on real observations while borrowing the sim data’s coverage. Too much sim and the policy optimizes for simulator artifacts the domain randomization did not cover; too little and you have not bought any of the volume sim was supposed to give you. The chapter exercise does not go deep on this because a single low-cost arm rarely has a matching high-fidelity sim, but for setups that do, notably the SimplerEnv-covered embodiments, co-training with real weighted up is the setup that most reliably transfers.

You now have the pieces: a base model, a dataset built to cover and to recover, a fine-tuning method that fits your hardware, a loop that uses sim honestly, and a diagnosis for the three ways it fails. §16.5 compresses all of it into something you can pin above your desk, a one-page recipe card for pointing this whole process at a robot you have never fine-tuned for before.