§6.1 Why imitation is the dominant signal in modern robotics

The cost of a reward label vs. the cost of a demonstration

Ask yourself what it takes to train a policy to pick a strawberry off a plant and place it in a punnet without bruising it. The task requires detecting a ripe berry, planning a grasp that avoids the stem, applying exactly the right contact force, and releasing it gently. Defining a reward function for this is not impossible — you could reward task completion detected by a camera-based classifier — but writing a reward that is dense enough to get learning started, robust against reward hacking, and transferable across different berry sizes and positions is weeks of engineering. And even after all that work, the signal you get is a single bit per episode: success or failure.

A skilled human demonstrator, by contrast, provides a trajectory of continuous sensorimotor data: joint positions, end-effector forces, wrist camera images, and gripper state at 50 Hz for the fifteen seconds the task takes. That single demonstration carries roughly 750 labeled time steps, each with an action the expert considered correct for the state observed. The supervisory content per minute of human effort is orders of magnitude higher than a sparse reward. And the human already knows how hard to grasp, how to adapt when the berry slips, and when to abort — information that a reward signal does not transmit.

This is the core asymmetry: a reward signal labels outcomes; a demonstration labels every decision on the path to the outcome. For tasks where the path matters as much as the destination — which covers nearly all of robot manipulation — demonstration data is simply a richer supervisory signal per unit of collection effort.

A second asymmetry: demonstration data scales

Chapter 5 discussed sample complexity for reward-driven learning. Q-learning on a 16-cell gridworld converges in hundreds of episodes; Q-learning on a continuous 20-dimensional manipulation state requires millions. The sample count scales with the complexity of the state-action space, not with the number of tasks being learned, and there is no known way to avoid this.

Demonstration data has a different scaling curve. If you collect 100 demonstrations of pick-and-place, you have 100 data points. If you collect 200, you have 200. The relationship is linear and the collection is parallelizable: ten teleoperators working simultaneously produce data ten times faster than one. By 2022, the Open X-Embodiment collaboration (arXiv:2310.08864) had aggregated over 1 million demonstration episodes across 22 robot embodiments and 527 skill categories. There is no RL dataset with comparable coverage for real-robot manipulation, and no obvious path to creating one, because autonomous RL data collection at scale requires autonomous robots operating in the real world — which is the thing you are trying to build in the first place.

The data flywheel works in favor of imitation. Every hour of teleoperation produces new training data directly usable by a behavior-cloning objective. Every new robot that comes online can contribute to the same dataset, regardless of whether it shares reward semantics with the others. The aggregation is straightforward because demonstration labels are states and actions, which are physically meaningful across embodiments (at least approximately), not task-specific scalars.

What “imitation learning” covers

The phrase “imitation learning” is used broadly in the literature to cover several distinct approaches that share the property of using expert behavior as training signal. It helps to place them upfront:

Behavior cloning (BC) treats demonstration data as a supervised regression problem: given a state, predict the action the expert took. No explicit model of the expert’s intentions, no reward function, no environment interaction during training. BC is §6.2’s subject, and it is the simplest and most widely deployed variant.

DAgger and variants address BC’s main failure mode — compounding error — by interleaving policy execution with expert correction: the learner runs in the environment, and the expert labels the states the learner reaches, not just the states the expert visited. §6.3 covers this in detail.

Inverse reinforcement learning (IRL) inverts the MDP problem: rather than assuming a reward and computing the optimal policy, IRL assumes the demonstrations are (approximately) optimal and infers a reward function that would explain them. The inferred reward can then be used to run RL. §6.4 examines IRL and its adversarial formulation (GAIL).

Offline RL uses demonstration data as a fixed dataset and runs a modified RL algorithm that stays close to the demonstrated behavior while optimizing a reward signal. It sits between BC and RL in the learning landscape, and we visit it briefly in §6.5 when comparing the approaches.

The chapter focuses on BC and DAgger because those two are the workhorses of current VLA training. RT-1, OpenVLA, Octo, π0 — all are trained primarily with behavior cloning on large demonstration datasets, with no reward signal in the loop. Understanding why BC works at scale, and where it breaks, is prerequisite knowledge for reading any modern VLA paper.

The historical arc: ALVINN to modern VLAs

It is worth tracing how imitation reached this dominant position, because the arc is not a smooth one — it involves two distinct inflection points separated by decades.

Dean Pomerleau’s ALVINN system (1988) demonstrated autonomous car driving by training a three-layer neural network on 45 minutes of human steering data. Input: a 30×32 image from a forward-facing camera. Output: a single steering angle. Training signal: the human’s wheel position at each frame. The network generalized to novel roads at up to 55 mph. This was behavior cloning before it had a name, and it worked remarkably well for 1988.

ALVINN faded from view because it generalized poorly: a network trained in one lighting condition failed in another, and the small dataset meant the model had encountered very few of the situations it would face on deployment. The decade that followed was dominated by explicit rule-based autonomy and then, when DARPA’s Urban Challenge revived interest in autonomous driving, by a probabilistic pipeline of perception, mapping, and planning modules — the opposite of end-to-end learning.

The second inflection was the arrival of large, high-quality teleoperation datasets and the transformer architecture. The 2022–2023 period saw both at once: Open X-Embodiment aggregating cross-embodiment data at scale, and RT-1 demonstrating that a transformer trained on 130,000 demonstration episodes could learn a single policy covering over 700 tasks. The difference from ALVINN was not the algorithm — BC in both cases — but the data volume and the model capacity. Pomerleau’s network had fewer than a thousand parameters; RT-1’s had 35 million; OpenVLA’s has 7 billion. The gap in representational capacity is what allows modern systems to interpolate across diverse demonstrations rather than overfit to narrow distributions.

Why not just use RL then?

A reasonable objection: if RL and BC both work, why not use RL more? After all, RL does not require a human demonstrator on the critical path, and in principle it can discover solutions better than any human demonstration if given enough exploration time.

The answer is not that RL is broken — Chapter 7 shows that SAC and PPO work very well in simulation for tasks where reward design is tractable and sample counts are not a constraint. The answer is that the preconditions for RL are rarely met in practical robot deployment. You need a reward signal, a simulator, enough samples to explore effectively, and a sim-to-real transfer procedure that works for your task. All four preconditions must hold simultaneously, and for the overwhelming majority of tasks encountered in service and manufacturing robotics, at least one fails.

The Mandlekar et al. (2021) study, “What Matters in Learning from Offline Human Demonstrations for Robot Manipulation,” is instructive on the comparison. They found that a BC policy trained on 200 demonstrations outperformed SAC trained from scratch on the same tasks when the reward was sparse — which it is in most real settings. The implication is not that BC is generically better than RL; it is that BC’s performance curve reaches acceptable levels at demonstration counts that are actually achievable, while RL’s performance curve reaches acceptable levels at sample counts that are not.

The practical synthesis of the field, circa 2024–2025, is to use imitation as the primary training signal and RL as a fine-tuning signal when reward is available and the policy is already close to competent. π0 (arXiv:2410.24164) is trained with a BC-style flow-matching objective on demonstration data, then optionally fine-tuned with online RL for specific downstream deployments. This ordering is not accidental: a policy that has never seen a demonstration of the task will explore poorly under RL, because good exploration requires already knowing roughly what good behavior looks like. Starting from BC gives RL a sensible initialization.

What this chapter builds

The remainder of Chapter 6 develops the mechanics of imitation learning in order of increasing sophistication. §6.2 sets up behavior cloning formally and walks through the training loop. §6.3 analyzes the compounding-error failure mode — the reason BC alone is insufficient for tasks with long horizons — and presents DAgger as the fix. §6.4 examines what happens when you want to extract a reward function from demonstrations rather than a policy directly. §6.5 synthesizes the comparison, giving you the decision criteria for choosing between BC, IRL, offline RL, and online RL on a real task.

The framing throughout is practical: given a dataset and a robot and a task deadline, which algorithm and how much data. That framing matters because imitation learning is not one algorithm but a family, and picking the wrong member of the family for your constraints is a common and avoidable failure.

Section 6.2 opens the BC training loop and works through it line by line.