§6.3 Compounding error and DAgger

The feedback loop nobody trained for

Start with the oldest concrete example in the field. ALVINN (Pomerleau, 1988) learned to steer from 45 minutes of human driving. A human driver, being competent, keeps the car centered in the lane. Consequently the training data contains thousands of frames of well-centered road and essentially zero frames of the car drifting onto the shoulder, because the expert never let that happen. Now deploy the network. It is a function approximator, so it makes small errors; suppose it steers slightly wide on one curve. The camera now shows a view the dataset barely covers: lane markings at an angle the expert never produced. On this off-distribution input, the network’s output is less reliable, so the next steering command is likely worse, which produces a still-stranger view, and the loop runs away. One small error did not cost one small penalty. It relocated the policy to a region of state space where all its subsequent decisions are unreliable.

Pomerleau saw this on real roads and patched it with a hack that has aged remarkably well: he synthetically shifted and rotated the camera images to simulate the car being off-center, and computed the corrective steering label geometrically. ALVINN was trained on recovery behavior the human never demonstrated. Hold that thought; the modern equivalents of this trick still do a lot of quiet work in current pipelines.

The general structure is worth stating plainly, because it separates imitation learning from every supervised problem you have met so far. In image classification, your prediction does not change the next image you are shown. In control, it does. The policy’s inputs at time $t+1$ are a consequence of its output at time $t$. Errors do not average out over a trajectory; they accumulate, and worse, they correlate: each error increases the probability of the next.

Why the cost is quadratic in the horizon

Ross and Bagnell (2010) made the accumulation precise, and the argument fits in a paragraph. Suppose the learned policy disagrees with the expert with probability at most $\varepsilon$ on states drawn from the expert’s distribution; this is what your validation loss (imperfectly) measures. Walk the trajectory step by step. At each step where the policy has so far behaved like the expert, it errs with probability $\varepsilon$. But once it has erred, all bets are off: it is now off the expert’s distribution, where its error rate is unbounded, and the analysis can only assume the worst for the remaining steps. An error at step 1 can poison $T-1$ subsequent steps; an error at step $T$ poisons none. Summing over the horizon, expected total cost is bounded by $\varepsilon \cdot T^2$ in the worst case, and the bound is tight: there are MDPs that actually realize it.

Contrast the supervised baseline. If the problem really were i.i.d., if someone handed the policy states drawn from the expert distribution at every step regardless of its past actions, total cost would be $\varepsilon T$. The gap between $\varepsilon T$ and $\varepsilon T^2$ is the price of the feedback loop, and it explains an everyday observation from §6.2: BC policies look great on short-horizon tasks and degrade sharply on long ones. Lifting a cube is a 50-step problem; the policy that succeeds 90% of the time there will not survive a 500-step kitchen-tidying task with ten times the polish. The horizon enters squared.

It is worth being honest about what the bound does and does not say. It is a worst case; real environments are often forgiving, with recoverable states and self-correcting dynamics (a slightly misaligned gripper above a cube often still descends into a workable grasp). The quadratic is not destiny. But it identifies the right villain: not the size of $\varepsilon$, which more data and bigger models steadily shrink, but the support of the training distribution, meaning what the policy has never seen.

DAgger: ask the expert about the learner’s mistakes

If the problem is that the dataset only covers expert states, the fix suggests itself: get labels on the states the learner visits. DAgger, Dataset Aggregation, from Ross, Gordon and Bagnell (2011), turns that into an algorithm:

D ← initial expert demonstrations
π₁ ← train BC on D
for i = 1, 2, ..., N:
    roll out πᵢ in the environment
    for every state s visited, query the expert's action a*(s)
    D ← D ∪ {(s, a*(s))}          # aggregate, never discard
    πᵢ₊₁ ← train BC on D
return best πᵢ under rollout evaluation

Two details matter. First, the learner drives and the expert labels. The states in the new data are exactly the off-distribution states that the quadratic bound worried about, including the botched approaches and near-misses that no expert demonstration contains. After a few iterations, the dataset covers the learner’s mistakes and, crucially, the expert’s corrections from them. Second, the aggregation: each policy is retrained on the union of all data so far, not just the latest batch. This is what supports the theory. DAgger is analyzed as a no-regret online learning procedure, and the punchline is that the quadratic goes away: with enough iterations, expected cost is $O(\varepsilon T)$ plus terms that shrink with iteration count. Linear in horizon, like proper supervised learning. (In early iterations the rollout policy is sometimes mixed with the expert, executing the expert’s action with probability $\beta_i$, decayed toward zero, to keep the first, worst policies from spending whole episodes in useless corners of state space.)

On the robomimic lift setup from §6.2 the effect is easy to reproduce in simulation, because there the “expert” can be a scripted or pretrained policy that is queryable for free: a BC policy trained on 200 demonstrations and stuck at 90% will, with two or three DAgger iterations of a few dozen rollouts each, typically clear the failures caused by drift, the slow sideways slide of the gripper that ends centimeters from the cube, because the dataset now contains exactly those slides, labeled with the correction.

The catch: who is this expert that answers queries?

Now the bad news, which is the reason this section is not titled “DAgger: problem solved.” DAgger’s expert must label arbitrary states on demand. In simulation with a scripted expert, fine. With a human expert and a real robot, the query model is awkward in two distinct ways.

The first is cost. Expert effort is per-state, on the critical path, every iteration; the parallel, offline collection economics that §6.1 credited for imitation’s dominance do not apply.

The second is subtler and worse: humans are bad at the query itself. Shown a frozen frame of a mid-failure state and asked “what action would you take here?”, a teleoperator gives noisy, mutually inconsistent answers. Humans demonstrate well in closed loop, with the robot responding, and label poorly out of context. Practical variants therefore restructure the interaction: let the human watch the policy run and take over when it goes wrong (gated or intervention-based DAgger), so the expert provides closed-loop corrections only on the segments where the policy actually needs help. The takeover states are precisely the off-distribution states DAgger wants labeled, and the human supplies them by demonstrating, which humans are good at, rather than by annotating, which they are not.

If that interaction pattern sounds familiar, it should: it is a deployment data flywheel. A fleet of robots runs the current policy at customer sites; human operators intervene on failures; the interventions are logged and folded into the next training run. That is intervention-based DAgger at industrial scale, and it is, as far as public information allows one to tell, roughly how commercial robot fleets improve after deployment. The algorithm from a 2011 AISTATS paper survives as an operations playbook.

What VLA training actually does about compounding error

Look at the training recipe of RT-1 (arXiv:2212.06817) or OpenVLA (arXiv:2406.09246) and you will find no DAgger loop, just pure BC on a fixed dataset. Are the theorists wrong, or are the practitioners lucky? Neither. Modern pipelines attack the same villain, training support, by other means, and it is worth naming them, because they look like unrelated engineering choices until you see them through this section’s lens.

Coverage by brute diversity. A dataset like Open X-Embodiment (arXiv:2310.08864), with a million episodes from dozens of buildings, operators, and embodiments, has vastly wider state support than 200 demonstrations of one cube on one table, including plenty of accidental near-failure states, since across thousands of hours operators wobble, retry, and recover. The recoveries that Pomerleau synthesized geometrically, scale collects by accident.

Shorter effective horizons. Action chunking (§6.2, and properly in Chapter 10) predicts $k$ steps per decision, cutting the number of closed-loop decisions per episode by a factor of $k$. If the cost is quadratic in the number of decisions, chunking buys a $k^2$ improvement in the worst case, a large part of why ACT and Diffusion Policy hold up on long-horizon manipulation.

Robust features. A pretrained visual backbone maps superficially novel observations, new lighting, new clutter, near familiar ones in feature space, so the policy is effectively “on-distribution” in feature space more often than raw pixel statistics would suggest. This shrinks how much off-distribution drift it takes to reach truly undefined behavior.

None of these eliminates compounding error; they postpone it. Push any current VLA on a long enough task (Chapter 15’s evaluation chapter shows this quantitatively on LIBERO) and the familiar signature reappears: success rates that decay with episode length, failures that begin with one small slip followed by increasingly confident nonsense. When you see that signature, you now know its name, its scaling law, and the two families of remedy: widen the data, or shorten the loop. And when fine-tuning a VLA on your own robot in Chapter 16, the single highest-value data you can collect is intervention data, DAgger’s idea, wearing work clothes.

There remains a question BC and DAgger both dodge: they clone what the expert does without ever asking why. The next section takes up the alternative, inferring the reward the expert seems to be optimizing, and letting the policy pursue that instead.