§3.6 Summary

The four ideas worth carrying forward

Calculus is the API between a policy and an optimizer. §3.1 made the case that the gradient of the loss with respect to the weights is the only thing SGD ever sees. Everything else, the architecture, the tokenization, the data pipeline, exists to shape that gradient into something useful. The chain rule is the mechanism by which a scalar loss at the end of a 7-billion-parameter network produces a per-parameter update at the beginning. Two consequences follow. When a model fails to learn, the gradient is the first thing to inspect, not the architecture. And the manipulator Jacobian $J(q) \in \mathbb{R}^{6 \times 7}$ and the loss Jacobian $\partial L / \partial \theta$ are the same kind of object, computed by the same chain-rule decomposition; the only difference is whether the chain runs through joint angles to end-effector pose, or through network weights to a scalar loss. That’s why someone who can derive the Jacobian of a 2-link arm tends to pick up backprop quickly, and why someone who has trained a deep network on images tends to pick up inverse kinematics quickly. They’re reusing the same idea.

Cross-entropy and KL divergence are the same loss in two notations. §3.2 worked through this twice, once with the math and once with the OpenVLA action-bin example, because it’s the single most useful identity for reading the modern literature. RT-1 and OpenVLA describe themselves as training with “cross-entropy loss on discretized action bins”; π0 describes itself as training with a flow-matching loss equivalent to minimizing a particular KL divergence between forward and reverse processes. The descriptions sound different. They aren’t. Cross-entropy $H(p, q) = -\sum_i p_i \log q_i$ and KL divergence $D_{\mathrm{KL}}(p \| q) = \sum_i p_i \log (p_i / q_i)$ differ only by the entropy of $p$, a constant when $p$ is the target distribution. Minimizing one minimizes the other. Once that’s internalized, the loss function of essentially every supervised and self-supervised action model in the book reduces to one of three patterns: cross-entropy on tokens (RT-1, OpenVLA), MSE on continuous actions (ACT, Diffusion Policy, arXiv:2303.04137), or a denoising-style velocity loss (π0, arXiv:2410.24164). The variety in named loss functions is mostly notation.

A training loop is six lines doing the actual work and a few hundred lines of plumbing. §3.3 made this concrete by writing the SmallPolicy loop in 50 lines and annotating which six lines carry out the optimization step. The remaining 44 lines are data loading, logging, learning-rate scheduling, and device placement, necessary but mechanical. Read the OpenVLA training code, or the Diffusion Policy training code, or the π0 training code, and the six lines are present unchanged: forward pass, loss computation, zero gradients, backward pass, optimizer step, scheduler step. What differs between codebases is the plumbing, especially the data pipeline, and this matters because the plumbing is where almost all the bugs live. A correctly written six-line update running on a corrupted dataloader still produces a model that doesn’t work. The §3.5 diagnostic, printing action statistics before training a single step, exists precisely because the data pipeline is the default suspect.

The loss family determines the failure mode. §3.4 introduced the three families and §3.5 turned them into a diagnostic table. Supervised losses fail by overfitting, mode collapse on multimodal data, and label noise. Reinforcement-learning losses fail by reward hacking, sparse-reward stagnation, and reward miscalibration. Self-supervised losses fail by noise-schedule mismatch and integrator error at sampling time. Knowing which family your loss belongs to narrows the hypothesis space the moment a run starts misbehaving. The whole point of organizing the chapter this way is that “my model isn’t training” isn’t a question anyone can answer, while “my supervised behavior-cloning loss plateaus at 1.2 nats per token after step 5000 on a dataset with two demonstrators” is a question with two or three plausible answers and a clear next experiment.

What you should be able to do now

Four concrete capabilities, in increasing order of how much the rest of the book relies on them.

You should be able to read a model card or paper appendix and identify the loss function in one pass. The OpenVLA paper says “we train with the standard next-token cross-entropy objective over 256 discrete action bins per dimension.” That sentence packs three §3.4 commitments, supervised family, cross-entropy form, tokenized output, and it lets you predict the §3.5 diagnostic suite you’ll need when fine-tuning the model. The Diffusion Policy paper says “we train an $\epsilon$-prediction denoising network conditioned on observations, with a fixed cosine noise schedule.” That sentence packs four commitments: self-supervised family, MSE-on-noise form, continuous output, fixed schedule. Reading either sentence quickly saves the half-day you’d otherwise spend reverse-engineering the loss from the training script.

You should be able to write a 50-line PyTorch training loop from scratch without consulting a reference. Not a state-of-the-art one, a working one: the six update lines, plus a dataloader, plus a logging hook, plus a learning-rate scheduler. The SmallPolicy from §3.3 is the template, and copying it for a new architecture should be a 20-minute job, not a two-day job. This skill matters because every chapter from 6 onward asks you to train at least one variant of the model under discussion, and the hands-on exercises assume you can stand up a loop without ceremony. The handful of moving parts the loop needs, gradient zeroing, backward call, optimizer step, scheduler step, are universal across PyTorch projects. The §3.4 loss family is the only piece that changes from chapter to chapter.

You should be able to debug a non-converging run in under an hour by working the §3.5 checklist: print action statistics, verify timestamp alignment, run for 10 steps on a batch of 4, inspect gradient norms, add NaN assertions, read the loss curve, apply the loss-family diagnostic. This checklist exists because in practice the cause is almost always one of seven things, and identifying which of the seven saves you from the much longer tail of “it must be the architecture.” A modern VLA fine-tune costs on the order of hundreds of GPU-hours; misdiagnosing gets paid back in days. The checklist is cheap by comparison.

You should be able to predict the architectural consequences of a loss choice. A paper proposing a new VLA that uses MSE on continuous actions with a single deterministic head should immediately prompt the question: how does it handle bimodal demonstrations? A paper proposing RL fine-tuning on a real robot should prompt: what’s the reward function, and how dense is it? A paper proposing a flow-matching action head should prompt: how many integration steps at inference, and what’s the latency budget? These aren’t gotcha questions. They’re the questions the §3.4 taxonomy implies, and asking them surfaces the design decisions a paper has made and the trade-offs it has accepted. By the end of Chapter 10 you’ll recognize each of these trade-offs as the entry point to a specific later chapter, with the §3.4 vocabulary as the index.

Where the chapter has set up the rest of the book

Part 1 is now complete. Three forward references are worth naming again, since they structure all of Part 2 and most of Part 3. The three-loss-families framing from §3.4 is the spine of Chapter 5 (RL family in depth), Chapter 6 (imitation family in depth), Chapter 7 (deep RL), and Chapter 10 (the self-supervised family applied to action generation); each of those chapters elaborates one family into its own design space. The PyTorch loop from §3.3 is the substrate for every code listing in Chapters 6, 7, 10, 11, and 12; later chapters add larger models, bigger datasets, and richer logging, but the six-line update stays put. The debugging checklist from §3.5 is the substrate for Chapter 16’s fine-tuning recipes and Chapter 17’s deployment monitoring, both of which treat the checklist as a prerequisite rather than introducing their own.

The chapter hasn’t set up two things you might have expected. It hasn’t covered transformers, attention, or the architectural details of the models listed in Part 4; those wait for Chapter 8, where they get discussed alongside Decision Transformer and the lineage that produced RT-1. It also hasn’t covered the data side, Open X-Embodiment, LIBERO, CALVIN, SimplerEnv, which is the entire subject of Chapter 15. The §3.5 admonition to print action statistics before training a single step is a preview; the systematic treatment of robot datasets, their failure modes, and their benchmark properties sits twelve chapters away.

Part 1 closes here. Chapters 1, 2, and 3 together gave you the vocabulary of action models (Chapter 1), a working VLA you trained and broke yourself (Chapter 2), and the math, code, and debugging instinct that turn the rest of the book into a series of one-section variations on recognizable themes (Chapter 3). Part 2 starts in Chapter 4 with the classical-methods family, PDDL, IK, motion planning, computed-torque control, and walks the four-family taxonomy forward through Chapter 7. By the time you reach Chapter 11 and the VLA recipe proper, every component you encounter will be something you’ve already seen a smaller, earlier version of. That progression is the design of the book, and Part 1 is where the design gets set.

§3.x closes the chapter with one hands-on exercise, a debugging puzzle on an intentionally broken SmallPolicy run with the cause hidden in one of the seven places §3.5 named, and the full reading list for the chapter.