§13.1 The trouble with discrete action tokens

Three costs, and only one of them is obvious

The resolution ceiling from §11.3 is the cost everyone names first, so start there and then keep going, because it is the least interesting of the three.

Uniform binning splits each action dimension into 256 buckets and reports the bin center. The reconstruction error is bounded by half a bin, roughly 0.002 in normalized units, and for a can slid across a table at 3 Hz that error vanishes under the robot’s own mechanical slop. Now thread a connector or seat a peg. The motion that distinguishes success from a jam is smaller than a bin, so the tokenizer rounds it away before the policy ever sees it. You can add bins, but the vocabulary grows and the classification the model has to learn gets harder, and you have started fighting your own representation. That much §11.3 already established.

The second cost is the one that quietly decides whether training even converges: sequence length. A transformer that predicts actions token by token has to emit one token per action dimension per timestep. A seven-DOF arm at 3 Hz over a two-second reach is manageable. A two-armed system at 50 Hz over a thirty-second folding task is thousands of timesteps times a dozen-odd dimensions, and now the autoregressive decode is a sequence in the tens of thousands of tokens. Attention cost grows with that length, training slows to a crawl, and the FAST paper (arXiv:2501.09747) reports the sharper failure: on some high-frequency dexterous datasets, per-dimension per-timestep binning does not just degrade, it fails to learn the task at all. The gradient signal drowns in the length of the sequence it has to model. FAST’s fix, compressing the action chunk with a DCT before tokenizing, buys back a lot of this, and it is a real fix, not a patch. But notice what it concedes. The moment you need a codec in front of your tokenizer to make the token count track the information in the motion rather than the raw sample rate, the “actions are just words” story has already sprung a leak. You are no longer reusing the language recipe unchanged; you are engineering around it.

The third cost is latency, and it is structural in a way the first two are not. Autoregressive decoding is inherently sequential: token $k+1$ cannot be sampled until token $k$ is out, because the model conditions on what it already emitted. A full action for a high-DOF robot might be dozens of tokens, and each one is a forward pass through the decoder. Put that inside a 7B-parameter VLM, as OpenVLA (arXiv:2406.09246) does, and every action costs you dozens of serial passes through a large network. RT-2 (arXiv:2307.15818) hit the same wall from the other side, serving a 55B model from a cloud TPU and landing control rates in the low single digits to about 10 Hz. For slow tabletop pick-and-place that is survivable. For contact-rich control where the arm has to react to a slipping grasp inside 20 milliseconds, a policy that thinks one token at a time is not in the running.

Why the costs cluster on exactly the tasks you care about

Here is the pattern worth internalizing, because it explains why the tokenization debate got loud right as the field moved toward capable robots rather than staying an academic footnote. The three costs above are cheap on easy tasks and expensive on hard ones, and they get expensive together.

Slow, coarse, single-stage manipulation, the RT-1 task distribution, is forgiving on all three axes at once: low control rate means short sequences, coarse motion means bin error is invisible, and a few hertz means latency has slack. Dexterous, high-frequency, multi-stage manipulation is unforgiving on all three at once: the fine motion needs resolution the bins do not have, the high rate and long horizon blow up the sequence length, and the tight reaction window leaves no room for serial decoding. There is no regime where tokenization is fine on two axes and only pinched on the third. The costs are correlated because they all scale with the same thing, the demand for fast precise motion over a long horizon, which is precisely the demand a useful home or factory robot places on its policy.

A concrete example fixes this. Take shirt-folding, one of the tasks π0 was built to demonstrate. It runs at 50 Hz on a two-arm setup, it lasts on the order of a minute, and the difference between a folded shirt and a wadded one lives in millimeter-scale adjustments of grip and tension. Every one of the three costs bites: the fold precision is below bin resolution, the minute-long 50 Hz two-arm trajectory is an enormous token sequence, and the cloth’s dynamics demand reactions faster than a dozen serial decoder passes allow. Try to fold a shirt with a per-dimension uniform tokenizer and you do not get a slightly worse fold; you get a policy that never learns to fold. That is the empirical wall FAST documents, and it is the wall π0 was designed to go around rather than climb.

The averaging trap you cannot simply regress your way out of

At this point the obvious rebuttal is: fine, drop the tokens and regress the continuous action directly, the way §11.3’s opening tension suggested. Squared error on a real-valued target, no bins, no vocabulary, no resolution ceiling. Why not?

Because naive regression walks straight into the multimodality problem from §10.4, and this is the reason discretization existed in the first place. When the same observation precedes a left turn in some demonstrations and a right turn in others, mean-squared-error regression minimizes its loss by predicting the average, which is straight ahead, into the obstacle both demonstrators avoided. Discretization sidestepped this for free: a distribution over 256 bins can put mass on the left bin and the right bin and leave the middle empty, and sampling gives you one real choice instead of a lethal compromise. Any continuous head that wants to replace the token head has to keep this property. It has to represent a truly multimodal distribution over actions, not collapse to a conditional mean.

That single requirement is why π0 does not use plain regression and instead reaches for flow matching. The action head has to be an expressive generative model over continuous actions, one that can carve out “either of these motions, not the thing between them” while still emitting smooth real-valued commands at full resolution. Diffusion policies and the ACT-style approaches from Chapter 10 are one family of answers to exactly this requirement; flow matching, which §13.3 develops, is the one π0 adopts, chosen partly because it generates an action in a handful of integration steps rather than the many denoising steps a diffusion sampler needs, which claws back the latency the token head was bleeding.

What this section sets up

So the case against discrete action tokens is not that tokens are bad. RT-1 and OpenVLA are proof they work, and FAST is proof that clever tokenization stretches them further than you would guess. The case is narrower and sharper: on fast, dexterous, long-horizon tasks the resolution ceiling, the sequence-length blowup, and the serial-decode latency all bite at once, and no amount of tokenizer engineering makes the third one go away, because autoregression is serial by construction. If you want a foundation model that folds laundry and buses tables at 50 Hz, you need an action head that emits a full continuous action chunk in one shot, represents multimodal action distributions honestly, and does it fast enough to close a real control loop.

π0 is one answer to that specification, and it is the answer this chapter takes apart. The next section lays out its architecture end to end: a pretrained vision-language backbone that carries the semantic knowledge from Part 4, feeding through cross-attention into a separate flow-matching “action expert” that does the continuous generation. Watch, as we go, for how cleanly the design splits the two jobs the token head was trying to do at once, understanding the scene and producing the motion, because that split is the whole idea.