§11.5 The data side: when does scale start to pay off

The scaling law you know does not transfer cleanly

The language-model scaling laws are clean because they hold across many orders of magnitude: loss falls as a smooth power of parameters, data, and compute, and you can read off how to trade one against another. Robot learning has nothing that clean, and the reason is not that roboticists are sloppier. It is that the axis that matters is not the raw count of demonstrations but their diversity, and diversity is much harder to put on the x-axis of a plot.

Consider what “one more demonstration” means. Collect a thousand more pick-and-place demos of the same block on the same table under the same lighting, and your model gets very good at that block on that table and learns almost nothing transferable. Collect a thousand demos spread across new objects, new surfaces, new phrasings of the instruction, and you buy robustness. Same count, wildly different payoff. RT-1’s own ablations (arXiv:2212.06817) make the point without any fancy analysis: when the authors cut the number of distinct tasks in the training set while holding the demo count roughly fixed, performance fell off harder than when they cut demos while keeping task variety. The model was living off breadth, not volume. That is the first thing to internalize before you buy a fleet of robots: counting demonstrations is measuring the wrong thing.

The floor, the knee, and the plateau

It helps to picture three regimes, even though the boundaries are fuzzy and dataset-dependent.

Below a certain diversity floor, you are in memorization territory. The model reproduces what it saw and falls over the moment the world shifts, exactly the compounding-error failure from §6.3, because it never saw enough variation to learn what to ignore. Adding more of the same data here does close to nothing for generalization. A specialist policy trained on a narrow task will often beat a “scaled” model in this regime, which is why so many working robots in factories are still narrow specialists and not foundation models. When someone tells you scale did not help them, the usual diagnosis is that they never left the floor.

Then there is a knee, the interesting part, where enough variety accumulates that the model starts inferring the structure behind the tasks instead of the tasks themselves. This is where RT-1’s headline results live: the robustness to unseen distractors, backgrounds, and instruction phrasings that §11.4 credited to breadth. You cannot predict exactly where the knee sits for a new problem, and that uncertainty is genuinely annoying, but you can recognize it after the fact because generalization to held-out conditions stops being flat and starts climbing with each increment of diversity.

Past the knee, returns compress again into a slow plateau. More data still helps, though each doubling buys less, and you are increasingly paying for the long tail of rare situations. Nobody has cleanly mapped where robot policies enter this plateau, partly because almost no group has enough data to get there on a single embodiment. RT-1’s 130,000 demonstrations, gathered by thirteen robots over roughly seventeen months, are enough to show the knee and not much more. That number is worth sitting with. Seventeen months of continuous fleet operation produced a dataset that a language model would consume before breakfast.

How do you tell which regime you are actually in, before spending another quarter on data collection? The practical test is cheap: hold out conditions, not just trajectories. Split your evaluation so that some objects, some backgrounds, and some instruction phrasings appear only at test time and never in training. Then plot success on that held-out split as you grow the dataset in increments. If the held-out curve is flat while the seen-condition curve is high, you are on the floor and collecting more of the same will not save you; you need variety or a specialist. If the held-out curve has started to climb with each increment, you have found the knee and further diverse data is worth the money. This is the single most useful measurement in the whole enterprise, and it is astonishing how often teams skip it and report only in-distribution numbers, which tell you nothing about whether scale is paying.

Why the demonstrations are so expensive

The economics are the whole story, so it is worth being concrete about them. A single teleoperated demonstration requires a human operator, a working robot, a reset of the scene between attempts, and time. Call it a minute or two of skilled human labor per usable episode once you count failures, resets, and the ones you throw away. There is no equivalent of scraping. You cannot download a manipulation trajectory the way you download a webpage, because the trajectory only exists if someone generated it on hardware that matches, or at least resembles, the robot you intend to deploy.

This is the constraint that bends everything else in the field. It is why simulation is attractive despite the sim-to-real gap (Chapter 7). It is why people chase learning from human video, where the data is abundant but the embodiment does not match, a thread we pick up in Chapter 18. And it is the direct cause of the two dominant strategies for getting scale without personally collecting millions of demos: pool everyone’s data, or borrow scale from a different modality entirely.

Strategy one: pool the data (Open X-Embodiment)

If your lab cannot afford to reach the knee alone, the obvious move is to combine your data with everyone else’s. That is precisely what Open X-Embodiment (arXiv:2310.08864) did. The project pooled robot data contributed by many institutions into a single corpus spanning more than twenty distinct robot embodiments and on the order of a million trajectories, then retrained RT-1-style and RT-2-style architectures on the mixture, producing the “RT-X” models.

The result matters more than the corpus size. A policy trained on the pooled multi-robot data generally outperformed the same architecture trained on any single contributor’s data alone, and the authors observed positive transfer across different robot bodies, meaning data collected on one arm improved performance on a mechanically different one. Read that against the diversity argument and it lands hard: pooling does not just add demo count, it adds exactly the kind of variety that pushes a model past the floor and toward the knee, and it does so cheaply for any individual participant because everyone contributes a slice and receives the whole. Open X-Embodiment is the field’s answer to the expense problem, and it works because diversity, not volume, was the binding constraint all along.

Strategy two: borrow the scale from the web

The second escape hatch is the one that defines everything in Chapter 12, so this section only plants the flag. If robot demonstrations are scarce and web image-text pairs are abundant, start from a model that already digested the abundant data and let the scarce data do only the last mile. That is the entire logic of building a VLA on top of a pretrained vision-language model. The web pretraining supplies the visual and linguistic breadth that would otherwise demand an impossible number of teleop episodes; the robot data only has to teach the mapping from that grounded understanding to motor commands.

OpenVLA (arXiv:2406.09246) is the concrete case. It is a 7-billion-parameter model fine-tuned for control on roughly 970,000 robot episodes drawn from the Open X-Embodiment collection, and it reaches strong generalization on a demonstration budget that would be hopeless if the model were starting from random weights. The robot dataset is large by robotics standards and still tiny next to the text and image corpora behind the backbone. That asymmetry is the point. You are not scaling the robot data to language-model sizes, which is infeasible; you are scaling a different, cheaper axis and spending your precious demonstrations only where nothing else will do.

The two strategies are not rivals; the strongest recent models use both at once. OpenVLA borrows web scale through its backbone and pooled-robot scale through its OX-E training mixture, and the mixture itself is weighted rather than uniform, oversampling the more varied datasets and downweighting the repetitive ones. That weighting is the diversity argument showing up as an engineering knob: given a fixed training budget, you spend gradient steps where the variety is, not where the raw episode count happens to pile up. When you fine-tune one of these models for your own robot in Chapter 16, you are effectively adding one more small, targeted slice to a mixture that already carries most of the breadth you need.

So when does scale start to pay?

Put the pieces together and the answer stops being mystical. Scale starts to pay when the diversity of your data crosses the knee for the distribution you actually care about, and not one demonstration sooner. Below that, you are buying memorization and a specialist would serve you better and cheaper. The knee arrives faster along the diversity axis than the raw-count axis, so a smaller, more varied dataset frequently beats a larger, more repetitive one. And you can move the knee dramatically closer by not starting from scratch: pooling with other robots (Open X-Embodiment) or standing on web pretraining (the OpenVLA move) both reduce how much of your own expensive data you need to spend before generalization switches on.

There is a deflating corollary that the RT-1 story from §11.4 already hinted at. For most people, the winning strategy is not to collect more data. It is to reuse data that already exists, from shared corpora and from pretrained backbones, and to spend a modest, targeted demonstration budget on the specific gap between what those give you and what your robot needs. Chapter 16 turns that into an actual fine-tuning recipe. The scaling question, framed this way, is less “how many demonstrations should I gather” and more “how little of my own data can I get away with collecting, given what I can borrow.” That reframing is the practical payoff of this whole chapter, and it sets up the summary in §11.6, where we consolidate the CLIP-to-RT-1 arc into the handful of ideas worth carrying into the rest of Part 4.