§11.4 What RT-1 changed and what it did not

What actually stuck

Start with the piece that outlived everything else: the framing. Before RT-1, the ambient assumption in manipulation was one policy per task, or at best one policy per narrow task family, each trained on its own carefully curated data. RT-1 made a different bet stick, which is that a single network absorbing hundreds of tasks from one big heterogeneous dataset beats a portfolio of specialists, and that its performance keeps climbing as you add data and task variety instead of plateauing. That claim had been made before in spirit. RT-1 backed it with numbers on real hardware across 700-plus instructions, and once the numbers existed the argument was over. Every model in Part 4 inherits this framing. When Chapter 12 talks about OpenVLA or Octo “generalizing,” it is standing on the ground RT-1 cleared.

It helps to attach numbers to the claim. RT-1 reported near-ceiling success on the tasks it was trained on and, more tellingly, held up well when the world shifted underneath it: performance stayed high with unfamiliar distractor objects cluttering the scene and with backgrounds and lighting the policy had not been trained in, and it completed a solid majority of instructions it had never seen phrased that way. Those robustness axes, not the raw seen-task score, are what convinced people. A specialist can memorize its way to a high number on tasks it trained on; staying upright when you add a distractor or swap the kitchen is the thing a portfolio of specialists never gave you for free, and it is what breadth of data bought.

The second durable contribution is more concrete: control as next-token prediction over a discretized action space, trained with the same cross-entropy loss a language model uses. Section 11.3 already argued why this matters and where it breaks, so the short version here is that RT-1 proved the trick works at scale on a physical arm. That proof is what let RT-2 (arXiv:2307.15818) fold robot actions into the token stream of a web-pretrained vision-language model a year later without redesigning anything. The tokenization scheme itself got improved on, but the idea that a robot policy could be a plain sequence model with an action vocabulary was RT-1’s to establish, and it held.

Third, and easy to overlook because it sounds like plumbing: RT-1 showed you could run a transformer policy on a real robot at a usable rate. A decoder-only transformer over six frames of history is not free, and a robot that thinks at 3 Hz is a robot that knocks the cup over while it deliberates. The TokenLearner compression from §11.2, squeezing each frame’s feature grid down to a handful of tokens, was the unglamorous engineering that kept inference inside the control loop. People remember the scaling story and forget that none of it would have mattered if the thing ran too slowly to close a gripper on a moving target. Real-time feasibility on real hardware was part of what RT-1 demonstrated, and it is why the paper reads as an engineering result and not just a benchmark.

What got replaced almost immediately

Now the parts that did not last. The most obvious one is the vision-language conditioning stack. RT-1 embedded instructions with a Universal Sentence Encoder and modulated a FiLM-conditioned EfficientNet, a perfectly reasonable 2022 design that looks quaint by 2023. The problem is the one §11.2 flagged: the language understanding is only as deep as the sentence encoder, which knows about English but nothing about the visual world. Ask RT-1 for something phrased far outside its 700 instructions and the conditioning quietly corrupts perception rather than admitting confusion.

RT-2’s whole thesis is a direct response to this. Instead of a small sentence encoder bolted to a vision network trained only on robot data, use a vision-language model that has already digested internet-scale image-text pairs, and let robot control ride on top of that far broader understanding. The gain is not subtle. RT-2 can act on concepts it never saw in any demonstration, “move the banana to the sum of two plus one,” because the backbone already knows what a banana is and what addition is from the web, not from teleop. RT-1’s conditioning was the right idea executed with the tools available; within a year those tools were the bottleneck, and the field routed around them. If you are building today, do not copy RT-1’s perception stack. Copy its framing and use a modern VLM, which is exactly the move Chapter 12 walks through.

The tokenizer is the second casualty, though a gentler one. RT-1’s uniform 256-bin, per-dimension scheme was fine for its task distribution and, as §11.3 showed, falls apart on high-frequency dexterous control. It did not get thrown out so much as outgrown; compression-based tokenizers like FAST and continuous diffusion heads (Chapter 10) now cover the regimes uniform binning cannot reach. Call it a contribution that aged into a special case.

The uncomfortable part: how much was the data?

Here is the question nobody wants on the first slide. RT-1’s headline generalization and robustness scaled with the 130,000 demonstrations behind it. So how much of “RT-1 works” is the architecture, and how much is just 17 months of a 13-robot fleet teleoperating in real kitchens? The honest answer is that the paper cannot fully separate them, and neither can you, because there is no ablation where someone trained RT-1’s exact architecture on a tenth of the data and a competing architecture on all of it under matched conditions. What the ablations do show is suggestive: RT-1’s performance degrades gracefully as you cut data diversity and task count, and it degrades hard when you cut them a lot, which is the signature of a model whose competence is downstream of its dataset breadth more than its parameter count.

The strongest evidence that data was doing much of the heavy lifting arrived after RT-1, in the Open X-Embodiment project (arXiv:2310.08864). That effort pooled data from many labs and robots into one corpus and retrained RT-1-style and RT-2-style models on it, the “RT-X” models, and found that a policy trained on the pooled multi-robot data outperformed the same architecture trained on any single robot’s data, sometimes by large margins, including positive transfer across different robot bodies. Same architectures, more and more varied data, better results. That is close to a controlled statement that the data axis was underexploited, and it reframes RT-1 in hindsight: not primarily a clever network, but the first serious demonstration that robot policies obey a data-scaling story at all, plus the industrial pipeline to feed it.

None of this diminishes RT-1. It relocates the achievement. The transformer and the tokens were competent, unsurprising choices; the surprising, expensive, hard-to-replicate part was proving that a real-robot dataset large and diverse enough to make scaling visible could be built and would pay off. That is a contribution most academic labs cannot copy, which is itself an uncomfortable finding, and it is one reason the community pivoted so fast toward shared datasets like Open X-Embodiment: if data is the lever, then pooling data is how everyone without a 13-robot fleet gets to pull it.

What RT-1 did not even attempt

Two limits are worth naming so you do not expect from RT-1 what it never promised. It does not reason. There is no chain of thought, no planning, no decomposition of “make me a sandwich” into steps; RT-1 maps a short instruction and a few frames straight to the next action, and long-horizon structure has to come from somewhere else, which in the original SayCan-style deployments meant a separate high-level planner picking the skills RT-1 then executed. We return to this split, high-level reasoning over low-level control, in Chapter 14 on dual-system architectures.

And it does not invent motor skills. Section 11.2 made this point about conditioning and it applies to the whole model: RT-1 generalizes across phrasings and recombinations of skills it practiced, not to mechanically novel behavior. The language covers the what; the 130,000 demonstrations still had to cover the how. A VLA is a mirror of its demonstration distribution with a language-shaped index on top, and RT-1 was the first model big enough for that description to sound like a strength instead of an excuse.

With RT-1 sorted into what lasted and what did not, one thread is still dangling, the one this whole chapter keeps deferring: the data. We have asserted that scale is where RT-1’s competence came from without saying when scale actually starts to pay, and that is the question §11.5 finally takes head-on.