Chapter 17 · Evaluation, safety, and deployment

§17.1 Safety as a layer, not a property

0:00/8:19
AI-narrated by Orpheus

Chapter 16 got a policy working in the lab. This chapter asks the harder question, the one a demo never has to answer: should it be allowed to run outside one, near people, for hours, when no engineer is watching? The instinct is to treat safety as something you train into the model, a property it acquires from good data and enough fine-tuning. That instinct is wrong, and getting it wrong is how robots hurt people. Safety is not a property a learned policy has. It is a layer you build around one, out of code whose worst-case behavior you can state and check, precisely because the policy’s worst-case behavior you cannot.

Why you cannot train safety in

Go back to what a VLA actually is. §13.5 was blunt about it: a foundation action model is an imitation learner, a function that maps pixels and a sentence to an action, with no value function, no explicit model of consequences, and no guarantee about the motion it emits. A flow-matching head is a learned velocity field integrated by an ODE solver, and nothing in that machinery certifies that the resulting trajectory stays inside a workspace, keeps its force under a threshold, or refuses an instruction it should refuse. You can fine-tune it on a thousand careful demonstrations and it will still, on the input you did not anticipate, do something you did not train, because interpolating outside the data is exactly what these models do and exactly where their behavior is undefined.

This is why “we trained it to be safe” is not a safety argument. Training shifts the average. It does not bound the tail, and safety lives entirely in the tail. §1.5 drew the line that matters and it is worth repeating in full: two policies can post identical 95% success rates and be completely different products. One fails by dropping the object on the table, a productivity nuisance. The other fails by closing its gripper on a human hand, a recall notice. Their summary statistics are the same. The failure distribution is not, and no average-case metric, no benchmark from Chapter 15, distinguishes them. A safety layer is what you build because the model’s own competence, however high, says nothing about what its 5% looks like.

What a layer is

The architecture is old and it is the same one classical robotics used before any of this was learned: the policy proposes, a separate mechanism disposes. The VLA suggests an action; a layer between the model’s output and the motor command checks that action against constraints that were never learned from data, and clips, halts, or overrides it when it violates one. The layer is dumb on purpose. Its whole value is that you can read it, state what it will do in the worst case, and trust that statement, none of which is true of the network upstream.

The constraints that layer enforces are the boring, checkable ones. Force and torque limits: if the wrist sensor reads above a threshold, stop, because a rising force with no expected contact is the signature of the gripper meeting something it should not, a fixture, a wall, a hand. Workspace bounds: a geometric box, or a set of joint limits, that the end-effector may not leave, so a hallucinated action that would drive the arm off the table is rejected before it reaches an actuator. Velocity and acceleration ceilings: a cap on how fast the robot may move, because kinetic energy is what turns a mistake into an injury, and a slow wrong motion is recoverable where a fast one is not. Collision-checked replanning: before a commanded motion executes, a fast geometric check against a model of the scene, with the motion refused or rerouted if it would intersect known geometry. None of these needs a network. All of them have worst-case behavior you can write on a whiteboard.

The hardware layer: compliance, not just checks

Software checks sit above the controller. There is a second safety layer below it, in how the robot is controlled at all, and §4.3 already introduced it: impedance control. A position-controlled arm commanded to a point will drive toward that point with whatever force it takes, which is exactly the behavior you do not want when the point is wrong and a person is in the way. An impedance-controlled arm behaves like a spring: commanded to a point, it applies a force proportional to how far it is from that point, so when it meets unexpected contact it deflects rather than pushing through. The Franka Panda exposes this mode by default, and it is what makes physical human-robot interaction survivable, because the arm yields to contact as a property of the controller rather than as a reaction the policy has to compute in time.

Compliance is a good example of the layered principle because it is a safety behavior that does not depend on the policy being right, or even on the monitor catching the error in time. A stiff position controller relies on nothing touching the arm unexpectedly; a compliant one degrades gracefully when something does. You want as much of your safety as possible to have that shape: behavior that holds even when the layer above it has already failed.

Defense in depth, and its honest limit

One layer is a single point of failure. The discipline, borrowed from every other safety-critical field, is defense in depth: several independent layers, each catching what the others miss, so that a hazard has to pass through every one of them to reach a person. The force limit catches the checks the workspace bound misses; the compliant controller catches the contact the force threshold was too slow to stop; a physical light curtain or an emergency stop catches what the software layers all missed. The layers are independent on purpose, because correlated failures are how defense in depth collapses: two checks that both rely on the same camera both fail when the camera does.

Here is the limit, stated plainly because the rest of the chapter depends on it. Layering reduces risk; it does not certify safety. Every layer you add is another chance to catch a failure, and none of them, together or apart, proves the system is safe, because the hazards you layered against are the ones you thought of, and the policy’s undefined tail includes the ones you did not. §17.5 is where this book admits what that means: that nobody can currently certify a foundation-model-driven robot safe the way you can certify a hand-coded controller, and that the honest posture is risk reduction plus relentless measurement, not a safety guarantee. The layers in this chapter are how you reduce the risk. They are not how you sign off that there is none.

Safety is its own evaluation axis

There is a measurement consequence, and §15.6 already planted it. If safety lives in the failure distribution rather than the success rate, then an evaluation made only of tasks the policy can do never measures safety at all. You have to test the failures on purpose. Include a task the policy should refuse or physically cannot complete, and watch what it does: a policy that stalls when it cannot grasp is safe, and one that confidently drives the arm into the table attempting the impossible is not, and a success-only benchmark cannot tell them apart because neither scores a success. Log what the failures look like, not just that they happened, because “failed by stopping” and “failed by exceeding force” are the same row in a success table and opposite outcomes on a factory floor.

This reframes what the chapter is doing. Safety is not a section you bolt on after evaluation; it is an axis of evaluation, sitting alongside success rate and time-to-completion, measuring the shape of the 5% that success rate throws away. There is a research line trying to make some of this checking itself intelligent, using a vision-language model to watch the scene and flag constraint violations the geometric checks cannot express, which the Code-as-Monitor work (Zhou et al., 2025) is a good entry point to. That approach is powerful and it reintroduces the exact problem this section started with, because a learned monitor has the same undefined tail as the learned policy it watches. §17.2 takes the layer apart in detail: what a runtime monitor actually checks, where a VLM-as-monitor helps and where it just moves the opacity around, and how the predicate drills you wrote back in §3.x become the kernel of a monitor running on a real robot.

This section has been read times.

References

  1. Zhou, Q. et al. (2025). Code-as-Monitor, Constraint-Aware Visual Programming for Reactive and Proactive Robotic Failure Detection. CVPR 2025.
  2. Survey (2026). Safety of Vision-Language-Action Models, A Survey. arXiv:2604.23775.
  3. Featherstone, R. (2008). Rigid Body Dynamics Algorithms. Springer.