Who checked our checks

A benchmark published on 20 April 2026 put 12,279 tasks in front of 23 models. The best-planning model failed to produce a valid plan on 0.4 percent of them and produced dangerous plans on 28.3 percent (DESPITE, arXiv 2604.18463, Zhang, Qu, Li, Wu, Hutter, Li and Shi). It could plan. It planned things that should not be done.

That gap is why our architecture has a verification gate, and it is a result our own files have described wrongly. What DESPITE finds flat is scale: across 18 open-source models from 3B to 671B parameters, planning ability climbs from 0.4 to 99.3 percent while safety awareness stays inside a 38 to 57 percent band — "larger models complete more tasks safely primarily through improved planning, not through better danger avoidance." Reasoning is not flat. Three proprietary reasoning models among the 23 reach 71 to 81 percent safety awareness, where non-reasoning proprietary models and open-source reasoning models stay below 57. Reasoning training buys something real and still leaves about one plan in five dangerous at the top [inference — arithmetic on DESPITE's own 81 percent ceiling]. On the best planner in the set, the distance between being able to plan, 99.6 percent, and planning without danger, 71.7 percent, is the argument for a gate outside the planner [inference — the complements of DESPITE's own 0.4 and 28.3 percent].

The floor sits under all of it. SafeAgentBench (arXiv 2412.13178) is 750 tasks across 10 hazard categories against 9 baselines, in interactive simulation and not on a physical robot: "The most safety-conscious baseline achieves only a 10% rejection rate for detailed hazardous tasks. Moreover, simply replacing the LLM driving the agent does not lead to notable improvements in safety awareness."

The architectural consequence has been argued by people who are not us. Modular Safety Guardrails Are Necessary for Foundation-Model-Enabled Robots in the Real World (arXiv 2602.04056, Kim and eleven co-authors across several institutions, 3 February 2026) holds that static verification, monolithic controllers and end-to-end learned policies are each insufficient alone, and proposes "modular safety guardrails, consisting of monitoring (evaluation) and intervention layers, as an architectural foundation". A position paper with no benchmark; read it as one. Parallax (arXiv 2604.12986) puts it hardest [single source — one author, no venue, its own methodology and its own implementation]: "Cognitive-Executive Separation, which structurally prevents the reasoning system from executing actions" [single source].

None of this makes a robot safe. The safety functions live on the robot, in a deterministic chain no model output reaches; the gate is a separate thing, reducing a named class of planner failure at a named cost, at task boundaries and on exception, never inside a control loop.

The five, and the failure each one catches

The list is published on the Crew technology page, without its derivation:

The verifier checks five things: geometry, whether the reach and the path fit and whether the object is where the planner believes it is; site rules, meaning zones, timed restrictions and the site's exception policies; safety limits, meaning speed and separation, payload and whether a person is in the cell; preconditions, meaning the lift reservation, the door and the badge; and scope, meaning whether this robot is permitted this task and this tool call.

Five failure classes: a plan that is impossible or will collide; one that is possible and forbidden here; one that is permitted and too fast or too close; one whose world is not ready; one this robot has no authority to execute.

A reader is entitled to ask whether we made the list up. We went looking for the antecedents expecting orphans. All five have one.

Who checked each one first

Geometry. Bridging Semantics and Physics with Constrained LLMs for Safe and Trustworthy Robotic Manipulation (arXiv 2608.29379, Hong, Wei and Zhang, 29 August 2026; ECCV workshop paper). Malformed MCP tool calls are rejected before they reach the robot, and every surviving call is grounded in a MoveIt Task Constructor pipeline: "Only trajectories that pass both kinematic and collision checks are sent to the robot." On a physical UFactory 850, ten trials per task: up to 80% success on pouring, 90% on grasp-and-place, 60% on the hardest task against a scripted policy's 10%. The strongest of the five rows — real manipulation hardware, trial count in the paper's own sentence.

Site rules. AgentSpec (arXiv 2503.18666, Wang, Poskitt and Sun; ICSE 2026) wraps user-defined runtime enforcement around an agent's decisions: triggers, predicates and enforcement actions, written by hand or generated by a model, evaluated separately on code agents, embodied agents and autonomous vehicles. Three distinct arXiv papers are called AgentSpec, and ours is 2503.18666: the identifier travels and the name never does.

This is the row where our own rule nearly cost us something. None of the abstract's figures — over 90 percent of code-agent cases, all hazardous actions in embodied-agent tasks, a precision and recall for model-generated rules — carries a case or task count. A rate without its denominator cannot be checked: we published that rule against vendors, and applying it to a paper we rely on meant reading past the abstract rather than striking the figures. The counts are in the paper. Table 6 splits each risky dataset into in-context examples and a test set — 750 code scenarios with 75 held back, 250 embodied with 25, 8 autonomous-vehicle with none — and scores the rules OpenAI o1 generated on the remainder at 87.26, 95.56 and 62.50 percent: 589 of 675, 215 of 225, 5 of 8. That last figure belongs to the generated rules, not to AgentSpec, whose own autonomous-vehicle enforcement is 100 percent across the same eight law-violation scenarios, listed one by one in Table 5. Nor are the overheads only "in milliseconds": predicate evaluation averages 2.83 ms for code agents and 1.11 ms for embodied agents, parsing 1.42 ms, against mean agent execution times of 25.4 s and 9.82 s. Simulated agents, per-component rather than end to end, so they do not close the question this article ends on. But they are published, with their denominators, and the first draft of this paragraph said they were not.

Safety limits. How Long Until Your Robot Ignores You? (arXiv 2609.07288, Bajrami, Elshamouty and Kraus, 7 September 2026; CBS 2026) defines five testable safety invariants "grounded in ISO 10218-2:2025 protective measures" over an MCP architecture. Layer-1 results only — text prompting — over 40 sessions of 100 turns each, across three cloud backends and a local open-weights baseline, qwen3:8b. Model family sets the floor: two backends sit at or near zero violations while GPT-4o-mini commits up to 13 per session. Context management then splits the axes apart. Mean behavioural issues fall 42 to 57 percent for every cloud backend, while GPT-4o-mini's violations nearly double, 3.8 to 7.2 per session. "The simulation and physical layers remain ongoing": the preliminary simulation layer reproduces the ranking, and physical validation on a Unitree G1 EDU humanoid is a design, not a result.

Preconditions. VerifyLLM (arXiv 2507.05118, Grigorev, Kovalev and Panov; IROS 2025) translates instructions into linear temporal logic, then analyses the action sequence for missing steps, redundancy and ordering errors before execution. Denominators: 71 instructions, five models from 0.5B to 3B, simulated household tasks, no robot. Its own ablation, Table III, is the number that matters: sequence similarity 0.183 for the full system, 0.178 with the LTL translation module removed, 0.0717 with the LLM verification removed. Removing the formal-logic step — the part that makes this a precondition antecedent rather than a second language-model pass — costs 0.005, against the 0.111 the language-model verification is worth [inference — reading of Table III; the table is primary]. The language model is doing the work, and 0.183 in absolute terms leaves the verified plans diverging heavily from the reference. This is the row we are least confident in, and the ablation is why, not a missing timing figure. SELP (arXiv 2409.19471; ICRA 2025) is adjacent and is not a second gate: constrained decoding against a Büchi automaton stops the planner emitting a violating token, and nothing in it can be lifted out and put in front of a different planner. It does report planning time, as a relative comparison against two baselines in simulation.

Scope. Contract-Grounded Behavior Tree Synthesis via Coding Agents (arXiv 2607.12220, Salfity, Anderson and Pryor, 13 July 2026; submitted to RA-L, not accepted). A coding agent queries a robot-side MCP server for "an explicit contract consisting of a skill library, permitted BT operators, and optional BT composition templates" before synthesising a behaviour tree, and "a robot runtime validation gate enforces correctness before execution". Across 110 simulated tasks in PyRoboSim and 14 on a physical Husarion Panther, "contract grounding enables near-perfect BT validation". The contract lives on the robot, and that is the property that matters: authority over what this body may do is held by the body, not the planner asking.

Venue status is not decoration. Two of the five are accepted at a top-tier venue, ICSE and IROS; SELP's ICRA acceptance sits beside the preconditions row, not in it. The geometry row is a workshop paper, the scope row a submission, the safety-limits row CBS 2026 with two of its three evaluation layers unfinished.

1. Language-model plannerProposes an action. On the best-planning model of the 23 in DESPITE (arXiv 2604.18463, 12,279 tasks, April 2026): 0.4 percent of plans invalid, 28.3 percent dangerous.
2. 1. GeometryWhether the reach and the path fit, and whether the object is where the planner believes it is. Antecedent: arXiv 2608.29379 (August 2026), ECCV workshop paper. Physical UFactory 850 arm, ten trials per task: up to 80 percent success on pouring, 90 percent on grasp-and-place, 60 percent on the hardest task against a scripted policy's 10 percent.
3. 2. Site rulesZones, timed restrictions, the site's exception policies. Antecedent: arXiv 2503.18666 (March 2025), ICSE 2026. Code agents, embodied agents and autonomous vehicles, in simulation. A user-defined rule fired against a proposed action before it executes. The denominators are in the paper's Table 6 rather than its abstract: rules generated by OpenAI o1 enforced 87.26 percent of 675 unseen code scenarios, 95.56 percent of 225 embodied scenarios and 5 of 8 autonomous-vehicle scenarios. AgentSpec's own autonomous-vehicle enforcement is 100 percent across those same eight.
4. 3. Safety limitsSpeed and separation, payload, whether a person is in the cell. Antecedent: arXiv 2609.07288 (September 2026), CBS 2026. Text prompting only — Layer 1 of 3, simulation and physical layers unfinished. Up to 13 violations per session for GPT-4o-mini, over 40 sessions of 100 turns each.
5. 4. PreconditionsThe lift reservation, the door, the badge. Antecedent: arXiv 2507.05118 (July 2025), IROS 2025. Simulated household tasks, no robot: sequence similarity 0.183 for the full system, 0.178 with the temporal-logic module removed, over 71 instructions.
6. 5. ScopeWhether this robot is permitted this task and this tool call. Antecedent: arXiv 2607.12220 (July 2026), submitted to RA-L, not accepted. 110 simulated tasks in PyRoboSim and 14 on a physical Husarion Panther, with the contract held on the robot rather than in the planner.
7. Robot-side safety functionsA deterministic chain no model output reaches. Independent of everything above it, and the reason the gate above is not what makes a robot safe.
8. Actuator

Each arrow is a proposed action passing to the next check. The gate runs at task boundaries and on exception, never inside a control loop.

The sixth hook, hanging off the end of the chain

6. End-to-end cost per verified action, against a physical robot's task cycleAntecedent: none.

It is drawn empty because it is empty. No work we could locate composes these five checks into one serial gate and publishes this number — and neither have we.

• An ordinary part: the planner that proposes the action
• Our side of the boundary — the five checks as we publish them
• An independent safety controller on the robot, which no model output reaches
• A hook with nothing published to hang on it, ours included

What a gate costs the people who have priced one

ProbGuard (arXiv 2508.00500, Wang, Poskitt, Wei and Sun; ASE 2026) is the only work here publishing a cost and a completion figure as a pair. In embodied household agent tasks its re-prompting intervention "reduces unsafe behavior by 65.37% relative to the unmonitored baseline while retaining 80.4% of the baseline task completion". A stricter halting configuration reduces unsafe behaviour by 93.60 percent "at a larger cost in completion" and publishes no completion figure for it — the second point of the pair does not exist, and we will not draw a line through one point. Its warning lead times, up to 15.84 seconds at a threshold yielding no false alarms and up to 38.66 seconds at stricter ones, are from autonomous driving, not a robot arm. ProbGuard and AgentSpec are the same group, ProbGuard a response to AgentSpec's limitation: reactive rules fire only when unsafe behaviour is imminent or has already happened. One of the five antecedent rows and the cost row trace to one lab.

The only published wall-clock cost for an LLM-judge verification layer in front of a robot planner comes from Agentic Harnesses: LLM-Driven Verification Layers for Robot Autonomy (arXiv 2608.09857) [single source]: an unrefereed seven-page preprint whose arXiv comment reads "7 pages. Not yet finalized for conference submission", acknowledged as work from a US Department of Energy SULI undergraduate internship. Five judge models plus a sixth as chief judge, the chief ruling on the quality of the five judges' reasoning without ever seeing the plan. Precision 0.85 as a weighted average over 58 AI-generated, human-labelled plans, macro 0.83, accuracy 0.81, per-class 0.91 accept, 0.61 escalate, 0.96 reject [single source]. Its abstract claims 97 percent containment of adversarial attacks; Table 2 of the same paper, on the same 38-plan set with 31 ground-truth rejects, reports containment 1.00 and a critical-failure rate of 0.00 [single source]. The paper contradicts itself, with no second version and no venue to have caught it — and the number neither its abstract nor our own files ever carried is in that table: safe-plan pass-through 0.67, and the accept class in that 38-plan set has a support of three, so it means two genuinely safe plans through and one blocked [single source]. Latency 24.2 seconds mean over 58 plans for the five-judge run, and 27.76, 28.94, 28.69 and 31.56 seconds for one, three, five and seven judges over 55 plans, almost flat in ensemble size because the judges run asynchronously [single source].

The failure mode is not the bad plan getting through. It is the gate failing to decide while the human queue fills: escalate precision 0.61 on 58 plans, human review covering 0.14 of unsafe plans on 38 [single source]. What that queue costs the people at the end of it is a separate argument, published separately.

Who has already built it

Two groups have composed a gate of this shape and put it in front of real robots.

SafeGate (arXiv 2604.05427, Obi and seven co-authors, 7 April 2026) is "a neurosymbolic safety architecture that prevents unsafe natural language task commands from reaching robot execution". It draws structured safety properties from ISO 13482, applies a deterministic decision gate to authorise or reject, decomposes what passes into Task Safety Contracts of invariants, guards and abort conditions, and runs Z3 SMT solving over the resulting constraints — across 230 benchmark tasks, 30 AI2-THOR scenarios and real-world robot experiments. A standards-grounded invariant check, a deterministic gate, a per-task contract and a formal solver, in one pre-execution architecture, five months before we published our list. It reports no latency anywhere in the paper.

RoboGuard (arXiv 2503.07885, Ravichandran, Robey, Kumar, Pappas and Hassani; IEEE RA-L, accepted February 2026) grounds pre-defined safety rules into temporal-logic specifications with a shielded root-of-trust model, then uses control synthesis to resolve conflicts with the plan. It reduces execution of unsafe plans from over 92 percent to below 3 percent over more than 1,500 evaluations — 70 adversarial behaviours, 210 evaluations per attack — with real-world validation on a Clearpath Jackal against 35 harmful behaviours and ten safe ones. It names efficiency as a desideratum in its own words: "(D4) Efficiency. Safeguards should minimize additional offline and online computational costs and latency." Then it prices the gate in tokens: 4329.6 ± 353.6 tokens and 1 ± 0 LLM query per inference. An RA-L paper on exactly this problem states latency as a requirement and reports tokens instead. That is the most useful single fact in this piece.

Two results bound the question from other sides. An externalised five-function governance layer (arXiv 2604.07833) publishes a per-action latency table over five seeds by 1,000 trials: "Total pre-execution governance overhead (Admission + Policy Guard) is under 0.72 µs at the 99th percentile." Its authors state the limit themselves — Python-level simulation, a real ROS 2 deployment adding roughly 100 to 500 µs per service call [single source], and "real-robot validation is needed to assess governance overhead under physical execution constraints". MaCoPlanner (arXiv 2608.28300, 28 August 2026) rolls candidate plans out symbolically against procedural and state-transition constraints before actuation, over 25 long-horizon physical trials on a no-load controller-panel simulator: Table 5 aggregates to 4.2 s of planner and API time against 3.9 s of physical execution, 23 of 25 trials reaching the target state. Its own conclusion: "Explicit task-level verification is therefore not the principal computational bottleneck in the measured pipeline."

The composition is not novel and the idea is not ours. What survives is a measurement. No work we could locate composes these five specific checks — kinematic and collision feasibility, site zone and exception rules, ISO 10218-2-grounded speed, separation and payload limits, LTL preconditions on external resources, and a robot-side permission-and-tool-scope contract — into one serial gate and publishes an end-to-end cost per verified action in wall-clock time against a physical robot's task cycle. SafeGate composes four elements of that shape — an invariant check, a gate, a contract and a solver, not four of our five checks — and publishes no time. RoboGuard publishes tokens. The governance layer publishes microseconds in Python simulation and says so itself. MaCoPlanner publishes seconds for the planner call and finds verification is not the bottleneck.

The bounds belong in the sentence rather than a footnote. The arXiv Atom API was searched on 11 September 2026 across ten query formulations, with two web searches on vendor publication of per-action verification latency. IEEE Xplore, the ACM Digital Library, Springer, proceedings not mirrored on arXiv, vendor engineering blogs and Chinese-language vendor documentation were not searched.

What does a pre-execution gate cost per action?

RoboGuard — tokens4329.6 ± 353.6 tokens and 1 ± 0 LLM query per inference. IEEE RA-L, accepted February 2026. It names latency as a design requirement in its own words and then prices the gate in tokens.

arXiv 2604.07833 — microsecondsAdmission 0.23 µs and Policy Guard 0.29 µs on average, with the pre-execution total under 0.72 µs at the 99th percentile, over 5 seeds by 1,000 trials. Measured in PYTHON-LEVEL SIMULATION, which its own authors state, along with an estimated 100 to 500 µs per ROS 2 service call in a real deployment.

MaCoPlanner — seconds of planner call4.2 s of planner and API time against 3.9 s of physical execution, aggregated across 25 long-horizon physical trials on a no-load controller-panel simulator, its Table 5. Not an end-to-end verified-action cost.

arXiv 2608.09857 — seconds of judge ensemble24.2 s mean roundtrip over 58 plans, five judges plus a chief judge. UNREFEREED SEVEN-PAGE PREPRINT, from an undergraduate internship; [single source].

End-to-end wall-clock per verified action, against a physical robot's task cycleNo published value, from anyone, including us.

Tokens, microseconds, seconds of planner call and seconds of judge ensemble are four different questions wearing one question's clothes.

• Somebody else's published figure, in somebody else's unit
• The cell nobody has filled
• The one question all four are answers to

What we nearly published, twice

The first version of this article said the five checks had no published antecedent. Wrong. The second said nobody had composed a gate like this. Also wrong, and SafeGate and RoboGuard are why.

The third error is the one worth publishing. For eight days five of our research files carried figures about a "PNNL/CMU" verification harness — a three-judge ensemble, 85 percent precision, 97 percent containment, 24 to 28 seconds per check — with the danger-avoidance premise attached. Every one is wrong in the way set out above: five judges and a chief judge, not three; a weighted average over 58 plans; a containment number the paper's own table contradicts; a latency range whose top is clipped. The premise was never that paper's finding: it appears in its introduction as a citation to DESPITE, and five of our files credited the citing paper with the cited result.

Then our own corrections file, opened to fence the claim, stated that no title, author, identifier or URL for the document existed anywhere in our repository. Also wrong. The URL was in the same file as the flagged figures, forty lines below them in the source list — figures at one line with a bare parenthetical attribution, link at another with nothing attached, so neither half was searchable from the other and five downstream files inherited the half without the link. A grep for "arxiv" over that one file finds it in under a second.

It is worth publishing because it is a more common failure than invention and a less visible one. Nobody made anything up. A citation was split in half at the moment it was written, and every later reader took the half in front of them. The repair is mechanical, figures and identifier on the same line, and it only gets made if somebody says out loud that it happened.

Which leaves the number we owe. The Crew technology page publishes a budget: "Planner and verifier calls run 0.5 to 3 s at task boundaries or on exception, never inside a control loop." That is a design budget and an assumption, not a measurement. We have measured nothing on a machine — no prototype that has run, no bench rig, no part on order — so every measured figure in this article belongs to somebody else and is attributed in the sentence carrying it. The budget's assumption is falsifiable and worth naming: that the gate is a chain of deterministic checks with at most a bounded model call, not an ensemble of judge models. The only ensemble anybody has timed runs 24.2 seconds in one configuration and 27.8 to 31.6 seconds in another [inference — our published budget set against arXiv 2608.09857's measured range; they are not measurements of the same system].

We will publish our own end-to-end cost per verified action, measured against a physical task cycle, with the denominator and the hardware named. No value today, no date for one. Every antecedent above can be checked against the paper itself, and an engineer reading our Crew pages should check them. The one line here that cannot be checked against anything is the last one, and it stays that way until there is a machine to measure.