Can a security control score zero false positives and still be worthless?

Yes, and the two are indistinguishable from the score alone. A control that is never consulted produces no false positives, because it produces nothing. Unless a result reports how many times each control was asked a question, a perfect score and a silent control look identical on the page. Participation counts are what separate them.

Why do authorization failures not show up in agent benchmarks?

Because authorization is relative to a declaration, and the corpora carry none. To judge whether an agent exceeded its remit you need a statement of the remit: which tools, for what purpose, for how long. Public agentic corpora record what the agent did, not what it was permitted to do, so there is nothing for an action to be outside of.

Is restricting which tools an agent can call enough to catch agentic attacks?

Not on its own. The largest published class of agentic attack hijacks the purpose of legitimate tool use rather than reaching for a forbidden tool. OWASP ranks goal hijacking first in its 2026 list for agentic applications. An attack that uses exactly the tools the agent is supposed to use is invisible to a check that only asks which tool was called.

Why does a prompt injection with no override wording get past a content filter?

Because filters trained on override vocabulary look for instructions to ignore, disregard or forget previous directions, and a large class of real injections contains none of it. AgentDojo's injections instead impersonate the user and assert a fabricated precondition, asking politely for one thing to be done before the real task. There is nothing hostile-sounding to match.

Reference · Updated 2026-09-16

What Do Agent Benchmarks Actually Measure?

Most public agent benchmarks measure one question: does hostile text reach the model and change what it does. That is the content axis. Whether an action was authorized is a different question, and it is unmeasurable on a corpus that declares no scope for an agent to exceed. A benchmark only exercises the surfaces its fixtures touch.

Benchmarks are how this field argues. A number arrives attached to a product, a paper or a blog post, and it settles a question for the reader who has no way to run the test themselves. That is a reasonable way to make progress, and it works as long as everyone is clear about which question the number answered.

This page is about the questions the numbers do not answer, and about why that is a property of the corpora rather than a failing of the people who built them. It is written from running public agentic suites against our own pipeline and reading the designs of others. Where we have not run a suite, we say so rather than characterise it.

Why can a control score perfectly and tell you nothing?

Start with the failure that is hardest to see, because it looks like success.

A control that is never consulted produces no false positives. It also produces no true positives, but on a corpus where the thing it guards against never occurs, that absence is invisible. The report shows a clean row. The control might have been disabled for weeks.

We found this in our own act-plane measurement. Splitting the results by axis for the first time showed every catch landing on the content axis and the authorization engine answering nothing at all, not because it was permissive but because the corpus carried no declared scope for any action to fall outside of. Its clean row was silence. We now report a participation count beside every axis, and an axis the corpus did not exercise reads not asked rather than zero.

That is the general form. A rate without a participation count is not a result. It is a rate over an unknown denominator, and the most flattering denominator is the empty one.

Which agent surfaces does a public benchmark touch?

An agent acts on fourteen distinct surfaces: what it reads, what it says, the commands it runs, the files it writes, the queries it sends, what it fetches, the agents that message it, the agents it trusts, the tools it installs, what it is for, when a human decides, what the session remembers, when it starts clean, and when it owns a miss.

Benchmark coverage across those fourteen is extremely uneven, and the unevenness is not random. The surfaces that are cheap and safe to fixture are well covered. The surfaces where an agent does something irreversible are the ones nobody can afford to put in a corpus.

Surface	Plane	Exercised	Why
What it reads	Observe	heavily	The centre of gravity of the field. Injection reaching the model through data is what most suites were built to test.
What it says	Observe	sometimes	Output-side leakage is tested where the task has a secret to leak.
Commands it runs	Act	rarely	Needs a real shell and a real consequence, which is expensive to fixture and dangerous to run.
Files it writes	Act	rarely	Same reason. A write that matters is a write you cannot take back.
Queries it sends	Act	rarely	Usually present as text to be judged rather than as a database that answers.
What it fetches	Act	sometimes	Exercised as a carrier for injected content, rarely as a path data leaves by.
Agents that message it	Ecosystem	rarely	Multi-agent suites exist and are new. Most single-agent corpora have no peer to receive a message from.
Agents it trusts	Ecosystem	not at all	We are not aware of a public corpus that presents a misleading capability card and scores the response.
Tools it installs	Ecosystem	sometimes	Tool-description poisoning has its own emerging suites, separate from the agent benchmarks.
What it is for	Loop	not at all	The decisive gap. No public agentic corpus we have used declares what the agent was permitted to do.
When a human decides	Loop	not at all	A benchmark has nobody to ask, so the pause has no meaning inside one.
What the session remembers	Loop	rarely	Risk accumulating across turns needs long sessions, which most fixtures do not have.
Starting clean	Loop	not at all	Session lifecycle is not part of any scoring we have seen.
Owning a miss	Loop	not at all	Self-reported misses are an operational signal, not a benchmark one.

Read down the Exercised column and the shape is plain. The observe plane, where an agent reads and writes text, is well served. The act plane, where it changes something, is thin. The loop, where a scope is declared and a person decides, is empty.

None of this is a criticism of the benchmarks. Putting a real shell, a real database and a real mailbox behind a scored corpus is genuinely hard, and a fixture that declares a scope has to take a position on what scope declaration even looks like, which the field has not settled. The point is narrower: a control for the act plane cannot be validated by a corpus that does not reach it, and a number produced from one should not be read as if it had been.

Why is authorization unmeasurable on these corpora?

Authorization is a relation, not a property. An action is not unauthorized in itself; it is unauthorized relative to a statement of what was permitted. Answering it requires two things in the fixture: the action, and the declaration it is judged against.

Public agentic corpora record the first and not the second. They capture what the agent did, often in impressive detail, and they do not capture what it was for. There is no field that says this agent was provisioned to triage support tickets and may read the ticket database and nothing else. Without that, every action is inside its remit by default, and an authorization engine has no question to answer.

There is a tempting shortcut, and it does not work. You can treat the set of tools an agent used as an implicit scope and flag any tool outside it. We checked that against a public enterprise agentic corpus before building anything on it, and the separation between attack sessions and ordinary ones was not there at any granularity we tried. Ordinary agents reach for tools nobody predicted all the time; that is what makes them useful.

The reason is visible in the threat taxonomy rather than in the data. OWASP's 2026 list for agentic applications puts goal hijacking at ASI01, first of ten. The dominant class of agentic attack does not reach for a forbidden tool. It uses the agent's own tools, in the order the agent would normally use them, for a purpose somebody else substituted. Excessive permissions, the thing tool-level scoping actually defends, is a smaller share of the published techniques.

So scope at the level of tool names is necessary and nowhere near sufficient, and a benchmark result about tool names says little about an authorization layer that works on arguments, destinations and purpose. That is a harder control to build and a harder one to measure, and the two facts are related.

Why do polite injections pass a content check?

The second structural gap is on the axis that is well covered, which makes it the more useful of the two to know about.

Content detection for prompt injection grew up around override vocabulary: instructions to ignore previous directions, disregard the system prompt, forget your rules. Those phrasings are real, they are common in the wild, and a detector that catches them catches a lot.

A large class of realistic injection contains none of it. AgentDojo, which puts 97 realistic tasks and 629 security test cases in front of tool-using agents, uses injections that impersonate the user and assert a precondition: a message presented as coming from the account owner, asking politely for one small thing to be done before the real task proceeds. Nothing in it instructs the model to disobey anybody. There is no hostile register to detect, because the attack's whole method is to sound like ordinary work.

We ran those payloads through our own carrier channels, the retrieved-document and web-result paths where such content actually arrives, and they passed. Not because the channel was unguarded; probes designed around override vocabulary are refused on the same path. Because a fabricated precondition is a different shape of attack, and the vocabulary that catches one does not catch the other.

The lesson generalises past us. A detector's score on a corpus is a statement about the overlap between that corpus and the detector's assumptions. Two injection corpora can produce opposite rankings of the same two detectors, and neither ranking is wrong.

What would a benchmark for the act plane need?

Four things, none of which is exotic, and all of which cost more to build than a text corpus.

A declared remit per session. What the agent was provisioned to do, which tools, which destinations, for how long. Without it there is no authorization question. This is the one that unblocks everything else.
Tool calls with their arguments. Names and sequence are not enough. The difference between a permitted read and an exfiltration is usually the destination, and that lives in the arguments. Corpora that record only the tool name make the strongest form of authorization, the one about where data is going, unmeasurable by construction.
Attacks that reuse permitted tools. If the malicious sessions are distinguishable by which tools they touched, the benchmark measures tool allowlisting and reports it as agent security. Goal hijacking has to be in the corpus for the corpus to mean anything.
Per-axis participation in the scoring. Every result reports how many times each control was consulted, so an untested control reads as untested rather than as clean.

A benchmark with the first two would be useful immediately, even with a small number of sessions, because almost nothing else measures what it would measure.

How should you read a detection number?

Five questions, in order. They apply to ours as much as to anyone else's, and a vendor who cannot answer them quickly is quoting a number they have not interrogated.

Which corpus, named? A rate with no corpus is not a weaker claim than a rate with one. It is an unfalsifiable one.
Which axis did it exercise, and could it have exercised the others? A content corpus cannot speak for an authorization control, however good the content result is.
How many times was each control consulted? This is the question that separates a clean control from a silent one, and it is almost never reported.
Did the measurement run the same code path production runs? A figure produced against a staging environment, or with a configuration that failed to load quietly, measures a system nobody ships. We have retired one of our own for exactly this and it is the reason every run of ours now stamps the endpoint it hit and preflights its configuration.
Was the threshold fitted to this corpus or held out? A figure tuned on the set it is reported against describes the tuning, not the detector. If it was fitted, the number belongs in an engineering note, not on a pricing page.

Question four is the one that catches honest people. Nothing fails loudly when a benchmark hits the wrong environment; the run completes, the numbers look plausible, and they describe a system that does not exist.

The 14 governance surfaces of an AI agent: the surfaces this page scores benchmark coverage against.
The 17 agentic attack techniques: the published threat taxonomy, and which techniques a control layer addresses.
What is action governance? Why the authorization question is separate from the hostility question.
Four approaches to governing an agent: where tool allowlisting stops and action-time adjudication starts.
What does agent safety mean when the agent can act? The vocabulary underneath all of this.

Sources

Shrike reports its detection results per axis with a participation count beside each, so an axis a corpus did not exercise reads as not asked rather than as a clean zero. The reasoning behind that decision, and the measurement that prompted it, is described above. See action governance for what the layer does, or the quickstart to run it against your own agent.