Is agent safety the same as AI safety?

No. AI safety, as the frontier labs and the International AI Safety Report use the term, is a property of the model: its alignment, its behaviour under evaluation, its potential for misuse. Agent safety is a property of the system built around the model, the harness, tools and environment that let it act. A well-aligned model holding the wrong credential and an ambiguous instruction can still do the wrong thing.

Is agent safety a security problem or a governance problem?

Both, and they are not interchangeable. Security defends the boundary against an adversary: prompt injection, poisoned tool descriptions, a compromised dependency. Governance is the accountability behind the boundary: the declared scope, the person with authority to widen it, and the record of each decision. Safety is the outcome when both hold.

Can a model keep its own actions safe?

Not on its own. The model is the component that may be mistaken or manipulated, so a check that lives inside the model has no independent view of it. The check has to sit outside the model, run before execution, and judge the action against what the agent was declared to do.

Reference · Updated 2026-09-15

What Does Agent Safety Mean When the Agent Can Act?

Agent safety is the property that an AI agent's actions stay inside what the people responsible for it intended, including when the model is wrong, manipulated, or confidently mistaken. For a chatbot, safety is about the words it produces. For an agent that can run commands, write files, or call APIs, it is about the actions it takes, decided before they run.

The word safety arrived in this field before agents did, and it was coined for a different object. Most of what is published under AI safety concerns the model: whether it is aligned, how it behaves under evaluation, what it will help someone do. Those questions do not go away when the model is given tools. They stop being sufficient.

How is agent safety different from AI safety?

AI safety, as the frontier labs, the International AI Safety Report and most of the academic literature use the term, is a property of the model: its alignment, its behaviour under evaluation, its potential for misuse, and the systemic effects of wide deployment. It is studied by training, evaluating and red-teaming the model itself.

Agent safety is a property of the system built around the model. The same model, placed in a harness with a shell, a database connection and a mailbox, can now do things rather than say them. Anthropic's own guidance on trustworthy agents describes an agent as built from four components, each of them "a potential point of oversight": the model, a harness, tools and an environment. Only the first of those belongs to the lab. The other three belong to whoever deploys the agent.

The distinction is one of location, not of importance. A well-aligned model holding the wrong credential and an ambiguous instruction can still delete the wrong thing, because the instruction was ambiguous and the credential allowed it. Nothing about the model's alignment was tested by that event. The system around it was.

Why does the word change when the agent can act?

Text can be reviewed after it is produced. An action cannot; it executes. Once the agent can act, the safety property has to be decided before execution, at the moment the model has chosen an action and before the runtime carries it out. Two consequences follow.

First, the decision cannot rest with the model alone. The model is the component that may be wrong or manipulated, so an arrangement in which the model is also the judge of its own actions has no independent check. This is the same reasoning that puts a code reviewer outside the author, and it is why an agent cannot govern itself.

Second, the decision has to be about the action, not the words. A command that exports a table is the same command whether it arrived from a clean prompt or an injected one. What makes it safe or unsafe is whether this agent, doing this job, was meant to run it. That is a question about intent and authority, and it is answered by a declared scope and a policy, not by a classifier over text.

What do safety, security and governance each mean for an agent?

Three words are used for what looks like one problem. They are three views of one system, and each answers a different question.

Word	The question it answers	Who usually asks it
Safety	Do the agent's actions stay inside what was intended, including when the model errs?	Everyone. It is the outcome the other two exist to produce
Security	Can an adversary make the agent act against its owner: prompt injection, poisoned tools, a compromised dependency?	Security engineers and the CISO
Governance	Who decided what this agent may do, who can widen that, and what record shows each decision?	Boards, compliance, auditors, and the CISO reporting to them

Security is the discipline that defends the boundary. Governance is the accountability structure behind it: the declared scope, the person with authority to widen it, and the record. Safety is what you have when both hold. A team can have strong security with no governance, every injection caught and nobody able to say what the agent was allowed to do. It can have governance with no security, a careful policy and a poisoned tool description that walks around it. Neither is safe on its own.

Which word do practitioners, boards and regulators actually use?

The vocabulary is split by audience, and the split is worth knowing before you search for a control or write a policy.

The frontier labs, the International AI Safety Report and the academic literature say AI safety, and mean model behaviour.
The security trade press and the security vendors say AI agent security, and mean the category of products that defend agents at runtime.
The guides written for CISOs and boards say AI governance or AI agent governance, and mean accountability, policy and evidence.
Gartner names the runtime enforcement layer guardian agents, inside its AI TRiSM framework for trust, risk and security management.
The US federal body that evaluates models dropped the word safety from its name in 2025. The AI Safety Institute became the Center for AI Standards and Innovation, and its stated focus moved to security and standards.

None of these is wrong. They are the same problem seen from the lab, the security team, the boardroom and the standards body. If you are looking for the layer that decides what an agent may do before it does it, the terms that find it are action governance, agent security and guardian agents. A search for AI safety finds the model.

What does agent safety require in practice?

The minimum is small and specific, and it starts applying at an identifiable moment in the code rather than at a maturity phase. When does an AI agent need governance? names the five triggers.

A pre-execution check on every action with side effects: a tool call, a command, a query, a file write, a message to another agent. The check judges the action in the context of the job, and it runs before the runtime does.
A declared scope: a statement of what the agent is for and what it holds, so that "outside the job" is decidable by policy rather than a matter of opinion per action.
A person with the authority to widen the scope, and no way for the agent to widen it itself. Narrowing can be free. Widening is a decision.
A record of every decision, allows included, so that a control which quietly stopped working is visible, and an auditor can see what operated rather than what was intended.

Everything above the minimum, semantic analysis of novel attacks, session-level correlation across turns, approval workflows, is what the broader action governance layer adds. Four approaches to governing an agent compares the ways teams build the check, and where each one stops.

What is action governance? The pillar definition this page hangs off.
When does an AI agent need governance? The five changes in a codebase that create the need.
Four approaches to governing an agent: credential scoping, rule engines, design-time allowlists, and action-time adjudication.
The AI agent security stack: where each control layer sits.
The agent was authenticated. Nobody asked if it should. Why the check has to live outside the model.
For coding agents: the defaults that make the check ship with generated code.

Sources

Shrike is the action-governance layer this page describes: your policy enforced on every tool call, command and query before it runs, a declared scope per agent that the agent cannot widen, a person who can, and a per-action record. See the quickstart or the playground.