Governing Agents the Way We Govern People (Almost)

Identity, observability, mandate and security, borrowed from how we already govern people.*

At this point, the agent is no longer an abstract concept. It is already running on your laptop, triaging your inbox, drafting the slides for your next keynote, or working hard in your database to produce insights. When the value becomes real, so do the risks. We are constantly exposed to the potential risks of prompt injection, credential theft and operational accidents, not to mention the high-profile Hugging Face incident recently [1]. Beyond that, there's also a far more ordinary risk: an exorbitant AI bill that can hardly be tied to an equally valuable outcome.

That is the moment a company enters the domain of agent governance, with a very simple goal:  Organise the agents so that the risk is contained, the cost is bounded, and the value is not merely asserted but proved. 

So let's look at it more closely, and let's do it by borrowing from things we already understand. A blueprint of governance, borrowed from the physical world

When you hear the word governance, what comes to mind first? For many people, it is paperwork: passport, visa, bank account, work badge, and contracts. All these documents and the processes around them are created by a central organisation (a government, a bank, an employer), mainly for three purposes:

• to prove that you are the person they know of, and provide the right level of access to resources according to your identity
• to be aware of the actions you take, especially when those actions have an economic or social impact, both positive and negative
• to limit your behaviour when it goes against the rules they have set, and to respond with legal sanctions when necessary

This is roughly what human governance has looked like for a very long time. And naturally, it is also what we need for agents.

To make it more concrete, let's look at governance at corporate level. Here the resemblance is even closer. Think of what a new hire receives in their first week: a badge, a login credential, a job description, a set of goals, maybe a corporate card and a code of conduct to sign. Each of these is a governance instrument:

• Badge and credentials establish the identity of the employee, provide access both in the physical and digital world, and enable centralised activity tracking.
• Budget and expense policy allow the employer to control cost in a transparent way without specifying it task by task.
• Job description, OKRs and KPIs define what the employee is supposed to work on, how their outcome and performance should be measured, and which behaviour should be rewarded.
• Code of conduct and the subsequent disciplinary process discourages behaviour that causes harm or inefficiency of the company. It also can be enforced in a strict way, via dismissal or legal pursuit.

Badge, ledger, mandate, sanction. Our experience and intuition with this toolkit can serve as a blueprint for agents. But before we copy anything across, let's look at a few important differences between an agent and a person, because these are the decisive factors of agent governance.

Three ways an agent is different from a person

1. An agent is ephemeral

First, an agent is ephemeral, while a person is stable, bound to a biological substrate. Unlike humans, the "life" and "death" of an agent are trivial. A code change, an update of the underlying model, a new set of skills or plugins, a redeployment: each of these can be considered a rebirth of the agent. People mostly go through their whole life with the same identity, anchored to recognisable physical traits like a fingerprint or a face. Tracking the "invariant" of an agent is almost certainly a futile mission. What we call an agent is really a runtime: an LLM wrapped in a harness, set in motion within a context shaped by skills and prompts. Run the same combination twice and there is no guarantee it will behave the same way. Keeping track of these runtimes relies largely on the deployer registering the agent when it is deployed. This is exactly what makes an agent on a laptop harder to govern than one living on a platform. There, the deployer is simply the user; the harness, skills and tools are chosen locally and change daily; and by default there is no central record of what is actually running and for what purpose [2]. We will come back to this case, because it is the one most companies are living with right now.

2. An agent acts on behalf of a person

An agent acts on behalf of a person, while people are usually the initiators of their own behaviour. Because people initiate their own acts, people can be held responsible for them and an agent cannot [3]. The flip side of this delegation structure is that an agent's behaviour should always be bounded by the user's intention, and any action that does not serve that intention should be limited or stopped. This implies that we should be able to (A) trace every action an agent takes back to a human intention, (B) assess whether the action is aligned with the user's intention, and (C) hold that person responsible for their intention when the agent's action is aligned. The most difficult part is (B): assessing the alignment between the agent's action and the human intention. It gets harder still when the agent delegates to a second agent, and that one to a third: the human intention should still be at the top of the chain, but each hop is an opportunity to lose it [4].

3. Agents share a common dependency on models

Many agents are powered by the same language model, either through the same endpoint of a model provider, or through different deployments of the same open weight model. This means that a fleet of agents have a common denominator, and often a common point of failure. The downtime of a popular model can cause accidents across fleets of agentsaround the world. This looks very much like supply chain risk in software engineering, but the usual remedies do not transfer cleanly. You can pin a model version, but you cannot fork it or patch it, the provider retires versions on its own schedule, and a version number says nothing about how the model's behaviour drifts from one call to the next. The same commonality also means that all agents share the inherited risk of a misaligned or malicious model: a low probability, very high impact event, closer to a natural disaster than to a bug. On a positive note, this common dependency also allows us to track agents more easily by logging the request and responses between the harness and the model, often directly through the implementation of an LLM gateway.  In short, agents are ephemeral where people are consistent across time and space; agents only act on behalf of a person, where people are driven by intentions and motivations for which they, and not the agents, can be held responsible; and agents share a common dependence on a handful of models, which exposes them to incidents with a very large blast radius.

With that in mind, we can go through the toolkit, starting with the badge.

The four elements of agent governance

The Badge: identity

The most fundamental element of governance is agent identity [5]. It is the equivalent of your employees' badges and credentials, which prove who they are and give them access to the working environment. But agent identity is different, because of the differences we mentioned above. Let's first think about what an agent identity should enable us to do:  To limit cost, it needs to let us attribute every LLM request, every tool call and hence every unit of spend to a person and to the task they asked for.  To minimize risk, it needs to give us a handle to stop the agent in motion when it misbehaves.  To maximize value, it needs to allow us quantify the output of agents in terms of business value. 

This list does not require the agent to stay the same over time. It only requires that, while the agent is running, we know which run it is, who sponsors it, what it is for and what its outcome was.

So the agent's identity should be bound to a runtime, a running workload that is assigned to an objective [6][7]. It has a much shorter life span than a human identity, but it is also independent of the physical machine and the platform it happens to run on. It is a process that we want to identify. Luckily, cloud platforms have already converged on a few rules for exactly this: one dedicated identity per workload, never a borrowed human account; short lived credentials handed to the process when it starts and revoked when it ends, rather than a key copied into a file (there is an open standard for this, SPIFFE [8], and the big clouds implement the same idea under their own names, such as service accounts and workload identity on Google Cloud or IAM roles on AWS); and the workload registered at creation, by automation, with an owner's name and a purpose attached. Redeploy the agent and you get a new runtime, with a new credential, but under the same registration. The agent is reborn, and the registry stays the same, because the registry never tried to hold on to the invariant in the first place. The same rules give us a handle on the laptop agent. The user cannot be asked to register anything, but the gateway can do it for them: if the only way to reach a model or a corporate tool from a laptop is through a gateway that issues a short lived credential per user and per session, then every run is registered as a side effect of simply working. What the gateway cannot see remains ungoverned: the local files, the skills and plugins pulled from public registries (a supply chain of their own [9]), and the tools that never leave the machine. It is worth being honest about that boundary rather than pretending the badge covers it.

That is the badge. The rest of the toolkit builds on it.

The budget and expense: observability and cost

To keep the cost bounded, we next need a transparent view of what actions the agent has taken and what requests it has sent to the model. Here the common dependency on the model, our third difference, works to our benefit. Every agent, however it was built and wherever it runs, has to call a model. We can quite easily log the input and output of every model request an agent submits, count the tokens, and roll them up at the agent level. 

Tool usage is the other source of cost, and a major source of risk as well. Since it is shaped by both the LLM and the harness design, we want to keep it under control, making sure the agent neither causes a security breach nor accidentally blows up the budget on a downstream task, such as an extremely expensive database query. In practice, this means the agent should be bounded by a set of predefined actions and access permissions, and ideally by a budget that limits cost at the task level.

None of this is new to anyone who has run a cloud platform [10][11]. The closest ancestor of an agent task is the pay-as-you-go query job: it is ephemeral, it runs under a named principal, and its cost depends on the content of the query rather than on anything provisioned in advance. The controls that grew up around it are a good source of inspiration for governing an agent's tool usage, whether it is a per-user quota, a per-job ceiling that fails the job rather than billing past it, a dry run that estimates the cost before anything runs, or a jobs log tied to the principal. What the agent adds is narrower than it first appears: the "query" is now written by the model rather than by the person it runs as, and the task can spend money outside the platform, through tools, where no quota is watching.

The job description: mandate

Beyond identity and observability, the most important element is the job description, and the task-level mandate that goes with it. The job description defines the agent's role and objectives at a high level, once, when the agent is registered. The mandate pins down one task: who asked, for what outcome, within which limits, recorded before the task starts. Without these two things, we cannot correctly evaluate whether an agent is doing the right thing, and it would be difficult to detect anomalies. 

Picture an HR agent whose mandate for the current task is "answer this employee's question about parental leave". Its job description lets it read HR policies and the employee's own record. If, halfway through, it starts querying the salary table for the whole department, nothing about its access necessarily stops it. Only the mandate makes the anomaly visible: that data query does not serve the user's intention that was recorded before the task began. The mandate is the agent equivalent of the power of attorney a person signs before letting someone act on their behalf [12][13], and it is what allows anyone, human or machine, to later judge whether an action served the intention or wandered away from it.

Binding agents to their job description also helps us quantify the value they create, just as OKRs and KPIs are set against a role profile. An agent that is supposed to produce marketing creatives can be evaluated by the number of images and videos adopted in a campaign, combined with the time that the marketing team used to spend on those tasks. It is in general easier to quantify the value of a specialist agent built for one type of task, and much more difficult to quantify a generalist agent which may be helpful in many ways but rarely with a measurable outcome. Hence, a job description and a few KPIs can make thevalue measurable; a recorded mandate makes every action judgeable. Together they are the most sensible way to bound an agent's behaviour, for both security and efficiency.

The bouding and sanction: security

Translate "bounding and sanction" into the vocabulary of the AI world and you get security. Security is a massive topic and deserves a proper deep dive of its own. Here we will only introduce a few key ideas, enough to see where it sits within the framework of agent governance. Identity, observability and mandate already provide a good framework to manage agents. But beyond passively tracking all the actions your agent carries out, you need a mechanism, with some intelligence in it, to identify abnormal behaviour and act on it: refuse a tool call, interrupt the agent, or shut down the workload. This is the step from detective work to preventive work, and it is the most difficult piece of agent governance.

It helps to think of the reaction mechanism as three levels:

The first level is fixed rules: which tools the agent may call, which data may leave the network boundary, how much a single task may spend, which actions are simply forbidden. These rules are simple and deterministic, and that is their virtue; a large share of the accidents people worry about never happen because the door was never there [14]. The job description and the mandate help here too: if the agent declared, before starting, which tools and which resources it would need, then any call outside that declaration is an anomaly by definition, and can be refused directly.

Technically, this rests on two checks every time the agent reaches for a tool. The first is authentication: the tool interface verifies who the agent is, from the short lived credential it received at start (the badge). The second is authorisation, and here there are two scopes to check against. One is the agent's own scope, defined once when the agent is registered: the access to tools and data which matches its "job description". The other is the delegated scope, granted for this task by the user, and never wider than the user's own access permission. The agent may only take an action that falls inside both scopes. Moreover, when the agent hands part of the task to a second agent, the delegated scope should travel with it, and each hop may narrow it but never widen it [4]. That is how the human intention at the top of the chain survives the hops our second difference warned about.

The second layer is a watcher: a separate model that reads the whole trace of the agent's plans and actions (this is where the observability pays off) and checks it against the user's intention. The watcher catches what the rules cannot: a sequence of individually permitted steps that, taken together, drifts away from what the user asked for. To do this, the watcher has to understand the intention semantically and make a judgement call about whether a trajectory is harmless or not. In other words, the watcher has to be a language model itself. Google calls this approach "hybrid defence in depth" in its introduction to AI agent security [15]: deterministic rules enforce the hard limits at runtime, and a reasoning model covers the grey zone the rules cannot express. But a watcher inherits the weaknesses of every language model. It can be fooled by the same tricks as the agent it watches, and the same paper is explicit that it can never be the only line of defence. The usual remedy is to keep it on a very short leash, letting it answer only "allow", "refuse" or "escalate" rather than pass free text back into the agent, so that a compromised watcher can at worst be wrong, never persuasive. And a watcher that refuses too often trains people to override it, which brings us to the third layer.

The third layer is a person. Escalation to human approval is reserved for actions that are high risk and cannot be undone: money leaving the company, a message going to a customer, a record deleted, a contract signed. The limit of this layer is human, not technical. Ask a person to approve forty actions a day and by the second week they approve all of them; approval fatigue is now a recognised attack surface in its own right [16]. So the third layer only works if the first two keep it rare.

Back to the goal

Let's go back to where we started. Agent governance exists to do three things: contain the risk, bound the cost, and make sure the value is proved rather than asserted. Everything in the toolkit serves one of those three, and it is worth saying which.

The badge and the ledger are what cap the cost. Once every call to a model or a tool carries an agent identity, an end user and a task, we get a clear picture of where the money has gone and for what purpose. This is also the part that is cheapest to build, because the shared dependency on a handful of models and tools hands you a natural place to log everything: the gateway that every agent has to pass through, no matter how it was built.

The mandate is what proves the value. An agent with a job description and a few measurable objectives can be judged the way a new hire is judged after six months: did it deliver what it was brought in for, and what did that cost? Without those objectives, the value of an agent is very hard to evaluate. This is also an argument for building specialists rather than generalist agents. A generalist that helps in many small ways is hard to tie to any concrete business outcome.

The sanction is what contains the risk. The three layers give you more than they may seem to: fixed rules and scope checks that refuse most bad actions before they happen, a watching model for the grey zone the rules cannot express, and a person for the few decisions that cannot be undone. None of the three is complete on its own, and the tools behind each of them are improving month by month. Regulators are starting to draw the outer boundary of the acceptable risk too [17] [18] [19], so the choice is not entirely a company's own.

This brings me to my last point. There is no universal setting for agent governance. Each company has to decide for itself how much risk it is willing to carry, put in place the controls and guardrails that match that decision, and then let its people build. The badge, the ledger, the mandate and the sanction are not there to slow anyone down; they are what makes it safe to say yes. The company that governs its agents well will not be the one with the most rules. It will be the one where an employee can spin up a fleet of agents to amplify their talents without worrying about the cost and security issues.

References

[1] OpenAI, The Hugging Face incident and the road ahead (August 2026). openai.com/index/hugging-face-incident-and-the-road-ahead

[2] Chan et al., Visibility into AI Agents (2024). arxiv.org/abs/2401.13138

[3] Kolt, Governing AI Agents (2025). arxiv.org/abs/2501.07913

[4] Tallam, Authorization Propagation in Multi-Agent AI Systems: Identity Governance as Infrastructure (2026). arxiv.org/abs/2605.05440

[5] Chan et al., Infrastructure for AI Agents (2025). arxiv.org/abs/2501.10114

[6] Cloud Security Alliance, Agent Identity Governance Framework v1 (2026). labs.cloudsecurityalliance.org/agentic/agentic-identity-governance-framework-v1

[7] Otsuka, Toyoda and Leung, AI Identity: Standards, Gaps, and Research Directions for AI Agents (2026). arxiv.org/abs/2604.23280

[8] SPIFFE, the Secure Production Identity Framework for Everyone, specification. spiffe.io

[9] Jiang et al., SoK: Agentic Skills, Beyond Tool Use in LLM Agents (2026). arxiv.org/abs/2602.20867

[10] FinOps Foundation, The FinOps Framework. finops.org/framework

[11] Google Cloud Architecture Framework (cloud.google.com/architecture/framework), AWS Well-Architected Framework (aws.amazon.com/architecture/well-architected), and Google Cloud, Shared responsibility and shared fate.

[12] South et al., Authenticated Delegation and Authorized AI Agents (2025). arxiv.org/abs/2501.09674

[13] Google, Agent Payments Protocol (AP2) documentation, intent and cart mandates. ap2-protocol.org

[14] Debenedetti et al., Defeating Prompt Injections by Design (2025). arxiv.org/abs/2503.18813

[15] Díaz, Kern and Olive, An Introduction to Google's Approach to AI Agent Security, Google (2025).

[16] OWASP GenAI Security Project, Top 10 for Agentic Applications (December 2025). genai.owasp.org

[17] European Commission, AI Act Service Desk, How are AI agents addressed within the AI Act? (2026). ai-act-service-desk.ec.europa.eu

[18] Singapore IMDA, Model AI Governance Framework for Agentic AI (2026). imda.gov.sg

[19] AEPD, Orientaciones sobre Inteligencia Artificial agéntica desde la perspectiva de protección de datos (2026). aepd.es