How to govern AI agents in production

Why agents need different governance

An agent acts. It calls tools, changes records and triggers workflows, so the risk moves from what the system says to what it does. Governance therefore has to cover permissions and delegated authority as well as output quality. The OWASP Top 10 for LLM Applications lists this risk as "LLM06:2025 Excessive Agency".

A model that drafts text for a person to read has a natural checkpoint: the person. An agent can remove that checkpoint. It may chain several tool calls in a single run, act on behalf of a user whose permissions it has borrowed, and repeat a mistake quickly. The questions a risk team asks change accordingly. What may the agent touch? Who authorised this action? What exactly happened on Tuesday at 14:02? Can we stop it now, and what happens to the work in flight?

Each of those questions needs an answer that exists before an incident, and evidence that can be produced after one. The controls below provide both. None of them depends on the model behaving well; they sit in the harness around the model, where the organisation sets and enforces them.

The control model at a glance

Boundaries and permissions

Give each agent the smallest set of tools it needs, with permissions scoped to the task and the user it acts for. Put tool calls through a policy check before they run, separate read from write, cap spend and rate, and make high-impact actions require approval. Enforce these limits in the harness, where the model cannot change them.

• Least privilege. Each tool the agent can call is listed in its permission manifest, with the operations allowed. Anything absent from the manifest is unavailable.
• Delegated authority. When an agent acts for a user, it acts with that user's permissions or fewer, and the audit trail records whose authority it used.
• Read and write separation. Reading a record and changing it are separate permissions. Many agents need only the first.
• Spend and rate limits. Cap the number of tool calls, model calls and external actions per run and per day, so a loop fails safely and cheaply.
• Data boundaries. Decide what data the agent may see and what must never leave your environment, and enforce it at the tool layer.
• Policy check. Before each tool call runs, a check confirms it is allowed for this agent, this user and this context, and logs the decision.

The audit trail

For every step, record who or what initiated it, on whose authority, the input and context used, the model and prompt or policy version, each tool call with its parameters and result, any human approval, and the final outcome, with timestamps and a trace ID. Store it tamper-evidently for a defined retention period.

A generic field list, to adapt to your own systems:

trace_id
step_id
timestamp
initiator            (user, schedule or upstream system)
acting_on_behalf_of
agent_id and agent_version
model_version
prompt_or_policy_version
input_reference      (or a redacted copy)
tool_name
tool_parameters
policy_decision      (allowed, denied, needs approval)
approver and approval_time
tool_result_summary
outcome
error

Decide retention with your legal and risk teams, and apply the same data protection rules to the trail as to the data it references.

Human oversight that works

A human should approve an agent's action when it is irreversible, affects customers or money, crosses a data boundary, or falls outside tested behaviour. Define these thresholds in advance, record who approved what, and make sure reviewers see enough context to make a real decision.

Approval design matters as much as the threshold:

• Show the reviewer what they need. The proposed action, the inputs that led to it, the agent's stated reason, and what will happen if they approve.
• Keep approvals rare enough to be read. If people approve dozens of actions an hour, they stop reading. Move low-risk actions below the threshold and tighten the agent's permissions instead.
• Give reviewers real options. Approve, reject, edit, or escalate, and record which they chose.
• Measure it. Track approval rates and time to decision. An approval rate close to 100% can mean the threshold is too low or reviewers are rubber-stamping.

Evaluating agents

Before release, run task-level test suites with expected outcomes, adversarial cases and structured-output checks as a release gate. After release, monitor task success, tool-error rates, escalations and drift, and replay regression suites on every model, prompt or tool change.

Evaluation for an agent tests behaviour across the whole run: did it pick the right tool, pass the right parameters, stay within its permissions, and stop when it should? Keep a regression set of real cases (with sensitive data removed), including the cases that failed in the past. Record the results per release so that the evaluation report becomes part of the evidence.

Stopping, rollback and incidents

Build the stop in from the start: a switch that disables the agent and falls back to the previous behaviour, credentials that can be revoked, and runbooks the operating team has rehearsed. If you cannot turn the agent off quickly, it is not ready for production.

• Switch. One setting turns the agent off without a deployment. Test it in every release.
• Fallback. Decide what happens to work in flight and which process takes over, whether that is a queue for people or the system as it worked before.
• Revocation. The agent's credentials can be withdrawn independently of anyone else's.
• Incidents. Treat agent incidents like other production incidents: detect, contain, investigate from the audit trail, fix, and add the case to the regression set.

Ownership

A named service owner, as for any production system, supported by risk or compliance for the control set. Ownership covers permissions, approvals, evaluation results, incidents and the decision to switch the agent off.

Review the permission manifest and the approval thresholds on a regular schedule and whenever the agent, its tools or its model change. Keep a short record of each review and what it changed.

Mapping to the EU AI Act, ISO/IEC 42001 and the NIST AI RMF

Human oversight (EU AI Act Article 14), record-keeping and logs (Article 12), and accuracy, robustness and cybersecurity (Article 15) correspond to approvals, audit trails and evaluation. ISO/IEC 42001 provides the management system around them, and the NIST AI RMF provides the Govern, Map, Measure and Manage vocabulary.

Articles 12, 14, 15 and 26 of the EU AI Act bind high-risk AI systems only. After the Digital Omnibus on AI, those obligations apply from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems. For other agents, treat the mapping as good practice. Whether a given agent is high-risk is a classification question; see does the EU AI Act apply to UK companies?

We map ISO/IEC 42001 at the level of its management-system themes. Map specific clauses and Annex A controls against the text of the standard your organisation holds. The NIST AI Risk Management Framework 1.0 (January 2023) is voluntary; NIST's Generative AI Profile (NIST AI 600-1, July 2024) applies it to generative AI. In the UK, the AI Risk Management Toolkit that DSIT published in September 2026 is a practical starting point for a risk register.

A starting checklist

• Every agent in production is listed, with a named owner.
• Each agent has a permission manifest listing its tools and allowed operations.
• Write operations are separated from read operations and granted only where needed.
• Tool calls pass a policy check before they run, and the decision is logged.
• Spend and rate limits are set per run and per day.
• Approval thresholds are written down, and approvals are recorded with the approver.
• The audit trail captures the fields above, with a trace ID across every step.
• Audit logs are tamper-evident and have a defined retention period.
• A regression set runs as a release gate on every model, prompt or tool change.
• Monitoring covers task success, tool errors, escalations and cost.
• The stop switch and fallback have been tested in the current release.
• Permissions and thresholds are reviewed on a schedule, and each review is recorded.

How Comonad helps

Comonad Limited, a London applied-AI consultancy, designs AI agent harnesses with these controls built in: permissions, policy checks, audit trails, evaluation and a tested stop. We also write the governance they must satisfy through our AI governance consultancy, mapped to the EU AI Act, UK regulatory guidance, ISO/IEC 42001 and the NIST AI RMF. Legal sign-off stays with your counsel.