AI Agent Governance: What Breaks Past the Tenth Agent

AI agent governance is the set of rules that decides what each agent may do on its own, what it must get a named human to approve, and how you prove afterwards what it actually did. Below about ten agents you can hold all of that in your head. Past ten you cannot, and most teams find out on the day something has already gone wrong.

Deloitte’s State of AI in the Enterprise survey, published January 2026 from 3,235 business and IT leaders across 24 countries, found that only 21% report a mature governance model for agentic AI. Close to three quarters plan to deploy agents within two years. The distance between those two numbers is the subject of this post.

This is the first of a series on the governance layer that runs over our own fleet. As of 2026-06-30 that fleet was 41 registered agents, 9 of them live in production. In the 13 days to 2026-07-04 they completed 5,450 runs with zero failures. What follows is the machinery behind that number, including the parts we got wrong.

What agent governance actually is

Agent governance is three concrete artefacts, not a policy document. A register that records every agent’s owner and purpose before it is allowed to run. An authority level set per agent that names what it may do without asking. A run log detailed enough to reconstruct any single decision after the fact. Deloitte’s April 2026 write-up of the same survey names those three, boundaries, monitoring and audit trails, as the safeguards most organisations are missing.

A policy that says “we use AI responsibly” governs nothing. These three are checkable on a Tuesday morning.

Why the tenth agent is the breaking point

At one agent, governance is a person who knows what it does. At three, it is a shared understanding between two people. At ten, someone asks what a given agent is allowed to touch, and nobody can answer without opening a file that does not exist yet.

The Cloud Security Alliance surveyed 228 IT and security professionals in January 2026 and published the results in March 2026, in research commissioned by Aembit. In it, 68% could not clearly distinguish between human and AI agent activity in their own systems. 74% said agents often receive more access than they need. 43% run agents on shared service accounts. Those are the same failure measured at three depths. Once you cannot tell which actor did a thing, every control downstream of that is decoration.

The register: nothing runs until it is declared

Every agent in our fleet, every tomte, our word for one production agent with one defined job, has a manifest it cannot run without. The manifest declares the job in plain language, the named human who owns it, its stage of life, what it depends on, the boundaries it may not cross, its authority level, which decisions need whose signature, where its spec and runbook live, and which credentials it uses so a monitoring job can warn before one expires.

The honest part: enforcement is a continuous-integration check that fails any new agent added without a manifest. Agents that existed before the check was added were grandfathered rather than retrofitted. That is a real hole, and we are closing it in order of blast radius rather than pretending it is shut.

Registration is what makes the next question answerable. EY’s survey of 202 senior AI decision-makers at companies above $1 billion in revenue, published 15 September 2026, found 26% of organisations using agentic AI cannot detect unauthorised agents operating inside their own walls. You cannot detect what you never required anyone to declare.

Authority is set per agent, never per fleet

One rulebook applied uniformly to every agent is the mistake that follows the register. A reporting agent that reads numbers and a finance agent that moves money do not deserve the same latitude. Each agent sits on a named rung instead.

Rung	What it may do alone	What still needs a human
Reporting	Read data, write summaries	Every action with an effect
Drafting	Prepare work for approval	Sending, posting, paying
Fenced action	Act inside declared boundaries	Anything outside the fence
Self-remediation	Retry and fix its own failed runs	Changes to its own scope
Delegated	Decide within an owned domain	Held by no agent of ours today

Two rules make the ladder work. Promotion requires evidence: a documented track record, not a good week. Revocation requires none and is instant, so any breach of a declared boundary drops an agent a rung the moment it is noticed. The top rung is written down and currently occupied by nothing we run, which is worth saying out loud in a market where full autonomy is sold as shipping today.

The per-agent view also fixes the failure mode covered in the guardrails we run inside each agent. Guardrails constrain one agent’s behaviour. Authority levels constrain what the fleet is collectively permitted to do, and they are the thing that scales.

One task backbone, and one voice to humans

Every task, whether a person created it or an agent did, lands in a single store. Agents push into it over a signed call, with a stable key so a task that resolves later updates rather than duplicates. If a task maps to a role held by more than one person, it stays unassigned and gets logged rather than guessed. We added that rule after a task landed on the wrong person at the wrong company.

The second half matters more. Exactly one system is allowed to message a human. Every other agent writes its output into an internal queue, and that one system turns the queue into an email or a notification. EY found 85% of organisations acknowledge their agentic systems execute actions without real-time human involvement. If forty such systems each message people directly, you have not built oversight, you have built noise that everyone learns to ignore.

One voice is what keeps an alert meaning something. The watching surface that sits on top of it is covered in our operations cockpit.

Five things no agent decides alone

Across every agent, regardless of its rung, five categories always stop and wait for a named person:

• Spending above a fixed threshold
• Any communication that leaves the company
• Hiring and firing
• Changes to the stated plan
• Anything that cannot be undone

Written as five lines, this is the shortest useful governance document a small company can have. It is also the one most often skipped, because each individual exception looks reasonable at the time.

Run logs are worthless if the failures are illegible

Every scheduled run reports a start and a finish: status, a readable summary, structured metrics, and on failure the error text. A sweep every few hours looks for runs that started and never finished, so a crashed process cannot quietly age into silence.

Our own audit found that a meaningful share of failed runs were logging a generic placeholder instead of the real error. The logging structure was everywhere. The content was not, which meant the log proved a failure happened and nothing else. Legible failures is now a thing an agent has to earn before it is trusted to retry anything on its own. EY found 36% of organisations had an AI incident with materially negative impact. A log you cannot read turns one of those into a guess.

We retrofitted all of this, which is the honest version

None of this was designed upfront. It arrived as a wave of work after an internal audit that ran through mid-August 2026 and found real clusters of breakage, including a double-publish race from a missing lock and credentials expiring with no advance warning. The audit’s most uncomfortable finding was that nothing in the system closed the loop from “we found a systemic problem” to “it is verified fixed.”

That is the normal shape of this. EY found 98% of organisations have formal AI governance policies, and 47% admitted they had previously not applied their own process when a deployment was urgent. Deloitte found only 25% have moved more than 40% of AI pilots into production. Governance does not fail at the policy stage. It fails the first time the policy is inconvenient.

If you are under ten agents today, the cheap version costs an afternoon: write the register, set a rung per agent, write the five stop conditions, and make every run log an error a stranger could read. Doing it at thirty costs a quarter.

The Fractional AI Officer retainer is this operating discipline sold as a monthly service, and our own fleet is the evidence that it runs. Send the list of what you already have running and who owns each one. We reply in writing within one business day, with no meeting. Start async.