Evaluating AI Agent Performance: One Scorecard, Two Blind Spots

To evaluate AI agent performance, score the work, not the model. Give every agent task the same record a human task gets: an owner, a due date, a condition for done, and evidence of what shipped. Then score agents and people with one formula. That covers execution. It does not cover judgment, and it does not cover failures nobody saw, so those two get their own instruments.

That is how we run it on our own companies, and the gap it closes is common. LangChain’s State of Agent Engineering survey, 1,340 responses collected between 18 November and 2 December 2025, found 89% of organisations have observability on their agents but only 52.4% run offline evaluations. Most teams can see what an agent did. About half can say whether it was any good.

Why agents and people get the same scorecard

An agent that ships late and gets reopened twice has cost the business what a person who did the same has cost. So our evaluation agent scores both from the same task records with the same formula. Owner type is a field on the record, not a separate rubric, and that was a decision written down before agent scoring went live.

The mechanics are in our post on AI employee performance tracking: base points, an impact multiplier, a timeliness multiplier, a quality modifier, and verification before anything scores. This post covers the design choices underneath that scorecard, and the two things it cannot see.

The practical case for one scorecard is that the alternative is two leagues. Measure agents on uptime and run counts while people are measured on outcomes, and nobody can compare them. Every “should an agent take this job” conversation turns into opinion.

Microsoft’s 2026 Work Trend Index, published 5 May 2026 from a survey of 20,000 knowledge workers in 10 countries, names “who reviews agent performance” as one of three questions organisations have to answer before agents scale. Our answer: the same system that reviews everyone else.

What we deliberately leave out of the score

Approval tasks are excluded from scoring. A task that is a human decision gate, such as approving leave or signing off a payment, is not performance work. Scoring it would reward whoever sits in the approval path. Our operating playbook says so in one line: action gates are excluded from performance scoring.

This matters more for agents than for people. Our agents raise a steady stream of approval requests, because the always-human decisions route through a person by design. If those counted, the approver would look like the most productive person in the building and the agent that correctly asked permission would look idle.

Excluding gates keeps the score about work done, and it keeps asking for approval free. An agent should never be nudged by its own scorecard toward skipping a check.

Why every score is an index and a raw total

Every score comes out as two numbers. A performance index, neutral to volume, answers “how well”. A raw contribution total answers “how much”. Publish only one and you lose something: a busy agent running routine jobs and a narrow one doing a few critical jobs either look identical or one looks unfairly bad.

Our design notes put the goal plainly: a workhorse and an owner of one easy task should not be indistinguishable. A small team doing high-stakes work can land in the same index range as a large team doing routine work, while their contribution totals still diverge.

For agents that split is most of the story. A reporting tomte, our word for one production agent with one defined job, might finish a task every night. An intake agent might close one task a week that matters more than all of those. The index lets you judge each on quality. The total stops anyone pretending they carry the same load.

Blind spot one: the scorecard cannot see judgment

A task scorecard grades execution: was the job done, on time, to the stated condition. It says nothing about whether an agent’s decisions were right. Some agents mostly make calls rather than finish tasks, and for those we run a second instrument, a trust meter built from ratings by the person on the receiving end.

Our chief-of-staff agent is the clearest case. Its job is deciding what reaches a human. Every decision card the owner answers gets rated real or noise, which gives a hit rate and a false-alarm rate, and that meter is what we check before widening what the agent may do alone.

The honest detail: the meter currently reports a fixed authority level. Authority does not grow by itself as trust is earned. A human reads the meter and decides. We are comfortable with that for now, but the design promises something the build does not yet do, so we say it.

The same survey backs the choice of grader. Among teams running evals, 59.8% use human review and 53.3% use a model as the judge. For judgment calls we lean on the human side, because the person who received the card knows whether it was worth their time.

Blind spot two: a clean record is not proof of health

Zero failures can mean an agent is good, or that its failures are invisible. So apart from the score, every agent sits on a maturity ladder driven by run-log evidence. It cannot climb past a certain rung until it has logged at least one real failure with real error text, because an agent never seen failing has never been seen failing legibly.

The ladder moves both ways. On one live recompute, two agents were demoted from a self-repairing stage back to a reporting stage because their logs did not support the rung they held. Their scores did not change. Their autonomy did.

The uncomfortable part is that our evaluation agent is held back by this rule too. Its own record shows no observed failure with real error text, and no merged improvement to itself. By the bar it helps enforce, the judge has not yet proven it can be judged.

Rules that stop agents gaming the score

Nothing grades its own work. That is the rule the rest depends on. A build agent’s self-check exists to catch it marking its own work correct when it isn’t, and a human merge check still sits behind it. Our code review agent runs on a different model and prompt from the one that wrote the code, because an executor re-reading its own output tends to agree with itself.

Add evidence-to-close and the approval exclusion, and the cheapest route to a good score is doing the job.

Three ways to evaluate an agent

	Observability only	Model evals	Task scorecard plus trust meter
What it answers	What the agent did	Is the output good on test cases	Did the job get done, and were the calls right
Comparable with people	No	No	Same formula as staff
Work claimed but not shipped	Invisible	Invisible	Verification runs before scoring
Bad judgment on live decisions	Invisible	Partly, on test data	Real-or-noise rating by the recipient
Failures nobody logged	Invisible	Invisible	Ladder blocks promotion until one is seen
Who grades	Nobody	A model or a reviewer	Task evidence, plus the person affected

Model evals still earn their place before launch. They just stop at the lab door, and the scorecard starts there.

Where our own evaluation layer is still weak

Three gaps are documented in our own files, and none is fixed yet. We publish them because a scorecard you cannot audit is a scorecard nobody should trust.

First, scores attach to people by name matching, not a stable ID, and that has put a score on the wrong person before. Second, the hourly verification pass has a recurring failure: it looks for a field on task records that does not exist, and that one error is the largest failure signature in its log.

Third, the gate that would block a task from closing without evidence is still a design proposal. It exists on paper because we found tasks marked complete with nothing behind them, some of them false claims. Until it ships, most completions carry no evidence field, so verification checks what it can and the rest pass quietly.

That is not unusual. Grant Thornton’s 2026 AI Impact Survey, released in April 2026 from nearly 1,000 senior US business leaders, found 78% lack strong confidence they could pass an independent AI governance audit within 90 days. The quickest way to pass one is to already know where you would fail.

What this costs to set up

Our fleet runs on this layer. As of 30 June 2026 it held 41 registered agents with 9 live in production, and the dated figures are in our agent fleet case study. Each one is scored on the same record as the people who work beside it.

For a client, evaluation is set up inside the Fractional AI Officer retainer at $2,900 a month, cancel monthly. We write your done-conditions with you in writing, point the scorecard at your task records, and you get the report plus the gaps, stated as plainly as the ones above. You own the rubric and the data.

If you run agents and cannot say which ones earn their keep, start async. Describe what they do in writing and we reply within one business day. No calls.