"It seems to work" is how most AI agents get put into use. It is not a measurement. Before an agent is trusted with more of the work, someone needs numbers: how often it is wrong, how badly, and whether that is changing. This page sets out what to measure, how to measure it without drowning in review, and how to use the results to decide how much freedom an agent gets.
The short answer: measure the agent on your own work, in conditions like real use, not on a demo. Track its error rate and the severity of its errors, the time people spend fixing its output, and how often it escalates. Keep measuring after launch, because performance drifts. Widen what the agent may do alone only when the numbers support it, and narrow it again when they do not.
Measure on your own work
The US NIST AI Risk Management Framework asks that an AI system's performance be measured and demonstrated for conditions similar to the setting where it will be used, and that the system to be deployed is shown to be valid and reliable, with the limits of how far its results generalize documented (NIST AI RMF 1.0, MEASURE 2.3 and 2.5).
In practice, a vendor benchmark tells you little about your contracts, your customers or your codebase. Build a test set from your own past work where you already know the right answer, and run the agent on it before it touches live tasks.
What to track
| Measure | What it counts | Why it matters |
|---|---|---|
| Error rate | Share of outputs a reviewer had to correct or reject | The core quality number; track it per task type |
| Severity of errors | How bad the mistakes were: cosmetic, costly or harmful | Ten typos matter less than one wrong payment |
| Escalation rate | How often the agent stopped and handed over to a person | Too low can mean it is guessing; too high means it is not saving time |
| Rework time | Time people spent fixing the agent's output | The honest measure of time saved |
| Turnaround | Time from task to approved result | Includes review, not just generation |
| Complaints and appeals | Problems reported by customers, staff or others affected | Catches what internal review misses |
NIST also asks organizations to document the test sets, metrics and tools they use (MEASURE 2.1), and to record which risks will not or cannot be measured, rather than quietly leaving them out (MEASURE 1.1).
Sampling instead of reviewing everything
Reviewing every output defeats the point of an agent doing high-volume work. Sampling gives a reliable picture at a fraction of the effort:
- Start with full review while the agent is new, until you have a baseline.
- Move to a regular random sample once the error rate is low and stable.
- Always review the high-stakes cases in full, whatever the sample says.
- Increase the sample after any change to the agent, its instructions, its model or its inputs.
Pick the sample at random. Reviewing only the outputs that look odd will miss the confident, plausible errors that matter most. Our guide to checking AI output covers how to review each item.
Keep measuring after launch
An agent that passed testing can still get worse. Inputs change, the underlying model is updated, and the cases it sees in real use differ from the test set. NIST calls for the functionality and behaviour of AI systems to be monitored in production, and for approaches to track existing, unanticipated and emergent risks based on intended and actual performance in use (NIST AI RMF 1.0, MEASURE 2.4 and 3.1).
For high-risk systems, the EU AI Act requires deployers to monitor the system's operation and to suspend use if they find a risk (EU AI Act, Article 26).
Bring in people who did not build it
The team that set up an agent is the team most likely to overlook its weaknesses. NIST recommends that internal experts who were not front-line developers of the system, or independent assessors, take part in regular assessments, and that domain experts and users are consulted (MEASURE 1.3). It also calls for ways for end users and affected people to report problems and appeal outcomes, with that feedback fed into the evaluation (MEASURE 3.3).
Using the numbers to set autonomy
Measurement is only useful if it changes decisions. A simple ladder:
- Shadow: the agent runs alongside the current process; nothing it produces is used.
- Approve each: its output is used only after a person approves it.
- Sample review: it acts alone on routine cases; a sample is checked.
- Exception review: it acts alone; people review flagged cases and complaints.
Set the error rate and severity limits for each step in advance, in writing, for each type of task. Move up a step only when the agent has met them over enough real cases. Move down immediately after a serious error or a rise in the error rate. NIST's MANAGE function expects mechanisms to supersede, disengage or deactivate AI systems whose performance or outcomes do not match intended use (MANAGE 2.4).
Measuring the measurement
Metrics can mislead. An agent can score well on the error rate by escalating every hard case, or by producing output so vague that it is never technically wrong. NIST asks that the effectiveness of the metrics themselves be evaluated (MEASURE 2.13), and that measurement approaches be connected to the real setting and informed by domain experts and users (MEASURE 4.1). Check now and then that the numbers still describe what you actually care about.
A starter scorecard
- Test set built from your own past work, with known answers.
- Error rate and severity, per task type.
- Rework time and escalation rate.
- Random sample size and how often it is reviewed.
- Written thresholds for each autonomy step.
- Date of the last review and who did it.
Questions and answers
How do you measure an AI agent's performance?
Run it on your own past work with known answers, then track its error rate, error severity, rework time and escalation rate in real use, using random samples.
Why not use the vendor's benchmark?
Benchmarks rarely match your work. NIST asks that performance be demonstrated for conditions similar to the setting where the system will be used.
Do I need to review every AI output?
Not once performance is established. Start with full review, then move to a random sample, but keep reviewing high-stakes cases in full.
When should an AI agent get more autonomy?
When it has met written error rate and severity limits over enough real cases. Reduce its autonomy immediately after a serious error or a rising error rate.
Does AI performance change over time?
It can. Inputs, cases and underlying models change, so NIST calls for monitoring AI systems in production, not just testing them before launch.