Human in the loop: oversight and accountability for AI agents

When an AI agent drafts an email, nobody worries much about who is responsible. When it sends a refund, changes a contract or rejects an application, the question matters a lot. Digital collar work is largely about this: deciding which steps a system may take on its own, which need a person to sign off, and making sure someone clearly answers for the result.

The short answer: an AI agent is never the accountable party. A named person or team is. Good practice, set out in the US NIST AI Risk Management Framework and, for high-risk systems, required by the EU AI Act, is to define who oversees each system, make sure they understand it and have the authority to override or stop it, keep logs, and match the level of human review to the risk of the task.

What "human in the loop" means

NIST describes a range of human and AI setups, from fully autonomous to fully manual. An AI system can make decisions on its own, hand the decision to a human expert, or serve a human decision maker as an extra opinion. Some systems, such as models that improve video compression, may need no human oversight; others specifically require it (NIST AI RMF 1.0).

In practice there are three common setups:

• Human in the loop: the agent proposes, a person approves each action before it takes effect.
• Human on the loop: the agent acts, a person monitors and can step in or stop it.
• Human out of the loop: the agent acts alone within fixed limits, and people review results afterwards.

Deciding what needs sign-off

A simple way to place a task is to ask four questions:

1. Can it be undone? A draft can be thrown away. A payment, a deletion or a sent message often cannot.
2. Who is affected? Decisions about people's money, health, jobs or rights call for closer review.
3. How often is the system wrong here? Measure it on your own work, not on a vendor demo.
4. Would anyone notice a mistake? Errors that surface quickly are less dangerous than silent ones.

Low-risk, reversible, easily checked work can run with light review. Irreversible or high-impact steps should wait for a person. The setup can loosen as a system proves itself on real work, and tighten again when it does not.

Who is accountable

NIST treats governance as a function that runs through all other risk management work. Its framework calls for roles, responsibilities and lines of communication about AI risks to be documented and clear, for executive leadership to take responsibility for decisions about AI risks, and for policies that define roles for human and AI setups and for oversight (NIST AI RMF 1.0, GOVERN 2.1, 2.3 and 3.2).

It also calls for mechanisms, with assigned responsibilities, to supersede, disengage or deactivate AI systems whose performance or outcomes do not match their intended use (MANAGE 2.4). Put simply: someone must own the off switch.

What the EU AI Act requires

For high-risk AI systems, the EU AI Act turns much of this into law.

• Systems built to be overseen. High-risk systems must be designed so that people can effectively oversee them while they are in use (EU AI Act, Article 14).
• Overseers who can act. The people assigned must be able to understand the system's capacities and limits, stay aware of automation bias (the tendency to over-rely on the output), interpret the output correctly, decide not to use it or override it, and intervene or stop the system (Article 14).
• The right people. Organizations using high-risk systems must assign oversight to people with the necessary competence, training and authority, and the support to do the job (EU AI Act, Article 26).
• Monitoring and logs. They must monitor the system, suspend use if risks emerge, report serious incidents, and keep the logs the system generates for at least six months unless other law says otherwise (Article 26).
• AI literacy for everyone. Since 2 February 2025, providers and deployers of any AI system must take measures to support AI literacy among staff who operate and use it (EU AI Act, Article 4).

Human oversight in practice

Oversight that exists only on paper does not catch anything. In day-to-day work, human oversight of an AI system comes down to a few routines that someone actually carries out:

• Named roles. Separate the owner (decides what the system may do and answers for it), the approvers (sign off high-impact actions) and the reviewers (check samples of routine work). One person can hold several roles, but each must be written down.
• Approval screens that show the evidence. An approver should see what the agent wants to do, why, and what it based that on, such as the order, the email or the record. Approving a one-line summary invites rubber-stamping.
• Sampling matched to risk. Check a share of routine outputs every week, and check more when the stakes are higher, after any change to the system, and after an error.
• Time to do it. Reviewing is work. If approvers have seconds per item, they are not overseeing; budget the time in their job.
• Watching for automation bias. Track how often reviewers change or reject what the system proposes. A rate near zero can mean the system is excellent, or that nobody is really looking; spot-check to find out.
• A clear escalation path. Reviewers need to know who to call when something looks wrong, and to be able to pause the step without asking permission.
• Regular review of the setup. Look at the logs, error rates and overrides at a set interval, and adjust what runs alone and what needs approval.

A simple test: if you asked the owner today what the system did last week, what went wrong and what was changed, could they answer from records rather than memory? If not, oversight is not yet working. For the training side, see how to train employees on AI; for measures, see measuring AI agent performance.

When an agent gets it wrong

Mistakes will happen. What separates a well-run setup from a risky one is what follows:

1. Stop the harm. Pause the agent or the affected step. The person who owns the system needs the authority to do this without waiting.
2. Fix the result. Correct what went out and tell the people affected.
3. Trace the cause. Use the logs to see what the agent was asked, what it saw and what it did. Without logs this step is guesswork.
4. Change the setup. Tighten the instructions, add a check, or move the step back under human approval.
5. Record it. Keep a short note of what happened and what changed, so the lesson outlasts the people involved.

"The AI did it" is not an answer. The question is always which person approved the setup that let it happen, and what they will change.

A starter checklist

• Every agent has a named owner.
• Each action the agent can take is marked: runs alone, needs approval, or not allowed.
• The owner can pause or stop the agent at any time.
• Actions and inputs are logged and kept.
• Error rates are checked on real work, not assumed.
• People who use the system know its limits and when not to trust it.