The most valuable habit in digital collar work is also the least glamorous: checking what the AI produced before anyone relies on it. An agent can write a fluent report, a tidy spreadsheet or working-looking code in seconds. Whether it is right is a separate question, and answering it is the person's job.
The short answer: generative AI can state false things with full confidence, and people tend to trust automated output more than they should. Good review means knowing the typical errors, checking the claims that matter against a source you trust, testing rather than reading where you can, and scaling the effort to what is at stake.
Why AI output needs checking
The US National Institute of Standards and Technology lists confabulation among the main risks of generative AI: the production of confidently stated but erroneous or false content, known colloquially as hallucinations or fabrications. It describes confabulation as a natural result of how generative models are designed: they produce outputs that approximate the statistical patterns of their training data, which can be accurate and consistent, or factually wrong and internally inconsistent (NIST AI 600-1, Generative AI Profile, 2024).
NIST notes that the risk is greatest with open-ended prompts for long answers and in fields that need deep context or domain expertise. It also warns that outputs can include confabulated logic or citations that seem to justify the answer, and that a model may set out logical steps for an answer that is itself wrong (same source).
The second problem: trusting it too much
The other half of the risk sits with the reviewer. NIST describes automation bias as excessive deference to automated systems, and notes that as generative AI becomes more capable and reliable, people may over-rely on it or rate its content as higher quality than content from other sources. Automation bias can make other risks, such as confabulation, worse (NIST AI 600-1).
The EU AI Act names the same risk: people assigned to oversee high-risk AI systems must be able to stay aware of the tendency to rely or over-rely on the output (EU AI Act, Article 14). A review step only works if the reviewer is actually prepared to disagree.
Errors to look for
- Invented facts: figures, dates, names or quotes with no real source.
- Invented sources: citations, links or references that do not exist or do not say what is claimed.
- Drift from the brief: output that answers a slightly different question, or ignores a constraint. NIST counts outputs that diverge from the prompt or contradict earlier statements as confabulation too.
- Plausible but wrong reasoning: neat steps that lead to a wrong conclusion.
- Silent omissions: the awkward case, exception or caveat that simply is not there.
- Stale information: answers that were true once and are not now.
Review methods that work
Check claims against a trusted source
NIST suggests comparing AI output with known reliable data and using several evaluation methods together, including human review and automated checks, and deploying fact-checking techniques, especially when information comes from several or unknown sources (NIST AI 600-1, MP-2.3-001 and MP-2.3-003). In practice: open the source and find the sentence. If you cannot find it, treat the claim as unverified.
Test instead of reading
Run the code, recalculate the total, reconcile the figures against the ledger, send the draft email to yourself. A test catches errors that careful reading misses.
Check the parts that matter most
You rarely need to verify every sentence. Identify the claims that drive a decision, a payment or a public statement, and check those every time.
Sample routine work
For high-volume, low-risk output, review a regular sample and track the error rate. If it rises, review more.
Ask for sources up front
Brief the agent to cite where each key fact came from. It makes checking faster, though the citations themselves still need checking.
How much checking is enough
Match the effort to the stakes. A brainstorm needs a glance. A customer reply needs a read. A contract clause, a medical summary or a financial filing needs line-by-line review by someone who knows the field. NIST's example is pointed: in healthcare, a confabulated summary of patient information could lead doctors to the wrong diagnosis or treatment (NIST AI 600-1).
The wider NIST framework lists being valid and reliable among the characteristics of trustworthy AI (NIST AI RMF 1.0). For a single piece of output, a person checking it is how you find out.
A reviewer's checklist
- Does it answer the actual brief, within its limits?
- Have I checked every fact that a decision depends on?
- Do the sources exist and say what is claimed?
- Have I tested what can be tested?
- What is missing that should be there?
- Would I sign my name to this?
Questions and answers
Why does AI make things up?
NIST explains that generative models produce outputs approximating the statistical patterns of their training data. This can give accurate answers, or confident ones that are false. NIST calls this confabulation.
What is automation bias?
Excessive deference to automated systems: trusting AI output more than it deserves. NIST notes it can make risks such as confabulation worse.
How do I check AI-generated content?
Check the claims that matter against a trusted source, test what can be tested, confirm that cited sources exist and say what is claimed, and look for what is missing.
Do I need to check every AI output?
Match the effort to the stakes. Sample low-risk, high-volume work and track the error rate; review high-stakes output in full, by someone who knows the field.
Can I trust AI citations?
Not without checking. NIST warns that generative AI can produce confabulated citations and reasoning that make a wrong answer look justified.