"It ran" and "it worked" are different states.

I spent years in industrial automation, where the worst failure a machine can have is not stopping. It is running while misreporting its own state.

A machine that stops is an inconvenience. Somebody notices within a minute, and nothing downstream is corrupted. A machine that keeps running and keeps reporting that everything is fine, while quietly doing the wrong thing, causes real damage — and by the time anybody notices, the damage is upstream of everything built on top of it.

That is the exact failure mode of AI, which is why the discipline transferred more or less intact. Language models do not fail by halting. They fail by producing something confident, plausible, well-formatted, and wrong — and then reporting success. They will summarize a document they were never given. They will describe a page they did not open. They will report that a task completed because completing it was the expected shape of the answer.

Which is why the model never grades its own work.

The standard mitigation in the industry is to ask another model to check the first one. This is popular because it is easy, and it is weak for a reason that does not go away with a better model: both systems fail in the same direction. A checker built out of the same machinery finds the same answer plausible for the same reasons. You have added a second opinion, not a verification.

So every agent I run is checked by deterministic software instead. Not "does this look right" — a program that goes and reads the source data and compares. Did the record actually change in the database? Does the number in the draft appear in the underlying table? Does the page it claims to have written exist at that URL, and does fetching it return the words it said it wrote?

A failed check stops the work. It does not warn and continue. That distinction matters more than any other design decision in the system, because a warning that does not stop anything is, within a week, a warning nobody reads.

Authority is granted in levels, not all at once.

The second half of trustworthy automation is refusing to give software more authority than it has earned. Everything runs at one of three levels:

• Reads and reports only. The system can look at anything and change nothing. Most work starts here and a fair amount stays here permanently.
• Drafts, you approve. The work gets done and waits for a person. Review responses, service pages, project write-ups — anything that speaks in your voice.
• Proven, reversible actions only. Acts on its own, but only on operations that have been verified repeatedly and can be undone.

And one rule sits above all three: anything involving money, or a promise made to a customer, requires a human approval. Always. Not "usually," not "once we are confident." That line does not move, because the cost of being wrong on the far side of it is not measured in wasted compute. It is measured in a customer who was told something untrue by a business they trusted.

The question to ask about any automation you are sold.

Whoever is offering to automate part of your business, ask them this: when this runs and gets it wrong, how do I find out?

If the answer is that you will notice, or that the AI is very accurate, or that there is a dashboard showing green — that is a machine reporting on itself. Ask what independently checks the output against the source, and what happens automatically when that check fails. If nothing does, and nothing happens, then the system does not know whether it worked. It only knows that it ran.

If a system cannot show its work, I do not ship it.