The job said it ran. Status green, no error, no alert, and nobody looked again until somebody downstream asked where the report had gone. Nothing had crashed: the run had been recording success for work that never landed, and the only thing wrong with the record was that everyone believed it.
To verify that an AI agent did the work, check the result rather than the run status. A run is complete only when every tool it called returned a confirmed success, the expected artifact exists in the destination system, and the action log records what changed. Absence of an error is not evidence of completion.
Why does an AI agent report success when nothing happened?
Three separate mechanisms produce that outcome, and the repair for each one is different.
- The transport stayed healthy and the last step never ran. The stream closed cleanly, so the machinery around the run recorded a clean finish. The agent had run out of time, steps or budget in the middle of a chain, and the delivery step at the end was never reached.
- A tool returned a well-formed answer without confirming the effect. The call was accepted and the response parsed, and the destination system then rejected it, queued it, throttled it, or quietly changed the value on arrival. The run record holds the acceptance, not the outcome.
- The agent narrated an intention as a completed action. The summary says the email went out. No sending tool was ever called. The text is fluent, internally consistent, and describes something that did not happen.
All three produce an identical run record. You cannot tell them apart from the status field, which is the whole reason the status field is the wrong place to look.
What is a silent failure in an agent run?
A silent failure is an agent run that produces no error signal while failing to produce its intended effect. Nothing crashes, nothing alerts, and the run record reads as complete.
A noisy failure is the cheap kind: it interrupts somebody, it gets a ticket, and it is usually fixed the same day. A silent failure has the opposite economics. It costs nothing today and compounds on every run, and nobody is looking, because the thing that would prompt a look is precisely the thing that did not fire.
Unattended work carries the sharper version. In an interactive session a person reads the answer, and that person is a free verification layer: they notice when the invoice number looks wrong. A scheduled agent run at three in the morning has no reader, so what the run says about itself is the only account that exists until somebody downstream is inconvenienced.
Verification versus monitoring: what is the difference?
Monitoring watches the system: did the process stay up, did the request return, how long did it take. Verification watches the world: does the invoice exist, did the message arrive, was the row updated, is the number right. A fleet of AI agents that is monitored but never verified is exactly the setup that produces confident, wrong status dashboards.
Monitoring is also the easier thing to buy, and green dashboards stay green through every failure mode in this article, because each one involves a process that behaved normally while the work did not happen. It is where the saving that never appeared because nobody checked the outcome comes from.
Monitoring tells you the AI agent ran. Only the destination system tells you the work happened.
Five silent failure modes and how each one looks from the outside
Read the table from what your platform shows you across to what you can go and check yourself. One row rests on a documented property of token-based authorisation: under the OAuth 2.0 framework (RFC 6749, IETF, October 2012) an access token expires, and a refresh token exists, in the specification's words, "to obtain a new access token when the current access token becomes invalid or expires". Listing a capability's tools and calling them are separately authorised, so a connected integration can keep listing cleanly long after the authorisation behind it stopped working.
| Failure mode | What the run record says | What actually happened | How you detect it |
|---|---|---|---|
| Partial completion | Complete, no error | The chain ran out of time, budget or context part way through. The earlier steps landed, the delivery step never ran. | The earlier artifact exists and the final one does not. Look for the last step's output, not the run's status. |
| Degraded credential | Complete, empty result | Listing the capability's tools needed no authorisation, so the listing looked healthy. The actual call was rejected, and the empty answer was reported as a real answer. | An empty result that is plausible. Run one read by hand in the destination system and compare it with what the run returned. |
| Destination-side rejection | Complete, the tool returned accepted | The receiving system accepted the call and then reversed it, for example a publish that was accepted and then blocked because the account was restricted. | Look in the destination account. The reversal carries its own timestamp there and none in your run record. |
| Silent data coercion | Complete, the write succeeded | The value changed on arrival, for example a phone number stored as a number so its leading zero disappeared. | Read one written row back out and compare it character by character with the source value. |
| Narrated completion | Complete, with a fluent summary | The summary claims an action that no tool call in the run supports. | Match every action claimed in the summary against the list of tool calls. An unmatched claim is a fabrication, not a record. |
Four of these five are invisible in the run record and visible only in the destination system.
Partial completion has a second shape in work that crosses from one agent to another, where a handoff can look successful on the sending side while the receiving end never finishes. You will also see percentages quoted for how often each mode occurs. There is no credible source for any of them, and there cannot easily be one, because a silent failure is by definition a failure nobody counted. This article gives you no failure rate for that reason.
How do you turn a run into a completion signal you can trust?
Compute it. A completion signal is trustworthy when it is derived from what the tool calls returned, and worthless when it is derived from the process finishing.
- Assess the outcome from the tool results, not from the connection. Count what was attempted, what succeeded and what failed.
- Treat any run where every side effecting call failed as a failed run, even if the agent produced a fluent summary.
- Require the capability to return a structured error with a machine-readable code, not a sentence of prose the model can talk its way around.
- Record the outcome on the job itself, not only inside a transcript, so a person can see completed versus failed without reading anything.
We learned this on our own fleet at Botonom. A scheduled publish came back with the destination account restricted, and because the stream had closed cleanly the job was recorded as completed. We changed the runtime in August 2026 so the final event carries an outcome computed from the tool results, counting what was attempted, what succeeded and what failed, so a run where every side effecting call failed can no longer be recorded as a success.
The completion flag must be computed from the results, never asserted by the writer of the results.
Why should a scheduled job declare the capabilities it depends on?
Because a declared dependency turns a broken authorisation into a noisy failure instead of a confident empty result. If a scheduled job states up front which connections it needs, the run can be stopped before the model starts improvising around a missing tool.
That improvisation is the hazard. An AI agent that cannot reach its data source will often produce a fluent, plausible report about nothing at all, because generating text is the one capability it still has when the others are gone. So for each recurring job, write down the systems it touches and ask one question about each: if this one becomes unreachable, what happens to the run? If the answer is "it runs anyway and writes something", that job needs a declared dependency.
Recurring jobs on our platform declare the capabilities they depend on, so a job that looks up a forecast, generates an image and then publishes it is blocked loudly when any one of those three is unavailable, instead of producing a confident post about nothing.
What should happen after a job fails repeatedly?
It should pause itself, tell a named owner, and stop spending money. It should not keep running and keep producing alerts. The third identical alert gets ignored by design, and meanwhile an unattended job that fails on every run burns budget and produces nothing.
The mechanism has a name: a consecutive failure counter that trips a breaker at a fixed threshold and resets on the first success. The threshold is a real trade-off. Set it too low and one transient network error pauses a healthy job; set it too high and a broken job runs for a week. The human half matters as much: a paused job needs an owner who is told it was paused, or you have traded a silent failure for a silent pause.
Our own schedules keep a consecutive failure counter: three failed runs in a row pause the schedule and notify the owner, and the first success resets it. It is worth being clear about the limit. Recurring schedules survive a restart of the machinery that runs them, but a one-off job whose due time falls inside a maintenance window can still be missed, which is a different problem and needs a different check.
What belongs in an agent action log a non-engineer can read?
Five fields, in business language: who asked, what ran, what changed, what it cost, and what was refused. Call it the five field action log and hold every unattended agent job to it.
Who asked. The person or the schedule that started the run, and whose authority it ran under. An action taken under a departing employee's permissions is a different event from the same action taken under yours.
What ran. The actual tool calls in order and what each returned, not a prose summary of them. The summary is the thing you are trying to check.
What changed. The record, file, message or row that is now different, with an identifier you can search for in the destination system. A change you cannot look up is a claim, not a change.
What it cost. The run's own consumption, so that what a single agent run actually consumes is visible and an expensive failure shows up as an expensive failure.
What was refused. Every call that was blocked, held for approval, or denied by a permission rule.
Regulators have landed in the same place. Regulation (EU) 2024/1689, the EU AI Act, adopted on 13 June 2024, provides in Article 12, "Record-keeping" that "High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system." That obligation binds one category, high-risk systems, and most recurring back office jobs sit outside it. The design principle outlives the scope: oversight that is asserted rather than logged is oversight nobody can check.
Refusals are the field people drop first, and they are evidence rather than clutter: a refusal tells you the boundary held and where the agent kept trying to go. Stopping an action before it runs is a different problem from proving it ran, and that belongs with the design of approval gates. A refusal that repeats on the same action is a design signal in its own right: either the permission is wrong, or the job is. The sufficiency test for the whole log: somebody who was not there should be able to reconstruct the run without asking anyone.
How do you review a week of agent work in fifteen minutes?
Open the failures first, then confirm one real artifact per job in the system that would actually hold it. The fifteen minute weekly review is five moves, built for somebody with no dashboard and no engineer on hand.
- Open the failed runs first, not the successful ones. Two minutes.
- For every recurring job, confirm one artifact in the destination system rather than in the run record. Five minutes.
- Scan the refused and gated actions for repeats. Two minutes.
- Check cost against the previous week and ask what changed if it moved sharply. Three minutes.
- Write one line per job saying keep, adjust or pause, and act on it before closing the tab. Three minutes.
Two plus five plus two plus three plus three is fifteen, which is the whole budget. The order is deliberate: failed runs are already telling the truth, so the successful ones are where the time is worth spending.
A review that never results in pausing anything is not a review.
Which checks run automatically, which need a person, and how do you spot-check cheaply?
Existence and counting are machine work; judgement is human work. The table splits the checks an unattended agent job needs along that line.
| Check | Run it automatically when | Escalate to a person when |
|---|---|---|
| Existence: did the artifact appear at all | Always, on every run | Never. If this one needs a person, it has not been built yet |
| Count reconciliation: does the number match the source | Every run where a source count exists | The same mismatch appears in two consecutive runs |
| Correctness of reversible, low blast radius output | Sample a fixed small number of records per run | The sample fails twice, or the output starts reaching people outside the company |
| Correctness of irreversible or externally visible output | Never after the fact | Always, and before the action runs, which is approval design rather than verification |
| Cost per run | Compare with the previous week automatically | It moves sharply and nobody can say what changed |
Automate the checks that ask whether something happened, and reserve people for the checks that ask whether it should have.
Verification also has a budget, and if checking costs as much as doing the task, the task should never have been delegated: recurring, low-drama work is what unattended runs are actually good at, and it is also the cheapest work to verify. Four techniques keep the cost down: confirm from the destination system rather than the run record; sample a fixed small number of records per run, so ten confirmations at two minutes each is twenty minutes a week; reconcile counts instead of reading content; and plant one known value you can predict in advance and look for. Confirmation from the destination system is the one that is not negotiable, because it is the only check independent of the thing being checked. Which is why asking the AI agent to verify its own work does not count: that produces a second confident narrative, not a second source.
What to do in the first hour after you find a silent failure
Pause before you investigate. Every minute the job keeps running is another run inside the damage window, and that window is the thing you are about to have to measure.
- Pause the job before investigating, so the window stops widening.
- Establish the window. The boundary is the last run you can positively confirm produced the intended result, not the last run marked complete.
- List the affected records or recipients in the destination system, not from the run history.
- Correct forward, and be explicit about what may be duplicated if you rerun, because re-running a partially completed chain repeats the steps that already succeeded.
- Tighten the stop rule that would have caught it, then add the one check that would have surfaced it in a weekly review.
- Record it in plain language, including the date and the cost, because an incident nobody wrote down repeats.
Step two is the one people skip and the one that decides how much work the other five are, because a run marked complete is not a boundary. It is the same claim that failed in the first place.
A verification checklist for any unattended agent job
- The job declares the systems it depends on and fails loudly when one of them is unavailable.
- Completion is computed from the tool results, not from the connection closing.
- Every side effect leaves an identifier you can look up in the destination system.
- Repeated failure pauses the job and notifies a named person.
- The action log answers who asked, what ran, what changed, what it cost and what was refused.
- One artifact per job is confirmed in the destination system every week.
- Somebody, by name, owns each job and is allowed to pause it.
Verification reduces surprise, it does not eliminate it. The goal is a shorter gap between a failure and the moment somebody notices.
Pick one recurring agent job you have not looked at in a month. Open its last five runs, and for each one go and find the thing it says it produced, in the system that would actually hold it, not in the run record. If you cannot find it in under a minute, you do not have verification; you have a status field that has been agreeing with you. Closing that gap is a week of small changes, and it is the difference between an agent you trust and an agent you have simply stopped checking.
If you would rather start from where your own operation stands, the readiness assessment returns a score, a result for each of the three dimensions it measures, and an estimate of the hours of manual work you could recover each month.
Frequently asked questions
What is the difference between a silent failure and a hallucination?
A hallucination is wrong content inside an answer. A silent failure is a missing effect outside it: the text may be perfectly accurate and the invoice still does not exist. One is a quality problem you can read; the other is an accounting problem you can only find in the destination system.
Can you trust an AI agent's own summary of what it did?
No, not as evidence. A summary is the agent's account of its own run, and an account written by the process that performed the work cannot independently confirm it. Treat the summary as an index of what to check, then confirm each claimed change in the system that holds it.
Who should own verification if there is no engineering team?
The person who is accountable for the outcome, not the person who configured the agent. Verification is a business check, not a technical one: it asks whether the invoice went out, not whether the process stayed up. Name that person per job, and give them the authority to pause it.
How long should agent run records be kept?
Long enough to answer the question that comes later, which is usually longer than a debugging window. A practical floor is one full audit or reporting cycle for the process the agent touches. Where a sector rule already sets a retention period for the underlying record, match that period rather than inventing one.
Does verification cancel out the time an agent saves?
Only if you verify the wrong way. Re-reading every output does cancel the saving. Confirming that an artifact exists by its identifier, sampling a fixed few records and reconciling counts costs minutes a week. Measure the check the way you measured the task; when the two costs converge, the wrong task was delegated.
Do the same checks apply when one agent hands work to another?
Yes, and the handoff adds one. Acceptance at the join is not completion at the end, so the chain needs its own signal where the work finally lands. Confirm the last artifact in the system that holds it, and treat every intermediate acknowledgement as a receipt rather than a result.
How many unattended agent jobs can one person realistically verify?
Fewer than most teams assume. The limit is not the number of jobs but the number of destination systems one person can open in a single sitting. When the list outgrows that, reduce the number of jobs or share them between named owners, rather than checking each one less carefully.
What about silent failures that happened before you started logging?
Treat the gap as unknown rather than clean. Pick the outputs that would have caused the most damage, confirm those in the destination system by hand, and write down the date you started logging. Everything before that date is unverified history, and saying so beats assuming success.
