The pilot worked. You have probably seen a version of this: a recording where the agent read the ticket, pulled the order, drafted the reply and got it right nearly every time. Six months later the process is still being done by hand, and nobody in the room can say which week it stopped mattering.
What counts as an AI agent pilot, and what counts as production?
An AI agent pilot is a limited trial in which an AI agent performs a real business process on a selected set of cases, under human observation, to establish whether it can do the work.
Production means the same agent doing the same work at full volume, on cases nobody filtered, with the authority to finish them and a monitored record of every action it took.
The short answer. AI agent pilots usually stall for one of three reasons: the process was never written down precisely enough for anything to follow, the agent was never allowed to complete the work, or the human review step became a queue at real volume. Diagnosing which one applies matters more than switching tools.
Production has four conditions, and all four must hold:
- Real volume. Work arrives at the rate it actually arrives, including the busy period.
- Unfiltered cases. Nobody removes the awkward ones before the agent sees them.
- Authority to finish. The agent completes a defined set of actions without a person taking the last step.
- A monitored record. Every action is recorded where a person reviews it, through a supervision layer that assigns the work and reports what happened.
Here is the trap: an AI agent pilot can satisfy the first condition and none of the other three and still be written up as a success. It ran at real volume, a person opened every case, and the reported number was a productivity estimate, not a completed outcome. That is the shape of most stalled work we have looked at.
This article does not define what an AI agent is. It assumes you have seen a demo that worked, and the question now is whether to fund the next stage.
Why do AI agent pilots fail?
AI agent pilots fail for three reasons, and from outside the three look identical: the project went quiet. The three have different fixes, and the fixes do not substitute for each other. Naming which one you are in is the whole job.
| Failure mode | How it is usually described | The symptom that identifies it | What actually fixes it |
|---|---|---|---|
| The undefined process | "The agent was inconsistent" | Two experienced people describe the same process differently | Write the process down: inputs, decision rules, stopping rules |
| The withheld last mile | "It worked, but the savings were not there" | Nobody can name one case the agent finished end to end | Completion authority on a defined subset, with a gate on irreversible actions |
| The review queue | "It got slower as we scaled it" | Quality holds, cycle time gets worse, work in progress piles up | Review a sample rather than every item, tier by damage, measure the queue |
The cost of getting this wrong is a second AI agent pilot that dies in the same place. A team with a definition problem changes platform and discovers four months later that the new agent is inconsistent in exactly the same decision, because the decision was never written down for either of them. Tool changes fix tool problems. They do not fix an unwritten process, a withheld last mile or a queue.
Failure mode one: the process was never actually defined
This is the undefined process. The AI agent pilot ran against one experienced person's version of the process, and that version was never written down. It lives in a head, it has exceptions nobody has enumerated, and it changed twice last year.
The symptom. Two experienced people describe the same process differently and both are right. Or a written version exists, and at the decision point that matters it says "depends on the case", and nobody can open that phrase into a rule.
An AI agent surfaces this faster than a new hire does, which is why it feels like an agent problem rather than a process problem. A new person fills the gap with judgement borrowed from a previous job and does not report the gap. An agent either asks or guesses, and both are visible in the record. The gap was always there. The AI agent pilot is the first thing that showed it to you.
The fix comes before any tool decision. Write down the inputs the work needs, the rules at each decision point, and the conditions under which the work stops and goes to a person. Not a policy document. One page, written by the person who does the job and checked by a second person who does it too.
This is not a call for better documentation habits. It is the cheapest diagnosis in this article: an afternoon, one page, and the disagreement it produces is the finding.
Failure mode two: the agent was never allowed to finish the work
This is the withheld last mile. The AI agent pilot was scoped to drafting or suggesting. A person still opens the case, still reads the context, still checks the draft and still performs the final action. The agent writes the middle of the work and a human keeps both ends.
Work the arithmetic. Take a case that costs nine minutes end to end: three minutes to find and read the context, four minutes to write the reply, two minutes for the closing actions, updating the record and sending it. In draft mode the person still pays the three minutes and the two, plus roughly a minute and a half to read and check the draft. That is 6.5 minutes against 9, a saving of 2.5 minutes, a little over a quarter, and only when the draft is right. On the subset where the agent is allowed to finish alone, the saving is the whole nine minutes.
The symptom. The savings in the pilot report are per item and small, and nobody can name a single case the agent took from arrival to completion.
The fix is not more autonomy everywhere. It is completion authority on a defined subset of cases, with a gate on the actions you cannot take back. In practice that means writing the subset down before the pilot starts and granting permissions against it, the way you would scope what any role is allowed to do rather than handing over everything or nothing.
Draft mode is a reasonable first week. It is not a reasonable sixth month.
Failure mode three: the review step became the queue
This is the review queue. Every item the agent produces goes to a person for review, and the review is quick, so nobody costs it. Two minutes an item at ten items a day is twenty minutes, which is nothing. At 240 items a day it is 480 minutes, which is eight hours, a full working day in which one person does nothing else. At 400 items a day it is more than thirteen hours, which no reviewer has.
Here is the part that surprises people. A reviewer at two minutes an item has a capacity of 30 items an hour. At 24 an hour arriving, which is 80 percent of capacity, the queue stays short and the AI agent pilot looks fine. At 28 an hour, which is 93 percent, the reviewer is still faster than the work on paper, but the wait per item multiplies, because work does not arrive at even intervals.
The relationship underneath this has a name and a date. John D. C. Little proved the link between the average number of items waiting, the rate at which they arrive and the average time each one spends in the system, in A Proof for the Queuing Formula: L = λW, Operations Research, volume 9, issue 3, pages 383 to 387, 1961.
The symptom. Quality metrics look good, cycle time gets worse, work in progress climbs, and reviewers start batching at the end of the day.
The fix. Review a sample rather than every item. Tier the review by what a wrong item can cost, so expensive cases keep a full check and cheap ones get sampled. Measure the queue, not the accuracy.
One secondary effect deserves its own line. Ceremonial review is worse than no review, because it produces the record of an inspection that did not happen.
How do you check whether a published AI failure rate is real?
There is no published failure rate specific to AI agent projects that comes with a traceable primary source and a stated method. The figures in circulation either count a much broader category of work or have no source at all. This article does not reprint them.
Here is the four-step trace. It takes ten minutes and works on any number somebody puts in front of you.
- Find the study, not the write-up. Follow the number back until you are holding the document that produced it. If the trail ends at an article citing another article, the number has no source.
- Read what was counted. A study of generative AI adoption across all uses is not a study of agent deployments.
- Check the date and the window. A figure gathered over half of one year, published as a preliminary finding, is a snapshot of that window and nothing else.
- Check what "failure" means. Cancelled, stalled, running but showing no measurable financial return, and never measured at all are four different findings, routinely reported as one.
The fourth step is where most of these numbers come apart. "No measurable return" and "the AI agent pilot did not work" are not the same claim, and the gap between them is often the whole story.
A number without a source is worse than no number, because it ends the conversation it should have started. If you use an analyst estimate, say the firm's name, the date and the word "estimate" in the same sentence.
Why do draft-only pilots lose to a baseline they were never allowed to beat?
Because the AI agent pilot was allowed to do part of the work and was then measured against the whole of it. The comparison was never fair, and the shortfall gets recorded as a capability problem.
Name the reasoning error in one line: you compared a partial system against a complete one and logged the difference as a limitation of the partial system.
The alternative is decided before the AI agent pilot starts, not after. Agree which actions the agent may complete on its own and which ones stop for approval, write both lists down, and measure the pilot only against that scope. A pilot scoped to drafting should be reported as a drafting result, with the human time it did not remove stated in the same table.
The usual objection to granting completion authority is that the agent will do something irreversible. That is a design question, not a reason to stay in draft mode for six months. Which actions stop for a person, at what threshold, and who holds the approval is a decision you make once and write down: that decision belongs in its own document, before the pilot.
The Afternoon Test: 10 questions to run on one process before you fund it
The Afternoon Test is ten questions you can answer about one process in a single afternoon, before you commit budget to an AI agent pilot on it. It is not a scoring model. It is an elimination.
Pick the candidate carefully; the choice does most of the work. Choose the boring, high-frequency process where a wrong answer is cheap to catch. The impressive one usually fails on verification, because what makes it impressive is that judging it correctly is hard.
Two traps are worth naming. The first is the process the general manager complains about, often complained about precisely because it is ambiguous. The second is the process with no owner, available exactly because nobody will defend it or extend permissions for it.
| # | Group | Question | What counts as a yes |
|---|---|---|---|
| 1 | Definition | Can one person write this process down end to end, on one page? | The page exists, and it was written before this meeting |
| 2 | Definition | Does every decision point name its condition and its outcome? | No branch resolves to "depends"; where judgement is needed, the page says whose |
| 3 | Inputs | Does the agent get every input the human uses, in readable form? | Systems and fields are listed, including the second screen somebody checks by habit |
| 4 | Inputs | Are the inputs there when the work starts, not gathered afterwards? | One real case walked through, showing where each input came from and when |
| 5 | Authority | Is there a written list of actions the agent may complete alone? | The list names actions, not intentions |
| 6 | Authority | Has someone who can grant those permissions agreed to the list? | A named person, not a role, not a committee, not the vendor |
| 7 | Verification | Can a wrong output be spotted faster than it can be produced? | You can state the check and its duration; longer than the work is a no |
| 8 | Verification | Can a wrong action be undone, by whom, within what window? | Every action on the authority list has an undo path or a gate |
| 9 | Volume | Does the process run often enough that one week shows the exceptions? | A count of cases per week pulled from the system, not an estimate |
| 10 | Ownership | Is there one named person who owns this process today? | You can say the name, and that person knows they own it |
The scoring rule is not a score. Any no in the definition group or the authority group is a stop, not a deduction. A process that fails question 1 or question 6 is the wrong candidate, and no amount of yes elsewhere changes that.
It takes an afternoon because it needs three people and no preparation: the person who does the work today, the person accountable for the result, and the person who can grant permissions. If you cannot get those three into a room for two hours, you have already learned something about question 10.
The Afternoon Test tells you whether a process is eligible, not how it ranks against other candidates, which turns on how fast an output can be verified and how cheaply a wrong one can be reversed.
What disqualifies a process outright? The five stop conditions
Whatever the rest of the Afternoon Test returns, five conditions mean you do not start the AI agent pilot at all. These are the five stop conditions, each independently fatal.
- Nobody can write the process down. Not "it would take a while". Nobody can produce the page.
- The output cannot be checked faster than it can be produced. If verifying an item costs more than doing it, review becomes the bottleneck as soon as volume is real.
- A wrong action cannot be undone and cannot be gated. If it is irreversible and there is no approval step you will keep, the process is not a candidate.
- The systems in the path have not been proven to execute one real call. Not documented. Executed, by a person, against the live system, with the result shown to you.
- There is no named owner who can widen the agent's authority. Without that person the pilot succeeds and then stops, because nobody turns a demonstration into permissions. That person belongs in the agent's written scope of work.
Elimination beats weighted scoring for one reason: a weighted score lets a fatal condition disappear into the average of four comfortable ones. A process that cannot be verified but scores well elsewhere comes out as a promising candidate, which it is not.
The fourth condition is the one people wave through. Treat any vendor integration list as a claim rather than as evidence, and that includes ours. When we audited our own capability catalogue we found that most of the listed entries had no running service behind them, and our own discovery tool was still recommending them to agents. We shipped a gate that hides anything without a live endpoint. Before you fund a pilot, make somebody execute one real call against every system in the path and show you the result.
How should you measure an AI agent pilot so the number means something?
Measure completed outcomes, queue length and cost per outcome. Model accuracy on sampled cases is the easiest number to collect and the least useful one you will report, because it describes the part of the work that was never the constraint.
Four measurements, each defined in one line:
- Completion rate without human touch. The share of cases the agent finished alone, end to end.
- Peak review queue length. The largest backlog waiting for a person, measured on the busiest day, not the average.
- Cost of a wrong output. What one incorrect result costs including the correction, not just the compute.
- Cost per completed outcome. Total cost divided by outcomes actually finished, not by cases attempted.
Set the baseline before the AI agent pilot starts, at the same volume. A baseline built afterwards is not a measurement, it is an argument, and everyone knows which way it will point.
On the fourth measurement: this is why recurring jobs in our own system carry a per-job cost meter rather than one account-level total. A pilot needs a cost per completed outcome, and an account total cannot produce one.
"The run did not error" is not a measurement.
A good pilot report is short. Four numbers, a before, an after, and the volume both were taken at.
What did the AI agent pilots that reached production do differently?
They narrowed the process, widened the authority, and named one owner. Each of those is an observable behaviour rather than a virtue.
Narrowing looks like cutting the case types in half and taking the two that arrive most often. Widening looks like a written list of actions the agent may finish alone, agreed by someone who can grant the permissions. Naming an owner looks like a person whose name is on the scope document, who can change it without convening anybody.
The fourth is less obvious: they ran the AI agent pilot at the volume that breaks things rather than on curated cases, and did it early enough that breaking was survivable. A pilot that only ever saw clean examples has not been tested, it has been rehearsed.
One honesty note, and it belongs here rather than in a footnote. This is a pattern drawn from accounts we could trace and from running our own systems, not a controlled study. Read it as a hypothesis to test against your own process, not as a finding.
What should you do if your AI agent pilot has already stalled?
Diagnose which of the three failure modes you are in before you change anything, because the three fixes are mutually exclusive uses of the same budget. Changing tools while the problem is a definition problem buys you a second stalled pilot.
Walk the decision path in order:
- Is the process written down? If not, stop and write it. You are in the undefined process, and no platform will resolve it.
- If it is written down, was the agent allowed to finish anything? If not, that is the fix, and it is a permissions decision rather than a technology one. You are in the withheld last mile.
- If it was allowed to finish work, did the review queue move? If the backlog grew as volume rose, that is the fix. You are in the review queue, and the answer is sampling and tiering, not a faster reviewer.
There is a fourth answer, and sometimes it is right: stop. If the process hits one of the five stop conditions, cancelling is correct, and at this point in the spend it is cheap. The expensive version of this decision is the one made after the second AI agent pilot.
An AI agent pilot that produces a clean elimination has produced a result. It told you which process not to fund, which is the same category of information as knowing which one to fund.
If you cannot answer the definition questions and the authority questions for the process you have in mind, that is your result, and it is the cheapest one this decision can produce. Write the process down, decide what the agent is allowed to finish, and run the Afternoon Test again next week.
If you would rather assess the whole operation than a single process, the AIQ assessment returns a readiness score, an archetype and a breakdown by dimension.
Frequently asked questions
How long should an AI agent pilot run before you decide?
Long enough to reach the volume that breaks things, not a fixed number of weeks. Run the AI agent pilot through one full cycle of the work, including the busy period and the awkward cases people normally route around. A pilot that only saw curated examples has told you nothing.
Who should own an AI agent pilot inside the company?
One named person who owns the process today, not a committee and not the vendor. They set the scope of authority, sign off on what the agent may finish alone, and answer for the result. AI agent pilots without a single accountable owner stall quietly, because nobody can widen the agent's permissions.
Should I use an AI agent or workflow automation for this process?
Ask one question: does any step require reading unstructured input or judging an ambiguous case? If not, rule-based workflow automation is cheaper and more predictable. If yes, an AI agent belongs on that step and rules belong on the rest. Most real processes need both, not one.
Can we run an AI agent pilot without connecting it to production systems?
You can, and the result will not transfer. A pilot on exported data proves the agent can reason about the work, not that it can do the work. The failures that stop rollouts live in the connections: permissions, rate limits, stale records. Prove one real call against every system first.
Our AI agent pilot succeeded and nobody will approve the rollout. Why?
Usually because the pilot proved capability and the rollout requires authority, and nobody in the room can grant it. Find the person who can widen the agent's permissions and give them what they need: the list of actions the agent would finish alone, and what happens when one is wrong.
