Table of Contents
- Pilot Purgatory
- The Demo Is the Easy 20%
- Why Accuracy Is Not the Blocker
- Building an ROI Case That Holds Up
- The Costs Nobody Budgets
- Choosing Problems That Actually Ship
- Workflow Redesign Beats Model Quality
- Measuring What Matters
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: Pilots die between demonstration and deployment because the demo proves technical feasibility while deployment requires integration, workflow ownership, and a business case — none of which the pilot tested.
Pilot Purgatory
The pattern is consistent enough to be predictable. An organisation runs an AI pilot. It performs well. Leadership is impressed. Then six months pass and nothing has shipped, while a second pilot begins elsewhere in the business.
This is not caused by the technology underperforming. It is caused by a pilot answering a question that was never the hard one.
A pilot asks: can a model do this task at acceptable quality? That question is now frequently answerable yes, quickly, with modest effort.
Deployment asks a different set of questions entirely. Who owns this system when it breaks at 2am? How does it connect to the record system that holds the data? What happens when it is wrong, and who is accountable? Which existing process changes, and who tells the affected team? What is the recurring cost, and whose budget carries it? What is the compliance position? None of these were tested by the pilot, and all of them can stop deployment independently.
The organisations that ship AI successfully are not those with better models. They are those that treat the pilot as a small part of a larger problem rather than as the main event.
The Demo Is the Easy 20%
A rough allocation of total effort in a successful deployment:
| Phase | Share of effort | What it involves |
|---|---|---|
| Prototype / demo | 10–20% | Model, prompt, evaluation on sample data |
| Data integration | 20–30% | Access, permissions, quality, pipelines |
| Workflow integration | 20–30% | Where it fits, who uses it, what changes |
| Reliability engineering | 10–20% | Monitoring, fallbacks, error handling |
| Change management | 10–20% | Training, adoption, incentives, trust |
| Ongoing operation | Continuous | Drift, cost, support, iteration |
Pilots typically address only the first row and are frequently evaluated as if they addressed all of them. The consequence is systematic underestimation — a project that looked two weeks from completion turns out to be six months, and momentum dies during the gap.
Two rows deserve particular attention because they are consistently underestimated.
Data integration is harder in real organisations than technically necessary, because the barriers are organisational. The data exists in a system owned by another team, with an access process, a data steward who must approve, quality problems nobody documented, and a schema that changed twice without notice. None of this is difficult engineering. All of it takes weeks of coordination.
Change management determines whether a deployed system is actually used. A tool that works and that nobody adopts has the same business value as one that does not work. This is where the most sophisticated technical projects fail most quietly.
Why Accuracy Is Not the Blocker
Teams stuck in pilot purgatory frequently respond by improving the model. This is usually addressing the wrong constraint.
The relevant comparison is not the model against perfection — it is the model against the current process, which is also imperfect. Human accuracy on document classification, ticket triage, or quality inspection is rarely as high as people assume, and it is inconsistent across individuals and across time of day. A system achieving 85 percent where the human baseline is 82 percent is an improvement, even though 85 percent sounds insufficient in isolation.
More importantly, the question is rarely whether the system is right. It is what happens when it is wrong.
A system with 85 percent accuracy and a well-designed fallback — flagging low-confidence cases for human review — can be more valuable than one with 95 percent accuracy and no fallback, because the first one’s errors are caught and the second one’s reach customers. Confidence-based routing converts an accuracy problem into a workflow problem, and workflow problems are more tractable.
This reframing changes what the pilot should test. Rather than maximising accuracy on a clean sample, test the error handling: what fraction of cases need human review, how long does that review take, and does the combined human-plus-system throughput beat the current process? Those are the numbers that determine whether deployment makes sense.
Building an ROI Case That Holds Up
Most AI business cases fail scrutiny because they claim benefits that never appear in any financial statement.
Weak: “This saves each analyst two hours per week.” Time saved is not money saved unless the freed capacity does something valuable or headcount changes. Otherwise it becomes slack, which may be fine and is not a return.
Stronger: “This reduces the resolution time on tier-one tickets from 14 minutes to 6, allowing the current team to absorb the 30 percent volume growth forecast for next year without the two additional hires currently budgeted.”
The difference is that the second version identifies a specific cost that will not be incurred, tied to a plan that already exists.
Categories that survive scrutiny, in rough order of credibility:
Avoided hiring against forecast growth. Concrete, budgeted, and verifiable.
Revenue from throughput that was previously capacity-constrained. If you were turning away work, processing more of it is directly measurable.
Reduced error cost. Requires knowing the current error rate and what errors cost — rework, refunds, penalties, churn.
Cycle time reduction with a revenue link. Faster quote turnaround improving win rate, for instance, where the link can be measured.
Capacity reallocation, if genuinely tracked. Analysts spending 30 percent less time on classification and measurably more on higher-value work. This is legitimate and requires actual measurement rather than assumption.
The discipline that matters most: measure the baseline before deploying. Teams that skip this cannot demonstrate improvement afterwards, and an undemonstrable improvement gets cut in the next budget cycle regardless of whether it worked.
The Costs Nobody Budgets
Business cases routinely count inference cost and stop there. The full picture:
Inference cost at real volume. Pilot volume is not production volume. Model this at projected scale, and note that per-request cost times high volume can exceed the labour cost it replaces.
Human review capacity. If 15 percent of cases route to a human, that reviewer’s time is a permanent operating cost, not a transitional one.
Monitoring and maintenance. Drift detection, periodic retraining, evaluation set upkeep. Typically a fraction of an engineer’s time indefinitely.
Integration maintenance. Upstream systems change. Every schema change is unplanned work.
Model and prompt iteration. Behaviour needs adjustment as usage patterns emerge. This is continuous, not a project phase.
Compliance and audit. Documentation, fairness evaluation, and record-keeping in regulated contexts.
Support burden. Users will ask why the system did what it did, and someone must be able to answer.
The pattern to internalise is that AI systems have higher ongoing cost relative to build cost than conventional software. Traditional software is largely stable once built. A model degrades against a shifting world and requires continuous attention. Business cases that treat deployment as the end of spending are wrong in a predictable direction.
Choosing Problems That Actually Ship
Problem selection predicts success more strongly than any implementation choice. Characteristics of good candidates:
Errors are recoverable. Someone can catch and correct a mistake before it causes harm. This permits deployment at imperfect accuracy, which is the only accuracy available.
Volume is high and cases are repetitive. Value scales with volume, and repetition means the distribution is learnable.
The data already exists and is accessible. If the required data needs a new collection process, add months.
One team owns the workflow end to end. Cross-departmental workflows multiply the approval and coordination burden.
Success is measurable with existing instrumentation. If you cannot measure the outcome today, you cannot demonstrate improvement.
The current process is a known pain point. Adoption is far easier when users already want the problem solved.
Characteristics of candidates that stall: errors are irreversible or high-stakes; the data is scattered across systems with unclear ownership; the workflow crosses several departments; success is subjective; volume is low; and — importantly — the process works acceptably today. Improving something nobody complains about generates no organisational energy to overcome deployment friction.
Workflow Redesign Beats Model Quality
The highest-leverage decisions in AI deployment concern how the system fits into human work, not how good the model is.
Confidence-based routing. Auto-process high-confidence cases, route uncertain ones to humans. This is the single most valuable pattern available, because it lets an imperfect model deliver value immediately while directing scarce human attention where it matters.
Draft-and-review instead of decide. Having the system produce a draft a human approves is dramatically easier to deploy than having it decide autonomously — lower risk, lower resistance, faster approval. It also generates correction data that improves the system.
Progressive autonomy. Start with the system suggesting and humans deciding. As measured accuracy justifies it, expand the range of cases handled automatically. This builds trust incrementally rather than requiring it upfront.
Explanation alongside output. Users who cannot see why the system produced an answer will not trust it, and untrusted systems get bypassed. Showing the source, the key factors, or the confidence level materially affects adoption.
Visible correction paths. When users can correct errors easily, they tolerate errors better and they generate exactly the data needed to improve the system. This is worth building early.
Measuring What Matters
Instrument three layers, because problems at each layer look different:
System metrics. Latency, error rate, availability, cost per request. Necessary and not sufficient.
Quality metrics. Accuracy on a labelled audit sample, human override rate, escalation rate. Override rate is the most valuable single number available — it directly measures whether users trust the output.
Business metrics. The outcome the project existed to change: cycle time, cost per case, throughput, error cost. This is what determines whether the project continues being funded.
Baseline everything before deployment. This is the most common and most consequential omission. Without a baseline, improvement is unprovable, and unprovable improvements do not survive budget review.
Common Pitfalls
Piloting without a deployment plan. Know who owns it, where it integrates, and what the business case is before proving technical feasibility.
Optimising accuracy instead of designing error handling. Fallbacks convert accuracy problems into workflow problems, which are easier.
Claiming time savings as financial return. Only counts if freed capacity produces value or headcount changes.
No baseline measurement. Makes improvement undemonstrable.
Treating adoption as automatic. A working tool nobody uses returns nothing.
Ignoring ongoing cost. These systems require continuous attention, unlike most software.
Starting with the most ambitious use case. Early wins build the organisational capability and credibility that hard projects require.
Conclusion
AI pilots stall because they answer the easy question. Model feasibility is now rarely the constraint; integration, ownership, workflow redesign, adoption, and a defensible business case are.
Select problems where errors are recoverable, volume is high, data is already accessible, and one team owns the workflow. Design the error path before optimising accuracy — confidence-based routing and draft-and-review make imperfect models useful today. Build an ROI case tied to costs that appear in a budget, and measure the baseline before you change anything.
Most importantly, plan the deployment before running the pilot. A pilot that succeeds into an organisation with no plan for it produces a slide deck. A pilot run inside a deployment plan produces a system.
Frequently Asked Questions
Should we build or buy? Buy where the capability is generic — transcription, translation, document extraction. Build where the value depends on your specific data or process. Building generic capability is a common and expensive misallocation.
How long should a pilot run? Long enough to see real variation, typically four to eight weeks. Shorter misses edge cases; longer usually indicates the pilot has become a substitute for a deployment decision.
What accuracy is good enough? Better than the current process, with an error path that catches the remainder. The human baseline is the relevant comparison, and it is usually lower than people assume.
Who should own an AI system in production? The team that owns the business process, with engineering support. Systems owned solely by a central AI team tend to lose the domain knowledge required to maintain them well.
How do we handle employee concerns about automation? Directly and early. Systems positioned as removing tedious work are adopted; systems perceived as replacing people are resisted, often effectively. Whether the framing is honest matters, because inaccurate reassurance destroys trust permanently.
What is a realistic timeline from pilot to production? Three to nine months for a well-scoped problem in an organisation with existing data infrastructure. Longer where data access requires new agreements or the workflow crosses departments.
Should we hire specialists or train existing staff? Both, in a specific ratio. Domain experts who learn the tooling frequently outperform specialists who lack domain context, because problem selection and workflow design require knowing the business. A small number of specialists supporting many domain-embedded practitioners is a pattern that works.
