Key takeaway: A pilot proves feasibility on curated inputs. Production requires everything the pilot deliberately excluded, and that remainder is usually the larger share of the work.
Why the Demo Was Easy
A pilot runs on selected examples, with the builder present to interpret results, and no requirement to handle failure. It answers one question: can the model do this at all?
That is a genuine and necessary answer. It is also a small fraction of what shipping requires.
What the Pilot Left Out
Inputs the pilot never saw. Real data includes empty fields, wrong formats, unexpected languages, corrupted files and adversarial content. Every one needs defined handling, and each represents a decision nobody has made.
Failure behaviour. What happens when the provider is unavailable, the response fails validation, or latency exceeds the timeout. A pilot retries by hand; production needs a policy.
Integration. The output has to arrive where work actually happens — the ticketing system, the CRM, the queue — with authentication, permissions and audit logging. This is frequently the longest single stretch of work and is invisible in a demo.
Support. When a user reports a wrong answer, someone must be able to investigate. That requires logged inputs, model version, retrieved context and reasoning — instrumentation nobody built.
Cost at real volume. Ten thousand daily requests cost differently from the fifty in the pilot, and the pilot’s prompt was not optimised because it did not need to be.
Workflow change. The largest factor and the least technical. If the output does not fit how people already work, they will not use it regardless of quality.
Structuring the Gap Honestly
| Stage | Question answered | Typical share of effort |
|---|---|---|
| Pilot | Is it feasible? | 10–20% |
| Hardening | Does it survive real inputs? | 25–35% |
| Integration | Does it reach the workflow? | 25–35% |
| Operations | Can we support and monitor it? | 15–25% |
Presenting a pilot as near-complete sets an expectation the remaining work cannot meet, which is how genuinely promising projects lose sponsorship at the point where they need it most.
Choosing What to Pilot
The strongest predictor of shipping is not model performance. It is whether the task has a clear owner, a measurable current cost, tolerance for imperfect output, and a workflow with an obvious insertion point.
A pilot on a task that scores well on those four ships. A brilliant pilot on a task without an owner does not, no matter how good the numbers are.
Tolerance for error deserves emphasis. Tasks where a wrong answer is cheap to detect and correct absorb current model limitations. Tasks requiring near-perfect accuracy need review capacity that must be budgeted from the start rather than discovered later.
The Bottom Line
Scope pilots to answer feasibility, then plan explicitly for hardening, integration and operations as the majority of the work. Select use cases with a named owner, a measured baseline cost and tolerance for imperfect output — those are the ones that reach production.
