Written by
John George K
Technical Business Analyst
You have probably seen the statistic. Some enormous proportion of AI projects never make it into production - the figure varies from survey to survey, and depending on who is counting and what they count as failure, it lands anywhere from around half to over eighty percent.
The precise number matters less than the shape, which is remarkably consistent: the demo works, everyone is impressed, and then the thing does not ship. Worth being straightforward about the evidence here - these surveys use different definitions and different samples, so treat any specific percentage as directional rather than precise.
What is far more useful than the statistic is the pattern underneath it. Across the pilots we have run, inherited, and been asked to rescue, the same five causes appear. Only one of them is technical.
1. No Falsifiable Question
"Can AI help with customer support?" cannot fail. It also cannot succeed, because there is no threshold to cross. A pilot built on that question produces something interesting, everyone agrees it is interesting, and nobody can say whether it justifies a production build.
A testable question names four things: the task, the data it will be tested on, the pass threshold, and the comparison. Something like: using our last six months of inbound tickets, can a system classify and route at least 85% correctly - matching our current human routing accuracy - at under a defined cost per ticket?
That version can be answered in weeks, and the answer changes a decision either way.
2. Data That Flattered the Demo
This is the most common technical cause, and it is entirely self-inflicted.
Pilots get run on clean, curated, representative-looking data because that data is easy to obtain and produces good results. Production then delivers the real thing: incomplete records, inconsistent formats, documents scanned at an angle, edge cases nobody documented, and the twenty percent of volume that follows a rule no one wrote down.
The gap between those two datasets is where most pilots quietly die. A system that only performs on the good cases has measured nothing, because the whole question is how it behaves on the ordinary ones.
Synthetic and cherry-picked data flatters every AI idea. Your messy, incomplete, real data is where feasibility is actually decided.
Where privacy or governance makes real data difficult - and it often does - the answer is anonymisation that preserves the mess, representative sampling, or processing inside your own environment. It is not cleaning the mess out, which defeats the purpose entirely.
3. No Owner for What Came Next
A pilot typically has a sponsor and a technical lead. Production needs something else: a business owner accountable for adoption, a budget line, a place on someone's roadmap, and an operational home.
We have watched genuinely successful pilots - measured, validated, clearly worth building - stall for six months because nobody could say whose team would run the thing. The technology was never the constraint.
The fix is unglamorous and works: before the pilot starts, name the person who will own the production system, and make sure they are in the room when success criteria are agreed. If nobody will accept that name, you have learned something important before spending anything.
4. No Way to Detect Silent Failure
Pilots are evaluated once, by humans, informally. Production systems change behavior constantly - a prompt is adjusted, a model version updates, the retrieved content shifts - and any of those can degrade quality without producing a single error in the logs.
Teams that have not built an evaluation suite discover problems the way their customers do. And because the failure is silent rather than loud, trust erodes before anyone knows there is an issue to fix.
What production actually requires:
- A golden test set built from real cases, including known past failures
- Automated regression runs on every change to prompts, retrieval, or model version
- Quality gates that block deployment when results regress beyond a threshold
- Sampling of live outputs with periodic human review
- Monitoring on inputs, because usage patterns drift even when your system does not
An AI system without an evaluation suite is not finished. It is unmeasured.
5. The Prototype Became the Product
A prototype optimizes for learning speed. It deliberately skips scale hardening, error handling, security posture, permissions, logging, and the rest of the work that separates a working demo from something you can put in front of customers.
Under pressure to show progress, teams try to harden the prototype instead of rebuilding on what it taught them. The four-week experiment becomes a four-month project that inherits every shortcut taken when nobody expected the code to survive.
Keep the boundary explicit. The prototype's real output is validated knowledge - that is what carries into production, and it makes the production build faster and considerably less risky.
Designing a pilot that ships comes down to six things, agreed before any work begins.
- A falsifiable question with a numeric threshold
- Real data, including the messy portion, with the access path resolved before the clock starts
- A named production owner, present when criteria are set
- A time box - two to six weeks for most feasibility questions
- Defined kill criteria, so stopping is a planned outcome rather than an admission
- An evaluation plan that will carry into production if the pilot passes
None of this is technically demanding. All of it is organizationally awkward, which is precisely why it gets skipped - and why the failure rate stays where it is.
A fast, cheap "no" is a genuinely valuable outcome. It is budget rescued for an idea that will work, arriving in weeks rather than after a year of sunk commitment. Teams that treat a negative result as a success are the ones that eventually ship something.
Questions We Get Asked
Two to six weeks for a well-scoped feasibility question. Longer usually means the question was too broad, or that data access was not resolved before the work started.
A small fraction of it. That is the entire economic argument - spending a few percent of a potential project's cost to establish whether the rest is worth spending is the cheapest decision-quality upgrade available to an AI budget.
Usually the success criteria were technical rather than commercial. A result showing 90% accuracy persuades engineers; a result showing hours saved per week, mapped to a cost line, persuades whoever controls the budget. Agree both at the start.
Rarely. Portfolios of simultaneous pilots dilute attention and produce a set of half-validated ideas with no owner. Sequence them, and let each one's result inform what you test next.
