Strategy
Why your AI pilot stalled
AI pilots rarely fail because the model was not good enough. They fail because nobody defined what working meant, the wrong answers were invisible, the workflow chosen was not the expensive one, or the thing was never wired into how people actually work.
There is a specific pattern we hear about constantly. A team ran an AI pilot last year. The demo was genuinely impressive. Everyone was enthusiastic. Then it went quiet, and now nobody wants to bring it up in a meeting.
It almost never failed because the model was not clever enough. Here are the four reasons that actually account for it.
1. Nobody defined what “working” meant
This is the big one.
The pilot started with “let’s see what AI can do with our data” and ended with a demo. Somebody asked “is it good?” and the honest answer was “it seems good?” There was no test set, no baseline, no threshold agreed in advance. So the decision to continue became a matter of vibes, and vibes do not survive a budget review.
What to do instead: before building the feature, write down twenty real cases from your business with the correct answer for each. That is your evaluation set. Now “is it working” has an answer, and so does “did that change make it better.”
Twenty cases takes an afternoon. Skipping it is what turns a technical question into a political one.
2. The wrong answers were invisible
A pilot that is right 90% of the time looks fantastic in a demo, because you demo the good cases.
In production, the 10% is the whole story. And the dangerous version is not the model refusing to answer. It is the model answering confidently and incorrectly, in a way that looks exactly like the correct output.
If nothing surfaced those cases, the system quietly poisoned a dataset or gave a customer wrong information for weeks, and trust collapsed the moment somebody found out.
What to do instead: design the confidence boundary before the feature. Above it, the system acts. Below it, the work goes to a human review queue. That queue is not a failure mode, it is the product. On a document-ingestion system we built, low-confidence rows are flagged for approval rather than guessed at, and nothing commits without a person agreeing.
Silent wrong answers are the failure mode that kills AI projects. Design them out first.
3. You automated the interesting workflow, not the expensive one
Pilots gravitate toward whatever is most fun to demo. That is rarely where the money is.
The expensive workflow in most businesses is boring: someone retyping values between two systems, someone reformatting a spreadsheet every Monday, someone manually checking a list for the one item that has gone wrong. It photographs badly. It costs a fortune.
What to do instead: count. How many minutes a day, times how many people, times how many days. The workflow with the highest number is the one to build, even if it is dull. Especially if it is dull.
4. It was never wired into how people actually work
The pilot lived in a separate tool. To use it, someone had to remember it existed, open it, paste something in, and copy the result back out.
People did that during the pilot because they were being watched. Then they stopped.
What to do instead: the AI goes into the system people already have open. If your team lives in a particular admin panel, that is where the output belongs. If your customers live on WhatsApp, an app they have to install is a tax you are choosing to charge them. This is the whole reason we talk about integration rather than AI products.
The pattern underneath all four
Each of these is a decision that had to be made before any code was written, and was not made.
That is why we run the sequence we do. One narrow workflow, chosen because it is expensive rather than interesting. An evaluation set before the feature. A confidence boundary with a human review path. And it ships into the tool your team already uses.
It is less exciting than a transformation programme. It also survives contact with a real business.
If you already have a stalled pilot
It is often recoverable, and usually cheaper than starting again. The questions worth asking:
- Was the underlying task actually valuable, or was it just demoable?
- Did it fail on accuracy, or on adoption? These have completely different fixes.
- Is there a review step that would make an imperfect model useful?
- Is the output landing where people work?
If it failed on adoption rather than accuracy, you may have a working system and a distribution problem, which is a much better place to be than you think.
We will look at a stalled pilot and tell you honestly whether it is worth reviving. If the answer is no, we will say so. And if you would rather see a new one work before paying for it, we build one at our cost and you run it free for three months.