A client showed us a prototype last spring that drafted clinical notes from consultation transcripts. In the demo it was extraordinary. In the clinic, roughly two in five drafts were rejected.
Nobody could say why. More precisely, nobody could say whether last week's prompt change had made things better or worse, because there was no way to measure it. Six engineers had been iterating for four months on vibes.
This is the normal condition of enterprise AI work, and it is the reason so much of it stalls between prototype and production.
The gap nobody budgets for
A demo answers one question: can this system produce a good output? Deployment asks a different one: can this system be relied upon not to produce a bad one?
Those questions need different evidence. The first needs an example. The second needs a distribution — and something that tells you, before you ship, where in that distribution the failures live and how severe they are.
Almost nobody budgets for the second. The proposal says "eight weeks to a working assistant" and the eight weeks are spent on retrieval, prompting and interface, because those produce visible progress. Evaluation gets scheduled for the end, where it becomes the thing standing between the team and the launch date, which is the worst possible position for it.
Start with the scorecard
On that clinical project we spent the first eleven weeks not shipping features.
We built a dataset of 2,400 real consultations with clinician-authored reference notes. Then — the part that actually mattered — we sat with three consultants and built a weighted scorer. A wrong medication counted forty times a clumsy sentence. A missing allergy counted more than either.
That weighting is the whole game. A generic accuracy score would have told us the prototype was 88% correct, which sounds deployable. The weighted score said 61%, because it noticed that most of the errors were clustered in the one category that could hurt someone.
What changes once you can measure
Three things, immediately.
Arguments end. "This prompt feels better" becomes a number that either moved or did not. We rejected six candidate changes during the build that the team was confident about, and shipped two that nobody expected to help.
Regression becomes visible. The scorecard went into CI as a release gate. Nothing ships that does not beat the current best. Twice this caught a retrieval change that improved average quality while making the worst cases considerably worse — exactly the trade you must never accept in a clinical system.
Risk conversations get short. The client's risk function had rejected two previous AI proposals. They approved this one in a single meeting, because we handed them a scorecard rather than a demo. They did not need to trust our judgement; they could inspect our evidence.
The uncomfortable part
Eleven weeks of no visible progress is a hard thing to sell, and a harder thing to sit through. The client's product lead asked twice whether we were stalling.
The honest answer is that this cost real time and would have been wasted on a project that was never going to production. If you are building a demo to raise money or win a mandate, skip all of this — you need something impressive on a screen, not a distribution.
But the moment a system will make a decision that affects somebody, the measuring stick is the product. Everything else is the thing you point at it.
Where to start
If you have a stalled AI project, the intervention is usually not a better model:
- Write down what a bad output costs you, by category. Not all errors are equal, and your scorer has to know that.
- Collect 200 real inputs with expert-authored reference outputs. Two hundred is enough to be useful; you can grow it later.
- Score your current system. The number will be worse than you expect.
- Put the score in CI. Nothing merges that lowers it.
Step three is where most teams stop, because the number is genuinely dispiriting. It is also the first honest information the project has produced, which makes it the most valuable thing on the roadmap.