What "good enough to ship" means when the output is clinical

Most AI evaluation writing is about models. The useful question in health is about gates: what specifically has to be true before a release reaches a patient-facing system, and who decided.

We run four kinds of check before anything ships. Answer accuracy against a held-out set. Citation accuracy, meaning every claim traces to a source document that actually says it. Hallucination rate, measured rather than assumed. Latency under stress, because a system that is correct and slow gets worked around, and a worked-around system is an ungoverned one.

Any of them failing blocks the deploy. That sounds obvious written down. In practice it is the part teams skip, because the gate is the thing that makes you late.

The reason we hold it in health specifically is that the failure is asymmetric. A wrong recommendation in a wealth product costs money and can be reversed. A wrong citation in a clinical protocol enters a record. The cost of being wrong is not symmetrical with the benefit of being fast, so the gate does not move.

What this does not do is make a clinical judgement. It checks that the system did what it said, and that a human can see how. The judgement stays with the clinician, which is the only place it can sit.