Audra Eval: How we hold our own AI work accountable

Building a quality engine for AI that we can rely on

Anyone can build an AI assistant. The harder question — the one that matters in a regulated business — is how you prove it’s right. Not in a demo. In production. Under scrutiny. When a compliance officer, a regulator, or a patient is relying on the answer.

That’s a question we kept running into on client engagements. There’s good tooling out there — LangSmith, RAGAS, DeepEval and others all solve real parts of this problem. But stitched together, none of them gave us the full picture we needed for regulated clients: enforceable gates that block a bad deployment automatically, a workflow domain experts could use without needing to be engineers, and an audit trail that would satisfy a compliance review. So we built a platform that wires those ideas together for the way we work. We called it Audra Eval — and it’s now the quality backbone of every AI engagement we run. It isn’t something we sell. It’s how we work.

Catching the drift

The AI industry has made it easy to build intelligent assistants. What it hasn’t done is make it easy to prove they work, that the responses are consistently aligned with official, approved source materials. We see the same pattern in regulated industries everywhere: a RAG-based AI goes live, manual testing suggests it performs well initially, then quietly drifts as model updates, data refreshes, and prompt changes degrade quality. Nobody notices until a user flags a wrong answer – or worse, acts on one. In traditional software, CI/CD and automated test suites solved this long ago. For AI systems, the tooling has matured rapidly — what’s far less settled is how that tooling gets wired into a workflow that satisfies a compliance function: enforceable gates, expert review, and an audit trail a regulator would accept.

And the stakes are very real. In aviation, a wrong answer can ground flights. In healthcare, a misquoted guideline risks patient harm. In financial services, a misinterpreted regulation means compliance violations. These aren’t hypothetical scenarios. They’re the reality compliance teams face every day when asked to sign off on AI-generated answers.

Working with our own quality platform

Audra Eval is an AI evaluation and quality assurance platform which we have built to support all our client AI engagements, and which has evolved over several real-life projects. Think of it as CI/CD for AI accuracy. It’s a systematic way to measure, validate and certify the quality of AI-generated answers before they reach users, and to keep measuring.

It sits alongside an AI product as a companion layer. When we are building for our clients we configure it with their knowledge bases, retrieval pipelines, and quality targets. The platform then runs automated benchmarks against expert-curated datasets, scores every response against several independent quality signals, enforces quality gates that block deployment when thresholds aren’t met, and routes failures and edge cases to human reviewers for expert validation.

Because we own it outright, it’s never standing still. Every engagement surfaces new edge cases, new failure modes, new things worth measuring — and because we’re not waiting on a vendor’s roadmap, we can fold that learning straight back into the platform. The version of Audra Eval we use today is sharper than the one we used three months ago, for exactly that reason, and it’ll keep being true.

The core philosophy is simple: if you can’t measure it, you can’t trust it. And if you can’t trust it, you shouldn’t ship it.

How it works: A classic five-step cycle

Audra Eval is built around a continuous improvement loop with five stages that mirror best practices from software engineering, adapted for the realities of non-deterministic AI systems.

Configure

We set up the knowledge bases, choose from multiple domain-specific retrieval pipeline strategies, define system prompts and establish the quality targets. While Windmill is building the AI solution with you, we use the Audra Eval platform as a testing ground, querying AI using different pipeline configurations, model choices and prompt strategies, with every result logged with full metrics. We compare the same question across multiple retrieval strategies side by side, inspect debug traces that show exactly where time is spent, and see citation sources at the chunk level.

Evaluate

The platform runs automated benchmarks against two tiers of evaluation data. The first is a baseline multiple-choice harness: hundreds of questions that validate foundational knowledge quickly and deterministically. The second is a golden dataset of domain-specific, expert-curated question-and-answer pairs, evaluated by an LLM-as-Judge that scores semantic equivalence rather than exact string matching. Every evaluation produces detailed per-question scoring with full reasoning, not just a pass or fail. We run systematic benchmarks, either quick random samples or targeted runs focused on problem areas, and the platform tracks regressions and improvements across runs so you can see at a glance whether changes are making things better or worse.

Gate

This is where Audra Eval really differentiates itself. Configurable quality gates enforce measurable thresholds: answer accuracy of 95% or above, citation accuracy of 98% or above, hallucination rates measured and tracked against domain-specific benchmarks – with deployment blocked when they exceed defined tolerances. If a model update or pipeline change causes quality to drop below these thresholds, deployment is blocked automatically. The gates are configurable per client, per pipeline and per environment.

Review

Failed or borderline results are automatically queued for subject-matter expert review. Reviewers see the full picture: the question, the AI’s answer, the reference answer, the judge’s reasoning and detailed confidence metrics. They can approve, reject or update the golden dataset creating a continuous feedback loop that makes the evaluation data itself better over time. For safety-critical queries, human experts review responses before they ever reach users.

Deploy and monitor

Once quality gates are met, our clients ship with a complete audit trail. But the process doesn’t stop. The platform continuously monitors production performance, detects regressions and alerts your team when accuracy drops, latency increases or citation quality degrades. Every change – model updates, data refreshes, pipeline modifications – triggers a fresh evaluation cycle. Production dashboards give us and you real-time visibility into query volumes, feedback trends, cache performance and active alerts. You know the health of your AI at all times.

The entire experience is browser-based, with role-based access controls for admins, evaluators and viewers. No specialist tooling, no command-line expertise required. Your domain experts can participate directly in the review and validation process with us without needing to be engineers.

Our default lens

Multi-dimensional scoring is increasingly standard practice in AI evaluation — and for good reason, since a single confidence number hides which part of the answer you should actually be worried about. Where we differ is in how we combine the dimensions. We call ours the Gamma score internally. Rather than averaging four quality signals into one figure, the way most composite scores do, we take the lowest of the four — so a confident, well-cited answer that fails on just one dimension doesn’t slip through on the strength of the other three.

The four signals are: whether every claim is backed by a source passage, whether the source actually supports the claim it’s cited for, how confident the retrieval step was in the evidence it found, and whether the right rule was applied when sources conflict or one takes precedence over another.

A strong score across all four means we’re confident in the answer. A weak score on any one of them sends the response to a human reviewer instead of out the door. And when none of the four are strong enough, the system doesn’t guess — it says it doesn’t know. In a regulated context, that’s usually the better answer.

Gamma is the default lens we apply everywhere, but it isn’t the only one. Each client project can also define its own custom metrics — domain-specific checks that matter for that engagement specifically, sitting alongside Gamma rather than replacing it. A financial services client might add a metric for whether a numeric figure matches a regulatory filing exactly; a healthcare client might add one for whether a dosage falls within an approved range. Gamma tells us whether we can trust the answer in general. Custom metrics tell us whether it meets the specific bar that engagement requires.

Why this matters across industries

We built Audra Eval while working with clients in demanding, accuracy-critical domains. But the underlying challenge is universal. Any organisation deploying AI to answer questions from authoritative document sets faces the same fundamental problem: how do you prove the AI is right?

Legal and compliance

Legal teams are increasingly using AI to navigate vast regulatory corpora. But without citation traceability, there’s no way to verify that an AI-generated interpretation actually reflects what the regulation says. Our citation-first approach means every claim is traceable to a specific source passage. Compliance officers get the audit trail they need and legal teams can verify any answer in seconds rather than hours.

Healthcare and pharma

Clinical documentation, drug interaction databases, treatment protocols: the stakes couldn’t be higher. A hallucinated dosage or a misquoted guideline could have serious patient safety implications. The dual human-in-the-loop strategy (background review for borderline cases and safety escalation for high-risk queries) ensures that expert oversight is applied exactly where it matters most.

Financial services

Financial institutions operate under intense regulatory scrutiny. When an AI assistant interprets a capital requirement or a sanctions rule, the margin for error is zero. The immutable audit trail we built into the tool with configurable retention periods gives compliance teams the evidence they need for regulatory examinations and internal audits. The regression detection we run catches quality degradation before it reaches clients.

Data centres and cloud infrastructure

Data centre teams navigate power distribution protocols, cooling specifications, SLA frameworks, and evolving sustainability mandates across multiple jurisdictions. A wrong answer about a failover procedure or uptime requirement can mean SLA breaches and millions in lost revenue. Audra Eval’s regression detection ensures that when standards are updated, answer quality is re-validated automatically and not left to degrade until a critical incident exposes the error.

Insurance, energy, and manufacturing

Whether it’s underwriting guidelines, safety procedures for energy infrastructure, or quality standards in manufacturing, the pattern is the same: complex document sets, high stakes, and a need for provable accuracy. Audra Eval is domain-agnostic by design. You bring your content, your expertise, and your quality targets, the platform handles the measurement, gating, and governance.

Accuracy you can evidence

Across our client deployments, Audra Eval has delivered answer accuracy above 95% with full citation traceability, citation accuracy above 98%. Results vary by domain and pipeline configuration, but the underlying pattern is consistent: measurable, provable accuracy in production.

But the numbers only tell part of the story. The real value is what they enable: the confidence to deploy AI in contexts where getting it wrong isn’t an option. The ability to tell your board, your regulator or your clients that your AI system is continuously measured, gated and governed, with a complete audit trail to prove it.

Windmill helped us with something we’ve never had before: proof that our AI is right. We went from manually spot-checking answers to having continuous, automated quality assurance with full audit trails. It’s transformed how our team thinks about deploying AI. We now ship with confidence, not hope.

— Head of Digital Transformation, global logistics organisation

We built Audra Eval because we needed it — and because we think the firms and teams we work with deserve better than hope as a quality strategy. If you’re working on an AI project in a regulated or accuracy-critical domain and want to work with a team that takes evaluation this seriously, we’d love to talk about what that looks like in practice. Get in touch at hello@windmill.digital, or visit windmill.digital to learn more.