The agentic journey to production

The Illusion of Simplicity

In every demo, deploying AI agents looks magical. A chatbot, assistant, or decision-maker springs to life, reasoning and responding like a pro. But when you move beyond a proof-of-concept and aim for production, the reality is very different.

We are very conscious that putting an AI agent into production means building and operating an entire ecosystem, not just launching some code

This ecosystem needs to watch costs, keep the system resilient, and help us and our clients sleep at night.

This blog sets out some of our current thinking and experiences from our real-world deployments—what actually matters when agentic AI hits real users, real systems, and real budgets

Latency Can Creep Up

Every agent we add, every reasoning step we include, adds time. What looks like instant magic in a demo often becomes a response time of several seconds or more in reality—and that can surprise our clients and their users.

In production, expect typical end-to-end agent response times to grow from ~300ms in a demo to 2–5 seconds depending on reasoning depth, chaining, and model locations

For one client, moving from prototype to prod added almost 4× latency. We had to rethink how much reasoning to chain before we hit user frustration thresholds.

The Cost Surprise

AI agents can become expensive very quickly. Every interaction with a model consumes tokens, and each token costs money. Every embedding you store takes up capacity. And when you chain agents together, both the costs and the delays add up fast.

Token consumption for a moderate-sized multi-turn chat agent can easily reach 50,000–100,000 tokens per user per day. That’s ~$10–$20/day/user at GPT-4 Turbo rates, unless there is proactive prompt tuning and caching.

In our early days we burned through hundreds of dollars in one week during tests because we hadn’t yet implemented prompt compression or caching. That was a painful lesson.

We have since built a cockpit that surfaces token and cost metrics daily, not monthly. This helps us track which agents are consuming tokens, how much they’re costing, and when usage patterns start to look unusual.

Mind the Data and Compliance Gaps

When AI agents go into production, they become part of a company’s trusted systems, which we can never forget.

Vector databases, graph databases, and language models all must align with our client’s data residency rules and user expectations. Even if the processing is temporary, users will want to know where their data is going and how it’s protected.

We have had a client demand to know our exact data flow before signing off. We had to produce diagrams and proofs on short notice but now we document fully and get sign-off as part of the intrinsic delivery process.

Evaluation Doesn’t End at Deployment

In traditional software, testing is something you finish before you go live. With agentic AI, that mindset doesn’t really hold.

Because these systems are non-deterministic, prompt-driven, and sensitive to even small changes in data, structure, or model behaviour, we have to treat evaluation as an ongoing activity—not a one-time gate.

It’s not just about accuracy—it’s about understanding and tracking how the agent behaves under pressure, across edge cases, with unexpected user inputs, or after a model version update.

We maintain structured evaluation flows for key use cases, and we aim to monitor prompt outcomes over time. A critical element is to replay known queries periodically to detect regressions and use feedback loops—explicit and implicit—to try to catch performance drifts early.

Basically, we can’t assume a working agent stays working. Evaluation is part of operations now.

Visibility is Protection

As with traditional software deployments, you can’t manage what you can’t see. But with agents it goes a bit further.

It’s not enough to have logs—you need usable, explainable traces for every decision path your agent takes.

For that, we aim to wrap all agents with structured observability:

• Prompt and model versioning
• Token consumption by component
• Per-request latency tracing
• User feedback signals (explicit and implicit)

With all of these in place, if a user says, “The answer made no sense,” we can trace back through the chain to understand which vector match, prompt instruction, or fallback model caused it.

The above is not meant to be the definitive list of everything our Ops team considers, but rather to illustrate some of the key elements driving the mindset we have adopted: in production, agentic pipelines must be treated as living, governed, and observable systems. This isn’t a one-time deployment—it’s a long-term operating model.

That mindset, combined with the right tooling and discipline, is what gets our clients to reliable, cost-conscious agentic systems they can trust.