A demo is not an agent in production

A working POC is still not ready for production. It is missing the system the demo did not need: identity and permissions, state and recovery, traces and evaluation, a cost cap, and an owner with a name.

A demo is fine, but does it scale? Almost never on the first try, and rarely because of the model. A POC (proof of concept) answers well because it plays at home: clean data, open permissions, and someone next to it who rephrases the question when the model drifts. Taking an AI agent to production means adding everything the demo did not need: identity and permissions, state and recovery, traces and evaluation, a cost cap, and an owner with a name. If you are missing any of those pieces, the next step is not to deploy. It is to cut scope until the system can be operated.

The pattern we see in almost every meeting

The script repeats so often we already recite it from memory. Someone spots an opportunity, someone puts together a test in two weeks, the test convinces the whole committee, and then the question that freezes the room arrives: and we plug this into real customer data?

That is when you see what it actually was: not an agent, but a lucky prototype. It lives in an environment with full access, a friendly dataset, and permanent human supervision. It works because the conditions are soft. The demo does not lie: it is designed to look good, and it does that very well.

What comes next almost never makes it into the presentation. Permissions per tenant, traces for audit, regressions when the model changes, a bill per run, someone on call. The team estimates that work, compares it with the roadmap it already committed to, and decides it will wait until next quarter. The pilot sits in limbo for months. The model did not fail. The system around it was missing.

It is not an AI-only problem either. DORA software delivery research has been measuring the same thing across every technology wave for years: what determines whether something reaches production and stays there is the team's delivery capabilities, not the tool that was purchased. With AI agents in production it is exactly the same, except the bill arrives sooner.

Demo, pilot, and production: three different states

Before you argue architecture, name the state you are in. The AI agent demo vs production conversation is not a vocabulary debate: each state has a different risk level, a different way of being measured, and a different owner.

Demo. Goal: credibility. Controlled data, a known path, and a person watching. It authorizes more time on the case. It does not authorize touching customer data.

Pilot. Goal: learn with bounded risk. One tenant, one flow, one metric defined before you start, supervision, and a go/no-go date on the calendar. This is why an AI pilot fails about half the time: enthusiasm was measured instead of outcome, or a cutoff date was never set. If your POC has been calling itself a pilot for five months, it is not a pilot: it is limbo with a budget.

Production. Goal: operate. Real customers, writes possible, rollback tested, cost that fits the SaaS margin, and someone who answers the phone when something breaks at nine in the morning.

A short way to remember it: the demo convinces, the pilot measures, production answers. If your AI agent POC is still in the first state, do not present it as the third. Naming the state out loud, in front of business and engineering, saves weeks of crossed conversation.

The layers that show up after the MVP

When you try to cross from pilot to production, layers appear that no prompt solves. Anthropic puts it well in Building effective agents: what makes an agent reliable is the system around it, not the model on its own. These are the four layers that have blocked the most deployments for us.

Authentication and permissions

What identity the agent uses, what it can read, what it can change, and how tenants are isolated from each other. The prompt is not an access control, no matter how well it is written. If the user cannot delete an invoice, the agent acting on their behalf should not be able to either.

The rule is least privilege, with scopes defined per tool and per tenant, and the agent's own identity so every action is auditable. If your tools have more access than the human's role, you have not added a feature: you have opened a hole with a very good interface. That layer is exactly what Devic's agent harness covers.

State, errors, and recovery

An agent that dies mid-trajectory does not leave a clean error: it leaves work half done. Half an invoice issued, half a ticket escalated, half a record updated. That requires an explicit contract for each tool (what goes in, what comes out, what happens when it fails), state that survives the cut, idempotency (repeating the same call does not duplicate the effect), and rollback.

Without that layer, every incident becomes an urgent ticket and an apology email. And the real cost is not the incident, it is the trust of the customer who had already started to delegate. If you also need to build the interface for that flow, that is where Devic UI's React components come in.

Observability and evaluation

If you cannot reproduce yesterday's bad answer, you do not have an agent in production: you have a black box on a lucky streak. You need traces of the tool calls, prompt and model version recorded, and evals (automated, repeatable evaluations) that run in staging on anonymized real cases.

The approval threshold is agreed by product before deploy, not by the engineering team on the afternoon of the rollout. And "it seems to be going well" is not an acceptance criterion: it is a feeling. When you change models, that same suite is the only thing that will tell you whether you improved or broke three cases that already worked.

Costs and ownership

The price per token is not the cost of your agent. What a SaaS needs is cost per run and per tenant, with a cap, an alert before the cap, a kill switch, an incident runbook, and an on-call person with a name.

Without a cap, a poorly closed loop eats a month's margin in one afternoon. Without an owner, the pilot rots on its own: nobody kills it and nobody improves it. To land caps and a cost model, look at Devic's plans and their tool call limit.

What must be proven before allowing writes

Moving from read to write under commercial pressure is the most expensive shortcut in the industry. Writes are the point of no return: the agent stops suggesting and starts changing data the customer invoices, audits, or presents to their board.

Before you open writes, at minimum:

1. Permissions and scopes of each tool validated per tenant, with no more access than the role of the user who invokes it.

2. Evals green in staging, with anonymized real cases and a threshold agreed with product in writing.

3. Rollback runbook, kill switch, and on-call owner, tested at least once for real, not in theory.

If one of the three is missing, the agent stays in read mode. That is not a lack of ambition. It is not wanting to explain to a customer why four hundred records were changed on a Friday afternoon.

Production readiness checklist

This is the production agent checklist for SaaS we use as a go/no-go in the meeting, not as a filler slide.

• You know whether you are in demo, pilot, or production, and you have said it out loud in front of business and engineering.

• The agent has its own identity and scopes per tenant, reviewable by someone who did not write them.

• You can reproduce yesterday's failure with a trace, a prompt version, and the sequence of tool calls.

• There are evals with an agreed threshold and a suite that runs every time the model or the prompt changes.

• You know the average cost per run and per tenant, and there is a cap with an alert before you hit it.

• There is an owner, a runbook, rollback, and a kill switch, and someone has tested them at least once.

"The demo proves the model can answer. Production proves the system can operate."

If you check fewer than five, writes are not due yet. A pilot is due, and saying so without decoration in the next product meeting is due.

Signs you are moving too fast

These signs show up when the team confuses demo speed with production maturity. If you recognize two or more, stop and reorder. They also explain why an AI pilot that looked fine in the demo fails.

• You do not know how much an average run costs per customer, and you justify it by saying it is "still low volume".

• The tools have more permissions than the user's role, because that made development faster.

• Nobody on the team can reconstruct a bad answer from last week.

• The pilot has been running for months with no agreed metric and no go/no-go date.

• The case was chosen because it shone in the demo, not because it moves a business metric.

Most of them get fixed in days if there is a platform and judgment. They turn into months when the team rewrites scripts blindly and calls trial and error iteration. Speed without measurement, in agents, is technical debt with very good copy.

When starting small is the right answer

Not every case deserves AI agents in production right now, and saying so in time is part of the job. Moving to production does not by itself fix missing data, a missing metric, or missing ownership: it only makes the gap more expensive and more visible.

If the process is stable and verifiable, a deterministic workflow is enough and costs less to maintain. If write risk is high, a read-only copilot that suggests and lets the human confirm is usually the sensible first step. And if the sources of truth are broken, fix the data first: garbage in, garbage out, now with a token bill.

If you want an open base to build on, look at Devic's open source tools for agents. On both paths the recipe is the same: one task, one flow, one pilot tenant, with permissions, traces, evals, and a cost cap from day one.

Checklist: is your agent ready for production?

Go through the six lines of the checklist with your team and be honest about the ones that fail. If they are green, you can propose writes with a net underneath. If not, you already know what to build this week, and it is not a better prompt.

To land it in your stack, look at which layers a harness with observability and cost control already covers. If what is blocking you is fitting it in the margin, compare prices by plan and the included tool call volume. And if you would rather look at it with your case on the table, tell us and we will look at it with the numbers in front of us.