The model is not the agent: the layers you still need

Besides the model, an application that acts reliably needs tools with a contract, state, permissions, a place where the code runs, evaluation, and someone who owns it when it fails. The model reasons; the rest is the system. If you only buy the engine, you do not have a car.

It is relatively easy, and impressive, to ship a demo assistant: you pick a model, give it some context, the answers come back coherent, and the board in the room applauds. A week later the copilot calls your API, stalls halfway, and nobody knows how many retries it made, which tool calls it tried before giving up, or which decisions it took while trying to resolve the question. That is the thesis: the model is the visible 20%; the rest is the system around it.

An LLM (large language model) predicts the next token. An agent is an application that uses that model to decide and act: it calls tools, keeps state, and operates on a customer's account (the tenant) in your SaaS. Google Cloud draws it that way: the model is the reasoning engine, and around it sit tools, memory, a runtime, and a place where code actually runs. If you only buy the engine, you do not have a car.

Anthropic is blunter: the building block is not "the model", but the augmented model (retrieval, tools, memory). They also say to start simple. A workflow with a fixed path is often enough; a self-directing agent costs more and fails in a cascade if the tools are sloppy.

This is the map. It is not the deep dive on permissions, and it is not MCP versus API. It is knowing which layers exist, what they are for, and which ones you can postpone in a pilot without lying to yourself.

What the demo hides

The demo is a chat and one happy tool: the user types, the model calls get_ticket, a summary appears, and the meeting ends.

What you do not see is who that runtime is when it hits your API, what happens if the tool returns 500 twice, which tenant stores the summary, whether the next turn resends 40,000 tokens of context, and who gets paged at 3 a.m. if the loop does not stop. Same jump we already covered: a demo is not an agent.

The sales trick is to sell the model as if it were the product. What you usually hear is "we put a copilot in our product", when the reality is a wrapper around Claude or Codex. The committee hears innovation, a new product, and more revenue; platform hears a hole: tool contracts, state, isolation, logs, and an owner. The model does not fill that hole. You can swap vendors and keep the same gap.

Three things the demo hides and a pilot cannot:

1. The action: the model proposes and the tool executes, so if the tool exists, the risk exists.

2. Time: a chat turn dies when the tab closes, but an agent leaves residue (memory, half-finished tickets, spent tokens).

3. The neighbor: a SaaS has more than one customer, and the demo usually has one.

If you cannot name those three, you do not yet have an agent in production: you have a conversation wired to an API.

A map of the layers

An AI agent architecture is the list of parts that turn an LLM into something that acts without reinventing the system every Tuesday. It is not a framework, and it is not a slide that says "orchestration". It is this, in "if it is missing, it hurts" order:

Figure. The seven layers (from what the demo shows to what saves you when it fails)

Google Cloud splits the drawing into model, tools, memory, agent runtime, and model runtime, and that works. We fold runtime into sandbox, and we split state from memory, because in a SaaS that mix-up costs an incident.

You do not need all seven on day one, but you do need to know which ones you left out on purpose. A pilot that "postpones" permissions is not postponing: it is waiting for the first customer that is not you.

Tools and contracts

A tool is an API endpoint designed for agents. Without tools, an assistant can only answer what is already in its corpus. With tools it can solve problems; with tools on your API is when you ship AI features to your customers.

The contract is what you can audit: name, what it does, which resource, which arguments, and which limit. "Search help articles in the current tenant, max 5, no raw HTML" is a contract; "query the database" is not. Anthropic spends a whole appendix on this and says that, on SWE-bench, they spent more time polishing the tools than the prompt: format, absolute paths, and how hard it is to get wrong. They treat the agent-tool interface with the same care as a human one.

How you expose that tool (your API, a CLI, MCP) is a contract decision, not a fashion one. MCP (Model Context Protocol) helps when you want a reusable plug between agents and servers. Your own API helps when that action already exists in your product as a stable verb (create a ticket, read an order) and you want the same input to produce the same effect, without the model improvising the path. A CLI helps on a laptop or in a job. We are not picking here; we are requiring that, whichever you pick, the model does not see an open menu.

Three map rules, not an implementation guide:

- Allowlist: if it is not on the list for that flow, it does not exist.

- Narrow arguments: a ticket_id from that tenant, not SQL, not a free URL.

- A limit: rows, amount, calls. A loop with no limit is an invoice and a risk.

If the demo opened admin "so it looked good", the demo is not ready. The model is not the problem; the menu is.

State, context, and memory are not the same

Teams use these three words as synonyms, and they are not. If you cannot tell them apart, the pilot mixes "what happened in this run" with "what the model read" and "what we stored for next month".

State is what the system knows about this run right now: ticket_id, tenant, step in the flow, last tool result, and whether it is waiting on a human. It lives outside the prompt. A log can read it and a kill switch can clear it. If state only exists in the chat, you do not have state, you have a thread.

Context is what you stuff into this turn's window: instructions, tool schema, last messages, and the document chunk you cut. You pay for it and you pay again on the next step. It is not memory: it is this flight's carry-on. If you pad it "just in case", you pay on every hop and you mix tenants if the cut is sloppy.

Memory is what survives across runs: a preference summary, an article index, a fact the customer already resolved. It has to be partitioned by tenant and it has to be forgettable. A global vector store where customer B's contract shows up in A's answer is not smart: it is a leak.

The usual failure is storing "memory" by pasting the whole thread into the next prompt and calling it state. You cannot revoke that, you cannot audit it, and you cannot say which part belonged to the tenant. The model "remembered" because you told it the secret again.

In a pilot, explicit state (ids, step, result) beats fancy memory. Add memory when there is something worth remembering and a place to partition it.

Where the code runs

A sandbox is the enclosure and execution is the fact of running code or a tool inside it. Together they answer where this runs and what it cannot touch.

The model does not run your backend: it runs (or triggers) a runtime. That runtime has a network or it does not, it can write to disk or it cannot, it can reach the internet or only your APIs, and it has a time limit and a service account. Google Cloud calls that the agent runtime, distinct from the model runtime. The split matters: you can swap LLMs and still have a process that curls anywhere.

The demo usually runs on the presenter's laptop, or in a notebook with env keys. The pilot runs next to customer data. If the agent can execute code (a "code interpreter", a CLI, an MCP that exposes the filesystem), the sandbox is not a security extra: it is the difference between "it proposed a script" and "it wrote to the tenant's disk".

Three questions that fit in a standup:

1. Does this agent reach the internet, or only an allowlist of hosts?

2. Can it write, or only read?

3. How much time and how many calls does it get before it dies?

If the answer is "the model already knows not to", there is no sandbox, there is a wish. Anthropic tells you to test autonomous agents in bounded environments because errors compound. The enclosure is where that error does not eat the product.

Evaluation and logs

Evaluation is how you decide the run was acceptable; logs are the trail that lets you reconstruct it. Without both, "it worked in the demo" is your only test.

A good-looking output is not an evaluation. The agent can call the wrong tool, mix the tenant into context, and still write a clean paragraph. You do not see that in the chat: you see it in the trace (which tool, which arguments, which decision, which prompt and model version).

On the map, this layer asks for the minimum so you are not flying blind:

- A "correct" criterion per case (ticket closed, valid JSON, a human who did not rewrite).

- A correlated log: a run_id that ties user, agent, tenant, and tools.

- The ability to replay the failure. If you cannot reproduce it, you cannot fix it.

You do not need an eval lab on day one, but you do need to keep the trail. A log that dumps two customers into one export is not observability: it is another leak. If you cannot tell the run, you cannot run the agent.

When it fails: retries and who owns it

Recovery is what happens when the tool fails, the model invents an id, or the run stalls. Ownership is the name of the person (or role) who answers for this agent on this tenant.

Without recovery, the agent retries with no cap or stops in silence, and both outcomes are bad. A serious run has a timeout, a maximum number of retries, and an idempotency rule so the same request does not run twice.

Without ownership, the agent is an orphan process. When Slack lights up, the thread will be "is this platform, product, or the model vendor?". If nobody owns it, there is no kill switch with permission: there is a debate.

Ownership is not a 12-cell RACI. It is a sentence someone can say out loud: "This support copilot, on the pilot tenant, is Marta's. If it needs to go down, Marta takes it down." The model does not operate; people do.

In the pilot, name an owner before you open memory. The thing with no owner does not need more layers: it needs a human.

How to sequence layers in a pilot

Do not build the whole map to impress. Build what stops the first incident and what lets you measure; date the rest.

Day one (or there is no pilot). Tools with a contract and an allowlist, tenant on every call, a kill switch, and an owner. Without that, you are in the demo with a real customer.

The next two weeks. Explicit state (ids, step, last result). HITL only where failure is expensive to undo, as in permissions. Logs with a run_id. A cost limit and a call limit.

Later, if the case holds. Partitioned memory. Evaluation that lets you drop models without guessing. A tighter sandbox (no network, no disk) if the agent runs code. The MCP plug or the extra API, once the contract already exists and repeating it hurts.

Before you open the second tenant, the team should be able to answer yes to this.

Is the model separate from the runtime, and can you swap it without rewriting the tools?

Does every tool have a contract, an allowlist, and a limit, or does the model pick from the whole backend?

Do state, context, and memory have an owner and a place, or does everything live in the prompt?

Does the agent have an identity and a tenant, or does it reuse Maria's session?

Do you know where it runs and what it cannot touch?

Can you reconstruct the last run that went wrong?

Is there a name that can shut this off today?

If two or more are missing, you do not have an agent architecture: you have a model with accessories. The prompt can be good, but it is not the system.

Once those lines exist, the rest is engineering. At Devic those layers (tools, policy, trace, runtime) run in the harness: you change the model, not the contract. The plans are there to measure one case on one tenant before you pretend the map is complete.

The model that wins the demo talks well; the agent that survives in a SaaS has layers.