How to evaluate AI agent actions, not just answers

An agent can write a flawless reply and still not have done the work. It says it updated the order, created the refund, or closed the ticket, and the text sounds right, but the system is unchanged or, worse, something it should not have touched changed. If you only read what the agent says, you cannot tell whether it worked.

This article explains how to evaluate what the agent does in your product: what to measure, what to keep from each run, how to build a set of real cases, and how to check that a change of model or instructions does not break what already worked.

If you want to reconstruct a task after it happened, see agent traceability. If you want the program around the model that runs the actions, see the agent harness. Here the focus is measurement: how to know, with numbers, whether the agent did its job well.

A correct answer versus work done

Evaluating a chat and evaluating an agent are two different problems. In a chat, the result is the text: if the answer is correct, useful, and clear, it did its job. In an agent, the text is only the end of a chain of steps, and what matters is what changed in your product along the way.

Think of a support copilot asked for a partial refund. The final reply may say "Refund of 25 euros processed". To know whether that is true you have to check four things the text does not prove:

• That it chose the right action (create a refund, not just look up the order).

• That it sent the right data (the amount and the order of that customer, not another).

• That the system accepted the action and the refund really exists.

• That it did so without extra steps: without repeating the same call three times or reading data it did not need.

A reply that reads well with a refund that does not exist is worse than a clumsy reply with the refund done. So the underlying question changes: from "does it sound good?" to "is the final state of the system the one it should be?".

In Building effective agents, Anthropic recommends starting with simple systems and measuring before adding complexity. The idea is the same: without a way to measure, every change is a bet. OpenAI describes evals in its evals guide as repeatable tests against defined criteria, and that is the spirit worth carrying over to actions, not just to texts.

Metrics that matter in production

You do not need a dashboard with thirty indicators. For an agent in a SaaS, a few measured the same way every time, and looked at by someone each week, are enough. They fall into two families: whether the work came out right, and what it cost to do it.

Task success and first-pass success

The main metric is task success: of all the tasks the agent attempted, what percentage ended with the correct result in the system. Not "it answered something", but "the order ended in the expected state". It is calculated by comparing the product's final state with the expected one.

A very useful variant is first-pass success: what percentage was solved without the agent having to correct itself, retry, or ask a person for help. An agent that ends well but takes four attempts works in the demo and costs a lot in production. If task success goes up and first-pass success goes down, the agent is using retries to compensate for something wrong in the actions or the instructions.

Also measure two things that are not part of the percentage:

• Tasks escalated to a person, because an agent that hands everything off is safe and useless.

• Serious errors, meaning incorrect actions with real effect (an extra refund, an email to the wrong customer). Count them separately, because a single case can outweigh a hundred successes.

Cost and steps per task

The second family answers how much each well-done task costs. Two numbers are enough:

• Steps per task: how many actions and model calls the agent needs on average. If it rises without the tasks getting more complex, something is repeating.

• Cost per successful task: total spend divided by the tasks that ended well, not by all attempted. Failed tasks cost too, and that is exactly what this calculation exposes. How that cost breaks down is in the token price is not the agent's cost.

Looking at both families together avoids being fooled. A change that raises task success from 82% to 88% but triples cost per task may not be worth it. One that halves cost at the price of five points of accuracy may not be either.

What to capture in each trace

A trace is the step-by-step record of a run: what the agent did, in what order, and what each action returned. Without traces you cannot compute any of the above, nor understand why a specific task failed.

To evaluate actions, each trace should keep at least:

What should not go into the trace without care is customer data and secrets. Store identifiers and results, not the full content of documents, and apply the same retention rules as any other store. How to build this record in detail is in traceability.

A set of action cases, not only chats

To measure with numbers you need a fixed set of cases with the expected result. In the trade it is called a golden dataset: a collection of real situations, each with the correct outcome already decided, against which you test the agent again and again.

The usual mistake is building it only from conversations: a question and the ideal answer. That evaluates the text. For an agent, each case should describe three things:

1. The starting situation: the product state before beginning (a shipped order, an open ticket, a customer on a specific plan).

2. The user's request, exactly as it arrives.

3. The expected result in the system: which action should have been called, with what data, and what state everything should end in. And, just as important, what it must not have touched.

Some guidelines to build it without it becoming unmanageable:

• Start with 20 to 30 real cases, taken from tasks the agent has already handled, with customer data anonymized. A small, reviewed set beats a huge, doubtful one.

• Include the hard cases on purpose: ambiguous requests, missing data, a customer asking for something not allowed, an action that returns an error. A set of only easy cases gives a high score and teaches nothing.

• Add every real failure that happens in production. When the agent gets something wrong with a customer, that case enters the set with the correct outcome. That way the same error does not return unnoticed.

• Separate by customer when it matters. In a SaaS with several accounts, some case should check that one account's agent does not touch another's data.

Regression when you change the model, actions, or instructions

A regression is when something that used to work stops working after a change. With agents it happens more easily than it seems, because almost everything can change: the model, the instructions, the description of an action, or the schema of its data. A tweak meant to fix one case can break three others without anyone seeing it.

The defense is a simple routine:

1. Before changing anything, run the case set with the current version and save the numbers.

2. Make the change (another model, a new instruction, a rewritten action).

3. Run the same set with the new version.

4. Compare the two tables: task success, first-pass success, steps, cost, and serious errors.

5. Ship only if the new one does not worsen what matters, or if the worsening is a conscious decision.

Two caveats prevent misreadings. First, the model does not answer exactly the same twice, so repeat each case several times or look at trends instead of a single result. Second, an average improvement can hide a drop in one kind of task: always look at the breakdown by case type, not just the total.

This also applies when you change only the description of an action. As we saw in tool contracts for agents, the model reads that description at every step and decides by it, so touching it is the same as touching behavior.

Human review without turning it into a bottleneck

Automatic numbers do not cover everything. Some things only a person judges well: whether the tone was right, whether the decision was reasonable, whether the action made sense even if technically valid. The risk is that review becomes a queue nobody finishes.

Three rules keep it manageable:

• Review by sampling, not everything. A small, steady sample (for example 5% of the week's tasks) gives a reliable picture without overwhelming the team. Raise the percentage only for high-risk tasks.

• Prioritize the unusual. Tasks with many retries, with cost far above average, or escalated to a person teach the most. Review them first.

• Turn each review into a case. If a person spots a failure, that case goes into the test set. That way review feeds the automatic measurement instead of repeating every week.

For actions that cannot be undone, such as a charge or a message to the customer, review is not sampling: it is approval before running, and that is designed in the permission system, not in evaluation. It is covered in identity, permissions, and limits.

Evaluation checklist

Before treating an agent in production as evaluated, check that you can answer yes to this:

• You measure task success by looking at the product's final state, not at the text of the reply.

• You know first-pass success and how many tasks are escalated to a person.

• You count serious errors separately.

• You know steps and cost per successful task, not just the model price.

• Each run stores a trace with actions, data, results, version, and final state.

• You have a set of 20 or more real cases, with starting situation and expected result in the system.

• That set includes hard cases and every real production failure.

• You run the set before and after changing the model, instructions, or actions, and compare.

• You review a fixed sample of tasks each week and turn failures into new cases.

• Someone with a name looks at these numbers daily or weekly, and can pause the agent if they get worse.

If several are missing, the next step is not improving the model. It is measuring: without those numbers, every change is decided by intuition, and the pilot stays stuck arguing whether the agent "is doing better". When measurement exists, the discussion changes: it is no longer an opinion, it is a table. It is part of the jump we cover in a demo is not an agent in production.