AI agent memory: working memory, context, and state

An agent that solves multi-step tasks needs to know where it is, what it already tried, and sometimes what is worth remembering next time. That is not the same as putting the full chat history into every turn. In a SaaS product, each customer account also has its own limits on what may be stored, who may read it, and for how long.

This article explains what an agent should keep during a task, what it may store between sessions, and what should never be mixed into the context the model reads on each step.

If you want the general layer map (model, actions, permissions, and records), see The model is not the agent. If you want the loop that orchestrates tools and context, see What is an agent harness, and why it matters more than the model in an AI agent. Here the focus is memory: working memory, memories of earlier interactions, context, and state.

Three different things: context, state, and memory

In meetings they are often used as synonyms. In production they are not.

Context

Context is what the harness puts in front of the model on this turn: instructions, recent messages, tool results, and document snippets. It costs tokens and is rebuilt every step.

It is not a reliable store. It is this trip's carry-on: what the model needs now, but it should not be the system's source of truth.

State

State is structured information that shows where a task, session, or workflow stands. It can include order or ticket identifiers, the current step, tools used, the last result, a pending error, a required approval, or the next step to run.

Unlike context, state is not simply text the model reads again. It lives in an execution log, a database, or temporary storage, and can be recovered when the task pauses, resumes, or moves to another channel.

State can represent different things:

• Execution state: which step is running, which tools were used, and what is still pending.

• Session state: who started the interaction, which configuration is active, and which permissions apply.

• Flow state: whether a task is started, paused, waiting for approval, completed, or failed.

• Entity or process state: an order shipped, a ticket open, or an invoice dispute.

The agent should not invent these states. It may propose a transition such as update_ticket_status, but the system must validate the transition, check permissions, run it, and persist the result. The source of truth is the system that manages that state, not what the model believes happened.

Memory

Memory is information that can still help after the task closes: user preferences, summaries of past incidents, or facts the customer already confirmed.

It must be partitioned by customer account and have rules for retention, updates, access, and deletion.

Mixing context, state, and memory produces the classic symptom: "the agent remembered" because someone re-injected the whole thread, including another customer's data or stale instructions. In multi-tenant SaaS, that is not a product hack: it is an isolation and security risk.

Working memory: what it needs right now

Working memory is the bounded set of facts the agent needs during a task to choose the next step without asking the obvious again.

Examples in a support SaaS:

• The ticket_id and the validated customer account.

• That the last tool returned not_found for an order.

• That the user chose "partial refund" two messages ago.

That data may appear in the model's context, but the source of truth should not be chat alone. The harness can keep a working-memory object, update it when a tool responds, and extract a short summary for the next turn.

Practical rules:

• Use few fields and stable names: current_order_id, not narrative paragraphs.

• Apply a TTL per task. When the run ends or expires, working memory is archived or deleted.

• Do not duplicate persisted system state. If the order is awaiting_approval in the API, the model should not "remember" that only as free text.

• Do not replace a fresh read with an old recollection. If the agent needs the real order state, a permissioned tool is usually more reliable than a message from ten turns ago.

Working memory is temporary and operational. It is not automatically persistent memory or transactional state.

Memory of earlier interactions

Persistent memory stores recollections of earlier interactions: what was tried, what worked, what failed, or what the customer decided. It may be a structured record or a natural-language summary.

It can support continuity across sessions:

"Last week this customer rejected an automatic refund and asked for manual review."

"On similar projects, the ERP integration failed on a currency field."

It should not be a dump of the previous chat. A useful record includes metadata such as customer account, date, task type, confidence level, and a reference to the original trace if audit is needed.

Risks of poorly designed memory

• Cross-customer contamination: a history entry indexed without tenant_id can resurface on the wrong account.

• Stale truth: a stored note that says "the plan has no API" can keep steering the agent even after the product changed.

• Hidden cost: searching thousands of entries every turn increases latency and token use.

• Confusing hints with permissions: a memory that says the user approved something does not mean the system may execute it.

A healthier pattern is to write a summary when the task closes (success or failure), retrieve only records relevant to the current intent, and present that data to the model as auxiliary context, not as authorization.

LangChain documents taxonomies such as buffer, summary, and entities. In production SaaS, however, the main question is not which buffer class to pick, but what deserves to be stored, who may read it, when it expires, and how it is corrected.

State: how the system knows where it is

State represents the current situation of a task, session, flow, or entity. For example, a run may be running, paused, waiting_for_approval, completed, or failed. A product entity may be open, shipped, or dispute.

The model may propose a transition, but it must not decide on its own that the transition happened. The system must validate the operation, check permissions, run it, and store the new state.

When a task needs human intervention, state should reflect it with an explicit value such as pending_approval. Then another channel (email, admin panel, or a new session) can continue the flow without relying on the assistant remembering an earlier message.

For long tasks, the harness can rehydrate context from stored state:

"Task X, step 3, last tool get_order, result shipped."

That is execution recovery: the system rebuilds what the model needs to continue. It is not magic LLM memory.

The important distinction: context is prepared for the current turn, state shows where the system is, and memory keeps information that may help in future tasks.

Managing memory: compaction, context, and retrieval

On a long task, context is sent to the model every turn: instructions, tools, messages, results, files, and command output. A large window is a limit, not a goal. The more context you keep, the higher the cost and the harder it is to find what matters.

Compaction replaces part of the history with a structured summary. It may keep the goal, technical decisions, files changed, errors, and pending work, but lose literal text and full tool outputs. The summary is a derived representation, not a source of truth.

Memory policy must decide when to compact, what to keep, what to retrieve, and what to delete. Compacting too early forces re-reading information; doing it too late can produce context rot, mix old decisions, and reduce model accuracy.

Here it helps to separate a closed decision from memory management. As we explain in Jev AI: what it is, how to use it, and where it fits in a SaaS, Jev fits when there is a semantic, bounded, repeatable, evaluable decision: classify a request, route it to a flow, or choose among known actions. Compaction asks a different question. It is not enough to decide whether an isolated message still looks relevant:

"Is this message still useful?" is not the same as "what state must the agent keep to keep reasoning?".

Whether information stays relevant can depend on everything that happened before and everything the agent must do next. A line that looks minor can explain why a hypothesis was dropped twenty steps earlier. That is why we do not recommend using Jev as the main mechanism to decide, message by message, which parts of the history to keep during compaction. Turning that into many independent classifications can lose relationships and context that only show up when you look at the full task.

Jev can still help with bounded decisions inside this system (for example, classify task type, detect whether an action needs approval, or choose a retention policy), but compaction should rely on execution state, explicit instructions on what to preserve, recoverable traces, and a memory strategy designed for the full flow.

A simple way to keep important state out of the chat history:

const executionState = {
  taskId: "task_1842",
  goal: "Update the welcome template",
  filesChanged: ["templates/welcome.html"],
  lastTool: "template.update",
  lastResult: "success",
  pending: ["capture the page and verify the result"]
};

const executionState = {
  taskId: "task_1842",
  goal: "Update the welcome template",
  filesChanged: ["templates/welcome.html"],
  lastTool: "template.update",
  lastResult: "success",
  pending: ["capture the page and verify the result"]
};

const executionState = {
  taskId: "task_1842",
  goal: "Update the welcome template",
  filesChanged: ["templates/welcome.html"],
  lastTool: "template.update",
  lastResult: "success",
  pending: ["capture the page and verify the result"]
};

The harness can build a short context for the next turn from that state:

function buildWorkingContext(state) {
  return [
    `Task: ${state.taskId}`,
    `Goal: ${state.goal}`,
    `Last tool: ${state.lastTool} (${state.lastResult})`,
    `Files changed: ${state.filesChanged.join(", ")}`,
    `Pending: ${state.pending.join("; ") || "none"}`
  ].join("\n");
}

function buildWorkingContext(state) {
  return [
    `Task: ${state.taskId}`,
    `Goal: ${state.goal}`,
    `Last tool: ${state.lastTool} (${state.lastResult})`,
    `Files changed: ${state.filesChanged.join(", ")}`,
    `Pending: ${state.pending.join("; ") || "none"}`
  ].join("\n");
}

function buildWorkingContext(state) {
  return [
    `Task: ${state.taskId}`,
    `Goal: ${state.goal}`,
    `Last tool: ${state.lastTool} (${state.lastResult})`,
    `Files changed: ${state.filesChanged.join(", ")}`,
    `Pending: ${state.pending.join("; ") || "none"}`
  ].join("\n");
}

The summary should keep confirmed changes, reproducible errors, architecture decisions, and pending work. Repeated command output or full payloads should stay in the trace, not necessarily in active context.

Compaction does not turn the summary into state. If the system says approval is still pending, the agent must read that state with a tool. Likewise, memory retrieved from a vector index is contextual hint, not authorization.

What goes in front of the model and what lives outside

Practical rule: the model sees enough text to choose the next step, not a dump of everything the system knows.

The harness decides when to compact messages, when to call RAG, and when to request a fresh read with a tool instead of trusting an old summary.

Deletion, retention, and GDPR in SaaS

Memory can contain personal data and customer data, so it should follow the same architecture choices and policies as any other database.

Partitioning and isolation

Information must be isolated by customer account in records, vector indexes, and preferences. A tenant_id on the document is not enough: the query must apply a mandatory filter, and the tool must check identity and permissions before adding the recollection to context.

Retention and deletion

Policy must set how long summaries are kept and when they expire. For example, an organization might keep some support summaries for 90 days and test data for less, but the period depends on the use case.

If a customer requests deletion, the system must be able to remove their recollections and preferences. You also need to document what is sent to model providers and what retention rules apply to those APIs.

Minimization

Do not store in memory what already lives in the CRM or ticket. Store only the ticket identifier (ticket_id) and let the tool fetch detail when needed.

Traceability helps show which memory influenced a decision. Without records, "the agent remembered" is not auditable for a customer or a regulator.

Typical mistakes: mixing everything into the prompt

Memory equals infinite history

History grows, cost rises, model attention worsens, and different tasks bleed into one thread.

RAG as universal memory

Retrieved documents do not replace order state or permissions. RAG is information retrieval: it adds context, not authorization.

Recollections without expiry

If memory never expires or is never reviewed, the agent may act on policies from a year ago.

One vector index for all customers

If the tenant filter fails even once, there is a risk of retrieving another account's chunks. Isolation must be verified on write, indexing, retrieval, and final read.

Writing memory from the model without validation

If the assistant "summarizes" that the customer accepted a charge and that text is stored as fact, you create legal and product debt. The model may propose a recollection; the system must verify it, assign provenance, and decide whether to store it.

Decision map

When the map is clear, the next step is usually tightening what tools return and which history entries get indexed, not lengthening the system prompt.

If the agent is already in pilot, one real trace helps separate what was context, what was persisted state, and what became persistent memory. That shows whether the design holds in production or only worked because the thread was short.

Conclusion

Agent memory is not about storing everything.

• Context is the information the model needs to choose the next step.

• Working memory holds the bounded data for the current task.

• State represents the structured situation of a task, session, flow, or entity.

• Persistent memory keeps information useful for the future, with permissions, retention, and traceability.

Managing memory is not storing more data. It is deciding what information should come back, when, from which source, and under what authority.