07 Sep 2024 AI Engineering5 min read

Agent Memory Beyond the Context Window

Context windows provide working memory, not durable recall. We distinguish the two concepts, examine the three mechanisms used in production by LLM agents in 2024 — conversation summaries, vector memory, and structured state — and trace their failure modes. We also explain why deliberate forgetting improves correctness and how the GDPR constrains persistent agent memory.

A Context Window Is Not Memory

An LLM is stateless. Every API call starts from zero. What looks like memory in a chat interface is replay: the client resends the previous turns with each request. The context window — 128,000 tokens for GPT-4o, 200,000 for Claude 3.5 Sonnet, up to 2 million for Gemini 1.5 Pro since 27 June 2024 — is working memory at best. After the call it is empty.

For agents, this distinction is architectural. An agent operating for weeks across sessions and tasks accumulates more history than any window can hold — and must recover specific facts at unpredictable moments. Memory therefore lives outside the model: the system must decide what to store, how to retrieve it, and when to delete it. Three mechanisms are in production use for that purpose in 2024.

A task arrives — the agent plans its next step. 1/4

Why Larger Windows Do Not Solve It

Three arguments against "just use a bigger window". First, cost: input tokens are billed on every call, so a history replayed in full makes every turn more expensive than the last. Second, latency: prefill time grows with prompt length. Third, recall quality: Liu et al. showed in July 2023 ("Lost in the Middle", arXiv 2307.03172) that models use information from the middle of long contexts significantly worse than information at the beginning or end.

Even a perfect window would not persist anything. The window is an argument to a function call, not a store. When the session ends, its content is gone. Persistence needs a write path, a retrieval path and a deletion path — and those are system design questions, not model questions.

Conversation Summaries

The oldest mechanism is rolling summarization. When the transcript approaches the token budget, the model compresses older turns into a summary; the summary replaces those turns in the prompt. LangChain ships this as ConversationSummaryMemory; most agent frameworks have an equivalent. Cost stays bounded. The prompt stays short.

Summaries necessarily discard information, and the loss cannot be controlled. At compression time, the model decides what survives without knowing what next week's question will require. Precise details — an order number or a version constraint mentioned once — are therefore the first casualties. Summaries preserve gist and tone, but facts only unreliably. They are not a system of record.

Vector Memories

Vector memory applies retrieval-augmented generation to the agent's own history. Turns or extracted statements are embedded and written to a vector index; at query time the top-k nearest entries are injected into the prompt. Storage scales to years of history. Retrieval cost stays constant regardless of how much has accumulated.

The limits are the limits of embedding similarity. Semantic nearness is not relevance: "customer cancelled the contract" and "customer extended the contract" embed close together. Time is not modeled: a preference from January and its revision from June both match the same query, and the index does not know which one is current. Vector memory retrieves. It does not reason.

Structured State

The third mechanism stores facts as explicit, typed state: a profile, a task list, key-value pairs the agent reads and writes through tool calls. MemGPT (Packer et al., October 2023) formalized this with an operating-system analogy: a small core memory pinned in the context, larger stores paged in on demand, the LLM issuing the memory operations itself via function calls.

Products followed. OpenAI announced memory for ChatGPT on 13 February 2024 and extended it to Free, Plus, Team and Enterprise users on 5 September 2024 — as visible, editable, deletable entries. That visibility is the point: structured state can be audited. The price is schema design up front and extraction logic that decides what becomes a fact.

Mechanism	Write path	Read path	Weakest when
Conversation summary	Model compresses older turns	Summary prepended to every prompt	Specific details are needed later
Vector memory	Embed and upsert statements	Top-k similarity search at query time	Recency or negation decides relevance
Structured state	Extraction into a defined schema	Direct lookup or pinned in context	Facts do not fit the schema

Forgetting as a Feature

A memory that only grows degrades. Stale facts crowd out current ones, retrieval precision drops, contradictions accumulate. Park et al. showed the alternative in April 2023: the "Generative Agents" memory stream scores every entry by recency, importance and relevance, with recency decaying exponentially. Forgetting was load-bearing in that design, not an afterthought.

In practice this means: time-to-live on episodic entries, decay scores that demote what is never retrieved, and supersede-on-write rules so a new fact replaces its predecessor instead of coexisting with it. An agent that remembers a cancelled plan keeps acting on it. Deletion is not data loss here. It is state hygiene.

Privacy Boundaries

Persistent memory turns an agent into a personal data store, and in the EU that has defined consequences. Stored user facts are personal data under the GDPR: Article 15 grants access, Article 17 erasure. "Delete my data" must reach summaries, vector entries and structured state alike — including derived facts the user never wrote verbatim.

Architecture follows from this. Memory must be partitioned per user and per purpose, never shared across tenants, and erasable by key. A memory the user cannot inspect should not exist. We treat inspectable and erasable memory as a hard requirement in enterprise agents, not as a feature. OpenAI's memory controls and temporary chats point the same way.

Outlook From September 2024

We expect the three mechanisms to converge into managed memory layers: hierarchical stores in which summaries, vectors and structured facts are tiers of one system, with MemGPT-style paging logic moving into frameworks and platform APIs. Model vendors will offer memory as an API primitive, not only as a product feature in their chat frontends.

From this, we make two predictions for the next twelve months. First, memory quality becomes a benchmark category alongside reasoning; today it is barely measured. Second, deliberate forgetting becomes a compliance capability that enterprise buyers request explicitly. Larger context windows make none of these mechanisms obsolete: the window is where an agent thinks. Memory determines what remains.